Skip to content

About

No description, website, or topics provided.

Resources

Stars

0 stars

Watchers

0 watching

Forks

 
 

Repository files navigation

Corestone

A Git-backed storage platform for building information management applications — requirements, issues, PLM, documentation — where the domain model is configuration, not code.

CI Go Frontend Projection Source of truth License

Corestone web client: the overview of a research folder with HIDs, workflow states and type-specific columns, and a measurement specification with schema-generated fields (dropdowns, tags) and a rich-text field with inline and display formulas, a striped table and a figure

The generic client renders itself from the repository's schemas: navigation, overview columns, detail sections, workflows and relationship types all come from configuration.


What it is

The Foundation stores, versions, organizes and relates structured information. It knows four native artifact kinds and nothing about any business domain:

Kind Purpose Stored at
Entry Reusable structured object (requirement, ticket, part, …); may be an overlay of another entry <folder>/<guid>/.corestone.json
Document Hierarchical composition: sections, text, media and references to entries <folder>/<guid>/.corestone.json
Link Directed, typed relationship between any two artifacts <scope>/.corestone/links/<guid>.json
Comment Threaded annotation on any artifact (whole artifact, a field, a document element) <scope>/.corestone/comments/<guid>.json

Everything domain-specific lives in JSON configuration inside the repository:

  • The type system lives in <scope>/.corestone/config/, with one directory per kind (entries/, documents/, links/, comments/): types.json defines the kind's types (HID generation, workflow assignments, link-type endpoint rules and cardinality, presentation hints), fields.json and <type>-fields.json the fields with the Foundation's generic field types, enums/ the enumerations and workflows/ the state machines — an artifact can be in several workflows at once. Definitions resolve lexically, nearest folder first; a kind's types.json can set "inheritance": "off" to give a subtree its own type system for that kind. The server enforces it: types and fields must be defined where an artifact lives. See docs/CONFIGURATION.md.
  • Scanner configuration (.corestone/scanner.json) tells the repository scanner which markers to look for (GUID files, configuration folders, indexers), so applications can extend the format.

How it works

Never trust metadata when the primary data already exists.

Git is the single source of truth. A bare repository holds every artifact and every piece of configuration; each logical operation is exactly one commit with a structured message (Entry REQ-42 in specs/boot created, plus Corestone-Op / Corestone-Guid trailers). Commits are built with plumbing only — no working directory — and published with a compare-and-swap update-ref.

PostgreSQL is a rebuildable projection. Plain SQL, no ORM. It holds the GUID → path table, the hierarchy as ltree, artifact metadata, an indexed key/value field index, a GIN full-text index, HID history, deleted-artifact records and the processed_hash of the revision it represents. Missing commits are replayed sequentially; when replay is impossible the projection is rebuilt in the four phases of the design guide (identity → fields → full text → history scan) while the API reports maintenance mode and progress.

Every write runs the repository update transaction (design guide §10.1):

sync projection to HEAD → validate & build the commit object (unpublished)
→ begin DB transaction → project the commit (not processed_hash)
→ [mutex] publish with CAS update-ref → advance processed_hash (CAS) → [release]
→ commit DB transaction        (stale CAS: rollback, rebuild on the new head, retry)

Validation therefore always sees the head it publishes on: HID uniqueness, overlay acyclicity, link endpoint and cardinality rules, reference integrity and workflow legality hold under concurrent writers and across several server processes sharing one repository and database.

Repository files are stable. An order-preserving JSON codec re-writes unchanged objects byte for byte, keeps property positions, appends new properties, and preserves indentation, line endings and trailing-newline style, so Git history shows logical changes only — and files edited by hand or by other tools (including their unknown properties) survive API updates.

Quick start

Requirements: Go 1.24+, git, Node 22+ (web client), PostgreSQL 14+ (16 recommended) with the ltree extension available.

createdb corestone
make build                      # web/dist + bin/corestone
./bin/corestone -repo data/corestone.git -addr 127.0.0.1:8080 -web web/dist -auth none \
  -db "postgres://user:pass@localhost:5432/corestone?sslmode=disable"
./examples/seed.sh              # a demo requirements domain

Open http://127.0.0.1:8080. -auth none skips signing in, for development and the seed script; without it (the default -auth local) the server asks for an email and password, and a new server first offers to create the admin account. Or with containers: docker compose up --build. Full instructions, including deployment notes, health probes and the systemd unit, are in docs/INSTALL.md.

The client

A Lit + TypeScript single-page application with a central store, a URL router (every state is a deep link), a REST client and a WebSocket session client. It contains no domain knowledge:

  • Navigation — search, subtree toggle, the physical folder tree, and artifacts grouped by type.
  • Overview — a table whose columns come from schema presentation metadata, with kind filters and the + New flow: choose a type visible at the current folder, fill an empty schema-generated form, create.
  • Entry detail — one scrollable page with quick links: general, fields, workflows (state dropdown with available transitions and a diagram), relationships (with allowed link types), threaded comments, attachments, overlay composition (chain and per-field origin), history, and pass-through metadata. Field edits, transitions, added or removed links, comments and attachments are staged (shown as unsaved) and published together by Save as a single commit; a comment posted with nothing else pending is published right away. Saves use If-Match; concurrent edits are announced over the session channel and rejected with 412 rather than silently overwritten.
  • Document view — a distraction-free block editor (sections, paragraphs, entry reference cards, lists, code, images) with optional sidebars: properties, current entry, relationships, comments, versions.

REST API

Artifact APIs ({guid} routes are kind-checked: an entry GUID is 404 under /documents):

POST /api/entries | /api/documents        {path, type, title, hid?, base?, fields?, content?}
POST /api/links                           {type, source, target, fields?}
POST /api/comments                        {subject, text, parent?, author?, type?, anchor?, fields?}
GET  PUT  DELETE /api/{entries|documents|links|comments|artifacts}/{guid}   ETag / If-Match
GET  PUT  DELETE /api/artifacts/{guid}/files/{name}                          attachments
POST /api/artifacts/{guid}/save           If-Match; one commit for all changes (see below)

POST /api/artifacts/{guid}/save takes {patch?, transitions?: [{workflow, to}], addLinks?, removeLinks?: [guid], addComments?, removeFiles?: [name]} as JSON, or as multipart with that object in a changes part plus one file part per attachment to store. Links must start or end at {guid}; comments are attached to it. All steps are validated together — any failure rejects the whole save — and published as one commit (Kind HID modified; dev open -> review, 1 link added, …; a single change keeps its usual subject). It returns the updated artifact and ETag. Each attachment may be up to 32 MiB, all of one save together up to 128 MiB.

If-Match names the version the user started from. When the artifact has changed since, the save is merged: the user's changes are applied on top of the current version, and changes by others to anything else stay, without a warning. Only when both changed the same thing (a property, a field, a workflow, an attachment, or a link being removed — removeLinks entries may be {guid, etag} with the link version the user saw) is the save refused with 412 and {etag, conflicts: [{key, author, time, base, theirs, mine}]}. Sending the same save again with overwrite: <that etag> saves the user's values; a change made after that version conflicts again. A link deleted meanwhile counts as removed.

A combined save commit carries Corestone-Op: update, the Corestone-Guid of {guid} and Corestone-Changes with the operations it combines (e.g. update,transition,create,attachment). Links and comments created in it get no Corestone-Kind trailer of their own: tools that read trailers find them by their files, as the history index does.

Service APIs:

GET  /api/health                          liveness/readiness: 200 while the database answers, 503 otherwise; version, sessions
GET  /api/repository                      head, projection status, statistics, scanner config
GET  /api/repository/tree?path=&subtree=  folders and artifacts
GET  /api/repository/search?q=&kind=&type=&path=&subtree=&hid=&field.<id>=&state.<wf>=&sort=&limit=&offset=
GET  /api/repository/types?path=          effective schemas visible at a folder
GET  /api/repository/values?type=&field=&path=   field values in use, most frequent first
GET  /api/repository/validate             repository invariants: dangling refs, self-links, duplicate HIDs, cycles, misplaced metadata, …
POST /api/repository/reindex              full phased rebuild (202; progress in status and over WebSocket)
POST /api/repository/folders/move         {from, to} — moves artifacts, attachments, configuration and metadata
POST /api/repository/maintenance/relocate-metadata     restore metadata locality after direct Git edits
GET  /api/repository/hids/{hid}           current and historical owners of a HID
GET  /api/repository/deleted?guid=        deleted artifacts with their deletion commit
GET  /api/artifacts/{guid}/schema | overlay | workflows | relationships | comments | history
                                          (history entries name links that repeat a deleted link: same type and endpoints)
GET  /api/artifacts/{guid}/versions/{blob}?commit=   a historic version: links, comments, endpoints and subject as they were then,
                                          and what the commit changed (created, restored after a deletion, properties, links, comments)
POST /api/artifacts/{guid}/transition     {workflow, to}      POST /api/artifacts/{guid}/move  {path}
GET  /api/config?scope=[&path=]           a folder's configuration files, or one file (ETag)
PUT  DELETE /api/config?scope=&path=      write or remove a file (If-Match); returns unresolved references as warnings
GET  /api/schemas/effective?type=&path=   the composed type    GET /api/workflows/{id}?type=&path=   a resolved workflow
GET  /api/icons/{name}?path=              an icon (SVG) as seen from a folder, nearest folder first
WS   /api/ws                              session service: commits, status, presence

Errors are JSON {"error": …, "status": …}: 400 validation, 404, 409 conflict (duplicate HID, cardinality, contention), 412 stale If-Match, 503 maintenance mode or projection unavailable (reads fail closed), 504 when a request exceeds its deadline. Bodies are capped at 4 MiB (attachments 32 MiB). WebSocket sessions are same-origin only unless origins are allowed explicitly.

Repository layout

cmd/corestone       server binary: flags, security headers, access log, health probe, graceful shutdown
internal/gitx         bare-repository plumbing: CAS commits, batched reads, first-parent history walks
internal/ojson        order-preserving, format-preserving JSON
internal/model        kinds, GUIDs/HIDs, folder rules, field types, schema composition, workflows, overlays, content
internal/scanner      configurable repository scanner and the foundation indexer
internal/projection   PostgreSQL projection: sync/replay, phased reindex, queries, validation
internal/foundation   the service: operations, the §10.1 transaction, service views
internal/httpapi      REST layer and WebSocket hub
web/                  Lit + TypeScript client and Playwright tests

Testing

export CORESTONE_TEST_DSN=postgres://postgres:postgres@127.0.0.1:5432/corestone_test?sslmode=disable
make test        # go vet, gofmt gate, all packages with -race (PostgreSQL-backed tests skip without the DSN)
make lint        # golangci-lint (errcheck, staticcheck, govet, unused, ...); CI runs it too
make fuzz        # fuzz smoke: JSON codec fixed point, folder validation envelope, scanner classification
make e2e         # REST end-to-end script against a temporary server
make test-ui     # Playwright smoke suite; make test-ui-all runs every browser suite (needs the corestone_e2e database): the + New flow, every field type through the
                 # generated form, attachments, the block editor, workflows, relationships, overlays, HID renames, moves,
                 # search and deep links, keyboard shortcuts, responsive and dark mode, reindex, two users in separate
                 # contexts (presence, live updates, conflicts), plus adversarial cases (script-looking content,
                 # javascript:/data: URLs, over-long input, double submit, garbage deep links)

Adversarial suites live next to the regular ones: internal/foundation/adversarial_test.go (hostile inputs, validation races between concurrent writers, hostile direct pushes, writes during reindex, branch rewinds), internal/httpapi/adversarial_test.go (limits, injection-shaped parameters, path tricks, WebSocket abuse) and web/tests/adversarial.spec.ts.

Integration tests compare the live, incrementally maintained projection with a full rebuild after every scenario (including concurrent writers in two server processes), so "rebuildable from Git" is verified on every run rather than assumed.

Documentation

Document Contents
docs/INSTALL.md Install, run, define a domain, deploy
docs/ARCHITECTURE.md Source tour and how a request flows
DESIGN_NOTES.md Design guide vs. implementation: adopted, adapted, deferred, and what testing found
docs/ROADMAP.md What is still missing, ranked by how much it blocks a deployment, with a plan for each item
docs/Corestone_Roadmap.docx (PDF) The roadmap as a specification document in the format of the design guide, for sharing

License

Corestone is released under the Apache License 2.0.

About

No description, website, or topics provided.

Resources

Stars

0 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages