diff --git a/PACKAGING-REWORK.md b/PACKAGING-REWORK.md new file mode 100644 index 0000000..c06cf85 --- /dev/null +++ b/PACKAGING-REWORK.md @@ -0,0 +1,245 @@ +# SPEAR packaging rework — a company commons base (build system included), then project deltas + +Status: **draft for review**. No code changed yet. + +## 1. Goal + +The dividing line is **common across a company's projects vs specific to one +project** — not "small/public vs large/private". + +- **The base image is a company's commons.** It carries what every project of + that company needs: the shared **build system** (`infrabase` and its family), + the cross-project corpora, the common skills and rules, the standards, and the + common knowledge seed. `infrabase` belongs here — it underpins OpenCN, sye, + pos and edgem1 alike — and so do the other corpora that are not tied to a + single project. + +- **Project content is the delta.** What is specific to one project + (OpenCN's maths and algorithms, for instance) lives in that project's + `spear-private` tree, **mounted on the base at run time** by default, and + **optionally baked** into a project-pinned image later. + +- **Multi-tenant.** A company X does the same: it builds its own base from X's + commons (X's build system, X's corpora), then mounts or bakes X's projects. + The base is what makes "a private company SPEAR image" reusable across all of + that company's work. + +Because the base carries the build system and private corpora, **the base is +private**. The only redistributable image is the content-free harness. + +``` + spear:-harness SPEAR app only — the one public, GitHub-shippable image + | + spear:-base a company's commons: build system (infrabase) + common + (per company) corpora + skills + rules + standards + knowledge seed + | (private; the "company SPEAR image") + mount (default) or bake + a project's spear-private + | + project-pinned image optional: base + one project's spear-private baked in +``` + +## 2. Where we are today, and the gap + +`scripts/docker/build.sh` has three **flat** profiles built independently from +one `docker/Dockerfile`: `public` (harness + only git-tracked content, no index, +no baked corpus), `reds` (corpora named by `--bake` + cut index, HEIG registry), +`private` (everything the host has). The reusable machinery already exists: + +- **Per-corpus visibility** `"public": true` in `projects.json` + (`gen-registry.py`); unmarked ⇒ private (fail-safe). +- **Per-document provenance** `source_origin` PUBLIC | LICENSED_STANDARD + (`stage-standards.py`); unknown ⇒ licensed (fail-safe). +- **Commons vs per-user state already separated**: `/opt/spear/common` + (`SPEAR_COMMON_STATE_DIR`, read-only seed) vs `/opt/spear/state` + (`SPEAR_STATE_DIR`, writable, one run's history/memory/audit). +- **Baked trees land writable**: `COPY --from=baked --chmod=0777 . /corpora/`, + shadowed by a bind mount when the host has the tree. +- **Named, lazy build contexts** keep the 4.3 GB embedder and venv cached. + +The gap: nothing today builds a **company commons** image — harness + the shared +build system (`infrabase`) + common corpora + skills + rules + standards + +knowledge, as one reusable private image that projects then extend. `reds`/ +`private` are flat, HEIG-specific (`ch.heig-vd.reds.spear.*`), and rebuild the +whole image each time instead of `base + delta`. + +## 3. Target model + +Three links in a `FROM` chain, so each layer is reused by the next: + +| tier | name | FROM | carries | redistributable | +|---|---|---|---|---| +| 0 | `harness` | `ubuntu` | SPEAR app, venv, embedder, empty content dirs | **true** (public, GitHub) | +| 1 | `base` | `harness` | a company's commons: common corpora **and the build system `infrabase`** (writable + indexed), common skills, rules, standards, knowledge seed | **private** (carries private trees) | +| 2 | project image | `base` | base + one project's `spear-private` + project corpora, baked | false | + +- **`base` is per company** and is the main deliverable — the private "company + SPEAR image". It bakes the common corpora **including full build trees like + `infrabase`** (`ib_so3`, `ib_u-boot`, `ib_avz`), writable under `/corpora` so + the assistant can build and modify them in the container. Cross-toolchains + stay **mounted** from the host (decision 6), so the base carries the build + *tree*, the host provides the compiler. + + ``` + spear-image build base --entity edgemtech \ + --commons infrabase,ib_so3,ib_u-boot,ib_avz, \ + [--knowledge ] + ``` + +- **Project content stays out by default.** A project's `spear-private` + (OpenCN's maths, its project corpora, its rules/skills) is **mounted** on the + base at run time via `SPEAR_CONTENT_TREES` / the `spear-private` mount, so it + stays live and editable with no rebuild. Baking it is the optional tier-2 step + for a self-contained project image: + + ``` + spear-image build project --from base --tree /path/to/opencn-ng/spear-private \ + [--bake ] [--standards ] [--licensed] + ``` + + A project also adds its **own standards**: `--licensed --standards ` + bakes the licensed documents of that store over the base's. So an edgemtech + project image is the **reds base + a private standard** -- the base is one entity's + commons, the project adds another's licensed standard, and the result is a + non-redistributable project image owned by whoever built it. + +- **Redistributability is derived, not a tier.** An image is `redistributable= + true` only if every baked corpus is `public:true` and every standard is + PUBLIC. A commons base that carries `infrabase` (private) is therefore private. + A company whose commons happen to be all-public would get a public base; the + guardrails decide, the tier does not. + +## 4. Concrete changes, file by file + +- **`docker/Dockerfile` → a FROM chain of three.** + - `Dockerfile.harness`: ubuntu + apt + venv + embedder + the app code + empty + `rules.d`/`skills`/`benches`/`corpora`. Public, content-free. This is today's + Dockerfile minus the content COPYs. + - `Dockerfile.base`: `FROM ${HARNESS_IMAGE}`, then bake the commons — the + `index`, `rules`, `skills`, `benches`, `standards`, `common` (knowledge), + the baked common corpora **including `infrabase`** into `/corpora`, the + generated registry, and the base manifest. + - `Dockerfile.project`: `FROM ${BASE_IMAGE}`, then COPY one project's delta + (its `spear-private` content merged over the base, its project corpora into + `/corpora`, its registry overlay). + +- **`scripts/docker/build.sh`.** Add `--profile harness | base | project`. + - `base`: bake the corpora named `--commons` (which may be full build trees + like `infrabase`, `public:false`) + their index + common content + standards + + knowledge; label `redistributable` by the derived rule and + `entity=`. + - `project`: `FROM` the base tag, bake `--tree`'s content and `--bake` corpora. + - Keep `public`/`reds`/`private` as thin aliases for one release (migration). + +- **`gen-registry.py`.** A `base` registry = that company's commons corpora + (common set, `infrabase` included); a `project` registry overlays the + project's own corpora on top. + +- **`stage-standards.py`.** The base takes the standards its company may carry + (PUBLIC always; a company's licensed docs when it owns them); a project adds + nothing new unless it owns standards of its own. + +- **`stage-content.py` / `stage-corpora.py` / `stage-knowledge.py` / + `stage-index.py`.** Unchanged logic, invoked once for the base commons and + once for a project delta. `stage-corpora.py --bake` already copies a tree + writable into `/corpora`; `stage-index.py --bake` cuts the index to a set. + +- **`scripts/spear-image`.** New verbs: `build harness`, `build base --entity + --commons [...]`, `build project --from base --tree [...]`. `run`/`save`/ + `load`/`push` unchanged; `push` still refuses a `redistributable=false` image + without `--allow-push`. + +- **Base content manifest.** `Dockerfile.base` bakes + `/opt/spear/common/base-manifest.json` (baked corpus list incl. `infrabase`, + index build date, standards manifest); `spear-image inspect base` prints it, a + project image records the base tag it came from. + +- **Labels.** Neutral `spear.*` namespace + `spear.entity=`; + `redistributable` governs `push`. + +## 5. Redistributability guardrails + +- A corpus is baked redistributable **only** if `projects.json` marks it + `"public": true`; `infrabase` is `public:false`, so any image carrying it is + `redistributable=false`. Unmarked ⇒ private. +- A standard travels into a redistributable image **only** if `source_origin` is + PUBLIC; unknown ⇒ licensed ⇒ withheld. +- A run's own `state` (history, memories, audit) is **never** baked; only the + common seed (`/opt/spear/common`) travels, read-only. + +## 6. Layer reuse, distribution, migration + +- **Reuse.** `harness` holds the expensive venv + embedder, shared by every + base; a base holds the commons, shared by every project image. N projects of a + company cost `base + N small deltas`. +- **Distribution.** A base builds `FROM harness`, a project `FROM base`; publish + or `spear-image save`/`load` each. Tags are pinned to `spearversion.sh`. +- **Migration.** `public → harness`; `reds`/`private → base --entity ` + (a company's commons, private); project-pinned images are the new tier 2. Keep + the old words as aliases for one release. + +## 7. Decisions taken (2026-11-10) + +1. **Names.** `harness` (public, content-free) / `base` (a company's private + commons) / `project` (optional, base + a baked project delta). +2. **The base carries the build system.** `infrabase` and its family are + **commons**, baked into the base (trees **and** index, writable), not left to + a per-project image. The base/delta line is "common across the company's + projects vs specific to one project", and `infrabase` is common. +3. **A `FROM` chain**: `Dockerfile.harness` → `Dockerfile.base` → + `Dockerfile.project`, so each tier reuses the one below. +4. **Labels** renamed to neutral `spear.*` + `spear.entity=`. +5. **Base content manifest** baked into every base. +6. **Toolchains stay mounted**, not baked: the base carries the build *tree* + (`infrabase`), the host provides the cross-toolchains at run time, as + `spear-docker.sh` does today. (A later `--toolchains` flag could bake them for + a fully standalone image; out of scope for now.) +7. **Redistributability is derived** from content, not a tier: a base carrying + `infrabase` is private; only `harness` is public. + +## 8. Implementation status (branch `spear-packaging-rework`) + +A first implementation is on this branch, built to be **additive and safe**: the +legacy flat build (`build.sh`, `docker/Dockerfile`, `gen-registry.py`, +`stage-standards.py`) is **untouched**, so nothing existing breaks. The new tier +build lives in its own script. + +Added / changed: + +- `docker/Dockerfile.harness` — tier 0 (app only, empty content dirs, empty + registry, public). +- `docker/Dockerfile.base` — `FROM ${HARNESS_IMAGE}`, bakes the commons + (index, content, corpora incl. `infrabase`, standards, knowledge, manifest). +- `docker/Dockerfile.project` — `FROM ${BASE_IMAGE}`, bakes one project's delta: + its content, corpora, index overlay, and **its own standards** appended over + the base's (the reds-base + a private standard case). +- `scripts/docker/build-tiered.sh` — orchestrates `harness` / `base` / + `project`: stages via the existing `stage-*` scripts (unchanged), maps the + profile for `stage-standards.py` (base → public, or private with `--licensed`; + project → private) so no Python needed a new profile value, derives + `redistributable` from the baked corpora, and writes the manifest. +- `scripts/spear-image` — `build harness|base|project` dispatch to + `build-tiered.sh`; `run`/`inspect`/`save` resolve the new tier names. +- `scripts/docker/push.sh` — decides on `redistributable` + `entity` instead of + the profile name, reads the neutral `spear.*` labels (falling back to the old + `ch.heig-vd.reds.spear.*`), and routes a private image to its entity's own + registry (`SPEAR__REGISTRY`; `reds` → reds-gitlab, `edgemtech` → + gitlab.edgemtech). A base/project headed for its entity's registry pushes + without `--allow-push`; anywhere else still needs it. Legacy `reds`/`private` + behaviour is preserved. + +Verified: `bash -n` on all scripts, `--help`, the argument-validation paths, and +the manifest / redistributable-check Python in isolation. **Not yet run** is a +real `docker build` (it needs the configured build host — the one with +`client/machine.env`, `client/projects.json` and the index — e.g. +`/opt/llm/spear`). To validate end to end there: + +```sh +spear-image build harness +spear-image build base --commons infrabase,ib_so3,ib_u-boot,ib_avz, --entity edgemtech +spear-image build project --from spear:0.3.0-base --tree /path/to/opencn-ng/spear-private --bake +``` + +Still legacy-only and worth folding in next: a neutral `spear.*` label on the +old profiles too, and retiring `public`/`reds`/`private` once the tiers are +proven. diff --git a/docker/Dockerfile.base b/docker/Dockerfile.base new file mode 100644 index 0000000..9735a47 --- /dev/null +++ b/docker/Dockerfile.base @@ -0,0 +1,51 @@ +# A company's commons, built on the harness. +# +# It carries what every project of the company needs: the shared build system +# and corpora (infrabase and its family included), the prebuilt index, the +# common rules / skills / benches, the standards and the common knowledge seed. +# It is private whenever it bakes a non-public tree -- which it does, so it is. +# Dockerfile.project builds one project's delta on top of this. +# +# Every addition overlays the empty directory the harness created, from a named +# build context staged by scripts/docker/build.sh. + +ARG HARNESS_IMAGE +FROM ${HARNESS_IMAGE} + +ARG APP=/opt/spear/client + +# The prebuilt index, cut to the commons corpora. ChromaDB opens its sqlite +# read-write even to query, so it must be writable by whichever uid runs. + +COPY --chmod=0777 --from=index . $APP/chromadb/ + +# The common content, merged over the harness's empty directories. + +COPY --from=rules . $APP/rules.d/ +COPY --from=skills . $APP/skills/ +COPY --from=benches . $APP/benches/ + +# The commons registry, and the repo's own corpora -- the API references and the +# shared notes. A bind mount on /corpora/spear shadows these for a host that has +# the spear checkout. + +COPY --from=repo docker/projects.docker.json $APP/projects.json +COPY --from=notes . /corpora/spear/claude/ +COPY --from=repo corpora/ /corpora/spear/corpora/ + +# The baked common corpora, the build system (infrabase) among them, writable so +# the assistant can build them. Shadowed by a bind mount when the host has them. +# Toolchains are NOT baked: spear-docker.sh mounts them from the host. + +COPY --from=baked --chmod=0777 . /corpora/ + +# The common knowledge seed and the standards store (which documents travel is +# decided per document -- see scripts/docker/stage-standards.py). + +COPY --from=common . /opt/spear/common/ +COPY --from=standards --chmod=0777 . /opt/spear/common/standards/ + +# What this base contains, for the companies and projects that build on it: the +# baked corpus list, the index build date and the standards manifest. + +COPY --from=manifest base-manifest.json /opt/spear/common/base-manifest.json diff --git a/docker/Dockerfile.harness b/docker/Dockerfile.harness new file mode 100644 index 0000000..5557962 --- /dev/null +++ b/docker/Dockerfile.harness @@ -0,0 +1,100 @@ +# SPEAR harness — the application only, no content. Public and shippable. +# +# First link of the FROM chain: Dockerfile.base builds a company's commons on +# top of this, and Dockerfile.project bakes one project's delta on top of that. +# This image carries the venv, the embedder and the harness code -- the +# expensive layers every image above reuses -- and nothing a deployment could +# not publish: no index, no corpus, no standards, no rules or skills content. +# +# Default build context is client/ (the harness); `repo` is a named context for +# the two files that live outside it. See scripts/docker/build.sh. + +FROM ubuntu:24.04 + +ENV DEBIAN_FRONTEND=noninteractive + +# bubblewrap is not optional: the harness runs every command in it. The build +# tools are there because a sandboxed command is often `make`. The second block +# is bitbake's host prerequisites; without a generated en_US.UTF-8 bitbake +# refuses to start. + +RUN apt-get update && apt-get install -y --no-install-recommends \ + python3 python3-venv python3-dev \ + bubblewrap ca-certificates curl git openssh-client \ + build-essential cmake ninja-build pkg-config \ + ripgrep less file \ + locales \ + gawk diffstat chrpath socat zstd lz4 texinfo \ + python3-pexpect python3-git python3-jinja2 \ + bison flex bc cpio unzip rsync device-tree-compiler u-boot-tools \ + && sed -i "s/^# *en_US.UTF-8/en_US.UTF-8/" /etc/locale.gen \ + && locale-gen en_US.UTF-8 \ + && rm -rf /var/lib/apt/lists/* + +ARG APP=/opt/spear/client +WORKDIR $APP + +# Dependencies first; they change far less often than the harness code. CPU +# torch before sentence-transformers, so pip never pulls the CUDA build. + +COPY deploy/requirements.txt deploy/requirements.txt +RUN python3 -m venv "$APP" \ + && "$APP/bin/pip" install --no-cache-dir --upgrade pip \ + && "$APP/bin/pip" install --no-cache-dir \ + --index-url https://download.pytorch.org/whl/cpu torch \ + && "$APP/bin/pip" install --no-cache-dir -r deploy/requirements.txt + +# The embedder (bge-m3) and Chroma's default MiniLM, baked at build time so a +# fresh container neither downloads 4.3 GB on first run nor fails offline. + +ENV HF_HOME=/opt/spear/hf +ENV HF_HUB_DISABLE_XET=1 +RUN "$APP/bin/python" -c "\ +from sentence_transformers import SentenceTransformer; \ +SentenceTransformer('BAAI/bge-m3')" \ + && find /opt/spear/hf -name '*.onnx' -delete 2>/dev/null || true + +RUN HOME=/opt/spear/chroma-home "$APP/bin/python" -c "\ +from chromadb.utils.embedding_functions import ONNXMiniLM_L6_V2; \ +ONNXMiniLM_L6_V2()(['warm'])" \ + && chmod -R a+rX /opt/spear/chroma-home + +# The harness code. + +COPY *.md $APP/ +COPY agent/ $APP/agent/ +COPY cli/ $APP/cli/ +COPY context/ $APP/context/ +COPY evidence/ $APP/evidence/ +COPY harness/ $APP/harness/ +COPY models/ $APP/models/ +COPY normative/ $APP/normative/ +COPY retrieval/ $APP/retrieval/ +COPY runtime/ $APP/runtime/ +COPY standard/ $APP/standard/ +COPY training/ $APP/training/ +COPY eval/ $APP/eval/ +COPY deploy/ $APP/deploy/ + +# Empty content, corpus and state directories, and an empty registry, so the +# harness finds a directory rather than nothing. A base image fills them. + +RUN mkdir -p "$APP/rules.d" "$APP/skills" "$APP/benches" "$APP/chromadb" \ + /corpora /opt/spear/common/standards /opt/spear/state \ + && printf '{}\n' > "$APP/projects.json" \ + && chmod -R 0777 /corpora /opt/spear/common /opt/spear/state "$APP/chromadb" + +# The launcher a recipient copies out to run the image, and the entrypoint. +# Both live in the repo, not in client/, so they come from the `repo` context. + +COPY --from=repo scripts/docker/spear-run.sh /opt/spear/spear-run.sh +COPY --from=repo docker/entrypoint.sh /opt/spear/entrypoint.sh +RUN chmod +x /opt/spear/entrypoint.sh /opt/spear/spear-run.sh + +ENV SPEAR_COMMON_STATE_DIR=/opt/spear/common +ENV SPEAR_STANDARDS_ROOT=/opt/spear/common/standards +ENV SPEAR_STATE_DIR=/opt/spear/state +ENV SPEAR_CORPUS_ROOT=/corpora +ENV PATH=$APP/bin:$PATH +WORKDIR /corpora +ENTRYPOINT ["/opt/spear/entrypoint.sh"] diff --git a/docker/Dockerfile.project b/docker/Dockerfile.project new file mode 100644 index 0000000..4fb192a --- /dev/null +++ b/docker/Dockerfile.project @@ -0,0 +1,41 @@ +# One project's delta on a company's base. +# +# It merges a project's spear-private content over the commons and bakes the +# project's own corpora, so the image is self-contained for that project. Always +# non-redistributable. The default is NOT to build this at all: a project's +# spear-private is mounted on the base at run time, and this image exists only +# when a self-contained, nothing-mounted project image is wanted. +# +# The index context is the FULL index for the commons plus this project's +# corpora (stage-index.py bakes it), so it replaces the base's rather than +# merging into it. + +ARG BASE_IMAGE +FROM ${BASE_IMAGE} + +ARG APP=/opt/spear/client + +# The project's content, merged over the base's commons. + +COPY --from=rules . $APP/rules.d/ +COPY --from=skills . $APP/skills/ +COPY --from=benches . $APP/benches/ + +# The registry overlay (commons + project corpora) and the full index. + +COPY --from=repo docker/projects.docker.json $APP/projects.json +COPY --chmod=0777 --from=index . $APP/chromadb/ + +# The project's baked corpora, writable, at their registry-resolved paths. + +COPY --from=baked --chmod=0777 . /corpora/ + +# The project's own standards, appended over the base's -- a licensed document +# this project is entitled to (a private/commercial standard). Empty unless the build +# was given --standards / --licensed. + +COPY --from=standards --chmod=0777 . /opt/spear/common/standards/ + +# This image's own manifest, recording the base it was built on and its delta. + +COPY --from=manifest project-manifest.json /opt/spear/common/project-manifest.json diff --git a/scripts/docker/build-tiered.sh b/scripts/docker/build-tiered.sh new file mode 100755 index 0000000..5af9499 --- /dev/null +++ b/scripts/docker/build-tiered.sh @@ -0,0 +1,261 @@ +#!/bin/bash +# Build the tiered images: harness -> base -> project. +# +# build-tiered.sh harness +# build-tiered.sh base --commons a,b,c [--entity NAME] [--licensed] +# [--knowledge p,q|all] +# build-tiered.sh project --from spear:-base --tree DIR +# [--bake a,b] [--standards DIR] [--licensed] [--tag TAG] +# +# A project adds its own standards too: --licensed takes the licensed documents +# of --standards DIR (a store holding, say, only one private standard), appended over the +# base's. So an edgemtech project image = the reds base + a private standard. +# +# This is the FROM chain of PACKAGING-REWORK.md. The legacy flat profiles +# (public|reds|private) stay in build.sh, untouched; this script only knows the +# three new tiers and leaves everything else alone. +# +# It reuses the staging scripts as they are. gen-registry.py takes any non- +# "public" profile as "the whole map", and stage-standards.py's profile is +# mapped here (base -> public, or private with --licensed; project -> private), +# so neither needs a new profile value. + +set -e + +REPO="$(cd "$(dirname "$0")/../.." && pwd)" +[ -r "$REPO/scripts/common/banner.sh" ] && . "$REPO/scripts/common/banner.sh" +HERE="$REPO/scripts/docker" +APP="${SPEAR_APP:-$REPO/client}" +VERSION="$("$REPO/scripts/spearversion.sh")" +LABEL=spear + +die() { echo "build-tiered: $*" >&2; exit 1; } + +TIER="${1:-}"; shift || true +case "$TIER" in + harness|base|project) ;; + ""|-h|--help) sed -n '2,20p' "$0" | sed 's/^# \{0,1\}//'; exit 0 ;; + *) die "tier must be harness, base or project, not '$TIER'" ;; +esac + +COMMONS="" BAKE="" KNOWLEDGE="" ENTITY="local" FROM_BASE="" TAG="" +STANDARDS_STORE="" +LICENSED=0 +TREES=() + +while [ $# -gt 0 ]; do + case "$1" in + --commons) COMMONS="$2"; shift 2 ;; + --bake) BAKE="$2"; shift 2 ;; + --knowledge) KNOWLEDGE="$2"; shift 2 ;; + --entity) ENTITY="$2"; shift 2 ;; + --from) FROM_BASE="$2"; shift 2 ;; + --tree) TREES+=("$2"); shift 2 ;; + --standards) STANDARDS_STORE="$2"; shift 2 ;; + --licensed) LICENSED=1; shift ;; + --tag) TAG="$2"; shift 2 ;; + *) die "unknown option: $1" ;; + esac +done + +[ -f "$APP/cli/rag_chat.py" ] || die "no harness under $APP — set SPEAR_APP" + +# The machine's own content locations and content trees, exactly as build.sh and +# the launchers read them. Without it a base ships no rules and no skills. + +if [ -r "$APP/machine.env" ]; then + echo " settings $APP/machine.env" + # shellcheck disable=SC1091 + . "$APP/machine.env" +fi + +EMPTY="$(mktemp -d)" +STAGE="$(mktemp -d)" +trap 'rm -rf "$EMPTY" "$STAGE"' EXIT + +exists() { docker image inspect "$1" >/dev/null 2>&1; } + +# redistributable=true only if every baked corpus is public:true and no licensed +# standard travels. Reads projects.json; unknown or unmarked corpus => private. + +all_public() { + local list="$1" + [ -n "$list" ] || return 0 + python3 -I - "$APP/projects.json" "$list" <<'PY' +import json, sys +reg, names = sys.argv[1], [n for n in sys.argv[2].split(",") if n] +try: + data = json.load(open(reg)) +except Exception: + sys.exit(1) +for n in names: + s = data.get(n) + if not isinstance(s, dict) or s.get("public") is not True: + sys.exit(1) +sys.exit(0) +PY +} + +write_manifest() { + # $1 = out file, $2 = corpus list + local out="$1" corpora="$2" + mkdir -p "$(dirname "$out")" + python3 -I - "$out" "$corpora" <&2 + + IDX="${SPEAR_INDEX_DIR:-$APP/chromadb}" + if [ -d "$IDX" ]; then + "$HERE/stage-index.py" --bake "$COMMONS" --index "$IDX" "$STAGE/index" + else + echo " index (absent — base carries no prebuilt index)"; mkdir -p "$STAGE/index" + fi + + CONTENT=() + [ -n "${SPEAR_RULES_DIR:-}" ] && CONTENT+=(--rules "$SPEAR_RULES_DIR") + [ -n "${SPEAR_SKILLS_DIR:-}" ] && CONTENT+=(--skills "$SPEAR_SKILLS_DIR") + [ -n "${SPEAR_BENCH_DIR:-}" ] && CONTENT+=(--benches "$SPEAR_BENCH_DIR") + if [ -n "${SPEAR_CONTENT_TREES:-}" ]; then + IFS=: read -r -a _t <<<"$SPEAR_CONTENT_TREES" + for t in "${_t[@]}"; do [ -d "$t" ] && CONTENT+=(--tree "$t"); done + fi + "$HERE/stage-content.py" --bake "$COMMONS" "${CONTENT[@]}" "$STAGE/content" + + STD_PROFILE=public; [ "$LICENSED" = 1 ] && STD_PROFILE=private + "$HERE/stage-standards.py" --profile "$STD_PROFILE" "$STAGE/standards" + + SPEAR_COMMON_STATE_DIR="${SPEAR_COMMON_STATE_DIR:-}" \ + "$HERE/stage-knowledge.py" --workspaces "$KNOWLEDGE" "$STAGE/common" + + "$HERE/stage-corpora.py" --bake "$COMMONS" --restrict-registry "$STAGE/baked" || true + + NOTES="${SPEAR_NOTES_DIR:-$REPO/claude}"; [ -d "$NOTES" ] || NOTES="$EMPTY" + + write_manifest "$STAGE/manifest/base-manifest.json" "$COMMONS" + + REDIST=false + if [ "$LICENSED" = 0 ] && all_public "$COMMONS"; then REDIST=true; fi + + echo "== building $TAG (base, entity=$ENTITY, redistributable=$REDIST) ==" + docker build -f "$REPO/docker/Dockerfile.base" \ + --build-arg "HARNESS_IMAGE=$HARNESS" \ + --build-context "repo=$REPO" \ + --build-context "index=$STAGE/index" \ + --build-context "rules=$STAGE/content/rules.d" \ + --build-context "skills=$STAGE/content/skills" \ + --build-context "benches=$STAGE/content/benches" \ + --build-context "notes=$NOTES" \ + --build-context "standards=$STAGE/standards" \ + --build-context "common=$STAGE/common" \ + --build-context "baked=$STAGE/baked" \ + --build-context "manifest=$STAGE/manifest" \ + --label "$LABEL.profile=base" \ + --label "$LABEL.entity=$ENTITY" \ + --label "$LABEL.built=$(date -Iseconds)" \ + --label "$LABEL.redistributable=$REDIST" \ + -t "$TAG" "$APP" + echo "built $TAG (base)" + ;; + +project) + [ -n "$FROM_BASE" ] || FROM_BASE="spear:$VERSION-base" + exists "$FROM_BASE" || die "$FROM_BASE not found — build the base first: spear-image build base ..." + [ "${#TREES[@]}" -gt 0 ] || die "project needs --tree DIR (the project's spear-private content)" + TAG="${TAG:-spear:$VERSION-project}" + HARNESS="" + + "$HERE/gen-registry.py" --profile private || echo " (some corpora skipped)" >&2 + + # The index is the FULL set the base already has plus this project's corpora, + # so it replaces the base's rather than merging into it. + ALLBAKE="$BAKE" + IDX="${SPEAR_INDEX_DIR:-$APP/chromadb}" + if [ -d "$IDX" ] && [ -n "$ALLBAKE" ]; then + "$HERE/stage-index.py" --bake "$ALLBAKE" --index "$IDX" "$STAGE/index" + else + mkdir -p "$STAGE/index" + fi + + CONTENT=() + for t in "${TREES[@]}"; do [ -d "$t" ] && CONTENT+=(--tree "$t") || die "not a directory: $t"; done + "$HERE/stage-content.py" --bake "$BAKE" "${CONTENT[@]}" "$STAGE/content" + + if [ -n "$BAKE" ]; then + "$HERE/stage-corpora.py" --bake "$BAKE" --restrict-registry "$STAGE/baked" || true + else + mkdir -p "$STAGE/baked" + fi + + # The project's own standards: a licensed document it is entitled to + # (--licensed), from its own store (--standards), appended over the base's. + # Empty otherwise, so the base's standards are kept untouched. + + if [ -n "$STANDARDS_STORE" ] || [ "$LICENSED" = 1 ]; then + STD_PROFILE=public; [ "$LICENSED" = 1 ] && STD_PROFILE=private + STORE_ARG=(); [ -n "$STANDARDS_STORE" ] && STORE_ARG=(--store "$STANDARDS_STORE") + "$HERE/stage-standards.py" --profile "$STD_PROFILE" "${STORE_ARG[@]}" "$STAGE/standards" + else + mkdir -p "$STAGE/standards" + fi + + write_manifest "$STAGE/manifest/project-manifest.json" "$BAKE" + + echo "== building $TAG (project, FROM $FROM_BASE) ==" + docker build -f "$REPO/docker/Dockerfile.project" \ + --build-arg "BASE_IMAGE=$FROM_BASE" \ + --build-context "repo=$REPO" \ + --build-context "index=$STAGE/index" \ + --build-context "rules=$STAGE/content/rules.d" \ + --build-context "skills=$STAGE/content/skills" \ + --build-context "benches=$STAGE/content/benches" \ + --build-context "baked=$STAGE/baked" \ + --build-context "standards=$STAGE/standards" \ + --build-context "manifest=$STAGE/manifest" \ + --label "$LABEL.profile=project" \ + --label "$LABEL.entity=$ENTITY" \ + --label "$LABEL.built=$(date -Iseconds)" \ + --label "$LABEL.redistributable=false" \ + -t "$TAG" "$APP" + echo "built $TAG (project)" + ;; +esac diff --git a/scripts/docker/push.sh b/scripts/docker/push.sh index 440532f..8b33aa3 100755 --- a/scripts/docker/push.sh +++ b/scripts/docker/push.sh @@ -1,24 +1,20 @@ #!/bin/bash # Publish an image, unless it is one that must not be published. # -# The two profiles differ in exactly one way that matters once the image -# exists: whether its contents may leave. A tag does not carry that -- it gets -# retyped, shortened and reused -- so the answer is a LABEL, written at build -# time, travelling with the bytes through `docker save`, a registry and back. +# Whether an image's contents may leave is a LABEL written at build time, not +# the tag -- a tag gets retyped, shortened and reused. `redistributable=true` +# (the harness, a public image) goes anywhere. A private image (a base or a +# project, or a legacy reds/private) goes to its entity's own registry, and +# anywhere else only on --allow-push. # -# scripts/docker/push.sh ghcr.io//spear:-public -# scripts/docker/push.sh reds-gitlab.heig-vd.ch:5050//spear:-reds -# scripts/docker/push.sh --allow-push ghcr.io//spear-private:-private +# scripts/docker/push.sh ghcr.io//spear:-harness +# scripts/docker/push.sh reds-gitlab.heig-vd.ch:5050//spear:-base +# scripts/docker/push.sh --allow-push //spear:-project # -# A reds image goes to the institute's registry and nowhere else without -# --allow-push: the image name must start with that host (any port), and -# SPEAR_REDS_REGISTRY overrides the default host. -# -# --allow-push is not a formality. A private image carries a licensed -# normative corpus -- the original document included, where the store retained -# it -- and a customer's source tree. Publishing one is a decision about a -# licence and a contract, not about a registry, and it should cost a -# deliberate word. +# An entity's registry is SPEAR__REGISTRY, with reds and edgemtech +# defaulted. --allow-push is not a formality: a private image carries a build +# system, a customer's source tree, or a licensed normative corpus, and +# publishing one is a decision about a licence and a contract. set -e # The release this is: "[spear vX.Y.Z]