From 190423f9fc1e34d9860902cc34bc8c97d3b07dc2 Mon Sep 17 00:00:00 2001 From: Daniel Rossier Date: Mon, 16 Nov 2026 08:28:04 +0100 Subject: [PATCH] spear-image: add get/update lifecycle verbs for base and project `get ` fetches a tier's image without the caller knowing where it lives -- the registry (SPEAR_REGISTRY_IMAGE) first, else a shared tarball directory (SPEAR_SHARE_DIR, spear--.tar.zst). `update ` rebuilds from local.conf, refreshes that tarball and tries to push; re-indexing stays a separate, deliberate step. Both are option-free -- client/local.conf holds every setting. Document the two verbs and the two new settings in docker/README.md, the container chapter and local.conf.example. --- client/local.conf.example | 15 ++++++ doc/source/start/container.rst | 32 ++++++++++++ docker/README.md | 32 ++++++++++++ scripts/spear-image | 91 ++++++++++++++++++++++++++++++++++ 4 files changed, 170 insertions(+) diff --git a/client/local.conf.example b/client/local.conf.example index 3ebf03a..929afdb 100644 --- a/client/local.conf.example +++ b/client/local.conf.example @@ -82,3 +82,18 @@ export SPEAR_BENCH_DIR=/path/to/spear-private/benches # # export SPEAR__REGISTRY= #export SPEAR_ACME_REGISTRY=registry.example.com + +# ========================== get / update ================================= +# `spear-image get ` fetches an image without your having to know +# where it lives; `spear-image update ` rebuilds it from the +# settings above, refreshes the shared tarball and tries to push. Two paths let +# them find an image, tried in this order: +# +# SPEAR_REGISTRY_IMAGE -- the full registry repository, no tag: what get pulls +# from and update pushes to. Unset => the tarball is the only path. +#export SPEAR_REGISTRY_IMAGE=reds-gitlab.heig-vd.ch:5050//spear/spear +# +# SPEAR_SHARE_DIR -- a directory of tarballs a colleague can reach, each named +# spear--.tar.zst: get loads from it when the registry misses, +# update writes it. Defaults to the SPEAR tree's parent. +#export SPEAR_SHARE_DIR=/path/to/shared/dir diff --git a/doc/source/start/container.rst b/doc/source/start/container.rst index cc792d0..c013542 100644 --- a/doc/source/start/container.rst +++ b/doc/source/start/container.rst @@ -55,6 +55,38 @@ Only ``scripts/`` goes on ``PATH``. ``scripts/docker/`` holds a ``build.sh``, and a shell that has also sourced another tree's ``env.sh`` with a ``build.sh`` of its own would then have two. +Base and project: get and update +--------------------------------- + +The tiered images — ``harness`` → ``base`` → ``project`` — have two +option-free verbs over ``build``, so that neither you nor a colleague has to +remember where an image lives or which flags built it. Everything they need is +in ``client/local.conf``. + +.. code-block:: console + + $ spear-image get base # pull from the registry, else load the shared tarball + $ spear-image run base --reds # start a session against the model endpoint + $ spear-image update base # rebuild from local.conf, refresh the tarball, try to push + $ spear-image get project # base + a deployment's spear-private + $ spear-image update project + +``get`` is idempotent: an image already present is left alone. It tries the +registry first (``SPEAR_REGISTRY_IMAGE``, the full repository path), then a +directory of tarballs a colleague can reach (``SPEAR_SHARE_DIR``, default the +tree's parent), each named ``spear--.tar.zst``. + +``update`` **rebuilds only** — from the contents ``local.conf`` names +(``SPEAR_COMMONS`` for the base, ``SPEAR_PROJECT_TREES`` and ``SPEAR_BAKE`` for +the project) — then refreshes that tarball and, if a registry is set, tries to +push. Re-indexing a corpus is a separate, deliberate step +(`Refreshing the index`_): it takes hours, and a rebuild is not the place for +it. ``update base`` builds the harness first if it is missing; ``update +project`` needs the base present and says so if it is not. + +The lower-level ``build`` commands, and the flat ``public``/``reds``/``private`` +profiles, follow. + .. code-block:: console $ spear-image build private # spear:-private diff --git a/docker/README.md b/docker/README.md index 006ac29..6ee8840 100644 --- a/docker/README.md +++ b/docker/README.md @@ -18,6 +18,38 @@ and have the same state you do. | corpus trees | **mounted**, or baked with `--bake` | working copies change daily; baked is for the container that is handed over with nothing to mount | | model weights | **neither** | the harness talks to an endpoint (`--endpoint`), it does not host a model | +## Get, run, update + +The two commands a colleague needs, and the one a maintainer adds. None takes +an option: `client/local.conf` holds the settings — the registry, the shared +tarball directory, and what each tier bakes. + +```sh +. ./env.sh # once per shell, from the tree root + +spear-image get base # pull from the registry, else load the shared tarball +spear-image run base --reds # start a session against the model endpoint + +spear-image update base # rebuild from local.conf, refresh the tarball, try to push +spear-image get project # the project image: base + a deployment's spear-private +spear-image update project +``` + +`get` is idempotent — an image already present is left alone. `update` rebuilds +**only**; re-indexing a corpus stays a separate, deliberate step (`spear-index` +on the host, then `update`), because it takes hours and is not what a rebuild +is for. + +Two settings in `local.conf` say where an image lives, tried in this order: + +- `SPEAR_REGISTRY_IMAGE` — the registry repository `get` pulls from and + `update` pushes to. Unset, both fall back to the tarball. +- `SPEAR_SHARE_DIR` — a directory of `spear--.tar.zst` tarballs + a colleague can reach. Defaults to the SPEAR tree's parent. + +The tiers themselves — `harness` → `base` → `project` — and what each carries +are below under the lower-level `build` commands. + ## Managing images From the repository root, once per shell: diff --git a/scripts/spear-image b/scripts/spear-image index 428712b..1ad504d 100755 --- a/scripts/spear-image +++ b/scripts/spear-image @@ -4,6 +4,8 @@ # spear-image build ... (tiered, see build-tiered.sh) # spear-image build [--bake a,b,...] [--knowledge a,b,...|all] # [--tag TAG] (legacy flat profiles) +# spear-image get (pull from the registry, else load the shared tarball) +# spear-image update (rebuild from local.conf, refresh the tarball, try to push) # spear-image list # spear-image inspect # spear-image run [spear-docker options] [-- harness args] @@ -56,6 +58,18 @@ image_of() { exists() { docker image inspect "$1" >/dev/null 2>&1; } +# Where images travel when a registry cannot or will not take them: a directory +# of tarballs a colleague can reach (SPEAR_SHARE_DIR, default the SPEAR tree's +# parent), each named spear--.tar.zst. +SHARE_DIR="${SPEAR_SHARE_DIR:-$(cd "$HERE/../.." && pwd)}" + +# The registry repository to pull from and push to -- the full path, no tag +# (e.g. reds-gitlab.heig-vd.ch:5050/reds-general/spear/spear). Left unset, get +# goes straight to the tarball and update does not push. +REGISTRY_IMAGE="${SPEAR_REGISTRY_IMAGE:-}" + +tarball_of() { printf '%s/spear-%s-%s.tar.zst' "$SHARE_DIR" "$1" "$VERSION"; } + # Labels moved to the neutral `spear.` namespace (build-tiered.sh); read it # first, falling back to the old one so images built before the rename still # show their profile. @@ -100,6 +114,83 @@ case "$action" in exec "$DOCKER_SCRIPTS/build.sh" ${tag:+"$tag"} --profile "$profile" "${bake[@]}" \ "${knowledge[@]}" ;; + get) + # Obtain a tier's image without having to know where it lives: the + # registry if one is set and reachable, otherwise the shared tarball. + tier="$1" + case "$tier" in + harness|base|project) ;; + "") die "get needs a tier: base or project" ;; + *) die "get takes a tier (base or project), not '$tier'" ;; + esac + tag="spear:$VERSION-$tier" + + if exists "$tag"; then echo "already present: $tag"; exit 0; fi + + if [ -n "$REGISTRY_IMAGE" ]; then + ref="$REGISTRY_IMAGE:$VERSION-$tier" + echo "pulling $ref ..." + if docker pull "$ref"; then + docker tag "$ref" "$tag" + echo "got $tag (from $ref)"; exit 0 + fi + echo "registry pull failed; trying the shared tarball" >&2 + fi + + tb="$(tarball_of "$tier")" + if [ -f "$tb" ]; then + echo "loading $tb ..." + zstd -dc "$tb" | docker load + exit 0 + fi + + die "cannot get $tag: no registry hit and no tarball at $tb. +Set SPEAR_REGISTRY_IMAGE or SPEAR_SHARE_DIR in client/local.conf, or put the tarball there." ;; + + update) + # Rebuild a tier from this host's current state (local.conf drives the + # contents -- no flags), then refresh the shared tarball and, if a + # registry is set, try to push. Indexing stays a separate, explicit step. + tier="$1" + case "$tier" in + base|project) ;; + "") die "update needs a tier: base or project" ;; + *) die "update takes base or project, not '$tier'" ;; + esac + tag="spear:$VERSION-$tier" + + case "$tier" in + base) + harness="spear:$VERSION-harness" + exists "$harness" || "$DOCKER_SCRIPTS/build-tiered.sh" harness + "$DOCKER_SCRIPTS/build-tiered.sh" base ;; + project) + exists "spear:$VERSION-base" \ + || die "no base image yet -- run 'spear-image get base' or 'spear-image update base' first" + "$DOCKER_SCRIPTS/build-tiered.sh" project ;; + esac + + command -v zstd >/dev/null || die "zstd is not installed" + tb="$(tarball_of "$tier")" + mkdir -p "$SHARE_DIR" + echo "refreshing $tb ..." + docker save "$tag" | zstd -T0 -19 -f -o "$tb" + [ "$(label "$tag" redistributable)" = false ] && echo \ + "!! $tb carries licensed and customer material: not redistributable" >&2 + + if [ -n "$REGISTRY_IMAGE" ]; then + ref="$REGISTRY_IMAGE:$VERSION-$tier" + docker tag "$tag" "$ref" + echo "pushing $ref ..." + if "$DOCKER_SCRIPTS/push.sh" "$ref"; then + echo "pushed $ref" + else + echo "registry push failed -- the tarball at $tb is the usable copy" >&2 + fi + fi + + echo "updated $tag" ;; + list) # The label, not the repository name: an image retagged for a registry # is still one of ours, and says which profile it is. Both namespaces: