Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
15 changes: 15 additions & 0 deletions client/local.conf.example
Original file line number Diff line number Diff line change
Expand Up @@ -82,3 +82,18 @@ export SPEAR_BENCH_DIR=/path/to/spear-private/benches
#
# export SPEAR_<ENTITY>_REGISTRY=<registry-host>
#export SPEAR_ACME_REGISTRY=registry.example.com

# ========================== get / update =================================
# `spear-image get <base|project>` fetches an image without your having to know
# where it lives; `spear-image update <base|project>` rebuilds it from the
# settings above, refreshes the shared tarball and tries to push. Two paths let
# them find an image, tried in this order:
#
# SPEAR_REGISTRY_IMAGE -- the full registry repository, no tag: what get pulls
# from and update pushes to. Unset => the tarball is the only path.
#export SPEAR_REGISTRY_IMAGE=reds-gitlab.heig-vd.ch:5050/<group>/spear/spear
#
# SPEAR_SHARE_DIR -- a directory of tarballs a colleague can reach, each named
# spear-<tier>-<version>.tar.zst: get loads from it when the registry misses,
# update writes it. Defaults to the SPEAR tree's parent.
#export SPEAR_SHARE_DIR=/path/to/shared/dir
32 changes: 32 additions & 0 deletions doc/source/start/container.rst
Original file line number Diff line number Diff line change
Expand Up @@ -55,6 +55,38 @@ Only ``scripts/`` goes on ``PATH``. ``scripts/docker/`` holds a ``build.sh``,
and a shell that has also sourced another tree's ``env.sh`` with a
``build.sh`` of its own would then have two.

Base and project: get and update
---------------------------------

The tiered images — ``harness`` → ``base`` → ``project`` — have two
option-free verbs over ``build``, so that neither you nor a colleague has to
remember where an image lives or which flags built it. Everything they need is
in ``client/local.conf``.

.. code-block:: console

$ spear-image get base # pull from the registry, else load the shared tarball
$ spear-image run base --reds # start a session against the model endpoint
$ spear-image update base # rebuild from local.conf, refresh the tarball, try to push
$ spear-image get project # base + a deployment's spear-private
$ spear-image update project

``get`` is idempotent: an image already present is left alone. It tries the
registry first (``SPEAR_REGISTRY_IMAGE``, the full repository path), then a
directory of tarballs a colleague can reach (``SPEAR_SHARE_DIR``, default the
tree's parent), each named ``spear-<tier>-<version>.tar.zst``.

``update`` **rebuilds only** — from the contents ``local.conf`` names
(``SPEAR_COMMONS`` for the base, ``SPEAR_PROJECT_TREES`` and ``SPEAR_BAKE`` for
the project) — then refreshes that tarball and, if a registry is set, tries to
push. Re-indexing a corpus is a separate, deliberate step
(`Refreshing the index`_): it takes hours, and a rebuild is not the place for
it. ``update base`` builds the harness first if it is missing; ``update
project`` needs the base present and says so if it is not.

The lower-level ``build`` commands, and the flat ``public``/``reds``/``private``
profiles, follow.

.. code-block:: console

$ spear-image build private # spear:<version>-private
Expand Down
32 changes: 32 additions & 0 deletions docker/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,38 @@ and have the same state you do.
| corpus trees | **mounted**, or baked with `--bake` | working copies change daily; baked is for the container that is handed over with nothing to mount |
| model weights | **neither** | the harness talks to an endpoint (`--endpoint`), it does not host a model |

## Get, run, update

The two commands a colleague needs, and the one a maintainer adds. None takes
an option: `client/local.conf` holds the settings — the registry, the shared
tarball directory, and what each tier bakes.

```sh
. ./env.sh # once per shell, from the tree root

spear-image get base # pull from the registry, else load the shared tarball
spear-image run base --reds # start a session against the model endpoint

spear-image update base # rebuild from local.conf, refresh the tarball, try to push
spear-image get project # the project image: base + a deployment's spear-private
spear-image update project
```

`get` is idempotent — an image already present is left alone. `update` rebuilds
**only**; re-indexing a corpus stays a separate, deliberate step (`spear-index`
on the host, then `update`), because it takes hours and is not what a rebuild
is for.

Two settings in `local.conf` say where an image lives, tried in this order:

- `SPEAR_REGISTRY_IMAGE` — the registry repository `get` pulls from and
`update` pushes to. Unset, both fall back to the tarball.
- `SPEAR_SHARE_DIR` — a directory of `spear-<tier>-<version>.tar.zst` tarballs
a colleague can reach. Defaults to the SPEAR tree's parent.

The tiers themselves — `harness` → `base` → `project` — and what each carries
are below under the lower-level `build` commands.

## Managing images

From the repository root, once per shell:
Expand Down
91 changes: 91 additions & 0 deletions scripts/spear-image
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,8 @@
# spear-image build <harness|base|project> ... (tiered, see build-tiered.sh)
# spear-image build <public|reds|private> [--bake a,b,...] [--knowledge a,b,...|all]
# [--tag TAG] (legacy flat profiles)
# spear-image get <base|project> (pull from the registry, else load the shared tarball)
# spear-image update <base|project> (rebuild from local.conf, refresh the tarball, try to push)
# spear-image list
# spear-image inspect <public|reds|private|TAG>
# spear-image run <public|reds|private|TAG> [spear-docker options] [-- harness args]
Expand Down Expand Up @@ -56,6 +58,18 @@ image_of() {

exists() { docker image inspect "$1" >/dev/null 2>&1; }

# Where images travel when a registry cannot or will not take them: a directory
# of tarballs a colleague can reach (SPEAR_SHARE_DIR, default the SPEAR tree's
# parent), each named spear-<tier>-<version>.tar.zst.
SHARE_DIR="${SPEAR_SHARE_DIR:-$(cd "$HERE/../.." && pwd)}"

# The registry repository to pull from and push to -- the full path, no tag
# (e.g. reds-gitlab.heig-vd.ch:5050/reds-general/spear/spear). Left unset, get
# goes straight to the tarball and update does not push.
REGISTRY_IMAGE="${SPEAR_REGISTRY_IMAGE:-}"

tarball_of() { printf '%s/spear-%s-%s.tar.zst' "$SHARE_DIR" "$1" "$VERSION"; }

# Labels moved to the neutral `spear.` namespace (build-tiered.sh); read it
# first, falling back to the old one so images built before the rename still
# show their profile.
Expand Down Expand Up @@ -100,6 +114,83 @@ case "$action" in
exec "$DOCKER_SCRIPTS/build.sh" ${tag:+"$tag"} --profile "$profile" "${bake[@]}" \
"${knowledge[@]}" ;;

get)
# Obtain a tier's image without having to know where it lives: the
# registry if one is set and reachable, otherwise the shared tarball.
tier="$1"
case "$tier" in
harness|base|project) ;;
"") die "get needs a tier: base or project" ;;
*) die "get takes a tier (base or project), not '$tier'" ;;
esac
tag="spear:$VERSION-$tier"

if exists "$tag"; then echo "already present: $tag"; exit 0; fi

if [ -n "$REGISTRY_IMAGE" ]; then
ref="$REGISTRY_IMAGE:$VERSION-$tier"
echo "pulling $ref ..."
if docker pull "$ref"; then
docker tag "$ref" "$tag"
echo "got $tag (from $ref)"; exit 0
fi
echo "registry pull failed; trying the shared tarball" >&2
fi

tb="$(tarball_of "$tier")"
if [ -f "$tb" ]; then
echo "loading $tb ..."
zstd -dc "$tb" | docker load
exit 0
fi

die "cannot get $tag: no registry hit and no tarball at $tb.
Set SPEAR_REGISTRY_IMAGE or SPEAR_SHARE_DIR in client/local.conf, or put the tarball there." ;;

update)
# Rebuild a tier from this host's current state (local.conf drives the
# contents -- no flags), then refresh the shared tarball and, if a
# registry is set, try to push. Indexing stays a separate, explicit step.
tier="$1"
case "$tier" in
base|project) ;;
"") die "update needs a tier: base or project" ;;
*) die "update takes base or project, not '$tier'" ;;
esac
tag="spear:$VERSION-$tier"

case "$tier" in
base)
harness="spear:$VERSION-harness"
exists "$harness" || "$DOCKER_SCRIPTS/build-tiered.sh" harness
"$DOCKER_SCRIPTS/build-tiered.sh" base ;;
project)
exists "spear:$VERSION-base" \
|| die "no base image yet -- run 'spear-image get base' or 'spear-image update base' first"
"$DOCKER_SCRIPTS/build-tiered.sh" project ;;
esac

command -v zstd >/dev/null || die "zstd is not installed"
tb="$(tarball_of "$tier")"
mkdir -p "$SHARE_DIR"
echo "refreshing $tb ..."
docker save "$tag" | zstd -T0 -19 -f -o "$tb"
[ "$(label "$tag" redistributable)" = false ] && echo \
"!! $tb carries licensed and customer material: not redistributable" >&2

if [ -n "$REGISTRY_IMAGE" ]; then
ref="$REGISTRY_IMAGE:$VERSION-$tier"
docker tag "$tag" "$ref"
echo "pushing $ref ..."
if "$DOCKER_SCRIPTS/push.sh" "$ref"; then
echo "pushed $ref"
else
echo "registry push failed -- the tarball at $tb is the usable copy" >&2
fi
fi

echo "updated $tag" ;;

list)
# The label, not the repository name: an image retagged for a registry
# is still one of ours, and says which profile it is. Both namespaces:
Expand Down
Loading