Skip to content

feat(krater): blue/green portal deploys, on for staging - #104

Merged
jaspermayone merged 1 commit into
claude/krater-pull-imagefrom
claude/krater-blue-green
Nov 13, 2026
Merged

jaspermayone merged 1 commit into
claude/krater-pull-imagefrom
claude/krater-blue-green

Conversation

@Adambomb210

Copy link
Copy Markdown
Collaborator

Stacked on #103; merge that first. This brings Krater deploys with no gap, on staging only.

Today: a deploy runs systemctl restart krater-web, which stops the portal, migrates, then starts the new release. That's about 60 s of 502s per deploy.

With blueGreen.enable, krater-web becomes an idempotent oneshot (krater-web-converge, a writeShellApplication, so ShellCheck runs on it). It:

  1. writes the env files, checks settings, waits for the database, and migrates while the running release keeps serving. Krater's migrations only add, and the old release already has to run on a newer schema;
  2. starts the new release in the idle slot: krater-web-blue on port (3010) or krater-web-green on blueGreen.standbyPort (3020), as Docker containers with --restart unless-stopped and logs to journald, tag krater-web;
  3. waits for an HTTP 200 from that slot's /healthz. An open port isn't enough, because docker-proxy accepts connections before uvicorn listens;
  4. writes /var/lib/krater/nginx-upstream.conf and reloads nginx. The staging vhost now proxies to upstream krater-web { include …; }, and a reload lets requests already in flight finish;
  5. after 5 s, stops the old slot with docker stop --time 30, so uvicorn finishes its requests.

If the new release never answers, it's removed and the old one keeps serving. Run again with the same release (a reboot, a repeat deploy), it changes nothing.

Unchanged: the CI deploy command, the sudo rules, and the follow timer's restore-and-restart path. The follow health check now asks whichever slot is serving.

Worker: it now wants web instead of requires, so a failed release doesn't keep the worker down; it comes back on the old image. It also gets docker stop --time 120 (TimeoutStopSec 150), so a scheduler pass or price check finishes before it stops.

Production is untouched. I built the unit files from main and from this branch and compared them. krater-prod-web, krater-prod-worker and krater-prod-storage are byte-identical. krater-prod-build and krater-prod-follow change but have restartIfChanged = false. Production gets its own small PR once staging has deployed this way.

One-time effects of this switch:

  • nginx gets enableReload = true, so later vhost changes reload instead of restart. This switch restarts nginx once, which is a blink for both sites.
  • Staging's portal is down once (about a minute) while the old krater-web unit stops and the first slot starts. Every staging deploy after that has no gap.

To stop the staging portal now: docker stop krater-web-blue krater-web-green (systemctl stop krater-web leaves it up; documented in modules/krater/default.nix).

Tested: nix build .#checks.x86_64-linux.krater-weft passes. It includes new checks for the vhost, the upstream include, the tmpfiles seed, the worker's wants and stop timeout, the converge script, and that production isn't blue/green. Formatted with the flake's nixfmt-tree.

🤖 Generated with Claude Code

A deploy restarted krater-web, which stopped the portal, then migrated and started the new release: about
60 s of 502s each time. With blueGreen.enable, krater-web is a oneshot that:
- migrates while the running release keeps serving (Krater's migrations only add);
- starts the new release in the idle slot (blue on `port`, green on `blueGreen.standbyPort`);
- waits for an HTTP 200 from its /healthz (not an open port: docker-proxy accepts before uvicorn listens);
- points nginx at it (an upstream file the vhost includes, then a reload);
- then stops the old slot with 30 s for uvicorn to finish its requests.
A release that never answers is removed and the old one keeps serving. Run again with the same release, it
changes nothing. The deploy command, the sudo rules and the follow timer's restore path are unchanged; the
follow health check asks the slot that serves. The worker `wants` web (so a failed release doesn't keep it
down) and gets 120 s to finish a task when stopped.

On for staging only. Production's web, worker and storage units are byte-identical to main (checked by
comparing the unit derivations); its build and follow units change but don't restart on a switch.

nginx on weft gets enableReload, so later vhost changes reload it instead of restarting it. This switch
restarts nginx once, and staging's portal is down once while its old unit stops and the first slot starts.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@jaspermayone
jaspermayone added this pull request to stack #106 November 13, 2026 12:38
@jaspermayone
jaspermayone merged commit 2ede8e1 into main Nov 13, 2026
5 checks passed
jaspermayone added a commit that referenced this pull request Nov 13, 2026
* origin/main:
  feat(krater): blue/green portal deploys, on for staging (#104)
  feat(krater): pull CI's image instead of building on weft (#103)
  feat(ansible): run ad converge build from ad.yml (#105)
  feat(selvage): add a staging instance on twill (#102)
  feat(selvage): pass the admin web ui paths through nginx on twill (#100)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants