Repository navigation
feat(krater): blue/green portal deploys, on for staging - #104
Merged
Merged
Conversation
A deploy restarted krater-web, which stopped the portal, then migrated and started the new release: about 60 s of 502s each time. With blueGreen.enable, krater-web is a oneshot that: - migrates while the running release keeps serving (Krater's migrations only add); - starts the new release in the idle slot (blue on `port`, green on `blueGreen.standbyPort`); - waits for an HTTP 200 from its /healthz (not an open port: docker-proxy accepts before uvicorn listens); - points nginx at it (an upstream file the vhost includes, then a reload); - then stops the old slot with 30 s for uvicorn to finish its requests. A release that never answers is removed and the old one keeps serving. Run again with the same release, it changes nothing. The deploy command, the sudo rules and the follow timer's restore path are unchanged; the follow health check asks the slot that serves. The worker `wants` web (so a failed release doesn't keep it down) and gets 120 s to finish a task when stopped. On for staging only. Production's web, worker and storage units are byte-identical to main (checked by comparing the unit derivations); its build and follow units change but don't restart on a switch. nginx on weft gets enableReload, so later vhost changes reload it instead of restarting it. This switch restarts nginx once, and staging's portal is down once while its old unit stops and the first slot starts. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
jaspermayone
added a commit
that referenced
this pull request
Nov 13, 2026
* origin/main: feat(krater): blue/green portal deploys, on for staging (#104) feat(krater): pull CI's image instead of building on weft (#103) feat(ansible): run ad converge build from ad.yml (#105) feat(selvage): add a staging instance on twill (#102) feat(selvage): pass the admin web ui paths through nginx on twill (#100)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #103; merge that first. This brings Krater deploys with no gap, on staging only.
Today: a deploy runs
systemctl restart krater-web, which stops the portal, migrates, then starts the new release. That's about 60 s of 502s per deploy.With
blueGreen.enable,krater-webbecomes an idempotent oneshot (krater-web-converge, awriteShellApplication, so ShellCheck runs on it). It:krater-web-blueonport(3010) orkrater-web-greenonblueGreen.standbyPort(3020), as Docker containers with--restart unless-stoppedand logs to journald, tagkrater-web;/healthz. An open port isn't enough, because docker-proxy accepts connections before uvicorn listens;/var/lib/krater/nginx-upstream.confand reloads nginx. The staging vhost now proxies toupstream krater-web { include …; }, and a reload lets requests already in flight finish;docker stop --time 30, so uvicorn finishes its requests.If the new release never answers, it's removed and the old one keeps serving. Run again with the same release (a reboot, a repeat deploy), it changes nothing.
Unchanged: the CI deploy command, the sudo rules, and the follow timer's restore-and-restart path. The follow health check now asks whichever slot is serving.
Worker: it now
wantsweb instead ofrequires, so a failed release doesn't keep the worker down; it comes back on the old image. It also getsdocker stop --time 120(TimeoutStopSec150), so a scheduler pass or price check finishes before it stops.Production is untouched. I built the unit files from main and from this branch and compared them.
krater-prod-web,krater-prod-workerandkrater-prod-storageare byte-identical.krater-prod-buildandkrater-prod-followchange but haverestartIfChanged = false. Production gets its own small PR once staging has deployed this way.One-time effects of this switch:
enableReload = true, so later vhost changes reload instead of restart. This switch restarts nginx once, which is a blink for both sites.krater-webunit stops and the first slot starts. Every staging deploy after that has no gap.To stop the staging portal now:
docker stop krater-web-blue krater-web-green(systemctl stop krater-webleaves it up; documented inmodules/krater/default.nix).Tested:
nix build .#checks.x86_64-linux.krater-weftpasses. It includes new checks for the vhost, the upstream include, the tmpfiles seed, the worker's wants and stop timeout, the converge script, and that production isn't blue/green. Formatted with the flake's nixfmt-tree.🤖 Generated with Claude Code