Skip to content

fix(deploy): retry transient boot fetches, fail the smoke fast on start failure - #913

Open
efizzybot wants to merge 1 commit into
mainfrom
t3/deploy-boot-resilience
Open

efizzybot wants to merge 1 commit into
mainfrom
t3/deploy-boot-resilience

Conversation

@efizzybot

Copy link
Copy Markdown
Collaborator

Summary

The 2026-11-20 rc.46 deploy (run 44750821150) failed its post-publish smoke on one instant undici fetch failed while the container loaded the host module from the CDN — the post-mortem probe from the same container 4 minutes later hit every URL 200. The config auto-rolled back (gate worked as designed), but the incident exposed three seams:

  1. No retry on the boot fetches. The orchestrator's remote boot path (host pin resolve, manifest fetch, loadRemoteModule) failed fatal on the first socket blip, even though fetchWithRetryEff already ships the house retry policy — the manifest fetch explicitly pinned retries: 0.
  2. The smoke burned its full 240s deadline after the boot was already dead. The start stack swallows a per-service start failure (session continues via orElseSucceed), so the container stays Running, the exit watch never fires, and the pollers poll a port nothing will ever listen on. The smoke docstring's claim that "the start stack exits non-zero when a service fails to boot" doesn't hold on this path.
  3. "fetch failed" hid the cause. pollUntil captured only error.message, dropping undici's error.cause (ECONNRESET / socket hang up), which cost diagnosis time.

Changes

  • orchestrator.ts — bounded, network-only retries on all three boot fetches: retries: 2 on the manifest fetch (house policy: capped exponential jittered, network TypeErrors/timeouts/5xx only), and an Effect.retry with while: isTransientBootFailure (cause is TypeError) on the pin resolve and module load. Skew/404-shaped failures still fail loud on the first attempt; the existing 180s remote-boot deadline still caps the whole sequence.
  • smoke.ts — race a docker logs watcher beside the exit watch; a Failed to start: line resolves the gate in seconds with the captured container logs instead of at the deadline. pollUntil now appends error.cause?.message to the last-failure line.
  • near-signer.ts — the gas-prepayment check validates integer strings before BigInt and classifies a decimal response (fastnear free RPC returned "1.28" in the same run's preflight) as an unexpected format instead of misreporting the RPC as unreachable.

Verification

  • pnpm run typecheck green; Biome + lint:effect clean on the changed files (only pre-existing warnings elsewhere)
  • pnpm --filter everything-dev test: 2199 passed; the one failure (fresh-init-dev consumer gate, AUTH_DATABASE_URL env refusal in a local boot) reproduces on the clean tree — pre-existing, unrelated
  • Changeset included (patch on everything-dev)

Follow-up (not in this PR)

The dev-session swallowing start failures is deliberate for bos dev UX, but it means the production start stack never exits non-zero on a dead service — worth revisiting for production mode specifically (issue to follow).

…rt failure

The 2026-11-20 rc.46 train lost its smoke gate to one instant undici
'fetch failed' blip while loading the host module from the CDN, then sat
4 minutes because the container stays Running after a swallowed start
failure. Three fixes:

- orchestrator: bounded, network-only retries on the remote boot fetches
  (host pin resolve, manifest fetch, module load) — the manifest fetch
  already had the house retry policy available and pinned retries: 0
- smoke: race a 'Failed to start:' log watcher beside the exit watch so a
  running-but-doomed container fails in seconds, not at the 240s
  deadline; surface undici's error.cause in poller failures
- near-signer: classify a non-integer gas-check response as an unexpected
  format instead of 'RPC unreachable'

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants