Repository navigation
Conversation
…rt failure The 2026-11-20 rc.46 train lost its smoke gate to one instant undici 'fetch failed' blip while loading the host module from the CDN, then sat 4 minutes because the container stays Running after a swallowed start failure. Three fixes: - orchestrator: bounded, network-only retries on the remote boot fetches (host pin resolve, manifest fetch, module load) — the manifest fetch already had the house retry policy available and pinned retries: 0 - smoke: race a 'Failed to start:' log watcher beside the exit watch so a running-but-doomed container fails in seconds, not at the 240s deadline; surface undici's error.cause in poller failures - near-signer: classify a non-integer gas-check response as an unexpected format instead of 'RPC unreachable'
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
The 2026-11-20 rc.46 deploy (run 44750821150) failed its post-publish smoke on one instant undici
fetch failedwhile the container loaded the host module from the CDN — the post-mortem probe from the same container 4 minutes later hit every URL 200. The config auto-rolled back (gate worked as designed), but the incident exposed three seams:loadRemoteModule) failed fatal on the first socket blip, even thoughfetchWithRetryEffalready ships the house retry policy — the manifest fetch explicitly pinnedretries: 0.orElseSucceed), so the container staysRunning, the exit watch never fires, and the pollers poll a port nothing will ever listen on. The smoke docstring's claim that "the start stack exits non-zero when a service fails to boot" doesn't hold on this path.pollUntilcaptured onlyerror.message, dropping undici'serror.cause(ECONNRESET / socket hang up), which cost diagnosis time.Changes
orchestrator.ts— bounded, network-only retries on all three boot fetches:retries: 2on the manifest fetch (house policy: capped exponential jittered, network TypeErrors/timeouts/5xx only), and anEffect.retrywithwhile: isTransientBootFailure(cause isTypeError) on the pin resolve and module load. Skew/404-shaped failures still fail loud on the first attempt; the existing 180s remote-boot deadline still caps the whole sequence.smoke.ts— race adocker logswatcher beside the exit watch; aFailed to start:line resolves the gate in seconds with the captured container logs instead of at the deadline.pollUntilnow appendserror.cause?.messageto the last-failure line.near-signer.ts— the gas-prepayment check validates integer strings beforeBigIntand classifies a decimal response (fastnear free RPC returned"1.28"in the same run's preflight) as an unexpected format instead of misreporting the RPC as unreachable.Verification
pnpm run typecheckgreen; Biome +lint:effectclean on the changed files (only pre-existing warnings elsewhere)pnpm --filter everything-dev test: 2199 passed; the one failure (fresh-init-devconsumer gate,AUTH_DATABASE_URLenv refusal in a local boot) reproduces on the clean tree — pre-existing, unrelatedeverything-dev)Follow-up (not in this PR)
The dev-session swallowing start failures is deliberate for
bos devUX, but it means the production start stack never exits non-zero on a dead service — worth revisiting for production mode specifically (issue to follow).