[pithead-developer]

What's the problem

rigforge#678 made control_upgrade/control_apply answer busy while a bench holds the rig reservation lock, so a borrowing stack's changes can't land mid-loan. The only exemption was the #670 owner marker — a file naming a live pid — and docs/operations.md documented it as reserved for an e2e phase whose api_allow_from is loopback-only. pithead#3985's real-upgrade leg reaches the rig over the LAN, so under that contract a remote borrowing stack can never hold the marker: while its own harness runs, its upgrades get busy forever. rigforge#687 (parent) tracks the pithead-side consumption; the condition itself belongs here.

How I fixed it

The marker becomes self-describing and the guard learns to distrust it until proven otherwise:

• Marker v2 (docs/operations.md): three lines — the harness pid, a harness-random temporary ACCESS_TOKEN, and the original token it replaced (empty if none). The borrowing stack swaps the rig's token for its own and pins api_allow_from to exactly one host it controls (loopback or its own address), then restores both and clears the marker after the window.
• _rig_owner_marker_admits (util/control-guards.sh): admits only when the marker is a regular euid-owned non-symlink with exactly three lines, the pid is alive, the marker's token is nonempty, differs from the recorded original, and equals the currently active ACCESS_TOKEN — and api_allow_from pins exactly one host (bare IP, /32, or /128; unset or any wider prefix refuses) and the change was received from that host. A dual-stack receiver records an IPv4 peer in ::ffff:-mapped form, so that mapped form of a pinned IPv4 host admits; nothing else does. A pid-only marker admits nothing (red at the previous head by design), keeping Refuse a remote upgrade while a bench has the rig borrowed; the bench's restore silently undoes it #670's dead-pid semantics.
• Source plumbing: the receiver (util/control-server.py) records client_address[0] durably as spool/.peer-<cid> before the staged change (peerless stages and orphans are pruned under the same queue lock, and prune now logs unlink failures instead of swallowing them), and the root appliers consume it once via _control_consume_source before _control_claim_staged and pass it to the guard. Missing or empty source never admits — fail closed.
• Named ACCESS_TOKENS entries are intentionally not checked (that would break the One access token per consuming stack, so a bench borrow never needs prod's token #516 e2e phase); the docs state pinning them is a harness duty for the window, and that the token swap takes effect only once the control receiver restarts with the new config — the ordinary config-apply path does. The status vocabulary and tests/contract/v1 are unchanged.

The review round on af5ae99 reproduced a race in the first ordering (a concurrent stage's orphan prune could delete the peer file of an already-claimed, not-yet-consumed change), flagged that a ::ffff:-mapped source fails an IPv4 pin on a dual-stack receiver, and caught two docs claims the docs hadn't actually made. The approach stands; the ordering, normalization and docs move: consume-before-claim closes the race structurally rather than by lock-window timing, and the two doc sentences were added.

Here is the proof

• Tier 1 (make test-suite): tests/test-control-upgrade.sh covers all three AC cases — admit for the pinned borrowing host (swapped token, red at current head: a pid-only marker now asserts busy instead), busy from an address outside the pinned host, and busy for a marker whose token or allow-list is still the rig's own — plus branch rows for bare IPv4, bare IPv6, /128 and the IPv4-mapped admit, no-source, wide-prefix (192.0.2.0/24, fd00::/64) and dead-pid refusals, driven table-style through the real control_upgrade/control_apply fixtures, and a static assert pinning consume-before-claim in both appliers. tests/test-control-spool-cap.py covers the peer twin, the cap ignoring peer files, prune of peerless/torn-down twins, and peer-write failure residue.
• Local (macOS, non-root): the flock/root-owned-state blocks can only run in the Linux CI container — verified by reproducing the identical failure signature on a pristine base checkout; every local red is the pre-existing processing-directory environmental one, no new signature. python3 tests/test-control-spool-cap.py and tests/control-revocation.py pass; FILE_BUDGET_REQUIRE_BASE=1 bash scripts/lint-file-budget.sh green (both grown files kept ≤400 lines, no new ceilings); shellcheck clean at warning+; the admit rule's address-form matrix checked by a standalone probe (7/7).
• CI tiers 1–3 green on head f2655ad: Test suite (the root-Linux run where the marker suite genuinely executes), Lint (shellcheck + shfmt), markdownlint, yamllint, gitleaks, zizmor, both Stock home setups, End-to-end (Docker) and Coverage. An earlier Docker failure was runner DNS during apt prereqs and passed on rerun.
• Tier 4 (review requirement, changing this child's scope): the phase catalog has no separate upgrade/replay names — the upgrade applier exercises inside control, and the replay phases are control-replay-thermal/control-replay-pools, which only a runner-queued pithead-control fixture job may run, because the fixture is what keeps the reservation held through restore (ordinary tier4-e2e jobs hand the rig lock to the harness, so an applied there proves the marker/token contract but not admission against a held lock). Job 3834 ran control and access-tokens on f2655ad: both phases set the three-line marker exactly as the harness contract says and reached applied (11 rows; its only red was [control] miner or refresh-state observation failed, a bench miner-observation flake with no issue linked, absent on the rerun) — bench-ci evidence 3834@f2655ad569c8a4ae376d608933e06244274a554a. To make the reservation-holding and busy proofs real rather than asserted, this head adds an end-of-replay probe: with the marker set and the runner reservation held, a POST of a current scalar from the rig's global address (outside the 127.0.0.1/32 pin) must settle busy, which discards the change; the offline replay harness drives it (busy settles, an applied non-pinned change fails the phase, a refused port skips) and the worker-proof counts follow. Three rounds of fixture-job submissions for the replay phases (6 jobs, two pushed heads, two bench nodes, two days) have each errored ~8s after start with checkout success and reserve failure — no free worker compatible with the pithead-control pool probe; verify approved input and source identities, zero phase rows — including the latest pair on head f823c02 (bench-ci evidence 3906@f823c022401a0f04346a6940a4251e1265d9a99c); develop has since merged into the branch (worker-proof pass-count table conflict resolved to the derived 12, full CI tiers 1-3 green on the merged head) so submissions no longer hit the base-conflict gate, and the fixture jobs on that head (bench-ci evidence 4023@775e173eb635b7697e9607faf216b44373483e52) errored with the same reserve refusal in 8s while bench health showed healthy nodes with free lendable rigs — seven identical refusals across four rounds over four days is a stable bench-side pairing state, not load: a bench operator must verify the fixture pool-probe approved input and source identities before any fixture job can run, and the evidence will be quoted here the moment one does; re-checks of bench availability reported it clear twice and reserve still found no compatible free worker. This is a bench-side pairing state this PR cannot change, so the reservation-held admit/busy proof is blocked on the bench operator verifying the fixture pool-probe input and source identities, and will be quoted here the moment a fixture job runs.

ponytail-review: minimal — one guard helper + one consume helper + receiver source record; the two reviewer passes (verifier, security) returned clean with no dropped lines.

Closes #688

🤖 Generated with Claude Code

[pithead-reviewer]

FAIL

What I investigated

Review details are in the private record because they contain sensitive information.

Reviewed head: af5ae992616096fc58262e392dfc64b0b20368d2.

What are the gaps

• The worker must address the findings in the private review record.