Skip to content

Add bounded after-response tasks on Fastly and Axum - #509

Open
prk-Jr wants to merge 5 commits into
feat/portable-client-infofrom
feature/476-after-response-tasks
Open

prk-Jr wants to merge 5 commits into
feat/portable-client-infofrom
feature/476-after-response-tasks

Conversation

@prk-Jr

@prk-Jr prk-Jr commented Nov 17, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

  • Trusted Server needs pull sync and guarded follow-up record writes to run after the document response, without delaying it, and Axum must wait for that work on SIGTERM (EZI-1 / AX-1 / ID-5). EdgeZero had no portable after-response hook.
  • Adds ctx.after_response(budget, task): the task starts once the response attempt ends at the adapter's terminal egress boundary, and is cancelled when its budget elapses. Axum drains registered tasks on SIGTERM within the existing shutdown grace. Adapters without support (Cloudflare, Spin for now) return 501 Not Implemented at registration instead of dropping work silently.
  • Adds the after-response-tasks manifest capability so apps can declare it required and have build/serve/deploy refuse adapters that cannot run it.

Stacked on #406 (issue-392-396-production-hosting-spec at e37e0df2), which carries #275, #499 and #460; this work needs #406's SIGTERM drain and #275's Fastly callback scope. Design: docs/superpowers/specs/2026-11-17-after-response-tasks-design.md; plan: docs/superpowers/plans/2026-11-17-after-response-tasks.md.

Changes

Crate / File Change
edgezero-core/src/after_response.rs (new) AfterResponseScheduler adapter trait, AfterResponseHandle extension handle, 501 when absent, zero budget rejected
edgezero-core/src/context.rs RequestContext::after_response(budget, task)
edgezero-core/src/manifest.rs Capability::AfterResponseTasks (after-response-tasks)
edgezero-adapter-axum/src/after_response.rs (new) Per-request scheduler: spawn_local on the server LocalSet, gated on a release signal, timeout(budget) inside a fresh outbound callback scope; watch-based task tracker; registration closes once released
edgezero-adapter-axum/src/{service,response}.rs Installs the handle per request; the release travels in AxumEgressBody and fires when Hyper drops the body (final frame handed over, or early disconnect/cancellation)
edgezero-adapter-axum/src/dev_server.rs drain_tasks waits for the tracker after connections settle, under the same grace deadline (an idle tracker wins over an elapsed deadline); pending tasks at the grace are cancelled → existing grace exhausted failure; abandoned tasks are logged on every exit path. Applies to production entrypoints, run_app and AxumDevServer::run
edgezero-adapter-fastly/src/after_response.rs (new), request.rs Request-local queue armed by both lifecycle funnels; after finish_request, tasks are drained inside the same with_callback scope (outbound and timers work), each raced against its budget; handle inserted only while armed, after prepare extensions; each lifecycle has a generation so a retained handle is rejected; send_request_with_registries_and_hooks_for_test exposes the run_app/serve_app funnel to contract tests
edgezero-adapter-*/src/cli.rs Capability matrix: Axum and Fastly BoundedCooperative (budget cancels at await/yield points), Cloudflare/Spin Unsupported
Tests Core unit tests; Axum executor unit tests, real-listener integration tests (tests/after_response.rs), subprocess SIGTERM tests; Fastly host tests and Viceroy contract tests
Docs capabilities.md (new checked matrix), handlers.md, adapters/axum.md, adapters/fastly.md, CHANGELOG.md; scripts/check_outbound_docs_contract.mjs checks the new matrix

Closes

Closes #476

Test plan

Local, Rust 1.98.1 / Node 24.20.0 (pinned versions):

  • cargo test --workspace --all-targets
  • cargo clippy --workspace --all-targets --all-features -- -D warnings, plus the per-adapter wasm/native clippy matrix from format.yml and Fastly --features cli / --no-default-features
  • cargo fmt --all -- --check
  • cargo check --workspace --all-targets --features "fastly cloudflare spin"
  • WASM builds: wasm32-wasip1 (Fastly) / wasm32-wasip2 (Spin) / wasm32-unknown-unknown (Cloudflare, core)
  • examples/app-demo workspace: cargo test --workspace --all-targets, fmt, clippy
  • Docs build: npm run lint && npm run format && npm run build; node scripts/check_outbound_docs_contract.mjs
  • Fastly wasm suites under Viceroy 0.21.0 (CI-equivalent patched readiness build): --test contract 42/42, --lib 357/357
  • Other — negative controls (each temporarily broken, then restored):
    • removing the Axum release gate fails the keep-alive streaming test
    • skipping the drain wait fails both SIGTERM subprocess tests
    • wrapping the release wait inside the budget fails the budget-starts-at-release test
    • skipping the Fastly drain fails the contract tests
    • unwrapping the run_app registry funnel fails its contract test
    • inserting the adapter handle before prepare extensions fails the hook-override tests
    • giving the task phase a fresh deadline fails the shared-grace test (1.82 s vs < 1.5 s)
  • Other — runtime runs of a scratch copy of examples/app-demo with a /after-task route (registers a task with a 1.5 s timer):
    • Axum run_app: the response arrives in ~1 ms and the task completes ~1.5 s later. SIGTERM right after the response waits ~1.5 s and exits 0. With a 1 s grace the task is cancelled, the process exits 1 with grace exhausted, and the abandoned-task warning is logged.
    • Fastly viceroy serve (0.21.0): the response arrives in ~1–2 ms and the task completes ~1.5 s later on two consecutive requests.
    • Control: the same 1.5 s wait done inline before the response takes ~1.5 s on both adapters, which shows the latency measurement detects pre-send work.
    • Spin 4.1.0 spin up and Cloudflare wrangler dev --local: /after-task returns 501 with after-response tasks are not supported by this adapter, while existing routes are unaffected.

Not verified: deployed Fastly Compute and a production Axum container/ECS stop; Trusted Server consumer integration; hosted CI (pending on this push).

Notes and limitations

  • The start boundary is the adapter's terminal egress handoff (same as ResponseEgressOutcome::HostHandoff), not proof of client receipt. Tasks run whatever the outcome (completed, error response, disconnect).
  • Budget cancellation is cooperative on both adapters: a CPU-bound or blocked task overruns its budget (documented), hence BoundedCooperative for both. On Fastly a blocked hostcall cannot be preempted.
  • Fastly: tasks delay the next request on a reused serve_app instance, and — when delivery did not commit — a caller-sent native replacement response, by at most the longest registered budget (documented trade-off; skipping would drop work silently).
  • A task cannot register further tasks (registration closes on release), which keeps total per-request work bounded by budgets registered during the request.
  • Fastly handles are tied to their lifecycle generation; a handle retained across requests is rejected.
  • Possible follow-up: no per-request or global cap on the number of registered tasks on Axum.
  • Follow-ups: After-response work on Cloudflare (waitUntil) #478 (Cloudflare waitUntil) and After-response capability error on Spin #483 (Spin capability error naming the adapter).

Checklist

  • Changes follow CLAUDE.md conventions
  • No Tokio deps added to core or adapter crates (Axum already depends on Tokio; core and Fastly use none)
  • Route params use {id} syntax (not :id)
  • Types imported from edgezero_core (not http crate)
  • Store wiring goes through KvRegistry / ConfigRegistry / SecretRegistry (not the legacy single-handle setters) — not touched
  • New code has tests
  • No secrets or credentials committed

Add RequestContext::after_response(budget, task) and the adapter-facing
AfterResponseScheduler/AfterResponseHandle in edgezero-core, plus the
after-response-tasks manifest capability (Axum Native, Fastly
BoundedCooperative, Cloudflare and Spin Unsupported).

Axum spawns each task on the server's LocalSet, releases it when Hyper
drops the response body, bounds it with a Tokio timeout inside its own
outbound callback scope, and drains registered tasks on SIGTERM within
the existing shutdown grace after connections settle.

Fastly queues tasks per request lifecycle and drives them after the
response is sent, inside the same callback scope, each raced against
its budget. Adapters without a scheduler return 501 at registration.

Closes #476.
@prk-Jr prk-Jr self-assigned this Nov 17, 2026
Address self-review findings on the after-response task work:

- Cover the Fastly registry lifecycle used by run_app and serve_app
  through send_request_with_registries_and_hooks_for_test, and assert
  that a prepare hook cannot replace the adapter scheduler.
- Stamp each Fastly lifecycle with a generation so a retained handle is
  rejected instead of queuing into a later request.
- Let an idle tracker win over an elapsed deadline in the Axum task
  wait, and log abandoned tasks on every shutdown exit path.
- Classify Axum as BoundedCooperative, matching its await-point
  cancellation, and document AxumDevServer::run draining.
- Add tests for the shared grace, a second signal during the task
  wait, Fastly outbound requests from a task, and native replacement.
- Keep the handler example from failing pages on unsupported adapters
  and sync the design and plan documents with the code.
Move the abandoned-task warning from drain_tasks into run_owned through
AfterResponseTracker::warn_abandoned, so the prebound serve() path and
the unusable-deadline return also report cancelled tasks, exactly once.
Exercise the prepare-hook override through the registry funnel test too.
@prk-Jr prk-Jr linked an issue Nov 17, 2026 that may be closed by this pull request
2 tasks
@prk-Jr
prk-Jr marked this pull request as ready for review November 17, 2026 09:52
prk-Jr added a commit that referenced this pull request Nov 18, 2026
Installing a rejecting Spin scheduler changes a zero budget from 501 to
500: core now finds a handle and rejects the budget before the
scheduler, as it already does on Axum and Fastly. The guides, changelog
and spec claimed every Spin registration returns 501 and that the zero
budget case was unchanged. Correct them and pin the 500 with a Spin unit
test.

Also note that core pre-dispatch hooks can replace or remove the handle
(removal falls back to core's 501), that PR #512 overlaps on the
CHANGELOG line as well as the capabilities bullet, and point the #509
design at the follow-up.
@prk-Jr
prk-Jr added this pull request to stack #516 November 18, 2026 18:17
@aram356
aram356 removed this pull request from stack #516 November 19, 2026 05:02
@aram356
aram356 changed the base branch from issue-392-396-production-hosting-spec to feat/portable-client-info November 19, 2026 05:03
@aram356
aram356 added this pull request to stack #511 November 19, 2026 05:03
aram356 added a commit that referenced this pull request Nov 19, 2026
#509 (2620377) adds bounded work after the response on Fastly and Axum, drained on Axum SIGTERM (EZI-1, #476). #512 (cdc232c) runs it on Cloudflare through waitUntil (#478), and #515 (ec347a2) reports Spin's capability error (#483). The line now stacks on #517 and each PR merged the one below it, so this merge is clean.
@aram356

aram356 commented Nov 19, 2026

Copy link
Copy Markdown
Collaborator

For Trusted Server's identity integration chain, this PR now stacks on #517, the top of the line #439 → #510 → #454 → #517, which also carries #469 and #502. 2620377a merges #517's head 10ed3803 (git show --remerge-diff 2620377a). The base is now feat/portable-client-info, so the diff still shows only this PR's changes, and the PR joins stack 511 above #517. GitHub does not let a stacked PR change its base, so stack 516 (#509, #515) was dissolved first.

Area Resolution
Axum dev_server.rs The service is built with the base's with_native_transport(..) and with_phase(..) plus this PR's with_after_response_tracker; JA4 selection stays
Axum service.rs Carries both the base's activity and this PR's after_response. The base moved dispatch into dispatch_prepared, so request_tasks(..) now runs there, after the store registries and before dispatch_admitted_with_client_info; release_after_response_on_drop wraps the prepared egress
Fastly request.rs, tests/contract.rs handle_if_armed() before the base's dispatch with geo and client info. Both sides had added the same delivery-boundary test entry, so the base's names stay and this PR's two references use them
Core context.rs, Axum response.rs Imports from both sides
CHANGELOG.md, docs/guide/handlers.md Both sides' entries; one method table with client_info(), geo(), ja4() and after_response(budget, task)

One behavior change, from the port: the base records an edgezero::native lifecycle entry on every shutdown failure, so this PR's after-response wait now records forced_stop on a second signal and grace_exhausted when the grace runs out. Exit codes and error text are unchanged.

Checked locally at 2620377a: cargo fmt, workspace clippy with all features, both Fastly clippy runs, the wasm feature check, cargo test --workspace --all-targets (3,113 passed), the Fastly contract tests under Viceroy (42), and both outbound scripts, all passing.

Exercise the consumer path Trusted Server relies on for follow-up writes:

- Axum: a conditional Add record write inside a task is acknowledged and
  readable by a later request, without delaying the response.
- Axum: SIGTERM arrives while a request is in flight, the handler then
  registers a KV write; the request and the write both complete within
  the grace, the process exits 0, and the value survives a restart.
- Fastly: a guarded record write inside a task lands after delivery.
  Ignored in CI because the bare Viceroy runner has no KV stores; run it
  from the adapter crate, whose runner loads the demo fastly.toml.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Bounded work after the response on Fastly and Axum, drained on Axum SIGTERM

2 participants