Repository navigation
Conversation
db9e915 to
bdd12a8
Compare
bdd12a8 to
9f1e764
Compare
ChristianPavilonis
left a comment
There was a problem hiding this comment.
Summary
This PR moves pull sync into after-response identity collection and adds a bounded, JavaScript-readable mirror of provider-written bidstream UIDs. One startup-validation issue needs addressing; there is also an opportunity to avoid a redundant KV read on steady-state navigations.
The inline comment is prose-only: the proposed fix and regression coverage touch lines outside the diff, so there are no one-click suggestions.
Blocking
🔧 wrench
- Reject malformed pull tokens during startup — see inline at
crates/trusted-server-identity-providers/src/pull_sync.rs:59-60.
Cross-cutting / body-level findings
♻️ Avoid the follow-up KV read when all lookups were skipped
Introduced by crates/trusted-server-identity-providers/src/lib.rs:169-182; affected helper: crates/trusted-server-core/src/ec/after_response.rs:76-79.
When every pull-sync source already holds a UID, dispatch produces Skipped(NotDue) results. Those results are nonempty, so the follow-up helper rereads KV despite having nothing to persist. Moving existing pull-sync partners into this path adds an avoidable read on every eligible returning navigation.
Import LookupStatus and SkipReason alongside LookupResults, then replace the helper's empty-results guard with:
if lookups.iter().all(|lookup| {
lookup.entries.is_empty()
&& lookup.cleared.is_empty()
&& lookup.status != LookupStatus::Skipped(SkipReason::OptedOut)
}) {
return AfterResponseOutcome::Unchanged;
}This preserves writes that remember provider opt-outs.
Apply manually — the helper, imports, and regression test are outside the diff.
Verification
- A focused scratch reproduction confirmed that a newline-containing pull token passes runtime config validation but prevents construction of the coordinator for both an invalid and an otherwise valid partner.
- A counting-store scratch reproduction confirmed that all-
NotDueresults still cause one KV read in the follow-up helper. - Both reproductions passed. Scratch changes were removed and the worktree was restored to the reviewed head.
- The proposed fixes have not been compiled or run through the full local gate.
CI Status
- integration tests: PASS
- browser integration tests: PASS
- integration tests (Fastly EC lifecycle): PASS
- trace runtime and browser acceptance: PASS
- cargo fmt: PASS
- cargo test (axum native): PASS
- cargo test (cross-adapter parity): PASS
- format-docs: PASS
- worker-deadlines (spin): PASS
- worker-deadlines (cloudflare): PASS
- ts dev lint domains (changed lines): PASS
- cargo test: PASS
- format-typescript: PASS
- cargo test (ts CLI, native) (macos-latest): PASS
- HTML EOF runtime (Fastly): PASS
- cargo check/build/test (spin native + wasm32-wasip1): PASS
- CLAUDE.md symlink guard: PASS
- docs-proposal scripts: PASS
- CI Node pin guard: PASS
- cargo test (ts CLI, native) (ubuntu-latest): PASS
- cargo check (cloudflare native + wasm32-unknown-unknown): PASS
- prepare integration artifacts: PASS
- vitest: PASS
No required checks were reported by the required-check query.
aram356
left a comment
There was a problem hiding this comment.
Summary
This PR moves pull sync onto the after-response provider path and adds a JavaScript-readable mirror of provider-written bidstream UIDs. The move keeps the live-row consent re-check and the bot gate. Before the mirror ships, it needs an operator opt-in and a per-vendor consent check. The after-response path also adds KV reads that the legacy dispatcher avoided.
3 of the 10 inline comments carry a one-click GitHub
suggestion. Use Commit suggestion, or Add suggestion to batch to apply several at once. The other comments describe the fix in prose, because those changes span files or touch lines outside the diff.
Blocking
🔧 wrench
- Make the EID mirror opt-in, separate from
bidstream_enabled: see inline atcrates/trusted-server-core/src/ec/finalize.rs:90 - Withhold mirrored UIDs whose provider lacks TCF vendor consent: see inline at
crates/trusted-server-core/src/ec/mirror.rs:77 - Skip the after-response row read when nothing is due: see inline at
crates/trusted-server-adapter-fastly/src/app.rs:453 - Cover the
enabled = falsebypass of the double-claim check (suggestion): see inline atcrates/trusted-server-core/src/config.rs:1455
❓ question
- Should pull-sync sources keep accepting browser UIDs?: see inline at
crates/trusted-server-core/src/ec/registry.rs:274
Non-blocking
♻️ refactor
- Delete
pull_enabled_partners; nothing calls it now: see inline atcrates/trusted-server-core/src/ec/registry.rs:358 - Keep one definition of "runs pull sync": see inline at
crates/trusted-server-core/src/ec/registry.rs:395
🤔 thinking
- The pull-sync cap is logged as
skipped_rate_limited: see inline atcrates/trusted-server-core/src/ec/coordinator/mod.rs:578
⛏ nitpick
- Describe where pull sync runs now (suggestion): see inline at
docs/guide/edge-cookies.md:268 - The
Nonecase cannot happen (suggestion): see inline atcrates/trusted-server-identity-providers/src/lib.rs:71
Cross-cutting / body-level findings
- 📝 "Behaviour changes from
main" is missing items. In addition to the four already listed:- Browser
ts-eidsandsharedIdUIDs are no longer stored for pull-sync sources. See the question atregistry.rs:274. - Pull sync runs only on Fetch Metadata navigations that get a page-load permit. It no longer runs after
/auctionresponses, on navigations without Fetch Metadata, or on prefetches. - Each pull used to have a 15-second first-byte timeout. Pull sync now shares the 2-second
identity_after_response_budget_mswith every after-response provider, so a partner slower than that is never filled. pull_sync_concurrency = 0used to mean one request at a time (.max(1), with every partner still pulled). It now means no limit.- A steady-state returning navigation makes up to two more KV reads after the response. See the inline comment at
app.rs:453and the open thread onpull_sync.rs:60.
- Browser
- 📝 Earlier findings are still open. ChristianPavilonis's two findings apply unchanged at this head, and that thread has no reply yet:
- 🔧 malformed pull tokens (
pull_sync.rs:60) - ♻️ the follow-up read when every result is
NotDue
- 🔧 malformed pull tokens (
Verification
- Scratch tests at this head did three things:
- reproduced the browser-ingest change (
trusted-server-identity-providers, native target) - reproduced the mirror's vendor-consent gap (
ec/mirror.rs,wasm32-wasip1under Viceroy) - confirmed that the double-claim bypass works
- reproduced the browser-ingest change (
- I applied each suggestion to a scratch copy, first alone and then together with the others:
- The config test passed rustfmt, the new test itself, all six adapter clippy aliases,
test-fastly,test-axum,test-cloudflare,test-spinand the parity suite. - The doc comment passed rustfmt and
clippy-fastly. - The guide line passed prettier.
- The scratch changes were then discarded.
- The config test passed rustfmt, the new test itself, all six adapter clippy aliases,
- The prose fixes (mirror opt-in, vendor filter,
any_due,runs_pull_sync) have not been compiled.
CI Status
- browser integration tests: PASS
- cargo check (cloudflare native + wasm32-unknown-unknown): PASS
- cargo check/build/test (spin native + wasm32-wasip1): PASS
- cargo fmt: PASS
- cargo test: PASS
- cargo test (axum native): PASS
- cargo test (cross-adapter parity): PASS
- cargo test (ts CLI, native) (macos-latest): PASS
- cargo test (ts CLI, native) (ubuntu-latest): PASS
- CI Node pin guard: PASS
- CLAUDE.md symlink guard: PASS
- docs-proposal scripts: PASS
- format-docs: PASS
- format-typescript: PASS
- HTML EOF runtime (Fastly): PASS
- integration tests: PASS
- integration tests (Fastly EC lifecycle): PASS
- prepare integration artifacts: PASS
- trace runtime and browser acceptance: PASS
- ts dev lint domains (changed lines): PASS
- vitest: PASS
- worker-deadlines (cloudflare): PASS
- worker-deadlines (spin): PASS
No required checks are reported for this branch.
9f1e764 to
15eb284
Compare
|
@aram356 @ChristianPavilonis review-body points, addressed in 7360a0e (rebased onto #1487 at f844a52):
|
prk-Jr
left a comment
There was a problem hiding this comment.
Summary
This round addresses every earlier finding: the mirror is opt-in per partner and honours TCF vendor consent, browser ingest still fills pull-sync sources, the after-response phase makes no KV read when nothing is due, malformed pull tokens fail at startup, and the concurrency cap logs as skipped_concurrency_cap with hourly rotation. HTTPS and allowlist checks for pull-sync URLs, the mirror's cookie encoding and __Host- attributes, the bot gate, the fail-closed limiter and the double-claim rejection also look right. Three non-blocking points remain.
Non-blocking
🤔 thinking
- Retiring the completeness marker brings back a pre-response KV read — see below.
♻️ refactor
provider_ownedhas no production reader after the ingest split — see inline atcrates/trusted-server-core/src/ec/registry.rs:276-283
⛏ nitpick
- Unused
logdependency — see inline atcrates/trusted-server-identity-providers/Cargo.toml:20
Cross-cutting / body-level findings
- 🤔 Retiring the completeness marker brings back a pre-response KV read. On the base branch, a valid
ts-ec-pull-completemarker let a permitted navigation skip the row preload (should_preload_ec_snapshot'smarker_can_skip) when no auction, browser EID cookie or withdrawal needed the row. With the marker gone,should_preload_ec_snapshot(publisher.rs:3466-3473) returnseligiblefor every permitted navigation. Deployments with pull-sync partners therefore read KV on steady-state returning navigations that have no auction and nots-eids/sharedIdcookie, where they previously read nothing for up to an hour. In the default inline mode the read overlaps the origin fetch, so the cost is mostly KV operations; with template-cache keys the origin waits for it. Dropping the marker also removes its periodicSet-Cookie, which is a fair trade. Could you add this to "Behaviour changes frommain", or confirm it is intended? The guide says the marker is retired but not that the read it avoided is back.
CI Status
All 29 check runs on 15eb284f4 passed.
ChristianPavilonis
left a comment
There was a problem hiding this comment.
Summary
This PR moves pull sync into after-response identity collection and adds an opt-in, bounded mirror cookie for provider-written bidstream UIDs. Two correctness issues remain: clearing an existing mirror after vendor-consent withdrawal, and preserving the default partner configuration schema during mixed-version deployment or rollback.
Both inline comments carry a one-click GitHub
suggestion. Use Commit suggestion, or Add suggestion to batch, to apply the proposed replacements.
Blocking
🔧 wrench
- Clear an existing mirror when vendor consent is withdrawn (suggestion) — see inline at
crates/trusted-server-core/src/ec/finalize.rs:99–114. - Omit default
mirror_enabledfrom serialized configuration (suggestion) — see inline atcrates/trusted-server-core/src/settings.rs:453.
Review scope and verification
- Reviewed the stacked delta against
feat/s2s-eid-collection-coreat head15eb284f495c5fb9554a04b178804a4776d63965. - Coverage limitation: large unchanged portions of
settings.rsandpublisher.rs, especially their tests, were not exhaustively examined end to end. This is not an exhaustive whole-file review. Full reads were completed for Fastlyapp.rs,main.rs,config.rs, and the deleted completeness-marker implementation. - Each suggestion was applied and verified independently against the clean reviewed head: rustfmt; all eight clippy aliases; Fastly, Axum and Cloudflare check aliases; Fastly tests with the repository-pinned Viceroy 0.21.0 (including the reusable-sandbox variant), Axum, Cloudflare and Spin tests; and cross-adapter parity tests.
- An out-of-tree native public-API harness exercised the mirror replacement with a real encoded cookie: vendor withdrawal on a non-navigation, withdrawal with an unavailable navigation snapshot, unchanged consent controls, no-cookie handling, and retaining another consented vendor on a readable navigation.
- The serialization replacement was checked with omitted, explicit-false and string-false inputs against the base partner key set; explicit
truesurvived serialization and deserialization. - Byte-exact pre/post-verification patch comparisons passed for both suggestions. Scratch edits were discarded and the review worktree was restored to the reviewed head. No commits or pushes were made.
- No manual browser or deployed-service validation was performed locally.
CI Status
- browser integration tests: PASS
- integration tests (Fastly EC lifecycle): PASS
- trace runtime and browser acceptance: PASS
- integration tests: PASS
- CodeQL: PASS
- docs-proposal scripts: PASS
- format-docs: PASS (required)
- Analyze (javascript-typescript): PASS
- cargo check (cloudflare native + wasm32-unknown-unknown): PASS
- format-typescript: PASS (required)
- cargo test: PASS (required)
- CI Node pin guard: PASS
- HTML EOF runtime (Fastly): PASS
- cargo test (ts CLI, native) (macos-latest): PASS
- cargo fmt: PASS (required)
- cargo check/build/test (spin native + wasm32-wasip1): PASS
- Analyze (actions): PASS
- Analyze (rust): PASS
- worker-deadlines (spin): PASS
- vitest: PASS
- worker-deadlines (cloudflare): PASS
- cargo test (axum native): PASS
- CLAUDE.md symlink guard: PASS
- cargo test (cross-adapter parity): PASS
- cargo test (ts CLI, native) (ubuntu-latest): PASS
- prepare integration artifacts: PASS
- ts dev lint domains (changed lines): PASS
- Analyze (python): PASS
- Analyze (javascript-typescript): PASS
aram356
left a comment
There was a problem hiding this comment.
Summary
This round fixes all eleven of my round-1 findings, and the rebase onto #1487's startup-built coordinators is clean. I found no new code defects. What still blocks are two documentation gaps: the design spec now describes the behaviour this PR replaced, and the CHANGELOG has no entry, including for one breaking change.
2 of the 4 inline comments carry a one-click GitHub
suggestion. Use Commit suggestion, or Add suggestion to batch to apply both at once. The other two describe changes outside the diff.
Blocking
🔧 wrench
- Update the design spec to match this PR: body-level, below
- Add CHANGELOG entries, including one Breaking item: body-level, below
Non-blocking
♻️ refactor
- Carry
runs_pull_syncontoPartnerConfigto simplify the double-claim check: see inline atcrates/trusted-server-core/src/config.rs:384
🤔 thinking
- Reject
mirror_enabledwithoutbidstream_enabled: see inline atcrates/trusted-server-core/src/settings.rs:454
⛏ nitpick
- Give
holds_live_uidits doc line back (suggestion): see inline atcrates/trusted-server-core/src/ec/coordinator/mod.rs:294-306 - Assert the specific token error (suggestion): see inline at
crates/trusted-server-core/src/config.rs:1502-1506
Cross-cutting / body-level findings
-
🔧 Update the design spec to match this PR.
docs/superpowers/specs/2026-10-30-s2s-eid-collection-design.mdwas added in #1487 (889159cd6) and is not onmainyet. It describes this PR as PR5 and PR6, but the behaviour changed in this round:- D6 (line 81) and §10 Contents (lines 1407-1408): say the mirror holds every
bidstream_enabledsource. It now needsmirror_enabledas well, and it leaves out UIDs whose owning provider's TCF vendor consent is denied. - §10 Set and Expired (lines 1412-1418): say the cookie is set on every consented navigation, and expired on any response without EID consent and on any navigation whose mirror would be empty. The code instead sets it only when the value changes, expires only a cookie the request carried, and leaves it untouched when the row cannot be read.
- PR5 row (line 1529) and acceptance scenario 21 (lines 1597-1600): say browser ingest skips pull-sync sources and batch sync rejects them, and that a browser
ts-eidsUID and a batch record are "ignored or rejected". Both now keep filling the source, and pull sync fills it only when it holds no UID. - PR6 row (line 1530),
IdentityCoordinator::mirror(line 1059) and theapply_cookiescall (line 1228): describe the mirror as a coordinator method. The code builds it withec::mirror::build_mirror(row, consent, partners, now), called fromec_finalize_response.
Since the spec lands with this stack, please update these sections in this PR, so reviewers and later work read the design that ships.
- D6 (line 81) and §10 Contents (lines 1407-1408): say the mirror holds every
-
🔧 Add CHANGELOG entries, including one Breaking item.
[Unreleased]carries #1487's provider entry but nothing for this PR's user-visible changes.- Breaking. On
main, a resolvedts_pull_tokenthat cannot form a header value made the old dispatcher's.expect("should build pull sync request")panic after the page was sent, so only pull sync failed. It now fails config validation, and the service does not start. A secret value with a trailing newline is the likely way to hit this. - A possible wording, under
### Changed:
- Pull sync now runs as a `pull-sync` identity provider built from the existing `[[ec.partners]]` settings, after the response on top-level publisher navigations from recognized browsers, within `ec.identity_after_response_budget_ms` shared with other after-response providers. It no longer runs after `/auction` responses, on prefetches or on navigations without Fetch Metadata, and a partner slower than the budget is not filled, where each pull used to wait up to 15 seconds. `ec.pull_sync_concurrency` now caps the partners pulled per navigation (`0` means no limit; it used to mean one at a time), partner order rotates hourly, and a partner past the cap logs `skipped_concurrency_cap`. The `ts-ec-pull-complete` marker is retired and leftover cookies are expired. `kind = "pull-sync"` in `[[ec.identity_providers]]` is rejected, as is a configured provider that claims a pull-sync partner's source. - Added `mirror_enabled` on `[[ec.partners]]` (default `false`). With `bidstream_enabled`, the partner's provider-written UIDs are mirrored to the JavaScript-readable `__Host-ts-eids-mirror` cookie, leaving out UIDs whose owning provider lacks TCF vendor consent. See [EID Mirror Cookie](docs/guide/edge-cookies.md#eid-mirror-cookie). - **Breaking:** For a partner with pull sync enabled, a resolved `ts_pull_token` that cannot form an `Authorization: Bearer` header, such as one ending in a newline, now fails config validation and stops startup. Before, only that partner's pull sync failed, after the page was sent. Check pull-token secrets before deploying.
- Breaking. On
-
📝 Earlier findings.
- Mine: all eleven round-1 findings are addressed at this head. I checked each in the code. The browser-ingest repro now stores the browser UID for a pull-sync source, and the vendor-consent and bypass cases have tests.
- Still open from other reviewers at this head:
- ChristianPavilonis's two 🔧: the mirror after a vendor withdrawal (
finalize.rs:114), and serializing a defaultmirror_enabled(settings.rs:453). - prk-Jr's ♻️ on
provider_owned(registry.rs:283). - prk-Jr's ⛏ on the unused
logdependency. - prk-Jr's 🤔 on the preload read that the retired marker used to skip.
- ChristianPavilonis's two 🔧: the mirror after a vendor withdrawal (
Verification
- At this head, the round-1 browser-ingest scratch test passes with the expected values: the UID is stored both with and without pull sync.
- I applied each suggestion to a scratch copy, first alone and then together:
- The assertion change passed rustfmt, its test, all six adapter clippy aliases,
test-fastly,test-axum,test-cloudflare,test-spinand the parity suite. - The doc-line move passed rustfmt and
clippy-fastly. - The scratch changes were then discarded.
- The assertion change passed rustfmt, its test, all six adapter clippy aliases,
- The prose proposals (
runs_pull_synconPartnerConfig, rejectingmirror_enabledwithout bidstream) have not been compiled.
CI Status
- Analyze (actions): PASS
- Analyze (javascript-typescript): PASS
- Analyze (javascript-typescript): PASS
- Analyze (python): PASS
- Analyze (rust): PASS
- browser integration tests: PASS
- cargo check (cloudflare native + wasm32-unknown-unknown): PASS
- cargo check/build/test (spin native + wasm32-wasip1): PASS
- cargo fmt: PASS (required)
- cargo test: PASS (required)
- cargo test (axum native): PASS
- cargo test (cross-adapter parity): PASS
- cargo test (ts CLI, native) (macos-latest): PASS
- cargo test (ts CLI, native) (ubuntu-latest): PASS
- CI Node pin guard: PASS
- CLAUDE.md symlink guard: PASS
- CodeQL: PASS
- docs-proposal scripts: PASS
- format-docs: PASS (required)
- format-typescript: PASS (required)
- HTML EOF runtime (Fastly): PASS
- integration tests: PASS
- integration tests (Fastly EC lifecycle): PASS
- prepare integration artifacts: PASS
- trace runtime and browser acceptance: PASS
- ts dev lint domains (changed lines): PASS
- vitest: PASS
- worker-deadlines (cloudflare): PASS
- worker-deadlines (spin): PASS
15eb284 to
29de3d4
Compare
|
aram356
left a comment
There was a problem hiding this comment.
Summary
This round addresses every earlier finding from all three reviewers, and the rebase onto #1487's latest head is clean. One defect remains, in the vendor-withdrawal expiry added this round: for a visitor who denies one mirrored vendor, the mirror is cleared after every page view and set again on the next navigation.
1 of the 2 inline comments carries a one-click GitHub
suggestion; the other describes a fix that spans two files.
Blocking
🔧 wrench
- Expire the mirror only when it holds a denied vendor's UIDs: see inline at
crates/trusted-server-core/src/ec/finalize.rs:144
Non-blocking
⛏ nitpick
- Drop "pull sync skips them" from the provider entry (suggestion): see inline at
CHANGELOG.md:24
Cross-cutting / body-level findings
- 📝 Earlier findings, checked at this head.
- Mine from round 2: all fixed. These were the spec drift, the CHANGELOG (including the Breaking pull-token entry),
runs_pull_synconPartnerConfig, rejectingmirror_enabledwithoutbidstream_enabled(at startup and atts config push, both tested), theholds_live_uiddoc line, and the exact token assertion. - ChristianPavilonis's two:
- The vendor-withdrawal expiry is in place; the 🔧 above is about its scope.
- A default
mirror_enabledis no longer serialized, with round-trip tests.
- prk-Jr's three:
provider_ownedagain means only an enabled configured provider.- The unused
logdependency is gone. - The pre-response row read is documented in the guide, the CHANGELOG and the description.
- Mine from round 2: all fixed. These were the spec drift, the CHANGELOG (including the Breaking pull-token entry),
Verification
- A scratch test at this head reproduces the cycle with two mirrored partners, one whose vendor is granted and one whose vendor is denied: navigation sets the mirror, a non-navigation expires it, and the next navigation sets it again.
- I applied the proposed fix in a scratch copy. It compiled and passed rustfmt and all 30 mirror and finalize tests (
wasm32-wasip1under Viceroy), and the cycle stopped. It has not been through the full adapter gate. The scratch changes were discarded. - The
CHANGELOG.mdsuggestion passes the repository's prettier check, and only ", pull sync skips them" changes. - The rebase onto #1487's split vendor gate holds:
keep_vendor_gatewrites the sameowner_tcf_vendor_idthe mirror reads, so a disabled provider's stored UIDs stay gated there as they are ineids.rsandidentify.rs.
CI Status
- Analyze (actions): PASS
- Analyze (javascript-typescript): PASS
- Analyze (javascript-typescript): PASS
- Analyze (python): PASS
- Analyze (rust): PASS
- browser integration tests: PASS
- cargo check (cloudflare native + wasm32-unknown-unknown): PASS
- cargo check/build/test (spin native + wasm32-wasip1): PASS
- cargo fmt: PASS (required)
- cargo test: PASS (required)
- cargo test (axum native): PASS
- cargo test (cross-adapter parity): PASS
- cargo test (ts CLI, native) (macos-latest): PASS
- cargo test (ts CLI, native) (ubuntu-latest): PASS
- CI Node pin guard: PASS
- CLAUDE.md symlink guard: PASS
- CodeQL: PASS
- docs-proposal scripts: PASS
- format-docs: PASS (required)
- format-typescript: PASS (required)
- HTML EOF runtime (Fastly): PASS
- integration tests: PASS
- integration tests (Fastly EC lifecycle): PASS
- prepare integration artifacts: PASS
- release scripts: PASS
- trace runtime and browser acceptance: PASS
- ts dev lint domains (changed lines): PASS
- vitest: PASS
- worker-deadlines (cloudflare): PASS
- worker-deadlines (spin): PASS
Signed-off-by: dhruv8sh <dhruv8sh@proton.me>
29de3d4 to
d1440d6
Compare
Signed-off-by: dhruv8sh <dhruv8sh@proton.me>
d1440d6 to
6ffb613
Compare
Brings PR #1489's pull-sync provider and EID mirror onto PR #1488's new head, which carries PR #1487 onto PR #1526 (EdgeZero cutover, local partner API answers, HTTP auction opt-in and awaitable EC persistence). How each area was resolved: - Pull sync: core `ec/pull_sync.rs` stays deleted, so the base's EdgeZero group sender and its tests go with it. Pull sync runs as PR #1489's `pull-sync` after-response provider, sent through the post-delivery client in the lookup batch. Post-delivery work is now telemetry, then after-response lookups; the separate pull-sync step, its graph factory and the unused clock are removed, and PR #1489's "nothing due" check runs before the live row read. `require_identity_graph`, now used only by tests, is test-only. - Coordinator: PR #1489's dispatch order and per-navigation pull-sync cap run inside PR #1487's batch-shaped dispatch; the cap counts each queued pull-sync request. - Fastly app: `EcFinalizeState` drops both the old `services` and `is_real_browser`; the bot gate moves into after-response capture as in PR #1489, with the base's request services and async batch sync. - Async KV and finalize: PR #1489's removals in `kv.rs` (snapshot partner upsert) and the pull-sync marker stand; the mirror and the inner finalize are awaited, and PR #1489's new tests await the store. - Publisher: keeps the base's EdgeZero origin request without the marker preload inputs PR #1489 removed. - Providers crate: PR #1489's pull-sync providers and PR #1488's SharedID cookie defaulting both apply when the phase coordinators are built. - Docs and CHANGELOG: the base's auction wording with PR #1489's identity bullets; the onboarding step names after-response lookups.
|
For the identity integration chain (#1608), this PR's base #1488 now carries #1487, #1557, #1526, #1525, #1614 and #1506's EdgeZero cutover.
Behavior that changes with the move:
Tests changed beyond the async conversion: The base stays Checked locally at the new head: |
…ntity #1489's head a7262c7 stacks on #1488 (8183c3c), which stacks on #1487 (64674f9); each merged the one below it and resolved its own conflicts, so this merge needs no resolution. It completes the provider stack (#1567): the vendor identity providers and cookie hooks (#1488), and the bounded identity mirror with pull sync as an after-response provider on EdgeZero's outbound client (#1489).
Core identity app. IdentityApp now carries the provider stack's identity state as well as its own: the partner registry, the page and after-response provider coordinators (#1487, #1489), the EC record store and the rate limiter. IdentityPlatform takes an optional IdentityProviders (the registry the coordinators were built against plus both coordinators); without one the app builds the registry with PartnerRegistry::from_ec and runs no provider, matching the base's non-Fastly adapters. The admin EC lookup awaits the async handler from #1526. Send and Sync. #1526 keeps EcKvStore free of Send and Sync bounds and tests a non-Send store, so the T21 supertrait bounds on EcKvStore and RateLimiter are dropped. The identity app instead holds its record store and rate limiter as Send and Sync trait objects, and IdentityApp::identity_graph hands out a graph over that one shared store through the new KvIdentityGraph::from_shared. IdentityPlatform's identity_graph field becomes record_store accordingly. Fastly app. AppState drops the base's partner_registry, page_identity and after_response_identity fields for the identity app. build_identity_app builds the kind-aware registry and both coordinators from the identity-providers crate, then the app with Fastly's record store and rate limiter. Page lookups and after-response preparation read their coordinators and rate limiter from the identity app; identify, batch sync, auction, page bids and the publisher fallback use its registry and graph. The admin lookups stay mounted by the identity app, so the base's async admin early dispatch is removed. Fastly entry point. finalize_response keeps the base's single block_on around the async EC finalize, which collects the page lookups first. FinalizeInputs keeps the base's AppState, and FinalizeState carries the identity app and the after-response lookups. maybe_identity_graph is gone; the test-only require_identity_graph reads the identity app's graph. Post-delivery. The after-response providers, pull sync included, take the partner registry, graph and rate limiter from the identity app carried in FinalizeState. The base's pull-sync tests build their state through the identity app. Axum, Cloudflare and Spin. The identity app mounts the admin lookups; the base's provider rejection at startup and #1614's local 501 and 405 partner API answers are kept. Tests. The core identity app tests use record stores and the async graph, and two tests cover the carried registry and coordinators. The Fastly identity fixture builds an identity app around its echo-provider coordinator. The adapter identity guard replaces dispatch_pull_sync, which #1489 deleted, with commit_after_response_lookups and also guards PartnerRegistry::from_ec.
The OpenRTB tests keep this pull request's EID helper imports alongside the base's scripted-client imports for the provider round-trip tests. The base's wire-level auction tests now read EIDs where this pull request writes them. The vendor-denied body EID test and the EC identity denied test in the auction endpoint, and the template cache reader-identity test in the publisher, read user.eids through the shared user_eids and assert_no_user_eids helpers instead of user.ext.eids, since the standard profile no longer writes the extension field. The APS no-EID test drops the transport_timeout_ms field that the base removed from AuctionContext.
The merge applied without conflicts. The Prebid integration config keeps this pull request's mode field next to the base's changes, and the example config, the Prebid guide and the tsjs Prebid sources take both sides unchanged. The server mirror cookie the identity mode reads is the same name and envelope in the base, so no code needed to change.
Fastly entry point: serve keeps the base's request-local identity lookup slot, dropped before and after the request, and sends through this pull request's with_finalization helper, so the conformance runner and production share one finalization callback. FinalizeInputs now carries the app state, and the conformance runner passes the retained app's state. The runner also drops leftover navigation lookups around each request, as serve does. EC store test support: the module opens to the test-utils feature as this pull request intended. Every test-only double, including the base's new write-failing store, stays behind cfg(test); only the in-memory store is shared. Conformance fixture: the recording store implements the async EcKvStore trait and awaits the in-memory store. Scenario::verify and the native rejection-control tests drive store reads and writes with block_on at their synchronous test boundary, because the adapter runner already blocks inside the Fastly lifecycle. The conditional-write test checks the recording store's create-only insert directly, since the base removed KvIdentityGraph::create_if_absent. Fastly runner: a document navigation now sends Sec-Fetch-Mode: navigate next to Sec-Fetch-Dest: document, and a subresource sends no-cors. The base's page-load write permit requires both headers, as a browser sends them. The integration-tests README says post-delivery work now covers the after-response identity lookups, pull sync among them.
Pull sync: the base deletes core ec/pull_sync.rs, so the merge takes the deletion. This pull request changed only that file's tests, adding the differential certificate-check test, so the test moves to where pull sync now sends: the Fastly post-delivery work, which builds the after-response pull-sync provider from the settings and sends its lookup through OutboundLookupTransport. The ported test runs post-delivery work with certificate_check true and false, each against its own fresh live row in the local identity store, and asserts the same method, URL, headers and body, the lookup transport's timeout, buffered cap, decoding and cache policy, and that both runs store the partner UID. The pull-sync test helpers take the EC ID so this test never writes the row the other pull-sync tests share. Docs: the certificate_check guide entry names identity providers, pull sync among them, since pull sync is one in the base. The config message, settings doc comment, guides and example config apply this pull request's wording unchanged.
Brings the provider stack, including the EdgeZero cutover (#1506), into the cross-adapter health and startup-failure contract, and keeps the contract on every adapter. Core: the probe pre-dispatch hook takes EdgeZero's PreDispatchRequest, and its tests drive the hook through a router with run_pre_dispatch_hook, because PreDispatchRequest cannot be built outside EdgeZero. Fastly: the cutover's handle_request, assemble_app and finalize_response stay. Liveness is built from the shared probe response and sent natively. A new initialize_dispatch runs the sandbox initialization, answers GET and HEAD /ready from its result, and answers a failed assembly on the seven publisher fallback methods with the fixed 503, logging the assembly error that EdgeZero's assembly-error response would have logged. Other methods keep EdgeZero's assembly-error response. finalize_response skips operational responses as it skips trace responses. The retained router wraps the trace hook in the probe hook. Fastly needs no failure router, so the PR's Fastly startup_error_router and its matrix test go; the matrix moves to main.rs and runs through initialize_dispatch and the native conversion. Axum, Cloudflare and Spin: Hooks::routes logs the report and returns the PR's failure router instead of failing assembly, so liveness stays 200 and /ready and every supported route answer the fixed 503. The routers keep the cutover's middleware and Axum's received-head preparer, with the probe hook around the trace hook. The cutover's EdgeZero managed logger already gives Cloudflare and Spin a production log sink, so the PR's adapter loggers are dropped. Tests: the cutover's startup tests are moved to the contract. Fastly's missing-blob test checks the fixed 503 for GET, the assembly error for TRACE, then recovery. The Cloudflare and Spin routes() tests serve the failure router. The Axum binary test now requires the binary to keep listening with live health and a 503 /ready and page, in place of a non-zero exit. The two Axum connection-entry tests move from /health, which now bypasses configured response headers, to the identify preflight. The probe parity test takes the cutover's forbidden-router tuple and also asserts that probes send no outbound request. Docs: the API reference, adapter guides, READMEs, AGENTS.md and CHANGELOG keep both sides. Startup statements now describe the contract: no Axum exit, no Spin /health 503, no Cloudflare JSON 500, with the Fastly logger-failure case and the Fastly assembly error for unsupported methods kept. The smoke-axum.sh negative cases stay on the PR's 503 contract with EdgeZero's port variable.
Summary
__Host-ts-eids-mirrorcookie so publisher-side code can read them, on a best-effort basis. The mirror is opt-in per partner (mirror_enabled, default off) and leaves out UIDs whose owning provider's TCF vendor consent is denied.Stacked on #1487; review only the commits on top of it.
Changes
trusted-server-identity-providers/src/pull_sync.rs; removes coreec/pull_sync.rs/ec/pull_sync_marker.rsand the old post-send pull sync path in the Fastly adapter. Pull-sync providers are built withcollect = "after_response", so they only run for a real browser's EC whose consented row already existsec/mirror.rs;ec/finalize.rs,ec/cookies.rs,constants.rswrite the__Host-ts-eids-mirrorcookieapi_tokenkeeps batch syncSet-Cookie, so returning visitors' pages stay cacheablekind = "pull-sync"in[[ec.identity_providers]]is rejected; pull sync comes only from partner config with its token and rate limit__Host-ts-eids-mirrorjoins the identity cookies the template cache key refusesdocs/guide/edge-cookies.mdandconfiguration.mdMAX_STORED_UID_LENGTHfor provider writesmirror_enabledopt-in and vendor-consent filter for the mirror; browser ingest restored for pull-sync sources; real-browser gate inprepare_after_response_lookups; no after-response KV read when nothing is due (ProviderCoordinator::any_due) and no follow-up re-read when nothing is to persist;skipped_concurrency_cap; startup rejection of ats_pull_tokenthat cannot form anAuthorizationheader;EcPartner::runs_pull_sync;pull_enabled_partnersremovedBehaviour changes from
mainidentity_after_response_budget_ms(shared with other after-response providers), and only for an EC whose consented row already exists. Each pull used to have a 15-second first-byte timeout; a partner slower than the budget is now never filled./auctionresponses, on navigations without Fetch Metadata, or on prefetches.ts-eidsandsharedIdUIDs still fill a pull-sync partner's source; pull sync only fills a source that holds no UID.pull_sync_concurrencyis a cap per navigation. It used to be a batch width with every eligible partner pulled; partners now rotate hourly so each gets turns under the cap, and partners past it logskipped_concurrency_cap.0used to mean one request at a time (.max(1), every partner still pulled); it now means no limit.pull_sync_rate_limitis enforced per minute with a minimum of one on Fastly, so the default of 10 per hour behaves like 60.bidstream_enabledandmirror_enabledthey reach the script-readable mirror cookie. Previously they lived only in KV. The mirror is off by default.Set-Cookieis gone.ts_pull_tokenthat cannot form anAuthorization: Bearerheader now fails config validation instead of failing the provider build.Closes
Part of #1246 and #1245; the provider implementations that close them come in follow-up PRs.
Spec
Spec: reviewed separately; not included in this PR.
Test plan
cargo test-fastly && cargo test-fastly-reuse && cargo test-axum && cargo test-cloudflare && cargo test-spincargo fmt --all -- --checkscripts/lint-wasm-clock.sh./scripts/test-cli.sh, domain lint against the base branch, CI Node pin guard, docs-proposal scriptsfastly compute serveChecklist
unwrap()in production code — useexpect("should ...")logmacros (notprintln!)Closes #1567