Skip to content

Mirror provider EIDs to a cookie and run pull sync as an identity provider - #1489

Open
dhruv8sh wants to merge 33 commits into
feat/s2s-eid-vendorsfrom
feat/s2s-eid-collection-providers
Open

dhruv8sh wants to merge 33 commits into
feat/s2s-eid-vendorsfrom
feat/s2s-eid-collection-providers

Conversation

@dhruv8sh

@dhruv8sh dhruv8sh commented Nov 2, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

  • Turns pull sync into an identity provider that runs in the after-response phase from Collect server-side identity provider EIDs in a single page-load write #1487, so pull-synced partner IDs flow through the same coordinator and KV writes as other providers without adding page latency.
  • Mirrors provider-owned bidstream UIDs to a JS-readable __Host-ts-eids-mirror cookie so publisher-side code can read them, on a best-effort basis. The mirror is opt-in per partner (mirror_enabled, default off) and leaves out UIDs whose owning provider's TCF vendor consent is denied.

Stacked on #1487; review only the commits on top of it.

Changes

Commit Change
Run pull sync as an identity provider on navigations New trusted-server-identity-providers/src/pull_sync.rs; removes core ec/pull_sync.rs / ec/pull_sync_marker.rs and the old post-send pull sync path in the Fastly adapter. Pull-sync providers are built with collect = "after_response", so they only run for a real browser's EC whose consented row already exists
Mirror provider-owned bidstream UIDs to a JS-readable cookie New ec/mirror.rs; ec/finalize.rs, ec/cookies.rs, constants.rs write the __Host-ts-eids-mirror cookie
Make mirroring best effort and fix pull sync validation Mirror failures no longer fail the response; the mirror cookie is expired only when the browser sent one; pull sync config validation fixes
Stop mirroring ownerless UIDs and re-pulling held pull-sync sources Only owned UIDs are mirrored; ownerless batch records count as held for pull sync
Keep batch sync for pull-sync partners Batch sync rejects only sources owned by a configured identity provider, so a partner with both pull sync and an api_token keeps batch sync
Set the mirror cookie only when its value changes An unchanged mirror emits no Set-Cookie, so returning visitors' pages stay cacheable
Skip identity lookups when the rate limiter cannot count A rate-limiter error now skips the lookup (as pull sync did before) for every provider
Reject the pull-sync kind in identity provider config kind = "pull-sync" in [[ec.identity_providers]] is rejected; pull sync comes only from partner config with its token and rate limit
Refuse to key the template cache on the EID mirror cookie __Host-ts-eids-mirror joins the identity cookies the template cache key refuses
Rotate pull-sync partners hourly Partners beyond the per-navigation cap still get turns
Docs Rate-limit rounding, the per-navigation cap and the mirror exposure callout in docs/guide/edge-cookies.md and configuration.md
Tests Pull-sync results mirrored with their owner, non-navigations leaving the mirror untouched, withdrawn rows expiring it
Test the pull-sync UID cap at the provider write limit Pull-sync UIDs follow #1487's MAX_STORED_UID_LENGTH for provider writes
Address review of pull sync and the EID mirror mirror_enabled opt-in and vendor-consent filter for the mirror; browser ingest restored for pull-sync sources; real-browser gate in prepare_after_response_lookups; no after-response KV read when nothing is due (ProviderCoordinator::any_due) and no follow-up re-read when nothing is to persist; skipped_concurrency_cap; startup rejection of a ts_pull_token that cannot form an Authorization header; EcPartner::runs_pull_sync; pull_enabled_partners removed

Behaviour changes from main

  • Pull sync timing. Pull sync now runs in the after-response phase, within identity_after_response_budget_ms (shared with other after-response providers), and only for an EC whose consented row already exists. Each pull used to have a 15-second first-byte timeout; a partner slower than the budget is now never filled.
  • Where pull sync runs. Only on Fetch Metadata navigations from recognized browsers that get a page-load permit. It no longer runs after /auction responses, on navigations without Fetch Metadata, or on prefetches.
  • Browser ingest unchanged. Browser ts-eids and sharedId UIDs still fill a pull-sync partner's source; pull sync only fills a source that holds no UID.
  • pull_sync_concurrency is a cap per navigation. It used to be a batch width with every eligible partner pulled; partners now rotate hourly so each gets turns under the cap, and partners past it log skipped_concurrency_cap. 0 used to mean one request at a time (.max(1), every partner still pulled); it now means no limit.
  • Rate limits. pull_sync_rate_limit is enforced per minute with a minimum of one on Fastly, so the default of 10 per hour behaves like 60.
  • Mirror exposure. Pull-sync UIDs now carry an owner, so for partners with bidstream_enabled and mirror_enabled they reach the script-readable mirror cookie. Previously they lived only in KV. The mirror is off by default.
  • Pre-response KV read on returning navigations. The pull-sync marker cookie is retired, so a returning permitted navigation reads the EC row before the origin response even when no auction, EID cookie or withdrawal needs it. In inline mode the read overlaps the origin fetch, and the marker's periodic Set-Cookie is gone.
  • Pull token validation. A ts_pull_token that cannot form an Authorization: Bearer header now fails config validation instead of failing the provider build.

Closes

Part of #1246 and #1245; the provider implementations that close them come in follow-up PRs.

Spec

Spec: reviewed separately; not included in this PR.

Test plan

  • cargo test-fastly && cargo test-fastly-reuse && cargo test-axum && cargo test-cloudflare && cargo test-spin
  • All eight clippy aliases
  • cargo fmt --all -- --check
  • Parity tests, native template build-digest test and clippy, scripts/lint-wasm-clock.sh
  • JS tests, format and lint
  • Docs format and Markdown prettier check
  • ./scripts/test-cli.sh, domain lint against the base branch, CI Node pin guard, docs-proposal scripts
  • HTML EOF checks (native harness, Axum and Fastly auction controls)
  • Fastly EC integration tests (left to CI)
  • Manual testing via fastly compute serve

Checklist

  • Feature PRs: spec linked above, ending in a review log of two or more agent review rounds
  • Changes follow AGENTS.md conventions
  • No unwrap() in production code — use expect("should ...")
  • Uses log macros (not println!)
  • New code has tests
  • No secrets or credentials committed

Closes #1567

@dhruv8sh dhruv8sh self-assigned this Nov 2, 2026
@dhruv8sh
dhruv8sh force-pushed the feat/s2s-eid-collection-providers branch from db9e915 to bdd12a8 Compare November 3, 2026 14:40
@dhruv8sh
dhruv8sh marked this pull request as ready for review November 5, 2026 16:26
@aram356 aram356 added this to the 202611 milestone Nov 5, 2026
@dhruv8sh
dhruv8sh force-pushed the feat/s2s-eid-collection-providers branch from bdd12a8 to 9f1e764 Compare November 6, 2026 12:47

@ChristianPavilonis ChristianPavilonis left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Summary

This PR moves pull sync into after-response identity collection and adds a bounded, JavaScript-readable mirror of provider-written bidstream UIDs. One startup-validation issue needs addressing; there is also an opportunity to avoid a redundant KV read on steady-state navigations.

The inline comment is prose-only: the proposed fix and regression coverage touch lines outside the diff, so there are no one-click suggestions.

Blocking

🔧 wrench

  • Reject malformed pull tokens during startup — see inline at crates/trusted-server-identity-providers/src/pull_sync.rs:59-60.

Cross-cutting / body-level findings

♻️ Avoid the follow-up KV read when all lookups were skipped

Introduced by crates/trusted-server-identity-providers/src/lib.rs:169-182; affected helper: crates/trusted-server-core/src/ec/after_response.rs:76-79.

When every pull-sync source already holds a UID, dispatch produces Skipped(NotDue) results. Those results are nonempty, so the follow-up helper rereads KV despite having nothing to persist. Moving existing pull-sync partners into this path adds an avoidable read on every eligible returning navigation.

Import LookupStatus and SkipReason alongside LookupResults, then replace the helper's empty-results guard with:

    if lookups.iter().all(|lookup| {
        lookup.entries.is_empty()
            && lookup.cleared.is_empty()
            && lookup.status != LookupStatus::Skipped(SkipReason::OptedOut)
    }) {
        return AfterResponseOutcome::Unchanged;
    }

This preserves writes that remember provider opt-outs.

Apply manually — the helper, imports, and regression test are outside the diff.

Verification

  • A focused scratch reproduction confirmed that a newline-containing pull token passes runtime config validation but prevents construction of the coordinator for both an invalid and an otherwise valid partner.
  • A counting-store scratch reproduction confirmed that all-NotDue results still cause one KV read in the follow-up helper.
  • Both reproductions passed. Scratch changes were removed and the worktree was restored to the reviewed head.
  • The proposed fixes have not been compiled or run through the full local gate.

CI Status

No required checks were reported by the required-check query.

@aram356 aram356 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Summary

This PR moves pull sync onto the after-response provider path and adds a JavaScript-readable mirror of provider-written bidstream UIDs. The move keeps the live-row consent re-check and the bot gate. Before the mirror ships, it needs an operator opt-in and a per-vendor consent check. The after-response path also adds KV reads that the legacy dispatcher avoided.

3 of the 10 inline comments carry a one-click GitHub suggestion. Use Commit suggestion, or Add suggestion to batch to apply several at once. The other comments describe the fix in prose, because those changes span files or touch lines outside the diff.

Blocking

🔧 wrench

  • Make the EID mirror opt-in, separate from bidstream_enabled: see inline at crates/trusted-server-core/src/ec/finalize.rs:90
  • Withhold mirrored UIDs whose provider lacks TCF vendor consent: see inline at crates/trusted-server-core/src/ec/mirror.rs:77
  • Skip the after-response row read when nothing is due: see inline at crates/trusted-server-adapter-fastly/src/app.rs:453
  • Cover the enabled = false bypass of the double-claim check (suggestion): see inline at crates/trusted-server-core/src/config.rs:1455

❓ question

  • Should pull-sync sources keep accepting browser UIDs?: see inline at crates/trusted-server-core/src/ec/registry.rs:274

Non-blocking

♻️ refactor

  • Delete pull_enabled_partners; nothing calls it now: see inline at crates/trusted-server-core/src/ec/registry.rs:358
  • Keep one definition of "runs pull sync": see inline at crates/trusted-server-core/src/ec/registry.rs:395

🤔 thinking

  • The pull-sync cap is logged as skipped_rate_limited: see inline at crates/trusted-server-core/src/ec/coordinator/mod.rs:578

⛏ nitpick

  • Describe where pull sync runs now (suggestion): see inline at docs/guide/edge-cookies.md:268
  • The None case cannot happen (suggestion): see inline at crates/trusted-server-identity-providers/src/lib.rs:71

Cross-cutting / body-level findings

  • 📝 "Behaviour changes from main" is missing items. In addition to the four already listed:
    • Browser ts-eids and sharedId UIDs are no longer stored for pull-sync sources. See the question at registry.rs:274.
    • Pull sync runs only on Fetch Metadata navigations that get a page-load permit. It no longer runs after /auction responses, on navigations without Fetch Metadata, or on prefetches.
    • Each pull used to have a 15-second first-byte timeout. Pull sync now shares the 2-second identity_after_response_budget_ms with every after-response provider, so a partner slower than that is never filled.
    • pull_sync_concurrency = 0 used to mean one request at a time (.max(1), with every partner still pulled). It now means no limit.
    • A steady-state returning navigation makes up to two more KV reads after the response. See the inline comment at app.rs:453 and the open thread on pull_sync.rs:60.
  • 📝 Earlier findings are still open. ChristianPavilonis's two findings apply unchanged at this head, and that thread has no reply yet:
    • 🔧 malformed pull tokens (pull_sync.rs:60)
    • ♻️ the follow-up read when every result is NotDue

Verification

  • Scratch tests at this head did three things:
    • reproduced the browser-ingest change (trusted-server-identity-providers, native target)
    • reproduced the mirror's vendor-consent gap (ec/mirror.rs, wasm32-wasip1 under Viceroy)
    • confirmed that the double-claim bypass works
  • I applied each suggestion to a scratch copy, first alone and then together with the others:
    • The config test passed rustfmt, the new test itself, all six adapter clippy aliases, test-fastly, test-axum, test-cloudflare, test-spin and the parity suite.
    • The doc comment passed rustfmt and clippy-fastly.
    • The guide line passed prettier.
    • The scratch changes were then discarded.
  • The prose fixes (mirror opt-in, vendor filter, any_due, runs_pull_sync) have not been compiled.

CI Status

No required checks are reported for this branch.

Comment thread docs/guide/edge-cookies.md Resolved
@aram356
aram356 added this pull request to stack #1515 November 9, 2026 05:49
@dhruv8sh
dhruv8sh force-pushed the feat/s2s-eid-collection-providers branch from 9f1e764 to 15eb284 Compare November 10, 2026 14:02
@dhruv8sh

dhruv8sh commented Nov 10, 2026 •

Copy link
Copy Markdown
Collaborator Author

@aram356 @ChristianPavilonis review-body points, addressed in 7360a0e (rebased onto #1487 at f844a52):

  • Follow-up KV read when every result is NotDue (ChristianPavilonis): commit_after_response_lookups now returns Unchanged without reading when no result has entries, cleared sources or an opt-out, so opt-out writes still land. Test: results_with_nothing_to_persist_skip_the_row_read. The first read is gone too when nothing is due (ProviderCoordinator::any_due, any_due_is_false_only_when_every_source_is_held), so a steady-state returning navigation whose pull sources are all held makes no after-response KV read.
  • "Behaviour changes from main" (aram356): the description now lists where pull sync runs (Fetch Metadata navigations from recognized browsers with a page-load permit; no longer after /auction, without Fetch Metadata, or on prefetches), the shared after-response budget replacing the 15-second first-byte timeout, pull_sync_concurrency = 0 now meaning no limit, and the ts_pull_token header validation. Browser ts-eids / sharedId ingest for pull-sync sources is restored rather than listed as a change (see the registry.rs:274 thread).
  • Earlier findings: the malformed-token thread is fixed and resolved.
  • Rebase note: pull-sync UIDs now follow Collect server-side identity provider EIDs in a single page-load write #1487's MAX_STORED_UID_LENGTH (1024) for provider writes, in a6f397f.

@prk-Jr prk-Jr left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Summary

This round addresses every earlier finding: the mirror is opt-in per partner and honours TCF vendor consent, browser ingest still fills pull-sync sources, the after-response phase makes no KV read when nothing is due, malformed pull tokens fail at startup, and the concurrency cap logs as skipped_concurrency_cap with hourly rotation. HTTPS and allowlist checks for pull-sync URLs, the mirror's cookie encoding and __Host- attributes, the bot gate, the fail-closed limiter and the double-claim rejection also look right. Three non-blocking points remain.

Non-blocking

🤔 thinking

  • Retiring the completeness marker brings back a pre-response KV read — see below.

♻️ refactor

  • provider_owned has no production reader after the ingest split — see inline at crates/trusted-server-core/src/ec/registry.rs:276-283

⛏ nitpick

  • Unused log dependency — see inline at crates/trusted-server-identity-providers/Cargo.toml:20

Cross-cutting / body-level findings

  • 🤔 Retiring the completeness marker brings back a pre-response KV read. On the base branch, a valid ts-ec-pull-complete marker let a permitted navigation skip the row preload (should_preload_ec_snapshot's marker_can_skip) when no auction, browser EID cookie or withdrawal needed the row. With the marker gone, should_preload_ec_snapshot (publisher.rs:3466-3473) returns eligible for every permitted navigation. Deployments with pull-sync partners therefore read KV on steady-state returning navigations that have no auction and no ts-eids/sharedId cookie, where they previously read nothing for up to an hour. In the default inline mode the read overlaps the origin fetch, so the cost is mostly KV operations; with template-cache keys the origin waits for it. Dropping the marker also removes its periodic Set-Cookie, which is a fair trade. Could you add this to "Behaviour changes from main", or confirm it is intended? The guide says the marker is retired but not that the read it avoided is back.

CI Status

All 29 check runs on 15eb284f4 passed.

@ChristianPavilonis ChristianPavilonis left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Summary

This PR moves pull sync into after-response identity collection and adds an opt-in, bounded mirror cookie for provider-written bidstream UIDs. Two correctness issues remain: clearing an existing mirror after vendor-consent withdrawal, and preserving the default partner configuration schema during mixed-version deployment or rollback.

Both inline comments carry a one-click GitHub suggestion. Use Commit suggestion, or Add suggestion to batch, to apply the proposed replacements.

Blocking

🔧 wrench

  • Clear an existing mirror when vendor consent is withdrawn (suggestion) — see inline at crates/trusted-server-core/src/ec/finalize.rs:99–114.
  • Omit default mirror_enabled from serialized configuration (suggestion) — see inline at crates/trusted-server-core/src/settings.rs:453.

Review scope and verification

  • Reviewed the stacked delta against feat/s2s-eid-collection-core at head 15eb284f495c5fb9554a04b178804a4776d63965.
  • Coverage limitation: large unchanged portions of settings.rs and publisher.rs, especially their tests, were not exhaustively examined end to end. This is not an exhaustive whole-file review. Full reads were completed for Fastly app.rs, main.rs, config.rs, and the deleted completeness-marker implementation.
  • Each suggestion was applied and verified independently against the clean reviewed head: rustfmt; all eight clippy aliases; Fastly, Axum and Cloudflare check aliases; Fastly tests with the repository-pinned Viceroy 0.21.0 (including the reusable-sandbox variant), Axum, Cloudflare and Spin tests; and cross-adapter parity tests.
  • An out-of-tree native public-API harness exercised the mirror replacement with a real encoded cookie: vendor withdrawal on a non-navigation, withdrawal with an unavailable navigation snapshot, unchanged consent controls, no-cookie handling, and retaining another consented vendor on a readable navigation.
  • The serialization replacement was checked with omitted, explicit-false and string-false inputs against the base partner key set; explicit true survived serialization and deserialization.
  • Byte-exact pre/post-verification patch comparisons passed for both suggestions. Scratch edits were discarded and the review worktree was restored to the reviewed head. No commits or pushes were made.
  • No manual browser or deployed-service validation was performed locally.

CI Status

@aram356 aram356 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Summary

This round fixes all eleven of my round-1 findings, and the rebase onto #1487's startup-built coordinators is clean. I found no new code defects. What still blocks are two documentation gaps: the design spec now describes the behaviour this PR replaced, and the CHANGELOG has no entry, including for one breaking change.

2 of the 4 inline comments carry a one-click GitHub suggestion. Use Commit suggestion, or Add suggestion to batch to apply both at once. The other two describe changes outside the diff.

Blocking

🔧 wrench

  • Update the design spec to match this PR: body-level, below
  • Add CHANGELOG entries, including one Breaking item: body-level, below

Non-blocking

♻️ refactor

  • Carry runs_pull_sync onto PartnerConfig to simplify the double-claim check: see inline at crates/trusted-server-core/src/config.rs:384

🤔 thinking

  • Reject mirror_enabled without bidstream_enabled: see inline at crates/trusted-server-core/src/settings.rs:454

⛏ nitpick

  • Give holds_live_uid its doc line back (suggestion): see inline at crates/trusted-server-core/src/ec/coordinator/mod.rs:294-306
  • Assert the specific token error (suggestion): see inline at crates/trusted-server-core/src/config.rs:1502-1506

Cross-cutting / body-level findings

  • 🔧 Update the design spec to match this PR. docs/superpowers/specs/2026-10-30-s2s-eid-collection-design.md was added in #1487 (889159cd6) and is not on main yet. It describes this PR as PR5 and PR6, but the behaviour changed in this round:

    • D6 (line 81) and §10 Contents (lines 1407-1408): say the mirror holds every bidstream_enabled source. It now needs mirror_enabled as well, and it leaves out UIDs whose owning provider's TCF vendor consent is denied.
    • §10 Set and Expired (lines 1412-1418): say the cookie is set on every consented navigation, and expired on any response without EID consent and on any navigation whose mirror would be empty. The code instead sets it only when the value changes, expires only a cookie the request carried, and leaves it untouched when the row cannot be read.
    • PR5 row (line 1529) and acceptance scenario 21 (lines 1597-1600): say browser ingest skips pull-sync sources and batch sync rejects them, and that a browser ts-eids UID and a batch record are "ignored or rejected". Both now keep filling the source, and pull sync fills it only when it holds no UID.
    • PR6 row (line 1530), IdentityCoordinator::mirror (line 1059) and the apply_cookies call (line 1228): describe the mirror as a coordinator method. The code builds it with ec::mirror::build_mirror(row, consent, partners, now), called from ec_finalize_response.

    Since the spec lands with this stack, please update these sections in this PR, so reviewers and later work read the design that ships.

  • 🔧 Add CHANGELOG entries, including one Breaking item. [Unreleased] carries #1487's provider entry but nothing for this PR's user-visible changes.

    • Breaking. On main, a resolved ts_pull_token that cannot form a header value made the old dispatcher's .expect("should build pull sync request") panic after the page was sent, so only pull sync failed. It now fails config validation, and the service does not start. A secret value with a trailing newline is the likely way to hit this.
    • A possible wording, under ### Changed:
    - Pull sync now runs as a `pull-sync` identity provider built from the existing `[[ec.partners]]` settings, after the response on top-level publisher navigations from recognized browsers, within `ec.identity_after_response_budget_ms` shared with other after-response providers. It no longer runs after `/auction` responses, on prefetches or on navigations without Fetch Metadata, and a partner slower than the budget is not filled, where each pull used to wait up to 15 seconds. `ec.pull_sync_concurrency` now caps the partners pulled per navigation (`0` means no limit; it used to mean one at a time), partner order rotates hourly, and a partner past the cap logs `skipped_concurrency_cap`. The `ts-ec-pull-complete` marker is retired and leftover cookies are expired. `kind = "pull-sync"` in `[[ec.identity_providers]]` is rejected, as is a configured provider that claims a pull-sync partner's source.
    - Added `mirror_enabled` on `[[ec.partners]]` (default `false`). With `bidstream_enabled`, the partner's provider-written UIDs are mirrored to the JavaScript-readable `__Host-ts-eids-mirror` cookie, leaving out UIDs whose owning provider lacks TCF vendor consent. See [EID Mirror Cookie](docs/guide/edge-cookies.md#eid-mirror-cookie).
    - **Breaking:** For a partner with pull sync enabled, a resolved `ts_pull_token` that cannot form an `Authorization: Bearer` header, such as one ending in a newline, now fails config validation and stops startup. Before, only that partner's pull sync failed, after the page was sent. Check pull-token secrets before deploying.
  • 📝 Earlier findings.

    • Mine: all eleven round-1 findings are addressed at this head. I checked each in the code. The browser-ingest repro now stores the browser UID for a pull-sync source, and the vendor-consent and bypass cases have tests.
    • Still open from other reviewers at this head:
      • ChristianPavilonis's two 🔧: the mirror after a vendor withdrawal (finalize.rs:114), and serializing a default mirror_enabled (settings.rs:453).
      • prk-Jr's ♻️ on provider_owned (registry.rs:283).
      • prk-Jr's ⛏ on the unused log dependency.
      • prk-Jr's 🤔 on the preload read that the retired marker used to skip.

Verification

  • At this head, the round-1 browser-ingest scratch test passes with the expected values: the UID is stored both with and without pull sync.
  • I applied each suggestion to a scratch copy, first alone and then together:
    • The assertion change passed rustfmt, its test, all six adapter clippy aliases, test-fastly, test-axum, test-cloudflare, test-spin and the parity suite.
    • The doc-line move passed rustfmt and clippy-fastly.
    • The scratch changes were then discarded.
  • The prose proposals (runs_pull_sync on PartnerConfig, rejecting mirror_enabled without bidstream) have not been compiled.

CI Status

@dhruv8sh
dhruv8sh force-pushed the feat/s2s-eid-collection-providers branch from 15eb284 to 29de3d4 Compare November 11, 2026 22:22
@dhruv8sh

dhruv8sh commented Nov 11, 2026 •

Copy link
Copy Markdown
Collaborator Author
  • Design spec (aram356): updated in 9de1d04. D6, §10 Contents/Set/Expired, the PR5 and PR6 rows, scenario 21, IdentityCoordinator::mirror (removed) and the apply_cookies call now describe what ships. That means the mirror_enabled opt-in, the vendor-consent filter, set-on-change, expire-only-when-carried, untouched on an unreadable row, the vendor-withdrawal expiry, and browser/batch ingest still filling pull-sync sources.
  • CHANGELOG (aram356): added in 9de1d04, mostly with your wording: the pull-sync provider entry, mirror_enabled (now noting that it requires bidstream_enabled), and the Breaking entry for a pull token that cannot form a header.
  • Pre-response KV read after retiring the marker (prk-Jr): this is intended. Without the marker, a returning permitted navigation reads the row before the origin response even when no auction, EID cookie or withdrawal needs it. In inline mode the read overlaps the origin fetch, and the marker's periodic Set-Cookie is gone. This is now in the guide (edge-cookies.md, under the retired marker) and the CHANGELOG. I'll add it to "Behaviour changes from main" in the PR body.

@aram356 aram356 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Summary

This round addresses every earlier finding from all three reviewers, and the rebase onto #1487's latest head is clean. One defect remains, in the vendor-withdrawal expiry added this round: for a visitor who denies one mirrored vendor, the mirror is cleared after every page view and set again on the next navigation.

1 of the 2 inline comments carries a one-click GitHub suggestion; the other describes a fix that spans two files.

Blocking

🔧 wrench

  • Expire the mirror only when it holds a denied vendor's UIDs: see inline at crates/trusted-server-core/src/ec/finalize.rs:144

Non-blocking

⛏ nitpick

  • Drop "pull sync skips them" from the provider entry (suggestion): see inline at CHANGELOG.md:24

Cross-cutting / body-level findings

  • 📝 Earlier findings, checked at this head.
    • Mine from round 2: all fixed. These were the spec drift, the CHANGELOG (including the Breaking pull-token entry), runs_pull_sync on PartnerConfig, rejecting mirror_enabled without bidstream_enabled (at startup and at ts config push, both tested), the holds_live_uid doc line, and the exact token assertion.
    • ChristianPavilonis's two:
      • The vendor-withdrawal expiry is in place; the 🔧 above is about its scope.
      • A default mirror_enabled is no longer serialized, with round-trip tests.
    • prk-Jr's three:
      • provider_owned again means only an enabled configured provider.
      • The unused log dependency is gone.
      • The pre-response row read is documented in the guide, the CHANGELOG and the description.

Verification

  • A scratch test at this head reproduces the cycle with two mirrored partners, one whose vendor is granted and one whose vendor is denied: navigation sets the mirror, a non-navigation expires it, and the next navigation sets it again.
  • I applied the proposed fix in a scratch copy. It compiled and passed rustfmt and all 30 mirror and finalize tests (wasm32-wasip1 under Viceroy), and the cycle stopped. It has not been through the full adapter gate. The scratch changes were discarded.
  • The CHANGELOG.md suggestion passes the repository's prettier check, and only ", pull sync skips them" changes.
  • The rebase onto #1487's split vendor gate holds: keep_vendor_gate writes the same owner_tcf_vendor_id the mirror reads, so a disabled provider's stored UIDs stay gated there as they are in eids.rs and identify.rs.

CI Status

Comment thread CHANGELOG.md Resolved
dhruv8sh added a commit that referenced this pull request Nov 12, 2026
Signed-off-by: dhruv8sh <dhruv8sh@proton.me>
@dhruv8sh
dhruv8sh force-pushed the feat/s2s-eid-collection-providers branch from 29de3d4 to d1440d6 Compare November 12, 2026 12:49
dhruv8sh added a commit that referenced this pull request Nov 12, 2026
Signed-off-by: dhruv8sh <dhruv8sh@proton.me>
@dhruv8sh
dhruv8sh force-pushed the feat/s2s-eid-collection-providers branch from d1440d6 to 6ffb613 Compare November 12, 2026 13:07
@dhruv8sh
dhruv8sh removed this pull request from stack #1515 November 12, 2026 13:08
@dhruv8sh
dhruv8sh changed the base branch from feat/s2s-eid-collection-core to feat/s2s-eid-vendors November 12, 2026 13:08
@aram356
aram356 removed this pull request from stack #1563 November 17, 2026 20:50
Brings PR #1489's pull-sync provider and EID mirror onto PR #1488's new
head, which carries PR #1487 onto PR #1526 (EdgeZero cutover, local partner
API answers, HTTP auction opt-in and awaitable EC persistence).

How each area was resolved:

- Pull sync: core `ec/pull_sync.rs` stays deleted, so the base's EdgeZero
  group sender and its tests go with it. Pull sync runs as PR #1489's
  `pull-sync` after-response provider, sent through the post-delivery
  client in the lookup batch. Post-delivery work is now telemetry, then
  after-response lookups; the separate pull-sync step, its graph factory
  and the unused clock are removed, and PR #1489's "nothing due" check runs
  before the live row read. `require_identity_graph`, now used only by
  tests, is test-only.
- Coordinator: PR #1489's dispatch order and per-navigation pull-sync cap
  run inside PR #1487's batch-shaped dispatch; the cap counts each queued
  pull-sync request.
- Fastly app: `EcFinalizeState` drops both the old `services` and
  `is_real_browser`; the bot gate moves into after-response capture as in
  PR #1489, with the base's request services and async batch sync.
- Async KV and finalize: PR #1489's removals in `kv.rs` (snapshot partner
  upsert) and the pull-sync marker stand; the mirror and the inner finalize
  are awaited, and PR #1489's new tests await the store.
- Publisher: keeps the base's EdgeZero origin request without the marker
  preload inputs PR #1489 removed.
- Providers crate: PR #1489's pull-sync providers and PR #1488's SharedID
  cookie defaulting both apply when the phase coordinators are built.
- Docs and CHANGELOG: the base's auction wording with PR #1489's identity
  bullets; the onboarding step names after-response lookups.
@aram356
aram356 added this pull request to stack #1621 November 17, 2026 21:29
@aram356

aram356 commented Nov 17, 2026

Copy link
Copy Markdown
Collaborator

For the identity integration chain (#1608), this PR's base #1488 now carries #1487, #1557, #1526, #1525, #1614 and #1506's EdgeZero cutover. a7262c7db merges #1488's head 8183c3cae; git show --remerge-diff a7262c7db shows the resolution.

Area Resolution
ec/pull_sync.rs Stays deleted, with the cutover's grouped send_all_until sender
Fastly post_delivery.rs Pull sync runs as this PR's pull-sync after-response provider through #1487's lookup batch on the post-delivery client: telemetry first, then after-response lookups, under the one block_on outside dispatch. The separate pull-sync step, its graph factory and the unused clock parameter are gone, and any_due's nothing-due return runs before the live row read
Fastly main.rs, app.rs EcFinalizeState drops services and is_real_browser; the bot gate sits in prepare_after_response_lookups; require_identity_graph is test-only now
ec/coordinator/mod.rs This PR's dispatch_order and pull-sync concurrency cap inside the batch dispatch
ec/kv.rs, ec/finalize.rs, publisher.rs This PR's removals stand (upsert_partner_ids_from_snapshot, keep_proven, apply_partner_id_updates, the marker preload); finalize_response_inner is async
Identity-providers lib.rs This PR's pull-sync provider configs pass through #1488's with_prebid_sharedid_cookie

Behavior that changes with the move:

  • Pull sync follows this PR's design on EdgeZero's transport: it runs within identity_after_response_budget_ms and writes once through after-response compare-and-swap. The cutover's 15 s per-group cutoff, grouped sender, bulk merge and per-partner failure lines go with pull_sync.rs.
  • The per-navigation pull-sync cap counts every queued pull-sync request, including one the transport later fails to build; before, it counted only sends that started.

Tests changed beyond the async conversion: pull_sync_reaches_partner_after_origin_aborts_mid_body runs through the after-response provider path with a new returning_reader_after_response_state helper, and keeps its assertion (telemetry first, then the partner pull URL); publish_overlapped sets a navigation permit, because only a navigation that may write its row preloads. The 29 tests of the cutover's pull_sync.rs went with the module; this PR's provider tests cover the reply parsing.

The base stays feat/s2s-eid-vendors (#1488), so the diff still shows only this PR's changes; it joins stack 1621 on top of #1488.

Checked locally at the new head: cargo fmt, all eight clippy aliases and the core build-digest clippy, scripts/lint-wasm-clock.sh, test-fastly (4,063), test-fastly-reuse (243), test-axum (64), test-cloudflare (66), test-spin (89), parity (33), core build-digest (10), host CLI tests (972), docs and Markdown format, and the URL-host lint against the PR below, all passing.

aram356 added a commit that referenced this pull request Nov 17, 2026
…ntity

#1489's head a7262c7 stacks on #1488 (8183c3c), which stacks on #1487 (64674f9); each merged the one below it and resolved its own conflicts, so this merge needs no resolution. It completes the provider stack (#1567): the vendor identity providers and cookie hooks (#1488), and the bounded identity mirror with pull sync as an after-response provider on EdgeZero's outbound client (#1489).
@dhruv8sh
dhruv8sh removed this pull request from stack #1621 November 18, 2026 16:50
@dhruv8sh
dhruv8sh added this pull request to stack #1630 November 18, 2026 16:50
aram356 added a commit that referenced this pull request Nov 19, 2026
Core identity app. IdentityApp now carries the provider stack's identity
state as well as its own: the partner registry, the page and after-response
provider coordinators (#1487, #1489), the EC record store and the rate
limiter. IdentityPlatform takes an optional IdentityProviders (the registry
the coordinators were built against plus both coordinators); without one the
app builds the registry with PartnerRegistry::from_ec and runs no provider,
matching the base's non-Fastly adapters. The admin EC lookup awaits the async
handler from #1526.

Send and Sync. #1526 keeps EcKvStore free of Send and Sync bounds and tests a
non-Send store, so the T21 supertrait bounds on EcKvStore and RateLimiter are
dropped. The identity app instead holds its record store and rate limiter as
Send and Sync trait objects, and IdentityApp::identity_graph hands out a graph
over that one shared store through the new KvIdentityGraph::from_shared.
IdentityPlatform's identity_graph field becomes record_store accordingly.

Fastly app. AppState drops the base's partner_registry, page_identity and
after_response_identity fields for the identity app. build_identity_app builds
the kind-aware registry and both coordinators from the identity-providers
crate, then the app with Fastly's record store and rate limiter. Page lookups
and after-response preparation read their coordinators and rate limiter from
the identity app; identify, batch sync, auction, page bids and the publisher
fallback use its registry and graph. The admin lookups stay mounted by the
identity app, so the base's async admin early dispatch is removed.

Fastly entry point. finalize_response keeps the base's single block_on around
the async EC finalize, which collects the page lookups first. FinalizeInputs
keeps the base's AppState, and FinalizeState carries the identity app and the
after-response lookups. maybe_identity_graph is gone; the test-only
require_identity_graph reads the identity app's graph.

Post-delivery. The after-response providers, pull sync included, take the
partner registry, graph and rate limiter from the identity app carried in
FinalizeState. The base's pull-sync tests build their state through the
identity app.

Axum, Cloudflare and Spin. The identity app mounts the admin lookups; the
base's provider rejection at startup and #1614's local 501 and 405 partner API
answers are kept.

Tests. The core identity app tests use record stores and the async graph, and
two tests cover the carried registry and coordinators. The Fastly identity
fixture builds an identity app around its echo-provider coordinator. The
adapter identity guard replaces dispatch_pull_sync, which #1489 deleted, with
commit_after_response_lookups and also guards PartnerRegistry::from_ec.
aram356 added a commit that referenced this pull request Nov 19, 2026
The OpenRTB tests keep this pull request's EID helper imports alongside
the base's scripted-client imports for the provider round-trip tests.

The base's wire-level auction tests now read EIDs where this pull
request writes them. The vendor-denied body EID test and the EC
identity denied test in the auction endpoint, and the template cache
reader-identity test in the publisher, read user.eids through the
shared user_eids and assert_no_user_eids helpers instead of
user.ext.eids, since the standard profile no longer writes the
extension field.

The APS no-EID test drops the transport_timeout_ms field that the base
removed from AuctionContext.
aram356 added a commit that referenced this pull request Nov 19, 2026
The merge applied without conflicts. The Prebid integration config keeps
this pull request's mode field next to the base's changes, and the
example config, the Prebid guide and the tsjs Prebid sources take both
sides unchanged. The server mirror cookie the identity mode reads is the
same name and envelope in the base, so no code needed to change.
aram356 added a commit that referenced this pull request Nov 19, 2026
Fastly entry point: serve keeps the base's request-local identity lookup
slot, dropped before and after the request, and sends through this pull
request's with_finalization helper, so the conformance runner and
production share one finalization callback. FinalizeInputs now carries
the app state, and the conformance runner passes the retained app's
state. The runner also drops leftover navigation lookups around each
request, as serve does.

EC store test support: the module opens to the test-utils feature as
this pull request intended. Every test-only double, including the base's
new write-failing store, stays behind cfg(test); only the in-memory store
is shared.

Conformance fixture: the recording store implements the async EcKvStore
trait and awaits the in-memory store. Scenario::verify and the native
rejection-control tests drive store reads and writes with block_on at
their synchronous test boundary, because the adapter runner already
blocks inside the Fastly lifecycle. The conditional-write test checks
the recording store's create-only insert directly, since the base
removed KvIdentityGraph::create_if_absent.

Fastly runner: a document navigation now sends Sec-Fetch-Mode: navigate
next to Sec-Fetch-Dest: document, and a subresource sends no-cors. The
base's page-load write permit requires both headers, as a browser sends
them.

The integration-tests README says post-delivery work now covers the
after-response identity lookups, pull sync among them.
aram356 added a commit that referenced this pull request Nov 19, 2026
Pull sync: the base deletes core ec/pull_sync.rs, so the merge takes the
deletion. This pull request changed only that file's tests, adding the
differential certificate-check test, so the test moves to where pull
sync now sends: the Fastly post-delivery work, which builds the
after-response pull-sync provider from the settings and sends its lookup
through OutboundLookupTransport. The ported test runs post-delivery work
with certificate_check true and false, each against its own fresh live
row in the local identity store, and asserts the same method, URL,
headers and body, the lookup transport's timeout, buffered cap,
decoding and cache policy, and that both runs store the partner UID.
The pull-sync test helpers take the EC ID so this test never writes
the row the other pull-sync tests share.

Docs: the certificate_check guide entry names identity providers, pull
sync among them, since pull sync is one in the base.

The config message, settings doc comment, guides and example config
apply this pull request's wording unchanged.
Takes #1488's new head ce8729f, which brings #1506's repin of
EdgeZero to rc/identity at 9d02d7b3. The merge is clean, and this PR's tree
changes only by that repin.
aram356 added a commit that referenced this pull request Nov 19, 2026
Takes #1489's new head e4ae9ca, which brings #1506's repin of
EdgeZero to rc/identity at 9d02d7b3. The merge is clean, and this PR's tree
changes only by that repin.
aram356 added a commit that referenced this pull request Nov 19, 2026
Takes #1489's new head e4ae9ca, which brings #1506's repin of
EdgeZero to rc/identity at 9d02d7b3. The merge is clean, and this PR's tree
changes only by that repin.
aram356 added a commit that referenced this pull request Nov 19, 2026
Takes #1489's new head e4ae9ca, which brings #1506's repin of
EdgeZero to rc/identity at 9d02d7b3. The merge is clean, and this PR's tree
changes only by that repin.
aram356 added a commit that referenced this pull request Nov 19, 2026
Takes #1489's new head e4ae9ca, which brings #1506's repin of
EdgeZero to rc/identity at 9d02d7b3. The merge is clean, and this PR's tree
changes only by that repin.
aram356 added a commit that referenced this pull request Nov 19, 2026
Takes #1489's new head e4ae9ca, which brings #1506's repin of
EdgeZero to rc/identity at 9d02d7b3. The merge is clean, and this PR's tree
changes only by that repin.
aram356 added a commit that referenced this pull request Nov 19, 2026
Brings the provider stack, including the EdgeZero cutover (#1506), into the
cross-adapter health and startup-failure contract, and keeps the contract on
every adapter.

Core: the probe pre-dispatch hook takes EdgeZero's PreDispatchRequest, and its
tests drive the hook through a router with run_pre_dispatch_hook, because
PreDispatchRequest cannot be built outside EdgeZero.

Fastly: the cutover's handle_request, assemble_app and finalize_response stay.
Liveness is built from the shared probe response and sent natively. A new
initialize_dispatch runs the sandbox initialization, answers GET and HEAD
/ready from its result, and answers a failed assembly on the seven publisher
fallback methods with the fixed 503, logging the assembly error that
EdgeZero's assembly-error response would have logged. Other methods keep
EdgeZero's assembly-error response. finalize_response skips operational
responses as it skips trace responses. The retained router wraps the trace
hook in the probe hook. Fastly needs no failure router, so the PR's Fastly
startup_error_router and its matrix test go; the matrix moves to main.rs and
runs through initialize_dispatch and the native conversion.

Axum, Cloudflare and Spin: Hooks::routes logs the report and returns the PR's
failure router instead of failing assembly, so liveness stays 200 and /ready
and every supported route answer the fixed 503. The routers keep the cutover's
middleware and Axum's received-head preparer, with the probe hook around the
trace hook. The cutover's EdgeZero managed logger already gives Cloudflare and
Spin a production log sink, so the PR's adapter loggers are dropped.

Tests: the cutover's startup tests are moved to the contract. Fastly's
missing-blob test checks the fixed 503 for GET, the assembly error for TRACE,
then recovery. The Cloudflare and Spin routes() tests serve the failure router.
The Axum binary test now requires the binary to keep listening with live
health and a 503 /ready and page, in place of a non-zero exit. The two Axum
connection-entry tests move from /health, which now bypasses configured
response headers, to the identify preflight. The probe parity test takes the
cutover's forbidden-router tuple and also asserts that probes send no outbound
request.

Docs: the API reference, adapter guides, READMEs, AGENTS.md and CHANGELOG keep
both sides. Startup statements now describe the contract: no Axum exit, no
Spin /health 503, no Cloudflare JSON 500, with the Fastly logger-failure case
and the Fastly assembly error for unsupported methods kept. The smoke-axum.sh
negative cases stay on the PR's 503 contract with EdgeZero's port variable.
aram356 added a commit that referenced this pull request Nov 19, 2026
Takes #1489's new head e4ae9ca, which brings #1506's repin of EdgeZero to
rc/identity at 9d02d7b3. The one conflict is CHANGELOG.md, where this PR's
health and readiness bullet sits next to the cutover's pin bullet; both are
kept, with the pin bullet naming the new commit.
aram356 added a commit that referenced this pull request Nov 19, 2026
The identity integration chain now runs #1489, #1612, #1623, so this PR
takes #1612's head 470f8e1, which sends EIDs in OpenRTB 2.6 user.eids. The
merge is clean, and this PR's tree changes only by #1612's own change.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Collect identity provider EIDs server-side for the server-side auction Write Lockr IDs to KV store

4 participants