Skip to content

Portable KV-backed Geo lookup - #454

Open
ChristianPavilonis wants to merge 24 commits into
feat/kv-record-contractfrom
feature/kv-geoip
Open

ChristianPavilonis wants to merge 24 commits into
feat/kv-record-contractfrom
feature/kv-geoip

Conversation

@ChristianPavilonis

@ChristianPavilonis ChristianPavilonis commented Nov 5, 2026 •

Copy link
Copy Markdown
Collaborator

What this PR delivers

This replaces the prototype-only serving story with opt-in, managed Axum KV Geo. The portable reader takes a checked IP and uses the existing KvHandle interface and a pinned immutable generation. It has no CSV, MMDB, native SDK, or backend-specific lookup dependency. Off remains the default; protected CloudFront and supplied-provider behavior are preserved.

The standalone operator tool builds GeoLite City or Country CSV artifacts, publishes with read-back and manifest-last ordering, and verifies the full tree through the serving binding against an independent CSV oracle. It supports ordinary stock celld and DynamoDB KV. The serving identity is read-only; the publisher can retire prior generations by inventory. No new service API, storage provisioning, or EC binding is required.

  • Design and acceptance: docs/superpowers/specs/2026-11-05-kv-geo-design.md
  • Configuration and proof commands: docs/guide/native-geo.md
  • Operator tool: examples/geo-kv-prototype/README.md

Review and landing order

This branch still targets #510 (feat/kv-record-contract). It includes the #439 provider foundation and a local reconciliation of #502. Please confirm that #439 lands first, that #502 and #510 are reconciled into the target before #454 lands, and that #454 need not carry their changes as its own feature diff. The PR base has not been changed.

Tracks #438. Maintainer decision requested before closure: accept a full-tree verification through the read-only serving binding before rollout, plus pinned manifest and nonempty root checks before Axum readiness, in place of the MMDB-specific malformed-file and mount checks. A required descendant missing or corrupt at request time is a provider failure, not a geographic miss or fallback.

Evidence

The local workspace run passed 3,051 tests with 9 existing ignores, alongside strict Clippy, feature checks, and the Spin WASM build. The earlier CLI test stall reproduces on both base and candidate only with terminal stdin. The workspace run used noninteractive stdin without adding any skips. The final standalone default and all-feature suites passed after the object-size decision. The Axum native feature lint matrix passed locally after the compiled-out fixture fixes.

The combined backend proof passed on stock celld and DynamoDB Local, including full-tree readback, managed Axum requests, and 4,096 IPv4/IPv6 oracle comparisons per backend and object size. The same 16 KiB artifacts avoided repeat KV reads in the sampled workload on both backends, so 16 KiB is now the default. 64 KiB remains an explicit option for fewer objects and fewer cold path reads. The report records which binaries were measured. Hosted IAM, production performance and durability are not claimed. No licensed source or derived data is checked in.

Trusted Server integration, deployment, and tags are out of scope for this PR.

@ChristianPavilonis
ChristianPavilonis marked this pull request as ready for review November 10, 2026 23:48

@aram356 aram356 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

PR Review

Summary

This PR adds a standalone KV Geo prototype: a CSV importer, the p1 binary index, digest-validated async lookup through KvHandle, the design spec and the results record. The lookup algorithm holds up under a randomized differential check: 872,448 lookups compared against naive range containment, covering both families, object budgets of 512–4096 bytes and cache budgets of 0, 2 KiB and 1 MiB, with zero mismatches. The blocking issues are about keeping the crate buildable and the evidence accurate. Nothing in CI builds it, its pins break on the next root bump, several tests can't fail, and the results doc's provenance doesn't match this head.

Findings

Blocking

  • 🔧 No CI coverage, and exact pins coupled to the root: a simulated root serde_json bump makes this crate unresolvable while CI stays green (examples/geo-kv-prototype/Cargo.toml:8)
  • 🔧 Object cache effectively untested: a cache that stores nothing passes all 10 tests (examples/geo-kv-prototype/src/index.rs:635)
  • 🔧 Overlap-rejection test can't fail: no build succeeds at 140 bytes, and deleting the overlap check still passes (examples/geo-kv-prototype/src/index.rs:763)
  • 🔧 Results doc provenance and coverage claims don't match this head (docs/superpowers/plans/2026-11-05-kv-geo-prototype-results.md:9)

Non-blocking

  • 🤔 Cache hit skips the family check, so warm and cold resolvers can disagree (src/index.rs:531)
  • 🤔 IPv4-mapped input misses (src/index.rs:501)
  • 🤔 Generation ID isn't derived from content (src/index.rs:274)
  • ♻️ Validation runs after publication (src/index.rs:291, src/main.rs:55)
  • ♻️ Second SHA-256 stack (Cargo.toml:20)
  • 🤔 load and ArtifactStore have no tests (src/import.rs:149)
  • ⛏ I/O errors lose the path (src/import.rs:132)
  • ⛏ oracle_mismatches is a constant (src/main.rs:128)
  • 🌱 An interrupted write blocks every rerun (src/store.rs:98)
  • 🌱 Before p1 is frozen:
    • The manifest serializes u128 bounds as JSON numbers and each digest as a 32-element array. IPv6 bounds above 2^53 lose precision in JavaScript and jq, and each digest takes about 115 characters as an array versus 64 as hex. Hex strings fix both.
    • Tests match anyhow errors by substring, which doesn't give the bounded failure categories spec §9 asks for. A typed error enum would.
    • Cache::get calls order.retain on every hit, which costs O(entries). A production cache needs O(1) LRU bookkeeping.
    • Resolver::lookup(&mut self) can't be shared across concurrent requests as written.

📌 Out of Scope

  • Dependabot's cargo entry covers only /. Standalone lockfiles such as this crate's get updated only when someone notices a root bump broke them. A tracking issue across the standalone crates would cover this.

CI Status

  • PR checks: 18/18 pass at c52bdaf3
  • Local workspace gates (fmt, clippy, test, feature check, spin wasm32): PASS
  • Local prototype checks: fmt PASS, tests PASS (10/10), clippy -D warnings PASS, release build PASS, cargo check for wasm32-unknown-unknown and wasm32-wasip1 PASS
  • None of the workspace gates or PR checks compile this crate.

How the findings were checked

  • Mutation testing: cargo mutants on index.rs, import.rs and store.rs produced 243 mutants: 146 caught, 71 missed, 20 unviable, 6 timeouts.
  • Probe tests in a scratch copy covered the cache/family divergence, IPv4-mapped input, the minimum budget and the randomized differential check.
  • CLI run: end-to-end build/verify/lookup on synthetic Country-shaped CSV.

Comment thread examples/geo-kv-prototype/Cargo.toml Resolved
Comment thread examples/geo-kv-prototype/src/index.rs Resolved
Comment thread examples/geo-kv-prototype/src/index.rs Resolved
Comment thread examples/geo-kv-prototype/src/index.rs Resolved
Comment thread examples/geo-kv-prototype/Cargo.toml Resolved
Comment thread examples/geo-kv-prototype/src/import.rs Resolved
Comment thread examples/geo-kv-prototype/src/import.rs Resolved
Comment thread examples/geo-kv-prototype/src/main.rs Resolved
Comment thread examples/geo-kv-prototype/src/store.rs Resolved

@dhruv8sh dhruv8sh left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

PR Review

Summary

Standalone, opt-in geo-KV prototype (own [workspace], no production impact) plus design/results docs. Lookup algorithm looks correct and is well validated; only non-blocking notes.

😃 Praise

  • Lookups are verified against a plain containment search over the original, unmerged source networks (import.rs:224-236). XK is kept with a core_country_supported flag rather than widening or dropping it, and the lookup path uses only KvHandle with block_on tests.

Findings

Non-blocking

  • 🤔 Generation check after write (src/main.rs:55) — inline
  • 🤔 O(n) cache hit bookkeeping (src/index.rs:427) — inline
  • ⛏ Silent edition fallback (src/import.rs:150) — inline
  • ⛏ Manifest size unbounded on read (src/index.rs:466) — inline
  • 🌱 Prototype not exercised by CI: its own [workspace] means root CI never builds or tests it, so it can rot as edgezero-core changes. Consider a cargo test --locked --manifest-path examples/geo-kv-prototype/Cargo.toml step or a follow-up issue.
  • 📝 Results doc predates the toolchain bump: it records Rust 1.95.0 and base 72ceed7e; the repo is now on 1.98.1 and the measurements weren't rerun. Keep that caveat visible.

CI Status

  • fmt: PASS
  • clippy (prototype, -D warnings): PASS
  • tests (prototype): PASS (10 passed, 0 failed)

Comment thread examples/geo-kv-prototype/src/main.rs Resolved
Comment thread examples/geo-kv-prototype/src/index.rs Resolved
Comment thread examples/geo-kv-prototype/src/import.rs Resolved
Comment thread examples/geo-kv-prototype/src/index.rs Resolved
Comment thread examples/geo-kv-prototype/src/index.rs Resolved
@ChristianPavilonis
ChristianPavilonis removed this pull request from stack #500 November 13, 2026 17:32
@ChristianPavilonis ChristianPavilonis changed the title Prototype portable KV-backed Geo lookup Portable KV-backed Geo lookup Nov 13, 2026
dhruv8sh and others added 8 commits November 14, 2026 02:28
Fastly reads geo_lookup for the client address and Cloudflare reads the
request's cf metadata. Both are always selected and pass their outcome
through dispatch_admitted_with_geo, so public extensions still cannot
replace it. Platform values that fail validation, such as Cloudflare's XX
and T1 placeholders, report missing metadata rather than an invalid
assertion. GeoSource and GeoTrustBasis gain platform variants and become
non_exhaustive; fastly and cloudflare are reserved provider IDs.
[adapters.axum.geo] carries provider, alb_peer_cidrs and
origin_credential_header_name defaults, each overridden by the matching
EDGEZERO__ADAPTER__GEO__* setting. Precedence is with_geo, then settings,
then the manifest, then off, and misconfiguration fails startup instead of
falling back. The dev run_app reads settings only. geo_capability() stays
explicit-only and geo_capability_for::<A>() reports the runner's view.
Takes #439's refreshed head c1cec8e. The one conflict is the Manifest
struct's schema validators: #502 adds the adapter geo placement check and
#439 brings #275's Fastly logging check. Both are kept, after the existing
case-unique adapter keys check.
@ChristianPavilonis
ChristianPavilonis removed this pull request from stack #501 November 17, 2026 17:14
@ChristianPavilonis
ChristianPavilonis changed the base branch from feature/pluggable-geo-providers to feat/kv-record-contract November 17, 2026 17:14
aram356 added a commit that referenced this pull request Nov 19, 2026
#517 (10ed380) tops ChristianPavilonis's line on #439:
- #510, the KvStore record contract with DynamoDB and celld providers on
  Axum (#472, #473);
- #454, portable KV-backed geo;
- #514, country and region through ctx.geo() on Cloudflare and Spin (#480,
  #486);
- #517, the portable ClientInfo contract with its Fastly, Axum, Cloudflare and
  Spin mappings (#475, #90, #91, #92, #485).

#517 also merged #469 and #502 itself, so its tree contains everything this
branch had. The only conflicts were JA4 lines that #517 refactors on top of
#469; this merge takes #517's version, and its tree is identical to #517's.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants