Skip to content

feat(fs): discover Filesystem regions from the published catalog - #92

Merged
srstack merged 5 commits into
mainfrom
feat/dynamic-fs-region-catalog
Nov 16, 2026
Merged

srstack merged 5 commits into
mainfrom
feat/dynamic-fs-region-catalog

Conversation

@srstack

@srstack srstack commented Nov 16, 2026 •

Copy link
Copy Markdown
Contributor

What changed

ti no longer compiles the TiDB Cloud Filesystem region list into the binary. Region availability, provider metadata, and the regional serviceAddress now come from the Filesystem open API:

GET https://fs.tidbapi.com/v1beta2/regions   (HTTP Digest + TiDB Cloud API keys)

A region published by the backend becomes usable without a ti release.

Resolution split

Command family Region source Network TiDB Cloud keys
Control plane: create/list/describe/delete-file-system, import-file-system-token, token lifecycle, AI provider config, list-regions published catalog (nextPageToken pagination, state=ACTIVE only) yes yes
Data plane: fs, fs-git, fs-journal, fs-vault, mount, layer, pack/unpack, ti exec service_address recorded in the File System credential, else the conventional https://<region-id>.drive9.ai no no

Control-plane commands still fail locally for a region the catalog does not publish (api.fs_endpoint_unsupported), and data-plane commands never request the catalog, so token-only and offline use is preserved.

Details

  • internal/fsregion (new) — catalog client with pagination, contract validation (canonical regionId, provider consistency, https address, duplicate/empty handling), ACTIVE eligibility, and DeriveServiceAddress.
  • internal/config/region — ParsePlacementCode becomes ParseCanonicalCode: structural parsing only. The TiDB Cloud Starter whitelist (Validate) is untouched, so ti db placement validation does not move; ValidateFilesystem and the Filesystem whitelist are gone.
  • internal/api/endpoints — builtInFSRegionManifest() and the manifest test control are deleted. ResolveFSProfile/ResolveFSRegion use the catalog; ResolveFSCredentialEndpoint uses the credential address with derivation fallback.
  • Credentials — ID-keyed credentials gain an optional service_address. create-file-system and import-file-system-token write it, ti fs list-regions backfills it on existing credentials, and mount locators record it so drain/unmount work without the catalog. Old credentials without the field keep loading.
  • ti fs list-regions (new) — read-only, permission fs.region.read, JSON (region_count, regions[].eligible) and aligned text output, --query supported, --dry-run rejected.
  • Installers — scripts/install.sh / install.ps1 no longer print region lists (they would drift); users read them from ti fs list-regions.
  • Hidden controls — TI_TEST_FS_MANIFEST_URL becomes TI_TEST_FS_REGIONS_URL (local builds with TI_ALLOW_TEST_ENDPOINTS=1 only). The release-build test still proves it is ignored.

Behavior changes to note

  • ti configure now validates the canonical region shape only; whether a Filesystem region exists is decided by the catalog when the command runs (no network at configure time).
  • import-file-system-token uses the catalog when TiDB Cloud keys are available and the derived conventional address when they are not, so a token-only machine can still register a File System without keys.
  • An unparsable service_address written by an older build is dropped instead of failing the credential load; the derived address or a later list-regions backfill takes over.

Testing

  • go test ./... — new unit coverage for internal/fsregion (pagination, repeated tokens, malformed pages, ACTIVE filtering, address validation, transport failures, derivation), endpoint resolution (catalog hit/miss/inactive, derived credential fallback, local-build override), region parsing vs Starter whitelist, list-regions output/backfill, credential round-trip, and token endpoint resolution.
  • make e2e — fake catalog through TI_TEST_FS_REGIONS_URL, a token-only data-plane command using the derived address, a recorded address winning over derivation, unpublished region rejection, import-file-system-token without keys, list-regions JSON/text/--dry-run/backfill, release build ignoring the test control, and installers carrying no region list.
  • Manual smoke with a local fake catalog confirmed JSON/text output, PENDING → eligible: false, and credential backfill.
  • make live-e2e-fs is not yet run (needs real credentials); it is recorded as the release gate in docs/spec/0037-dynamic-fs-region-catalog.md.

Review feedback addressed (round 2, 1aa027b)

  • Blocking — exit-code contract: the catalog request is a Digest-authenticated TiDB Cloud control-plane call, so 401/403/404 now map to auth.invalid_credentials (exit 3), authz.permission_denied (exit 4), and api.not_found (exit 5); only other statuses stay api.fs_regions_unavailable (exit 1). The taxonomy lives once in internal/auth and is rendered by both internal/api and internal/fsregion, so it cannot diverge. Unit tests pin all four buckets.
  • Cancellation: ResolveFSProfile/ResolveFSRegion take the command context, threaded through the fs control plane, token management, AI configuration, and import.
  • Routing-gate coverage: an endpoints-level test covers lookup → eligible → routable, and e2e now asserts an unpublished region fails with api.fs_endpoint_unsupported while an active unmapped-provider entry stays visible in ti fs list-regions and is rejected for routing.
  • Credential lifecycle: a keyless import no longer persists a derived address (only catalog-resolved addresses are authoritative), an address-only change never counts as an import conflict, a missing address never clears a recorded one, and the list-regions backfill re-reads and writes under the credential lock so a concurrent token rotation cannot be reverted.
  • Nits: the stale Client/Load comment is gone, Credentials.ProfileName is now used for auth error wording, and isFSPlacementError is wired into capability probing instead of being dead code.
  • Deferred with rationale (follow-ups): narrowing the resolver input / de-duplicating the resolver defaulting block (item 5) is a wider refactor of internal/api/endpoints and four callers; I kept the review's profile-based input for now and recorded it here rather than mixing it into the correctness fix.

Review feedback addressed (round 1, b2ef112)

  • Unmapped cloud provider (High, Copilot) — ResolveFSRegion no longer borrows the provider mapping from the region-code prefix when the catalog reports a cloudProvider this release cannot map. Such an entry stays visible in ti fs list-regions but fails with api.fs_endpoint_unsupported naming the provider; Catalog.Routable()/RoutableRegionCodes() carry the distinction, and both the listing and rejection paths are unit-tested (b2ef112).
  • Stale cache comment (Low, Copilot) — the internal/fsregion package comment, the ListRegions doc comment, and the spec now state that the catalog is fetched on demand and never cached locally (b2ef112).

Follow-ups

  • docs/pingcap-docs/docs/ is an uninitialized submodule in the authoring environment, so the matching English ti- pages were not updated here. They need the same region-discovery and ti fs list-regions edits when the docs submodule is checked out; spec 0037 now lists this next to the live run as a gate before the spec moves to done/.
  • make live-e2e-fs still has to be run against the real Filesystem API; TestLiveFSListRegions is in place and the spec tracks that gate.

Rollback

Reverting the commit restores the compiled-in tables. Credentials written with service_address remain readable by the previous release because the field is additive, and mount locators without the field stay valid.

Copilot AI balanced review requested due to automatic review settings November 16, 2026 07:39
@ti-chi-bot ti-chi-bot Bot added the size/XXL label Nov 16, 2026

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note

Copilot was unable to complete this review at the selected tier due to model policy or plan restrictions.

🟡 Changes recommended

A critical issue remains where ResolveFSRegion can incorrectly route unknown/unmapped providers by falling back to a region-code prefix contrary to spec, and there is also a documentation mismatch about catalog caching that needs correction.

2 open issues
What changed in this PR

Switches ti Filesystem region discovery from compiled-in tables to the published Filesystem regions catalog API, while keeping data-plane usage offline-capable via recorded (or derived) service_address.

Changes:

  • Add internal/fsregion paginated catalog client + validation and update endpoint resolution to use the catalog for control-plane operations.
  • Record/backfill service_address in stored FS credentials and mount locators so data-plane commands can run without catalog access.
  • Add ti fs list-regions, remove installer region lists, and update docs plus unit/e2e tests for the new behavior.
File Description
scripts/​install.sh Removes printing of the compiled-in Filesystem region list from the shell installer output.
scripts/​install.ps1 Removes printing of the compiled-in Filesystem region list from the PowerShell installer output.
README.md Updates documentation to reflect dynamic Filesystem region discovery and the new ti fs list-regions guidance.
internal/​runtimeexec/​service_test.go Updates runtime exec tests to align with catalog/credential-based Filesystem endpoint resolution.
internal/​fswrap/​runner.go Adjusts wrapper execution to resolve Filesystem endpoints via recorded/derived service addresses and related config plumbing.
internal/​fswrap/​runner_test.go Updates wrapper runner tests for the new Filesystem endpoint resolution behavior.
internal/​fsregion/​fsregion.go Introduces the Filesystem regions catalog model, validation rules, eligibility filtering, and service-address derivation helpers.
internal/​fsregion/​fsregion_test.go Adds unit tests covering catalog pagination, validation, filtering, and derived/normalized service address behavior.
internal/​fsregion/​client.go Implements a paginated HTTP Digest catalog client for fetching published Filesystem regions.
internal/​fs/​tokenmgmt/​service.go Reworks token management endpoint selection to prefer recorded credential service_address with catalog/derivation fallback.
internal/​fs/​tokenmgmt/​service_test.go Updates token management tests to cover the new endpoint resolution and service-address recording behavior.
internal/​fs/​tenant_control.go Updates control-plane filesystem operations to use catalog-based resolution and to persist service_address into credentials on create/import paths.
internal/​fs/​tenant_control_test.go Adjusts tenant control tests for catalog-based region/endpoint resolution and credential updates.
internal/​fs/​regions.go Adds/updates region-listing behavior and credential backfill of service_address based on the published catalog.
internal/​fs/​regions_test.go Adds tests for ti fs list-regions output, eligibility semantics, and credential service_address backfill behavior.
internal/​fs/​mountlocator/​locator.go Extends mount locator records to include an optional service_address so drain/unmount can work without catalog access.
internal/​fs/​fscred/​fscred.go Updates filesystem credential handling to support canonical region parsing and the optional recorded service_address field.
internal/​fs/​fscred/​fscred_test.go Updates filesystem credential tests to reflect canonical region parsing and the new credential field behavior.
internal/​fs/​fscred/​credential.go Adds optional service_address to stored FS credentials and validates/normalizes (or drops) invalid values during load.
internal/​fs/​drive9_companion.go Updates Drive9 companion flows to resolve endpoints via recorded/derived service addresses and to persist service_address where needed.
internal/​fs/​drive9_companion_test.go Updates Drive9 companion tests for catalog/credential-based endpoint resolution and recorded/derived address behavior.
internal/​fs/​control.go Separates control-plane (catalog) and data-plane (credential/derived) Filesystem endpoint resolution paths.
internal/​fs/​aiconfig/​service.go Updates AI configuration service calls to resolve Filesystem endpoints via the catalog-backed resolver.
internal/​fs/​aiconfig/​service_test.go Adjusts AI configuration tests for catalog-based Filesystem endpoint resolution.
internal/​config/​region/​region.go Replaces placement parsing with structural-only canonical parsing and removes Filesystem whitelist validation from config-time parsing.
internal/​config/​region/​region_test.go Updates region parsing/validation tests to match canonical structural parsing and preserved Starter whitelist semantics.
internal/​config/​profile.go Adds/adjusts profile credential resolution to support optional TiDB Cloud keys for keyless data-plane flows and related helpers.
internal/​config/​profile_test.go Updates profile tests for canonical parsing and optional-credentials behavior.
internal/​config/​configure/​wizard.go Updates the configure wizard to validate canonical region code shape without requiring catalog access at configure time.
internal/​config/​configure/​wizard_test.go Updates wizard tests to reflect shape-only canonical validation behavior.
internal/​cli/​root.go Updates CLI root wiring to use canonical region parsing for telemetry/placement handling.
internal/​cli/​root_test.go Adjusts CLI root tests for the new command set and updated region/endpoint resolution behaviors.
internal/​cli/​commands.go Registers and wires the new ti fs list-regions command and related control-plane behavior.
internal/​authz/​authz.go Adds the fs.region.read permission and maps it to ti fs list-regions authorization.
internal/​api/​endpoints/​endpoints.go Removes the built-in FS region manifest and switches Filesystem region/profile resolution to catalog and credential-based routing.
internal/​api/​endpoints/​endpoints_testcontrols_test.go Updates hidden local-build test controls to use TI_TEST_FS_REGIONS_URL and verifies release builds ignore it.
internal/​api/​endpoints/​endpoints_test.go Reworks endpoints tests around catalog fetching, eligibility filtering, and credential/derived endpoint selection.
e2e/​runtime_signal_unix_test.go Updates runtime-signal e2e setup to seed credentials with service addresses to avoid catalog dependency.
e2e/​runtime_live_test.go Updates live runtime e2e to route via recorded credential service addresses under the new model.
e2e/​release_build_test.go Updates release-build e2e assertions for the renamed test override and new routing behavior.
e2e/​pack_unpack_test.go Seeds filesystem credentials/service addresses in pack/unpack e2e tests to match data-plane routing changes.
e2e/​live_test.go Updates live e2e flows to include region catalog behavior and credential-based endpoint routing expectations.
e2e/​installer_test.go Updates installer e2e tests to assert region lists are no longer embedded/printed by installers.
e2e/​fsregions_test.go Adds e2e coverage for fake catalog serving, region listing, and credential backfill behavior.
e2e/​cli_test.go Updates CLI e2e tests to exercise the catalog override and recorded/derived endpoint routing behavior.
docs/​spec/​0037-dynamic-fs-region-catalog.md Adds a spec describing dynamic Filesystem region catalog discovery, routing split, and verification gates.
docs/​priciples.md Updates principles documentation to reflect dynamic Filesystem region publication and recorded service-address routing.
AGENTS.md Updates contributor/agent guidance to reflect the new dynamic Filesystem region flow and related commands.

Review effort: Balanced

Comment thread internal/api/endpoints/endpoints.go Resolved
Comment thread internal/fsregion/fsregion.go Resolved
srstack added a commit that referenced this pull request Nov 16, 2026
Address review feedback on #92.

- ResolveFSRegion no longer borrows the provider mapping from the region-code
  prefix when the catalog reports a cloudProvider this release cannot map. Such
  an entry stays visible in `ti fs list-regions` but fails with
  api.fs_endpoint_unsupported naming the provider, matching the spec.
- Catalog gains Routable()/RoutableRegionCodes() so "available regions" in
  errors only lists routable entries.
- internal/fsregion package comment no longer claims a local cache; the catalog
  is always fetched on demand.
- Tests cover the unmapped-provider listing and rejection paths.
@srstack
srstack requested review from mornyx and qiffang November 16, 2026 08:10

@mornyx mornyx left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review summary

This PR replaces the compiled-in ti fs endpoint table with a published Filesystem regions catalog (internal/fsregion) and keeps the data plane offline via a recorded/derived service_address — the right architecture, executed with real deletion rather than parallel maintenance. Both findings from the previous review round are verified fixed at b2ef112 (unmapped-provider rejection; no-cache wording), with one comment residue left. Under a strict bar, one merge-blocker remains: the new catalog client collapses every non-2xx response (including 401/403/404) into a generic exit-1 error, regressing the repo's documented exit-code contract on the most common failure path — invalid TiDB Cloud credentials. The rest are strong non-blocking fixes around cancellation, the exact test coverage the spec promises, and the service_address recording lifecycle.

Resolved since the last review round (re-verified on b2ef112):

  • ResolveFSRegion now rejects an active entry whose cloudProvider maps to no internal provider (Routable() gate, UnsupportedProviderError), keeping it visible in ti fs list-regions.
  • The package/spec wording no longer claims a local catalog cache (one stale sentence survives, see nits).

What looks good

  • Genuine subtraction: the built-in region table, manifest fetch, mode matching, and test-control plumbing are deleted, and endpoints.go ends smaller than it started while gaining dynamic behavior.
  • The control-plane (catalog) vs data-plane (recorded/derived address) split preserves token-only and offline use with zero new user-facing configuration.
  • The catalog client is defensively written for a brand-new external dependency: full-page validation (duplicate/malformed ids, provider consistency, https-only), repeated-page-token and page-count loop guards, bounded reads, key redaction through the existing digest/debug transports.
  • The "list-only, never routable" rule is implemented cleanly with error copy that names the provider and available regions, and spec 0037 honestly stays out of done/ pending the live gate.

Blocking (please address before merge)

1. Catalog fetch regresses the auth/authz/not-found exit-code contract
internal/fsregion/client.go:156 maps every non-2xx status to api.fs_regions_unavailable (api, exit 1). The catalog request is Digest-authenticated with the TiDB Cloud key pair and now runs before any tenant request on every fs control-plane command, so an invalid key pair on ti fs create-file-system, list-file-systems, token lifecycle, AI config, or list-regions produces exit 1 and "the Filesystem region catalog returned HTTP 401" instead of exit 3 / auth.invalid_credentials with the actionable "Check ~/.ti/credentials" guidance that the shared client returns for the identical condition (internal/api/client.go, 401→3, 403→4, 404→5). This breaks the documented contract ("API/auth errors must preserve categories and exit codes: 3 authentication, 4 authorization, 5 remote not found") and silently changes behavior for scripts keying on exit codes. Fix: before the generic unavailable bucket, map 401→authentication (exit 3, same actionable message as the shared client), 403→authorization (exit 4), 404→not-found (exit 5) — ideally by extracting/reusing the status→error mapping from internal/api so the catalog doesn't grow a second divergent taxonomy.

Should fix (non-blocking, strong)

2. Catalog fetch is uncancellable (context.Background())
internal/api/endpoints/endpoints.go:165 fetches the catalog with context.Background(). Resolution used to be offline; it is now a network hop (up to 15s/page, up to 20 pages) on every control-plane command, but ignores the command context wired for Ctrl+C in cmd/ti/main.go. Service.ListRegions already threads a real context — the resolver path is the gap. Thread the context through ResolveFSProfile/ResolveFSRegion (or the service-level resolvers, which mostly already receive one).

3. The routing gate this PR exists to enforce has no test at its layer, and promised e2e coverage does not exist
The unmapped-provider rejection branch (!entry.Routable() in ResolveFSRegion) is tested only in internal/fsregion (on Entry), not through ResolveFSRegion's lookup → eligibility → routable ordering — reordering or dropping the branch would not fail any test (endpoints_test.go covers unpublished and inactive regions, not unmapped providers). Meanwhile spec 0037's e2e section and the PR body ("unpublished region rejection") claim black-box coverage that does not exist: no e2e/ test asserts api.fs_endpoint_unsupported or the "ti fs is not available" message at all. Add TestResolveFSRegionRejectsUnmappedProvider at the endpoints layer plus one e2e case (unpublished region and unmapped-provider entry through the fake catalog, asserting exit code, error text, and that the unmapped entry still appears in list-regions output) — or trim the claims to match what lands.

4. Recorded service_address is write-once with no reconciliation, and the backfill has a race and a misleading conflict

  • Keyless import-file-system-token persists the derived conventional address (https://<region-id>.drive9.ai) into the credential as authoritative (internal/fs/drive9_companion.go:246 via importEndpoint); the list-regions backfill only fills empty addresses (internal/fs/regions.go:86), so a recorded-but-derived address is never corrected, and tokenmgmt prefers the recorded address over the catalog even when keys are present. If the backend ever publishes a different address, a keyless-imported machine mis-routes until a manual re-import. Consider storing an empty address on keyless import (derivation already covers routing dynamically) and stating one rule for when a recorded address outranks the catalog — spec 0037's table and the code currently disagree for token lifecycle.
  • The backfill rewrites full credential records with StoreCredentialRecord(..., replace=true) without fscred.WithCredentialLock, so a concurrent refresh-file-system-token rotation can be silently reverted by the stale APIKey write. Wrap each backfill write in the lock (the loop already ignores per-credential errors).
  • Because credentialsEqual now includes ServiceAddress (internal/fs/fscred/credential.go:243), re-importing the same token in the same region fails with fs.credential_import_conflict ("a different local token or region is already stored") when only the address changed (e.g. keyless import then keyed import). Treat an address-only difference as an updatable field, or at least name the address in the message.

5. Resolver contract grew profile coupling and forced caller-side defaulting duplication
internal/api/endpoints now imports the whole internal/config package (endpoints.go:14) because the new methods take *config.Profile, inverting the layering of low-level endpoint infra above the profile model when only a region code, optional address, and key pair are needed. In parallel, the NewResolver() + inherit-if-nonzero + debug-merge defaulting block is copy-pasted into four packages (internal/fs/control.go:284, internal/fs/aiconfig/service.go:47, internal/fs/tokenmgmt/service.go:801, internal/fswrap/runner.go:459), and the "keys → catalog, else recorded/derived" policy exists twice with different shapes (tokenmgmt vs fs.importEndpoint). Narrow the resolver input (e.g. ResolveFSRegion(ctx, regionCode string, keys fsregion.Credentials), ResolveFSCredentialEndpoint(placement)), put the defaulting on endpoints.Resolver itself, and extract one shared credential-vs-catalog helper next to ResolveFSCredentialEndpoint.

6. English ti- doc pages not updated in this change
The PR adds a user-facing command (ti fs list-regions) and changes published behavior (region discovery, import bootstrap, installer output), which AGENTS.md and spec 0037's own Documentation section require to move in the same change under docs/pingcap-docs/docs/ (plus TOC-ai.md / ai/_index.md). The PR body records this as a follow-up because the submodule is uninitialized in the authoring environment — please land it before the release that ships this, and consider adding it to the spec's verification gate alongside the pending make live-e2e-fs run so it can't be silently dropped after merge.

Nits

  • internal/fsregion/client.go:36 — the Client doc still says "caching and fallback policy live in Load"; no Load exists and this is a residue of the stale-cache wording the previous round asked to remove. Credentials.ProfileName (client.go:26) is declared but never set or read.
  • internal/fs/drive9_companion.go:1758 — isFSPlacementError is dead code; either wire it into the capability-probe error path its comment describes or delete it.

Recommendation

Request changes. Item 1 is a verified regression of the documented exit-code contract on the most common failure path (bad credentials) for every fs control-plane command, with a small local fix; everything else can follow as follow-ups.

srstack added a commit that referenced this pull request Nov 16, 2026
Address review feedback on #92.

Blocking:
- The catalog request is a Digest-authenticated TiDB Cloud control-plane call,
  so a credential, authorization, or not-found status now keeps the documented
  contract instead of collapsing into exit 1: 401 -> auth.invalid_credentials
  (3), 403 -> authz.permission_denied (4), 404 -> api.not_found (5). The
  classification lives in internal/auth and is rendered by both internal/api
  and internal/fsregion, so the taxonomy has one source; 5xx stays
  api.fs_regions_unavailable (1).

Should-fix:
- Resolver endpoint resolution accepts the command context, threaded through the
  fs control plane, token management, AI configuration, and import, so Ctrl+C
  cancels a slow catalog fetch.
- The routing gate is covered end to end: e2e asserts an unpublished region
  fails with api.fs_endpoint_unsupported, and an active entry with an unmapped
  cloud provider stays visible in list-regions while resolution rejects it.
- Keyless import-file-system-token no longer persists a derived address (only a
  catalog-resolved address is authoritative); an address-only change is no
  longer an import conflict, and a missing address never clears a recorded one.
- The list-regions backfill re-reads and writes each credential under the
  credential lock so a concurrent token rotation is not reverted.
- capability probing surfaces placement failures instead of reporting an
  outdated companion (isFSPlacementError is wired, not dead).

Docs:
- spec 0037 records the shared status taxonomy, cancellation, the address
  recording rule, the backfill lock, and the English-docs gate.
@srstack

srstack commented Nov 16, 2026

Copy link
Copy Markdown
Contributor Author

Thanks for the thorough pass — all blocking and should-fix items except item 5 are addressed in 1aa027b, with tests updated alongside.

1 (blocking) — exit-code contract. Fixed. The shared taxonomy now lives in internal/auth (StatusError, InvalidCredentials, PermissionDenied, NotFound) and is rendered by both internal/api and internal/fsregion, so there is one source rather than a second taxonomy: 401 → auth.invalid_credentials (exit 3, same wording/guidance as the shared client), 403 → authz.permission_denied (exit 4), 404 → api.not_found (exit 5). Other statuses (5xx) stay api.fs_regions_unavailable (exit 1). TestFetchKeepsTheDocumentedExitCodeContract pins all four buckets, and the api client's 401/404 branches now reuse the same helpers (internal/api tests unchanged).

2 — cancellation. Fixed by threading the command context: ResolveFSProfile/ResolveFSRegion take ctx and pass it to FetchFSRegions; the fs control plane (adminTenantClient/adminCreateInputs/adminDeleteInputs/adminItemInputs/resolveAdminFS), token management (resolveEndpoint/bearerClient/controlClient/resolveManagementAuth and the dry-runs), AI configuration (adminClient/embeddingInputs/extractInputs/dry-runs), and import-file-system-token all forward it. Ctrl+C now cancels a slow catalog fetch.

3 — routing-gate coverage / claims. The endpoints-level test now exists for exactly the ordering you described (TestResolveFSRegionRejectsUnmappedCloudProvider: lookup → eligible → routable), and e2e covers both claims through the fake catalog: unpublished aws-eu-central-1 fails with exit 2 + ti fs is not available for region …, and an ACTIVE aws-us-west-3 entry with cloudProvider: tencent still appears in list-regions JSON while --region aws-us-west-3 is rejected with “cloud provider "tencent" … does not support”. The PR body claims are corrected to match.

4 — credential lifecycle. Three changes:

  • keyless import-file-system-token no longer persists the derived address; only a catalog-resolved address is recorded (importEndpoint + HasTiDBCloudCredentials), and a keyless import then routes by derivation. This also resolves the “recorded wins over catalog” ambiguity: a recorded address is catalog-sourced by construction, so it always outranks the convention.
  • the list-regions backfill now re-reads and writes each credential inside fscred.WithCredentialLock, so a concurrent refresh-file-system-token rotation cannot be reverted.
  • credentialsEqual is split into credentialsIdentityEqual + an explicit address comparison: an address-only change updates the field instead of returning fs.credential_import_conflict, and an empty address never clears a recorded one. TestStoreCredentialUpdatesAddressWithoutReplace covers all three cases; spec 0037 states the rule.

5 — resolver coupling/defaulting. Deferred as a follow-up, with rationale rather than silence: both suggestions are right, but narrowing internal/api/endpoints inputs and moving the NewResolver() + inherit-if-nonzero + debug-merge block onto the resolver itself touches the four service packages and every resolver test, and I did not want to mix that structural change into a correctness fix under review. I kept the profile input (it is also the carrier for the catalog key pair and home directory) and recorded the refactor in the PR body as a follow-up.

6 — English docs. Agreed and it cannot be silently dropped: the submodule is uninitialized in this environment, so I recorded it in spec 0037's verification gate next to make live-e2e-fs — the spec now stays out of done/ until both the English ti- pages (TOC-ai.md, ai/_index.md included) and the live run land.

Nits. Done: stale Client/Load comment removed, Credentials.ProfileName is now used for the auth message, and isFSPlacementError is wired into incompatibleDrive9Error so a placement failure is not reported as an outdated companion.

Verification on 1aa027b: go vet ./..., go test ./..., and make e2e all pass. Please take another look when you have a moment.

@srstack
srstack requested a review from mornyx November 16, 2026 09:45

@mornyx mornyx left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review summary

Follow-up pass on 1aa027b. Every fix claim from the last round was re-verified against the source: the exit-code blocker is genuinely resolved — a shared status taxonomy in internal/auth now drives both the API client and the catalog client (401→exit 3, 403→exit 4, 404→exit 5, 5xx→exit 1, pinned by TestFetchKeepsTheDocumentedExitCodeContract); context threading, the keyless-import no-persist rule, the identity/address comparison split, the backfill lock, and all three nits are in as described, and both test suites pass. The resolver-input refactor deferral is accepted — the rationale is recorded and the correctness fixes belong in this change. What remains are defects introduced or left half-finished by the fix commit itself: a visible 403 message glitch, a credential lock that only some writers hold, the live gate Makefile target skipping the new live test, and a doubled catalog fetch on delete. None block the merge.

Resolved since the last review round (re-verified on 1aa027b):

  • Exit-code contract: single taxonomy in internal/auth, rendered by internal/api (401/404 reuse the helpers) and internal/fsregion; all four status buckets pinned by tests.
  • Cancellation: ResolveFSProfile/ResolveFSRegion take a context; control plane, token management, AI config, import, and the CLI dry-run handlers forward it; no context.Background() remains on live resolution paths.
  • Routing gate: unmapped-provider rejection covered at the catalog-unit level and through the binary via the fake catalog (ACTIVE tencent entry listed in list-regions, rejected for routing; unpublished region rejected with exit 2).
  • Credential lifecycle: keyless import persists no derived address; an address-only change updates instead of conflicting; an empty address never clears a recorded one; the backfill re-reads and writes under WithCredentialLock.
  • Resolver-input refactor: deferral accepted (recorded in the PR body); kept visible below as a follow-up, not a blocker.
  • English docs gate: recorded in spec 0037's verification gate next to the live run. Nits: all fixed.

What looks good

  • The internal/auth status taxonomy is exactly the right shape for the blocker fix: one classification source, two renderers, tests pinning every bucket — and the API client's 401/404 branches now reuse it instead of duplicating literals.
  • The control-plane/data-plane resolution split holds up under a second pass: token-only and offline data-plane use preserved by construction, and the unmapped-provider gate stays "listed but never routable".
  • The credential model change (address as placement metadata, not token identity) is consistently executed across store semantics, backfill, import, and mount locators, with the keyless no-persist and empty-address-preservation rules each pinned by tests.
  • Test depth is unusually good for this kind of change: binary-level e2e through a fake catalog (including a keyless import that must NOT persist the derived address, and a release build that must ignore TI_TEST_FS_REGIONS_URL), plus inverted installer assertions.

Should fix (non-blocking, strong)

1. The shared 403 classification renders a broken "in ." message
internal/auth/auth.go:113 calls authz.PermissionDenied(profileName, permission, action, "", ""), and authz.PermissionDenied (internal/authz/authz.go:137-152) builds location from provider/region with no empty fallback — so a catalog 403 renders "…is not allowed to read the Filesystem region catalog in . Ask an organization admin…". The API client's own permissionDeniedMessage (internal/api/client.go:532-545) guards exactly this case (location = "the selected scope"), but that fallback stayed in the renderer the refactor left behind: codes and exit codes are unified, the 403 wording is not, and TestFetchKeepsTheDocumentedExitCodeContract only asserts Contains("permission denied") so the glitch is untested. Fix: add the empty-location fallback to authz.PermissionDenied (or pass a real location such as "the Filesystem region catalog" from fsregion), make permissionDeniedMessage reuse it so there is exactly one 403 renderer, and pin the full message in the taxonomy test.

2. The credential lock only some writers hold
importFileSystemToken (internal/fs/drive9_companion.go:281) and createFileSystem (internal/fs/tenant_control.go:118) call fscred.StoreCredentialRecord without WithCredentialLock, while refresh/generate/delete and the new list-regions backfill all serialize through the lock. The stated invariant ("a concurrent token rotation is not reverted") holds only because rotation takes the lock — an unlocked import landing between the backfill's locked re-read and write can still be reverted (old token + new address restored after the import already reported success). The locking discipline currently depends on every caller remembering the lock, and three write-path families exist with only one covered. Fix: move the lock inside StoreCredentialRecord (it already does a read-compare-write, making it the natural choke point) or wrap the create/import stores — that closes the whole class rather than the rotation instance.

3. The live-e2e-fs gate silently skips the new live test
TestLiveFSListRegions (e2e/live_test.go:2664) exists, but the focused live-e2e-fs target (Makefile:73-74) runs an explicit name alternation that does not include FSListRegions, so the test only executes under the aggregate live-e2e. Spec 0037 and the PR body both name make live-e2e-fs as this feature's verification gate — run as written, the gate skips the only live check of list-regions and the credential backfill against the real API. Fix: add FSListRegions to the -run alternation (one line).

4. delete-file-system pays the catalog fetch twice
adminDeleteInputs (internal/fs/tenant_control.go:278-287) calls adminItemInputs, whose adminTenantClient already performs a full Digest-authenticated, paginated catalog fetch via resolveAdminFS and discards the endpoint (client, creds, _, err), then immediately calls s.resolveAdminFS again. With the old compiled-in table a double resolve was free; now each one is a network round-trip with its own timeout and failure modes, so every delete (and its dry-run) pays it twice. Fix: have adminTenantClient return the endpoint it already resolved (other call sites just ignore one more value) and drop the second call.

5. Resolver-input refactor (deferred) — accepted, but take one cheap slice now
The deferral is reasonable: narrowing internal/api/endpoints inputs touches four service packages and every resolver test, and it should not ride along with a correctness fix. But this PR grew the very coupling it defers — the identical NewResolver() + inherit-if-nonzero + debug-merge helper is copy-pasted in internal/fs/control.go:284, internal/fs/aiconfig/service.go:47, internal/fs/tokenmgmt/service.go:801, and internal/fswrap/runner.go:459, each copy now ~7 lines. Consider one exported helper (e.g. endpoints.ApplyDebug(base Resolver, debug bool, w io.Writer)) replacing the four copies now, and record the placement-defaulting asymmetry (ResolveFSProfile falls back to FSCloudProvider+FSRegionCode while ResolveFSCredentialEndpoint requires the placement code; key-based token management prefers a recorded — possibly stale — address over the live catalog) explicitly in the follow-up issue so it does not harden into contract.

Nits

  • docs/spec/0037-dynamic-fs-region-catalog.md:97 still says EligibleRegionCodes(); the code ships RoutableRegionCodes() (renamed by the unmapped-provider fix without updating this sentence).
  • The reply cites TestResolveFSRegionRejectsUnmappedCloudProvider at the endpoints layer; no such test exists there (the behavior is covered at the catalog-unit and binary-e2e levels). Add the endpoints-layer test or correct the record so the thread doesn't cite a test that isn't there.
  • internal/fswrap/runner_test.go:5 — the new fsregion import sits inside the standard-library import group against the repo's grouping convention.
  • internal/config/profile.go:297 — resolveOptionalTiDBCloudCredentials silently falls back to stored keys when only one of the TIDB_CLOUD_* pair is set; on the control-plane path this turns a CI misconfiguration into a silent success with different credentials. Worth a pinning test, or logging the ignored variable name.

Recommendation

Comment. The prior blocker and should-fix items are verified fixed or acceptably deferred, and both test suites pass; the five items above should land (or be ticketed) before or immediately after merge.

srstack added a commit that referenced this pull request Nov 16, 2026
Address the non-blocking findings on #92.

- Authorization wording: authz.PermissionDeniedMessage is now the single 403
  renderer with the empty-location fallback ("in the selected scope"), so a
  catalog 403 no longer renders "in ."; internal/api delegates to it and the
  taxonomy test pins the full sentence.
- Credential writes: create-file-system and import-file-system-token take the
  credential lock like token rotation and the region backfill, and
  StoreCredentialRecord documents that every racy writer must hold it.
- Live gate: the focused `make live-e2e-fs` target includes FSListRegions, so
  the gate named by spec 0037 actually runs the new live test.
- delete-file-system resolves the endpoint once: adminTenantClient hands back
  the endpoint it already resolved instead of fetching the catalog twice.
- Resolver defaulting: Resolver.WithDefaults/WithDebug replace the four
  copy-pasted NewResolver + inherit-if-nonzero + debug-merge blocks.

Nits: spec uses RoutableRegionCodes, the endpoints-layer unmapped-provider test
exists, the test import grouping follows the repo convention, and the partial
TIDB_CLOUD_* pair behavior of LoadLocal is pinned by a test.
@srstack

srstack commented Nov 16, 2026

Copy link
Copy Markdown
Contributor Author

Thanks again — everything from this round is in 8d54e29, and I added the one test the thread had claimed but did not have.

1 — 403 message. Fixed at the single renderer: authz.PermissionDeniedMessage now owns the wording and the empty-location fallback (location = "the selected scope"), authz.PermissionDenied and internal/api.permissionDeniedMessage both delegate to it, and TestFetchKeepsTheDocumentedExitCodeContract asserts the full sentence (… is not allowed to read the Filesystem region catalog in the selected scope …). No in . remains.

2 — credential lock coverage. create-file-system and import-file-system-token now wrap their store in fscred.WithCredentialLock, so create/import, token rotation, and the region backfill all hold the lock, and StoreCredentialRecord documents the requirement on the read-compare-write it performs. Deliberately not moving the acquisition inside StoreCredentialRecord: WithCredentialLock is a non-reentrant file lock, and rotation/backfill call the store while already holding it, so an internal lock would deadlock those paths. TestTenantControlCreatePersistenceFailureReturnsRecoveryData still pins the lock failure → warning branch.

3 — live gate. Fixed: the focused target now runs TestLive(FSRemoteInventoryLifecycle|FSCommandSurface|FSListRegions|…), so make live-e2e-fs actually executes the new live check and the backfill against the real API.

4 — double catalog fetch. Fixed: adminTenantClient returns the endpoint it already resolved (via a new adminItemEndpointInputs), and delete-file-system/its dry-run use that instead of resolving a second time.

5 — resolver defaulting, cheap slice taken. endpoints.Resolver.WithDefaults() and WithDebug(debug, writer) replace the four copy-pasted blocks. The remaining slice (narrowing the resolver inputs off *config.Profile, and the placement-defaulting asymmetry you called out — ResolveFSProfile falling back to FSCloudProvider+FSRegionCode versus ResolveFSCredentialEndpoint requiring the placement code, plus key-based token management preferring a recorded address) is recorded in the PR body as the explicit follow-up so it does not harden into contract.

Nits. docs/spec/0037 now says RoutableRegionCodes(); TestResolveFSRegionRejectsUnmappedCloudProvider now exists at the endpoints layer (you were right — it did not, the earlier round only added the catalog-unit and binary-e2e coverage); the fswrap test import moved into the third-party group; and LoadLocal's partial TIDB_CLOUD_* behavior is pinned by TestLoadLocalIgnoresAPartialTiDBCloudEnvironmentPair (documented as intentional: partial pairs never count as credentials, so a CI misconfiguration cannot silently authenticate with different keys on the data-plane path).

Verification on 8d54e29: gofmt, go vet ./..., go test ./..., make e2e all pass. Nothing in this round is blocking by your assessment; if the remaining follow-ups look acceptable as recorded, an approving review would unblock the merge.

@srstack
srstack requested a review from mornyx November 16, 2026 10:34

@mornyx mornyx left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review summary

Third pass, on 8d54e29. Every claim from the author's reply was verified against the source rather than the reply text: the 403 wording now renders from exactly one function with the empty-location fallback, create/import credential stores run under the lock, the live-e2e-fs gate actually executes TestLiveFSListRegions, delete-file-system resolves the catalog once, the four copy-pasted resolver-defaulting blocks collapsed into WithDefaults/WithDebug with semantics preserved, and all four nits from last round are in. Unit tests for every touched package pass at HEAD and the fix commit introduces no fresh defects. The lock-placement rationale is also correct engineering, not a dodge: WithCredentialLock is a non-reentrant O_EXCL file lock, and rotation/generate call the store from inside the closure, so moving acquisition into StoreCredentialRecord would self-deadlock — keeping it at the orchestration layer is the right structure, and the new doc comment pins the contract. What remains is nits only; the recorded follow-ups (resolver-input narrowing, live run, docs submodule) stay properly gated in spec 0037.

Resolved since the last review round (verified on 8d54e29):

  • 403 message: authz.PermissionDeniedMessage is the single renderer with the "the selected scope" fallback; authz.PermissionDenied and the API client both delegate; the taxonomy test pins the full sentence.
  • Credential lock: create-file-system and import-file-system-token wrap their stores in WithCredentialLock, matching the backfill and rotation sites; every racy StoreCredentialRecord writer is now covered.
  • Live gate: the live-e2e-fs alternation includes FSListRegions (and the $-anchoring still keeps the journal/vault/git families in their own targets).
  • Double fetch: adminTenantClient returns its endpoint via adminItemEndpointInputs; real and dry-run delete each resolve the catalog once.
  • Resolver defaulting: WithDefaults/WithDebug replace all four duplicated blocks, including the test-injection and debug-override semantics.
  • Nits: spec says RoutableRegionCodes(); TestResolveFSRegionRejectsUnmappedCloudProvider exists at the endpoints layer; the fswrap test import grouping is fixed; the partial TIDB_CLOUD_* pair behavior is pinned by a test.

What looks good

  • The single-renderer consolidation landed at the right layer: authz owns the 403 sentence, auth.PermissionDenied round-trips through the real error type so the taxonomy structurally cannot diverge, and tests assert rendered sentences rather than fragments.
  • The lock design keeps acquisition at the orchestration layer for a sound documented reason (non-reentrant lock; rotation/generate hold it across the store call), with warnings kept outside the critical section and the one-time token still surfaced on store failure.
  • WithDefaults/WithDebug retired the duplication with exact semantics preservation, and adminItemEndpointInputs fixed the double fetch by widening the existing helper instead of adding a parallel path.

Nits

  • internal/api/client.go:532 — permissionDeniedMessage is now a pure pass-through to authz.PermissionDeniedMessage with one caller; call the shared renderer directly and delete the wrapper.
  • internal/fs/tenant_control.go:286 — after the fix, adminItemInputs is a three-line discard wrapper with a single caller; have describeFileSystem call adminItemEndpointInputs directly and drop the hop.
  • internal/db/product/starter/cluster.go:575 — the Starter service still hand-rolls if s.Resolver.IsZero() { return endpoints.NewResolver() }; return s.Resolver, which is literally s.Resolver.WithDefaults(); adopting the helper that shipped in this PR is a one-liner.
  • internal/fs/fscred/credential.go:85 — the new lock-contract comment enumerates the writers that must hold the lock but omits the legacy migration writer (serialized by migrationMu, fail-closed, pre-existing); stating that exemption explicitly is better than leaving it discoverable only by grep.

Recommendation

Approve. All round-2 findings are verified fixed in source, tests pass, the remaining nits are optional polish, and the outstanding follow-ups are recorded and gated where they cannot be silently dropped.

@srstack

srstack commented Nov 16, 2026

Copy link
Copy Markdown
Contributor Author

All four nits from the approval pass are in 5b202da; nothing else changed (4 files, +6/−15):

  • internal/api/client.go — deleted the permissionDeniedMessage pass-through; the 403 branch calls authz.PermissionDeniedMessage directly, so the sentence has one renderer and one call site.
  • internal/fs/tenant_control.go — describe-file-system calls adminItemEndpointInputs directly; the three-line adminItemInputs discard wrapper is gone.
  • internal/db/product/starter/cluster.go — the Starter service now uses Resolver.WithDefaults() instead of its own IsZero/NewResolver block.
  • internal/fs/fscred/credential.go — the lock contract names the legacy name-registry migration exemption explicitly (serialized by migrationMu, fail-closed, runs before other writers for the file systems it migrates).

Verified on 5b202da: gofmt, go vet ./..., go test ./..., make e2e pass; CI test pass (2m0s) and CLA pass. The approval still stands on the new head.

Replace the two compiled-in Filesystem region tables (the region whitelist in
internal/config/region and builtInFSRegionManifest in internal/api/endpoints)
with the catalog published by TiDB Cloud Filesystem:
GET https://fs.tidbapi.com/v1beta2/regions (Digest auth with the API keys).

- internal/fsregion: paginated catalog client, contract validation (canonical
  regionId, provider consistency, https service address), ACTIVE eligibility,
  and the conventional https://<region-id>.drive9.ai address derivation.
- Control-plane commands (create/list/describe/delete, import, token lifecycle,
  AI provider configuration, list-regions) resolve the regional endpoint from
  the catalog and require TiDB Cloud API keys.
- Data-plane, mount, layer, pack/unpack, Git, journal, vault, and exec route
  from the service_address recorded in the File System credential, falling back
  to the derived conventional address. They never request the catalog and stay
  usable without TiDB Cloud keys.
- ID-keyed credentials gain an optional service_address. create-file-system and
  import-file-system-token write it, ti fs list-regions backfills it on stored
  credentials, and mount locators record it for drain/unmount.
- region.ParseCanonicalCode splits structural parsing from the static Starter
  whitelist, which stays unchanged so ti db placement validation does not move.
- New read-only ti fs list-regions command (fs.region.read) with JSON/text
  output and --query support.
- Installers stop printing compiled-in region lists; hidden test control
  TI_TEST_FS_MANIFEST_URL becomes TI_TEST_FS_REGIONS_URL.
- Docs, AGENTS.md, docs/priciples.md, README, e2e, live e2e, and
  docs/spec/0037-dynamic-fs-region-catalog.md updated.

Verification: go test ./... and make e2e pass. Live verification
(make live-e2e-fs) is still pending and is recorded as the release gate in the
spec.
Address review feedback on #92.

- ResolveFSRegion no longer borrows the provider mapping from the region-code
  prefix when the catalog reports a cloudProvider this release cannot map. Such
  an entry stays visible in `ti fs list-regions` but fails with
  api.fs_endpoint_unsupported naming the provider, matching the spec.
- Catalog gains Routable()/RoutableRegionCodes() so "available regions" in
  errors only lists routable entries.
- internal/fsregion package comment no longer claims a local cache; the catalog
  is always fetched on demand.
- Tests cover the unmapped-provider listing and rejection paths.
Address review feedback on #92.

Blocking:
- The catalog request is a Digest-authenticated TiDB Cloud control-plane call,
  so a credential, authorization, or not-found status now keeps the documented
  contract instead of collapsing into exit 1: 401 -> auth.invalid_credentials
  (3), 403 -> authz.permission_denied (4), 404 -> api.not_found (5). The
  classification lives in internal/auth and is rendered by both internal/api
  and internal/fsregion, so the taxonomy has one source; 5xx stays
  api.fs_regions_unavailable (1).

Should-fix:
- Resolver endpoint resolution accepts the command context, threaded through the
  fs control plane, token management, AI configuration, and import, so Ctrl+C
  cancels a slow catalog fetch.
- The routing gate is covered end to end: e2e asserts an unpublished region
  fails with api.fs_endpoint_unsupported, and an active entry with an unmapped
  cloud provider stays visible in list-regions while resolution rejects it.
- Keyless import-file-system-token no longer persists a derived address (only a
  catalog-resolved address is authoritative); an address-only change is no
  longer an import conflict, and a missing address never clears a recorded one.
- The list-regions backfill re-reads and writes each credential under the
  credential lock so a concurrent token rotation is not reverted.
- capability probing surfaces placement failures instead of reporting an
  outdated companion (isFSPlacementError is wired, not dead).

Docs:
- spec 0037 records the shared status taxonomy, cancellation, the address
  recording rule, the backfill lock, and the English-docs gate.
Address the non-blocking findings on #92.

- Authorization wording: authz.PermissionDeniedMessage is now the single 403
  renderer with the empty-location fallback ("in the selected scope"), so a
  catalog 403 no longer renders "in ."; internal/api delegates to it and the
  taxonomy test pins the full sentence.
- Credential writes: create-file-system and import-file-system-token take the
  credential lock like token rotation and the region backfill, and
  StoreCredentialRecord documents that every racy writer must hold it.
- Live gate: the focused `make live-e2e-fs` target includes FSListRegions, so
  the gate named by spec 0037 actually runs the new live test.
- delete-file-system resolves the endpoint once: adminTenantClient hands back
  the endpoint it already resolved instead of fetching the catalog twice.
- Resolver defaulting: Resolver.WithDefaults/WithDebug replace the four
  copy-pasted NewResolver + inherit-if-nonzero + debug-merge blocks.

Nits: spec uses RoutableRegionCodes, the endpoints-layer unmapped-provider test
exists, the test import grouping follows the repo convention, and the partial
TIDB_CLOUD_* pair behavior of LoadLocal is pinned by a test.
- Drop the api.permissionDeniedMessage pass-through; the 403 branch calls
  authz.PermissionDeniedMessage directly, so the sentence has one renderer with
  one call site.
- describe-file-system calls adminItemEndpointInputs directly and the
  three-line adminItemInputs discard wrapper is gone.
- The Starter product service uses Resolver.WithDefaults() instead of its own
  IsZero/NewResolver block.
- StoreCredentialRecord's lock contract names the legacy name-registry migration
  exemption (serialized by migrationMu, fail-closed, runs before other writers).
@srstack
srstack force-pushed the feat/dynamic-fs-region-catalog branch from 5b202da to 9142645 Compare November 16, 2026 12:22
@srstack
srstack merged commit 6e37676 into main Nov 16, 2026
2 checks passed
@srstack
srstack deleted the feat/dynamic-fs-region-catalog branch November 16, 2026 15:14
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants