Skip to content

fix(migrations): a migration with a row threshold decides by the current size of the table, not by the estimate of its last ANALYZE (migration 191) - #484

Merged
GottZ merged 1 commit into
rootfrom
issue/399-migration-threshold-helper
Nov 22, 2026
Merged

GottZ merged 1 commit into
rootfrom
issue/399-migration-threshold-helper

Conversation

@GottZ

@GottZ GottZ commented Nov 21, 2026 •

Copy link
Copy Markdown
Owner

What this changes

Migration 191 adds one SQL function, ctx_migration_rows(p_rel regclass, p_threshold bigint) RETURNS bigint: the size a migration with a row threshold compares with it — at or above the threshold, the out-of-band path. It measures rows, not the heap.

  • Estimate. For each relation with a heap in the tree under p_rel in pg_inherits (partitions, inheritance children, TimescaleDB chunks) it takes the largest of reltuples, the stored density applied to the current heap (reltuples / relpages × pg_relation_size / block_size, the issue's proposal) and n_live_tup, and sums them.
  • The estimate decides only "out of band". A sum at or above the threshold is the answer, without a read of the table. Below it the function counts the rows through p_rel and stops at the threshold — so no estimate ever decides "inline". The value is the exact row count below the threshold; at or above it the estimate, or the threshold itself where a count stopped there.
  • It also counts where no estimate can be trusted: never analysed or truncated (reltuples −1), reltuples 0, relpages below 32, no VACUUM/ANALYZE of the relation on record in the cumulative statistics (crash recovery, pg_stat_reset, pg_upgrade).
  • A missing relation (42P01), a relation that is not a table (42809) and a missing or non-positive threshold (22023) raise instead of returning a number a caller would read as small. Invoker rights, SET search_path = pg_catalog, pg_temp, VOLATILE.

The twelve shipped files that copy the reltuples decision stay as they are (frozen by checksum). docs/development.md gains "A migration with a row threshold" (the pattern, what the value means, rows vs. heap), docs/operations.md gains "Migration 191" (measurements, the ANALYZE before upgrading a large store from before one of the twelve, a bloated heap, way back). A unit test over go/migrations refuses any later migration whose code decides a size itself — reads reltuples, relpages, n_live_tup, pg_stat_get_live_tuples, pg_relation_size, pg_total_relation_size or pg_table_size outside a definition of the function (comments and string literals do not count as a definition). The schema-contract manifest gains the function.

Why

Closes #399.

The twelve files (115, 140, 171, 172, 173, 175, 176, 182, 184, 185, 186, 187 — the issue named eight; 183 has no threshold, and 115, 140, 173, 175, 176 copy the same query) decide by pg_class.reltuples. A table that grew since its last ANALYZE (autovacuum off, a bulk load right before the upgrade) or was analysed while empty passes as small, and the file builds inline above its threshold while writers wait.

No estimate may decide "inline", because all three measures can trail the table together: VACUUM sets reltuples and n_live_tup by extrapolation (vac_estimate_reltuples), counting the pages it skips as all-visible at the table's stored density. Measured (each a regression test): 100 000 rows made all-visible, every row beyond the first 10 000 but one in a thousand deleted, ANALYZE (10 090), VACUUM → all three read 1 103; refilled to 100 000 → 91 013; the oldest 90 % deleted, ANALYZE, VACUUM → 1 048 for 10 000; VACUUM only, never ANALYZE (half deleted, VACUUM, refilled, VACUUM) → 75 030 for 100 000; a refill whose counts a live session still holds back → 2 001 for 20 000. The three measures remain to let a table above the threshold go out of band without a read and to give the WARNING a size near the truth; n_live_tup sees growth the file does not show (a refill into space a VACUUM freed), the density growth no ANALYZE saw. Rejected: taking the heap size into the decision (the function measures rows; a threshold in pages would need a density the threshold does not carry) — a bloated heap is the operator's to compact before the upgrade, measured and documented; rejecting partitioned tables and hypertables (the tree sum covers them; the copied query read 0 on a hypertable parent and would build inline at any size); a predicate returning a boolean (the files print the size in their WARNING).

Behaviour changes (intended)

None for a running instance: no shipped migration and no Go code calls the function. The migration only creates it.

Consumers of the changed contracts

Changed Callers and readers Covered by
new SQL function ctx_migration_rows(regclass, bigint) none in code; future migrations (pattern in docs/development.md) TestMigration191RowsFunction, TestMigration191EstimateBelowIsCounted, TestMigration191IssueMeasurement (172 and 184 rebuilt on it)
schema-contract manifest (function entry, chain fingerprint) schemacontract.Check at boot and on recheck; rollback image v5.73.0 TestManifestNamesTheMigrationChain, TestCheck_FreshDB_NoDrift; rollback: the v5.73.0 manifest diffed against schema 191
migration chain (191 added) the runner, pre-commit Gate 3 the migration-test packages listed under Verification

Cost

The migration writes one catalog row and takes no table lock (it runs beside ACCESS EXCLUSIVE holders of all eight tables the twelve files measure). The function reads pg_class, pg_inherits and the cumulative statistics and calls pg_relation_size once per relation of the tree, which takes ACCESS SHARE and releases it before returning (it waits only behind an ACCESS EXCLUSIVE holder, up to the caller's lock_timeout). An estimate above the threshold costs 0.6 ms at 3 000 000 tokens. A count reads at most the threshold's live rows under ACCESS SHARE to the caller's commit — 204 ms for 500 000 rows, 226 ms for 1 000 000 (host load 11) — and runs only where the file then builds inline and reads the whole heap anyway; over 10 000 000 dead rows and 5 live ones (1 655 MB) it read 3.7 s the first time after the delete (setting the hint bits the delete left, as any first reader does) and 0.17–0.22 s after.

Verification

  • Red before, green after. On the base: the function does not exist (42883). On the first head of this PR: the five states above returned 1 103, 1 103 (without FREEZE), 91 013, 1 048, 75 030 and 2 001 — each below its threshold on a table above it; now each returns the threshold (a count). Green: the 27 subtests of TestMigration191RowsFunction, the six states of TestMigration191EstimateBelowIsCounted, the lock test and the rebuilt 172/184; new among them: per-relation record (a child whose counters are lost while its parent's stand), the page floor (a coarse density claiming 90 000 for 2 900), rounding at the threshold, a child under two parents measured once, VACUUM-only record.
  • The issue's measurement at full size (pgvector-timescaledb:pg18, autovacuum off, tokens analysed at 100 000 and grown to 3 000 000: reltuples 100 000, relpages 3 125, pg_relation_size 768 000 000 — the issue's numbers): copied query 100 000; file 184 as shipped built inline in 2.55 s, 3.00 s and 3.75 s (three runs) with a token writer waiting as long (host load 11–23); ctx_migration_rows 3 000 000, file 184 rebuilt on it warned and built nothing (25 ms in the last run).
  • A heap of dead rows (10 000 000 rows, 5 live, 1 655 MB): the function counts 5; an inline build over that heap took 2.1 s, and 0.21 s after a VACUUM, which left the heap at 1 655 MB.
  • Mutants: 34 of 34 killed. On the function (25): only reltuples; only density × size; each of the three terms dropped; density without the size; no pg_inherits tree; UNION ALL in the tree (a child under two parents measured twice); page floor removed; floor 32 → 2; relpages guard removed (division by zero); reltuples 0 trusted; count without LIMIT; count one row short; an estimate below the threshold trusted; an estimate at the threshold counted (> for >=); floor for ceil in the decision; relkind, missing-oid and threshold checks removed; empty-heap shortcut removed; pg_total_relation_size for the main fork; the no-record rule removed; VACUUM not taken as a record; the record read on the root instead of per relation. On the size gate (9): comments kept; string literals not tracked; 191 itself left out; a definition not cut out; only reltuples a size read; a definition reaching to the end of the file; definitions found inside string literals; case-sensitive reads; a definition anywhere in the raw source exempting the file.
  • Gates: go/gates.sh wave --final rc 0 (build, vet, lint, doccheck, short, census, race, integration 6/6 including internal/store and internal/handler, deadcode, deadcode-testonly); one earlier run of it failed once on TestMigration185KeyDeleteUsesTheIndexes (a planner pin of migration 185 on schema 184, which this change does not reach: 75 buffers on a bitmap scan where it allows 64), which passed five of five times run alone and in the package run before. Migration-test packages run on their own (go test -tags=integration -p 1): guard, blocktype, backends, llmlog, llmusage, overview, rrf, cmd/ctxd, events, store and schemacontract all ok. The new tests ran five times each, with and without -race, and once on the production image pgvector-timescaledb:pg18.

Not verified

  • File 185 at 10 000 000 write-log rows with the function, cold storage, 100 000 000 rows.
  • The v5.73.0 image itself against schema 191 was not started; its manifest was diffed against a migrated database (one breaking drift: the unknown function) and its boot handling read in cmd/ctxd/contract.go.
  • A real crash recovery or pg_upgrade (modelled with pg_stat_reset_single_table_counters).
  • What a count does not see, by design: rows of a transaction still open when it runs (the build that follows waits for that transaction's lock, at most the file's lock_timeout), and rows row-level security hides from a role subject to it (no table of ctx has row-level security; ctx migrates as the owner).

🤖 Generated with Claude Code

@GottZ
GottZ requested a balanced review from Copilot November 21, 2026 21:57
@coderabbitai

coderabbitai Bot commented Nov 21, 2026 •

Copy link
Copy Markdown

Warning

Review limit reached

You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository.

Next included review available in 44 minutes.

Check out review usage here.

View limit details

Limit details: You’ve used the included review currently available.

Learn how review limits work.

Review configuration:

Run configuration

  • Source mode: Auto
  • Repository: Unavailable
  • Branch: Unavailable
  • File: No configuration file from this repository applied
  • Commit: Unavailable
  • Layers: defaults
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: 4fae7163-e403-4c2f-81f0-5b01175bc8ab
Commits

Reviewing files that changed from the base of the PR and between 3d77ce3 and ffe2c2a.

Files selected for processing: 6

  • docs/development.md
  • docs/operations.md
  • go/internal/schemacontract/manifest.json
  • go/internal/store/migration191_integration_test.go
  • go/migrations/191_migration_rows.sql
  • go/migrations/threshold_test.go
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has insufficient remaining quota. Try requesting a review at a lower effort level.

@GottZ
GottZ force-pushed the issue/399-migration-threshold-helper branch from 71c9a34 to 9864710 Compare November 22, 2026 00:18
…ent size of the table, not by the estimate of its last ANALYZE (migration 191)

Closes #399.

Twelve migrations (115, 140, 171-173, 175, 176, 182, 184-187) build an index
or validate a constraint inline below a row threshold and decide by a copied
pg_class.reltuples query: a table that grew since its last ANALYZE
(autovacuum off, a bulk load right before the upgrade) or was analysed while
empty passes as small and is built inline above the threshold (file 184 at
3 000 000 tokens analysed at 100 000: 2.55-3.75 s, its writer waiting as
long). Migration 191 adds ctx_migration_rows(regclass, bigint): it estimates
(the largest of reltuples, the stored density times the current heap size
and n_live_tup, summed over partitions, inheritance children and TimescaleDB
chunks) and lets the estimate decide only "out of band"; below the
threshold, and where no estimate can be trusted, it counts the rows, stopped
at the threshold. All three measures can trail a table together (a VACUUM
extrapolates reltuples and n_live_tup over the all-visible pages it skips:
1 103 read for 10 090 rows), so no estimate decides "inline". The migration
takes no table lock. The frozen files stay; docs carry the pattern, the
ANALYZE before upgrading a large store and the advice for a heap far larger
than its rows, and a test over go/migrations refuses a later file that
decides a size itself.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@GottZ
GottZ force-pushed the issue/399-migration-threshold-helper branch from 9864710 to ffe2c2a Compare November 22, 2026 00:27
@GottZ
GottZ merged commit 53b8f59 into root Nov 22, 2026
25 checks passed
@GottZ
GottZ deleted the issue/399-migration-threshold-helper branch November 22, 2026 00:40
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

migrations: the inline-or-runbook threshold reads pg_class.reltuples alone, a trailing estimate builds inline above the threshold

2 participants