Repository navigation
fix(migrations): a migration with a row threshold decides by the current size of the table, not by the estimate of its last ANALYZE (migration 191) - #484
Merged
Conversation
|
Warning Review limit reachedYou've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository. Next included review available in 44 minutes. View limit details
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. |
GottZ
force-pushed
the
issue/399-migration-threshold-helper
branch
from
November 22, 2026 00:18
71c9a34 to
9864710
Compare
…ent size of the table, not by the estimate of its last ANALYZE (migration 191) Closes #399. Twelve migrations (115, 140, 171-173, 175, 176, 182, 184-187) build an index or validate a constraint inline below a row threshold and decide by a copied pg_class.reltuples query: a table that grew since its last ANALYZE (autovacuum off, a bulk load right before the upgrade) or was analysed while empty passes as small and is built inline above the threshold (file 184 at 3 000 000 tokens analysed at 100 000: 2.55-3.75 s, its writer waiting as long). Migration 191 adds ctx_migration_rows(regclass, bigint): it estimates (the largest of reltuples, the stored density times the current heap size and n_live_tup, summed over partitions, inheritance children and TimescaleDB chunks) and lets the estimate decide only "out of band"; below the threshold, and where no estimate can be trusted, it counts the rows, stopped at the threshold. All three measures can trail a table together (a VACUUM extrapolates reltuples and n_live_tup over the all-visible pages it skips: 1 103 read for 10 090 rows), so no estimate decides "inline". The migration takes no table lock. The frozen files stay; docs carry the pattern, the ANALYZE before upgrading a large store and the advice for a heap far larger than its rows, and a test over go/migrations refuses a later file that decides a size itself. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
GottZ
force-pushed
the
issue/399-migration-threshold-helper
branch
from
November 22, 2026 00:27
9864710 to
ffe2c2a
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this changes
Migration 191 adds one SQL function,
ctx_migration_rows(p_rel regclass, p_threshold bigint) RETURNS bigint: the size a migration with a row threshold compares with it — at or above the threshold, the out-of-band path. It measures rows, not the heap.p_relinpg_inherits(partitions, inheritance children, TimescaleDB chunks) it takes the largest ofreltuples, the stored density applied to the current heap (reltuples / relpages × pg_relation_size / block_size, the issue's proposal) andn_live_tup, and sums them.p_reland stops at the threshold — so no estimate ever decides "inline". The value is the exact row count below the threshold; at or above it the estimate, or the threshold itself where a count stopped there.reltuples−1),reltuples0,relpagesbelow 32, noVACUUM/ANALYZEof the relation on record in the cumulative statistics (crash recovery,pg_stat_reset,pg_upgrade).SET search_path = pg_catalog, pg_temp,VOLATILE.The twelve shipped files that copy the
reltuplesdecision stay as they are (frozen by checksum).docs/development.mdgains "A migration with a row threshold" (the pattern, what the value means, rows vs. heap),docs/operations.mdgains "Migration 191" (measurements, theANALYZEbefore upgrading a large store from before one of the twelve, a bloated heap, way back). A unit test overgo/migrationsrefuses any later migration whose code decides a size itself — readsreltuples,relpages,n_live_tup,pg_stat_get_live_tuples,pg_relation_size,pg_total_relation_sizeorpg_table_sizeoutside a definition of the function (comments and string literals do not count as a definition). The schema-contract manifest gains the function.Why
Closes #399.
The twelve files (115, 140, 171, 172, 173, 175, 176, 182, 184, 185, 186, 187 — the issue named eight; 183 has no threshold, and 115, 140, 173, 175, 176 copy the same query) decide by
pg_class.reltuples. A table that grew since its lastANALYZE(autovacuum off, a bulk load right before the upgrade) or was analysed while empty passes as small, and the file builds inline above its threshold while writers wait.No estimate may decide "inline", because all three measures can trail the table together:
VACUUMsetsreltuplesandn_live_tupby extrapolation (vac_estimate_reltuples), counting the pages it skips as all-visible at the table's stored density. Measured (each a regression test): 100 000 rows made all-visible, every row beyond the first 10 000 but one in a thousand deleted,ANALYZE(10 090),VACUUM→ all three read 1 103; refilled to 100 000 → 91 013; the oldest 90 % deleted,ANALYZE,VACUUM→ 1 048 for 10 000;VACUUMonly, neverANALYZE(half deleted,VACUUM, refilled,VACUUM) → 75 030 for 100 000; a refill whose counts a live session still holds back → 2 001 for 20 000. The three measures remain to let a table above the threshold go out of band without a read and to give the WARNING a size near the truth;n_live_tupsees growth the file does not show (a refill into space aVACUUMfreed), the density growth noANALYZEsaw. Rejected: taking the heap size into the decision (the function measures rows; a threshold in pages would need a density the threshold does not carry) — a bloated heap is the operator's to compact before the upgrade, measured and documented; rejecting partitioned tables and hypertables (the tree sum covers them; the copied query read 0 on a hypertable parent and would build inline at any size); a predicate returning a boolean (the files print the size in their WARNING).Behaviour changes (intended)
None for a running instance: no shipped migration and no Go code calls the function. The migration only creates it.
Consumers of the changed contracts
ctx_migration_rows(regclass, bigint)docs/development.md)TestMigration191RowsFunction,TestMigration191EstimateBelowIsCounted,TestMigration191IssueMeasurement(172 and 184 rebuilt on it)schemacontract.Checkat boot and on recheck; rollback image v5.73.0TestManifestNamesTheMigrationChain,TestCheck_FreshDB_NoDrift; rollback: the v5.73.0 manifest diffed against schema 191Cost
The migration writes one catalog row and takes no table lock (it runs beside
ACCESS EXCLUSIVEholders of all eight tables the twelve files measure). The function readspg_class,pg_inheritsand the cumulative statistics and callspg_relation_sizeonce per relation of the tree, which takesACCESS SHAREand releases it before returning (it waits only behind anACCESS EXCLUSIVEholder, up to the caller'slock_timeout). An estimate above the threshold costs 0.6 ms at 3 000 000 tokens. A count reads at most the threshold's live rows underACCESS SHAREto the caller's commit — 204 ms for 500 000 rows, 226 ms for 1 000 000 (host load 11) — and runs only where the file then builds inline and reads the whole heap anyway; over 10 000 000 dead rows and 5 live ones (1 655 MB) it read 3.7 s the first time after the delete (setting the hint bits the delete left, as any first reader does) and 0.17–0.22 s after.Verification
FREEZE), 91 013, 1 048, 75 030 and 2 001 — each below its threshold on a table above it; now each returns the threshold (a count). Green: the 27 subtests ofTestMigration191RowsFunction, the six states ofTestMigration191EstimateBelowIsCounted, the lock test and the rebuilt 172/184; new among them: per-relation record (a child whose counters are lost while its parent's stand), the page floor (a coarse density claiming 90 000 for 2 900), rounding at the threshold, a child under two parents measured once,VACUUM-only record.pgvector-timescaledb:pg18, autovacuum off, tokens analysed at 100 000 and grown to 3 000 000:reltuples100 000,relpages3 125,pg_relation_size768 000 000 — the issue's numbers): copied query 100 000; file 184 as shipped built inline in 2.55 s, 3.00 s and 3.75 s (three runs) with a token writer waiting as long (host load 11–23);ctx_migration_rows3 000 000, file 184 rebuilt on it warned and built nothing (25 ms in the last run).VACUUM, which left the heap at 1 655 MB.reltuples; only density × size; each of the three terms dropped; density without the size; nopg_inheritstree;UNION ALLin the tree (a child under two parents measured twice); page floor removed; floor 32 → 2;relpagesguard removed (division by zero);reltuples0 trusted; count withoutLIMIT; count one row short; an estimate below the threshold trusted; an estimate at the threshold counted (>for>=);floorforceilin the decision; relkind, missing-oid and threshold checks removed; empty-heap shortcut removed;pg_total_relation_sizefor the main fork; the no-record rule removed;VACUUMnot taken as a record; the record read on the root instead of per relation. On the size gate (9): comments kept; string literals not tracked; 191 itself left out; a definition not cut out; onlyreltuplesa size read; a definition reaching to the end of the file; definitions found inside string literals; case-sensitive reads; a definition anywhere in the raw source exempting the file.go/gates.sh wave --finalrc 0 (build, vet, lint, doccheck, short, census, race, integration 6/6 includinginternal/storeandinternal/handler, deadcode, deadcode-testonly); one earlier run of it failed once onTestMigration185KeyDeleteUsesTheIndexes(a planner pin of migration 185 on schema 184, which this change does not reach: 75 buffers on a bitmap scan where it allows 64), which passed five of five times run alone and in the package run before. Migration-test packages run on their own (go test -tags=integration -p 1): guard, blocktype, backends, llmlog, llmusage, overview, rrf, cmd/ctxd, events, store and schemacontract all ok. The new tests ran five times each, with and without-race, and once on the production imagepgvector-timescaledb:pg18.Not verified
cmd/ctxd/contract.go.pg_upgrade(modelled withpg_stat_reset_single_table_counters).lock_timeout), and rows row-level security hides from a role subject to it (no table of ctx has row-level security; ctx migrates as the owner).🤖 Generated with Claude Code