ADFA-6683: Store documentation pages in compressed blocks (schema 4.0.0) by davidschachterADFA · Pull Request #58 · appdevforall/OfflineDocumentationTools

davidschachterADFA · 2026-10-27T18:14:32Z

Why

The documentation database stores every page as its own Brotli row, so the dictionary can't share what neighbouring pages have in common. Storing pages in compressed blocks (64 pages per block, one Brotli stream against the shared dictionary) cuts the .br download by about 10 MB. This is the ship-time step that makes it, and it declares a new major, 4.0.0, because an app without the reader would show a blocked page as empty. The reader is CodeOnTheGo #2191 (base stage); a 4.0.0 database must not go to dev-assets or Drive before that PR is merged, and the Drive master is upgraded to 4.0.0 only after ADFA-6871 (how a 4.x master is built into a copy that ships: the ship-time order starts from a 2.x copy, as the master is today), ADFA-6806 (docs pipeline scripts refuse or handle 4.0.0), ADFA-6807 (DocDB Studio opens and edits a 4.0.0 database) and ADFA-6808 (check-tools tooltip-table failures on 3.x, blocked pages flagged empty) are done.

Change

scripts/store_pages_in_blocks.py: five sets, each in ORDER BY path COLLATE BINARY, never mixed in a block: a/ templated (not a/devsite/), j/ text/html, k/ templated, every Brotli row under a/devsite/, every Brotli row under i/. Case-sensitive GLOB.
Blocks of 64 items. An item over 1 MiB (decoded) stays a row. A block closes before it would pass 4 MiB decoded.
Version gate: refuses no version, a major below 3, and a major of 4 or above. Refuses a database that already has the tables.
Declares 4.0.0 in the same transaction (a dry run rolls it back).
Before a row is emptied, each item's slice is compared byte for byte with that row's decoded bytes. Before the commit, ContentBlocks must hold exactly the blocks written (by sha256) and ContentBlockItems exactly the items written (block, offset, length); no block is decoded twice. An empty Brotli row stays a row.
optimize_db_media.py: declared_major and table_exists (moved from this script), and BrotliCodec, which this script now uses, passes -q 11 -w 24 explicitly (the CLI's defaults for stdin, so the same bytes). migrate_tooltip_schema.declare_version takes the version to declare, and this script reuses it.
remint_dictionary.py and verify_remint_dictionary.py refuse a 4.x file, and any file that has the block tables; the verifier closes both databases when it does. B02 measured re-minting on blocks: it saves at most 30,880 B of .br, against 0.33 MB of seed noise and a 25,285 B worse 4-seed mean, so the order stays re-mint, verify, blocks, pack.
CLAUDE.md: a "Schema 4.0.0" section (tables, ship-time order, --pack Templates,Content,TooltipTexts,ContentBlocks --page-size 4096, which tools refuse 4.x, release order). python-tests.yaml runs the new test file.

Measured

On the 2026-10-20 ship-time build the packed .br goes from 66,283,003 to 55,914,367 bytes (10,368,636 smaller): 22,478 pages in 356 blocks, 18 kept as rows by the cap, largest block 4,172,562 bytes decoded and 559,581 bytes compressed. integrity_check, foreign_key_check and the pack's identity check are clean. On the shipped 3.0.0 database (B02) the saving is 10,231,079 bytes. On a Pixel 6 Pro, 12 of 13 sampled pages were byte-identical to the 3.x layout.

Tests

scripts/test_store_pages_in_blocks.py: each set (including unTemplated a/devsite/ and i/ rows), the cap, the 4 MiB block close, the default of 64, the refusals (no version, 2.x, 4.x) with nothing written, 4.0.0 declared after a run and not after a dry run, a byte mismatch that refuses and writes nothing, more than one decode window, the write-ahead path, and every check_stored check (a block lost with or without its items, a changed block, a moved item, a row not emptied), an empty row left a row, and each block decoded once.
test_remint_dictionary.py: the refusals on the function and the command line, and both connections closed after a refusal.
Mutants: each new test was run against a script broken the way it names (20 of 20 caught). Of the tests added for Hal Eisen's review of aa4f43f, four fail on the code before it: the block lost with its items kept (the run committed), the empty row (the run stopped), the verifier's connections (both left open) and the single decode.
Full CI pytest step: 715 passed, 1 skipped (667 passed on origin/main). The tests of the other scripts that use BrotliCodec pass (450).

Not covered

The compressed size of a block isn't capped. Android's default CursorWindow is 2 MiB, so incompressible Brotli rows (PDFs) in sets 4 and 5 could make a block the app can't read. None are in those sets on this build, and the reader PR reads big blocks in pieces; a cap here is a follow-up.
Ship-time steps other than the re-mint pair don't refuse 4.x, and they also run on the Drive master, which will become 4.0.0 and never go back to 3.x. sync_kdoc_json_to_db.py can silently corrupt a blocked database, insert_optimized_media.py crashes on the first blocked row, and migrate_content_to_dictionary_brotli.py exits 1 with an error per blocked row. Fixing or guarding them is ADFA-6806; DocDB Studio is ADFA-6807; check-tools is ADFA-6808. CLAUDE.md lists what each tool does on a blocked database. The Drive master is not upgraded before those and ADFA-6871 are done.
The full B05 build (fresh-process timing, off-device traffic check) waits until both PRs are merged.
Commit 670095f's message still says B02 hadn't run; later commits supersede it.

🤖 Generated with Claude Code

store_pages_in_blocks.py DB --pages-per-block N [--dry-run] writes the format in ADFA-6683-block-format.md, which the app reader follows too: - ContentBlocks(id, data): Brotli q11 w24 against CompressionDictionary of N pages' decoded bytes, concatenated. - ContentBlockItems(contentId, blockId, offset, length), WITHOUT ROWID with declared foreign keys, so rebuild_database.py --pack Content renumbers contentId. - Three sets, each cut in path order (BINARY), the last block of a set shorter, never mixed: a/ templated (not a/devsite/), j/ text/html, k/ templated. Brotli types only; a row with continuation rows and each continuation row stay as they are. - A blocked row keeps path, languageID, contentTypeID and templateId; its content becomes x''. Each block is decoded again before it is stored, and before the commit every item is sliced from its stored block and compared by sha256 with its row's decoded bytes. One transaction, then VACUUM; refuses a database that already has ContentBlockItems; --dry-run rolls every write back. test_store_pages_in_blocks.py covers the seven cases on a synthetic database. Each test was checked to fail against a script broken in the way it names: id order or NOCASE order, mixed sets, no continuation skip, a stuck offset, rows not emptied, templateId cleared, no refusal, a dry run that commits. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018CMZQuXT3GT7nEct2w4tg3

…bers blocks Measured on the 2026-10-20 G02 ship-time build (final.db, sha256 c7de1d8a), each database packed with rebuild_database.py --page-size 4096: - Release .br (brotli -q 11 -w 24) 7,904,471 bytes smaller for N = 16 and 10,236,536 for N = 64, of 66,283,003 for the base. - Adding ContentBlocks to --pack: file 823,296 / 200,704 bytes smaller; .br 34,947 bytes larger for 16, 101,421 smaller for 64. - j/'s Javadoc index pages decode to up to 5,065,005 bytes, so two j/ blocks decode to more than the app's 8 MiB cache for N = 16 (largest 14,169,872) and one for N = 64 (28,055,018). The docstring said block ids follow the sets in order. That holds as written, but --pack ContentBlocks renumbers them; nothing reads the order. Comments only. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018CMZQuXT3GT7nEct2w4tg3

From documentation version 4.0.0, store_pages_in_blocks.py stores most pages in ContentBlocks, compressed against the dictionary, and empties their Content rows. The re-mint recompresses Content rows and templates, not blocks, so a re-mint of a 4.x file would leave every blocked page undecodable, and the verifier compares no blocks either. Both now refuse a database whose newest DocumentationDatabaseVersion row, read as the app reads it, declares major 4 or later: the re-mint before training and before its backup, the verifier for either database. Each says so on one "error:" line naming the blocks step and the ship-time order: re-mint, verify, store_pages_in_blocks.py, then the pack with ContentBlocks. This is the plan's "re-mint before" variant. B02, which would measure whether a dictionary trained on the blocks stores less, has not run; if it pays, this refusal becomes recompress-and-verify code instead. Tests: ContentBlocksRefusalTest (majors 4 and 5 refused before training with nothing written; an older 4 under a newer 3 re-minted and verified; the verifier refusing either side) and two command-line cases (no backup; an error line and no "Byte-identical"). Each fails against the pair broken that way: no re-mint refusal, the threshold at 5, the greatest major read instead of the newest row, the refusal after training and backup, a message without the order, the verifier checking the re-minted file only, and no verifier refusal. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

… by default - Five sets, in this order, each in path order and never mixed in a block: a/ templated JSON (not a/devsite/), j/ HTML, k/ templated JSON, then every Brotli row under a/devsite/ and under i/. Paths are matched with GLOB, case-sensitive as paths are, where LIKE is not. - The cap: an item over 1 MiB decoded stays a row. A block closes at --pages-per-block items, now 64 by default, or before the next item would take it past 4 MiB decoded, so two blocks fit the app's 8 MiB cache. Rows are decoded in the thread pool ahead of the grouping walk, a window at a time, and blocks are built a few ahead of the writes, so no set is held decoded all at once. - Before a row is emptied, its block is decoded back and each item's slice compared byte for byte with the row's decoded bytes. The sha256 pass over the stored blocks before the commit stays. - The summary shows every set with its blocked content types, the pages the cap kept (named, up to 10 a set) and the rows left for their continuation rows. - Docstring: the five sets, the cap, where the step goes, and G02's figures with them, replacing the prototype's 16/64 figures and its reference to a format file the repo does not have. On the 2026-10-20 G02 ship-time build (the prototype's input, sha256 c7de1d8a): 22,478 pages in 356 blocks, rows 34,834,228 -> blocks 24,608,732 bytes; packed with --pack Templates,Content,TooltipTexts,ContentBlocks, the .br went from 66,283,003 to 55,914,367 (10,368,636 smaller). The cap kept 18 pages (5 a/, 12 j/, 1 k/); the largest block decodes to 4,172,562 bytes. integrity_check and foreign_key_check were clean before and after the pack, and the pack's identity check passed. Tests, on the synthetic database: untemplated, non-HTML a/devsite/ and i/ rows blocked; an A/ path left out; nine blocks in set order. On a second database: an item one byte over 1 MiB kept, one exactly at it blocked; a block of exactly 4 MiB, and one closed at 3.5 MiB; 65 k/ pages making 64 + 1 through main()'s default; a codec garbling only the decode-back, and one garbling only the decode before the commit, each refusing with nothing written. Each fails against the script broken that way: three sets, LIKE, sets reordered, a/devsite/ or i/ narrowed, no item cap, the cap at instead of over, on stored size, or at 2 MiB, no block cap, at instead of past, or at 8 MiB, a default of 16 or none, no byte-for-byte check, a length-only check, and no sha256 pass. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

Blocks are documentation version 4.0.0, a new major: an app without the reader would show every blocked page empty, and the app reads blocks by the declared major, never by whether the tables exist. 4.0.0 promises everything 3.0.0 does as well (keyed tooltips, compressed templates). - The step reads the major from the newest DocumentationDatabaseVersion row (ORDER BY changeTime DESC, rowid DESC, as the app and remint_dictionary.declared_major do) and refuses no version, a major below 3, or a major of 4 or more, writing nothing. The refusal of a database that already has the tables stays, and comes first. - In the same transaction as the blocks it replaces that table's rows with one, 4.0.0, who and comment naming ADFA-6683, as migrate_tooltip_schema.py declares 3.0.0. A dry run rolls it back with everything else. Tests: no version table, no version row, 2.x, 4.x and 5.x each refused with nothing written; the newest row deciding (an older changeTime on a later rowid, ties broken by rowid, an older 4 under a newer 3); one 4.0.0 row after a run and the old rows after a dry run. Each fails against the script broken that way: no gate, 4.x let through, no version let through, the newest row by rowid, the greatest major, no declaration, a declaration appended instead of replacing, and a dry run that declares. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

…E.md python-tests.yaml lists its test files one by one, so without this line CI never runs scripts/test_store_pages_in_blocks.py. CLAUDE.md gets a "Schema 4.0.0: content blocks, in the shipped copy only" section beside the 3.0.0 one: the two tables, the rule that a blocked Content row is empty, the five sets and the cap, the ship-time order (the step after the last re-mint and its verify, before --pack Templates,Content,TooltipTexts,ContentBlocks), which tools refuse 4.x and which run before the step because they do not know blocks, and the release order. The re-mint paragraph says it refuses 4.x, and the repository tour lists the script. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

…ored-block check Review of B03, F02-F04, F08, F09: - The decode window is read when a set is decoded, not bound as a default argument, so a test can shrink it. New tests decode ten rows four at a time (the next window only once the walk reaches it), and block the synthetic database with windows of 2 and 5, so a/'s seven pages span windows and a block straddles two. - One page to a block with 1 and 2 workers, so a/'s seven blocks pass the 2 * workers built ahead of the writes: every block is written, in path order, with consecutive ids. - check_stored now requires each block as stored to decode to exactly its items' lengths added up. New tests: a decode-back one byte longer (only the decode-back length check sees it), a stored block one byte longer (only the new check sees it), a lost block whose rows were emptied all the same (only the item count sees it), and a blocked row left full (only the "still hold their content" check sees it). Each writes nothing. - --workers below 1 is refused on an error: line; -1 raised a ValueError traceback and 0 ran as 1. - A database declaring no version, or a major below 2, is no longer told to run migrate_tooltip_schema.py, which converts 2.x only and refuses them. A 2.x database still is. Mutants, each failing the tests: the first window only, the whole set in one window, the window as a default argument, the write dropped or written last-in-first-out under the write-ahead limit, no decode-back length check, no item count, no stored-block length check, no "still hold their content" check, --workers unchecked, 0 run as 1, and the migrate_tooltip_schema.py advice for no version or for 1.x. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

…cord B02 Review of B03, F10, F01 and F06 (the re-mint's part): - remint_dictionary.py and verify_remint_dictionary.py refused a blocked database only by its declared major. A 4.x file whose version row a later step or a hand edit rewrote to 3 (or dropped) would be re-minted, leaving every block undecodable, and the verifier compares no blocks to notice. Both now also refuse a database holding ContentBlocks or ContentBlockItems, whatever it declares, writing nothing. - B02 has run since 670095f said it had not: re-minting after the blocks step, on the blocks, saved at most 30,880 bytes of .br (25,285 more on the mean of 4 seeds), well inside the docstring's 0.33 MB seed spread, so "re-mint before" stands. The docstring records the measurement, and that whole blocks as zstd samples lose 0.52 to 0.74 MB because zstd trains on a sample's first 128 KiB only. - The ship-time order in the refusal message ends with --page-size 4096, as the pack is run. Tests: the re-mint refuses a 3.x database with both tables, with ContentBlockItems alone, and a database with no version and ContentBlocks, before training and with nothing written; the verifier refuses either side holding ContentBlocks; the command line refuses a 3.x database with both tables, with no backup. The order check includes --page-size 4096. Mutants, each failing the tests: no table check, ContentBlocks only, the table check only with a version declared, the verifier checking the new file only, and the order without --page-size. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

…ader; record B02 Review of B03, F01 and F05-F07, F11: - B02 has run: re-minting after the blocks step, on the blocks, saved at most 30,880 bytes of .br (the mean of 4 seeds 25,285 larger), against 0.33 MB of seed spread, so the re-mint stays before the step. Both the 4.0.0 section and the re-mint paragraph say so, and the re-mint paragraph says the pair also refuses the block tables. - The other Content tools on a 4.x file: drop_unread_rows.py does read the version (3 or more means compressed templates) and does not refuse 4. None of them sees a blocked page as empty: x'' is not a Brotli stream ("corrupt input"), so each refuses, writes what it can read and exits 1, or stops with a traceback, as listed. migrate_tooltip_schema.py (nothing to do) and the master-only writers migrate_content_to_dictionary_brotli.py, insert_optimized_media.py and sync_kdoc_json_to_db.py (which would overwrite blocked k/ rows) are added. - Every ship-time order ends with its pack and --page-size 4096: the 3.0.0 section and the migrate_tooltip_schema.py entry with --pack Templates,Content,TooltipTexts, as compress_docdb.yml runs it (a 3.0.0 file has no ContentBlocks for the 4.0.0 pack to name), plus the 4.0.0 pack with ContentBlocks; the store_pages_in_blocks.py entry with the 4.0.0 pack. - The reader is ADFA-6683's CodeOnTheGo PR to stage, not yet numbered or merged, and a file declaring major 5 would be read as 4. - python-tests.yaml: the comment no longer names the brotli CLI twice. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

hal-eisen-adfa · 2026-10-28T00:58:11Z

scripts/store_pages_in_blocks.py:129: the j/ set takes only text/html, while a/ and k/ take templated rows (templateId > 0). sync_javadoc_json_to_db.py writes j/ pages as JSON with javadoc.peb, so if those land on the ship-time copy they would stay one row each, with nothing in the summary saying why.

On the copies I have (3.0.0 documentation.db of 2026-10-22, its 6740 test copy, and 2.0.0 documentation-v2.db), every j/ row has templateId = 0: 5,274 text/html plus 19 other Brotli rows (CSS, JS, Markdown, plain text), so there's no size impact today. Is leaving out templated j/ JSON intended? If so, could the docstring say so; if not, should the condition be t.value = 'text/html' OR c.templateId > 0?

hal-eisen-adfa · 2026-10-28T01:06:23Z

DocDB Studio and 4.0.0. The documentation team downloads the database from Google Drive, edits it in DocDB Studio, and pushes it back. Once the Drive copy is upgraded to 4.0.0, Studio has to change in these places (line numbers from this branch):

Version gate: docdb-studio/docdb_studio.py:191 (SUPPORTED_DATABASE_MAJOR_VERSION = 3), checked at :5219-5221. Studio refuses to open a 4.0.0 file at all, so tooltip editing stops too, even though the tooltip tables are the same as in 3.0.0.

Once the gate allows 4, everything below still reads or writes Content.content directly and knows nothing about ContentBlocks / ContentBlockItems:

Anchor lookup for button URIs: get_html_anchors_for_path, :406-445. It reads the row and its -N chunks and decodes them. A blocked page's content is x'', so it can't list that page's anchors. It needs to read the item from its block.
Local preview server: fetch_content_for_path, :846, serving through _ContentHTTPHandler, :894. It serves Content.content, so blocked pages come back empty or fail to decode. It needs the same block read.
Content tree sizes: get_paths_with_sizes, :643-647 (LENGTH(content)). Blocked pages show 0 bytes. It could use ContentBlockItems.length for those.
Content import, delete side: _delete_content_by_ids, :1672-1680, and the orphan pass in import_content_files, :1763 (delete at :1831). Deleting a blocked row leaves its ContentBlockItems row pointing at a Content.id that no longer exists, and the page's bytes stay in the block. That fails foreign_key_check and keeps the dead bytes in the shipped file. The import should delete the ContentBlockItems row with the Content row (or the schema could declare ON DELETE CASCADE).
Content import, insert side: :1867. A re-imported page gets a new id and is served from its own row, which works. But nothing removes the old item from its block, so with (5) unfixed the block keeps stale bytes for good. If anything ever updated a blocked row in place, the app's reader would reject a blocked row that also has its own content.

Not affected: the tooltip tables, tooltip CSV import/export, and Templates (get_template_text / save_template), which are the same in 4.0.0 as in 3.0.0.

hal-eisen-adfa · 2026-10-28T01:48:56Z

Workflows that use the Drive copy, once it is 4.0.0. The documentation team downloads the database from Drive, edits it, and pushes it back. If that copy is upgraded to 4.0.0, three workflows run these tools on it. Each was run on a 4.0.0 test copy built with this branch's store_pages_in_blocks.py (line numbers from this branch):

scripts/sync_kotlin_stdlib_docs/sync_kdoc_json_to_db.py (build-kotlin-docs.yaml): corrupts the database with no error. It has no version check and updates k/ rows in place (:147: UPDATE Content SET content = ? WHERE id = ?). On the test copy it exited 0 and left 3,234 blocked rows with their own content while their ContentBlockItems rows remained, a combination the app's block reader rejects. A page that grows past CHUNK_SIZE is deleted and re-inserted with a new id (:151-160), which leaves its ContentBlockItems row orphaned. It needs at least a refusal of major 4 and above before it writes anything.
ProcessKotlinWebsiteJSON/insert_optimized_media.py (build-kotlin-docs.yaml): crashes with a traceback in rewrite_pages (:289) on the first blocked row, after it has written its backup. Content is unchanged, but the workflow fails.
populate_db.py (build-kotlin-docs.yaml) and scripts/sync_java_docs/sync_javadoc_json_to_db.py (build-java-docs.yaml): both refuse a major above 3 and write nothing (populate_db.py:610-616, :798-806; sync_javadoc_json_to_db.py:274-281). Safe, but both workflows stop on a 4.0.0 Drive copy. The sync_javadoc refusal was run; the populate_db one was only read in the code.
check-tools/ (docdb-regression-test.yaml, daily): already fails on the 3.x tooltip tables (no such column: detail). Once fixed for 3.x, its empty-content check in db_health_checker.py (:261-280) would report every blocked row as a problem unless it learns about ContentBlockItems.

hal-eisen-adfa · 2026-10-28T01:50:00Z

CLAUDE.md, "Schema 4.0.0" section: two corrections.

"migrate_content_to_dictionary_brotli.py, insert_optimized_media.py and sync_kdoc_json_to_db.py write the master only, which never gets blocks." That holds only if the master stays 3.x. The documentation team downloads the Drive copy, edits it, and pushes it back, and the PR body plans for a 4.0.0 database on Drive. Once that copy is 4.0.0, the master does get blocks, and these three run on it: sync_kdoc_json_to_db.py exits 0 leaving blocked rows with their own content (see the comment above), and migrate_content_to_dictionary_brotli.py commits what it can read, then reports an error for each of the 22,472 blocked rows on the test copy. The section should either say the Drive master stays 3.x, or stop describing these as safe.
The 4.x tool list is incomplete. These tools read or write Content (or run on the same files) and aren't mentioned. Each was run on a 4.0.0 test copy unless noted:
- ProcessDocs/AndroidDocs/AndroidDocsToJson/verify_android_json_db.py: reads the version (through templates_compressed, :86), then crashes with a traceback on the first blocked a/ page (:107). So the claim that only drop_unread_rows.py and migrate_tooltip_schema.py read the version is incomplete.
- scripts/compare_svg_renders.py: traceback in svg_rows (:76) on a blocked SVG.
- scripts/set_content_type_compression.py: traceback at decode (:120), then it rolls back.
- scripts/list_database_documents.py: lists every blocked page as "Bytes: 0" with no warning (:19).
- DocsDbScripts/layout_xml_tooltips.py: its Tooltips(summary, detail) insert (:92-97) fails before its Content insert (:104). From reading the code; not run.
- scripts/TooltipManager.py: no such column: summary (:52), nothing written.
- check-tools/: no such column: detail on the 3.x tooltip tables. Once fixed for 3.x, its empty-content check would report every blocked row.
- scripts/DocumentationDatabase.py / ingest.py: refuses the unrecognised tables (:92-95), as on 3.x.
- scripts/rebuild_database.py: works on 4.x. --pack ...,Content renumbers ContentBlockItems.contentId through the declared foreign key, and both pack lists ran clean. Worth listing as the one tool that does handle 4.x.

hal-eisen-adfa · 2026-10-28T01:51:14Z

Other tools on a 4.0.0 database. Each was run on a 4.0.0 test copy built with this branch's store_pages_in_blocks.py, from the 2026-10-22 3.0.0 database and from a 2.0.0 master put through migrate_tooltip_schema.py, unless marked "from the code". A blocked row's x'' isn't a Brotli stream, so decoding it fails with brotli failed: corrupt input.

Write part of their changes, then exit 1:

scripts/bundle_remote_images.py: stored 19 files, rewrote 10 pages, skipped 14,448 rows it couldn't read (:305), committed (:400-403), then exited 1. Rows are updated in place, so no orphaned ContentBlockItems.
scripts/optimize_db_media.py: optimized 828 files and committed (:1867), with "error: failed to optimize" for each blocked SVG (:1745), then exited 1.
ProcessKotlinWebsiteJSON/migrate_content_to_dictionary_brotli.py: commits the rows it can read in batches (:331, :356, :393); 22,472 rows error with "decodes neither plainly nor with the dictionary" (:210); exits 1.
scripts/remove_remote_loads.py, remove_skin_font_comment.py, share_skin_sprite.py, update_media_references.py: skip blocked rows as unreadable and exit 1. They had nothing else to change on the test copies, so the partial write wasn't seen; it's inferred from the code.

Stop without writing:

scripts/drop_unread_rows.py: on the 2.0.0-derived copy, "14908 problem(s) found; nothing was changed", exit 1. On the ship-time copy it printed "Nothing to do" and exited 0, so the CLAUDE.md claim that it refuses holds only when it has rows to drop.
scripts/minify_text.py: "16 stylesheet(s) could not be read ... nothing was changed", exit 1.
scripts/svg_images_to_webp.py: "54 SVG row(s) cannot be read; nothing was written", exit 1.
ProcessDocs/AndroidDocs/AndroidDocsToJson/load_android_json_db.py (:339-346), scripts/android_pages_to_templates.py (:851-854), scripts/trim_kotlin_json.py (:427-429): their version checks refuse it.
scripts/hm_pages_to_templates.py: traceback on the compressed template (:904/:942) before any write.
scripts/set_content_type_compression.py: traceback at decode (:120), rolled back.

Read-only, or crash before writing:

scripts/check_remote_loads.py: no such column: summary (:429), as on any 3.x file.
verify_android_json_db.py: traceback on the first blocked a/ page (:107).
scripts/compare_svg_renders.py: traceback in svg_rows (:76) on a blocked SVG.
scripts/TooltipManager.py: no such column: summary (:52).
scripts/DocumentationDatabase.py / ingest.py: raises ValueError for the unrecognised tables (:92-95), as on 3.x.
DocsDbScripts/layout_xml_tooltips.py: its Tooltips(summary, detail) insert (:92-97) fails before its Content insert (:104). From the code; not run, since it calls ./clean.sh.
scripts/list_database_documents.py: doesn't fail, but lists every blocked page as "Bytes: 0" (:19) with no warning.

Most of these run only on the ship-time copy, before the blocks step, so these results matter mainly if the Drive copy becomes 4.0.0 or the order slips. The partial writers are the ones to watch: they leave a half-changed database and only an exit code to show it.

davidschachterADFA · 2026-10-29T00:04:08Z

Thanks for going through this in this much detail. I checked each point against the branch head (e213338). Where I ran something it was on a small 4.0.0 test database I built with this branch's own test helpers (about 30 pages, 20 of them blocked), not on your copies, so your row counts are unverified on my side. Your line numbers match within one to three lines.

1. j/ condition. Templated j/ JSON is already included. sync_javadoc_json_to_db.py keeps contentTypeID as text/html on the rows it writes and sets templateId to the javadoc.peb id (:375, :398; the header at :7-8 says so), so those rows match t.value = 'text/html' at store_pages_in_blocks.py:129. On the test database, templated and plain j/ pages were both blocked; only the element-list passthrough (other type, templateId 0) stays a row. Your docstring point stands: :66 says "j/ HTML" and no test has a templated j/ row. I can't tell from the repo whether including them was a deliberate choice, and its size effect is unmeasured, since no copy has one. Your proposed OR templateId > 0 would only add a j/ row with a non-HTML type, which nothing writes.

2. DocDB Studio. Everything you list is there as described: the gate at :191 and :5220-5221, direct Content.content reads in get_html_anchors_for_path, fetch_content_for_path / _ContentHTTPHandler and get_paths_with_sizes (a blocked row shows 0 bytes), and plain DELETE FROM "Content" in _delete_content_by_ids and the orphan pass, with no reference to the block tables anywhere in the file. The refusal of a major above 3 is deliberate and documented; this PR does not touch Studio. I have not run Studio.

3. Workflows on a 4.0.0 Drive copy. On the test database I called write_content from sync_kdoc_json_to_db.py on blocked k/ rows. A small page left the row with its own content and its ContentBlockItems row in place, and a page over CHUNK_SIZE deleted the row and left an orphan (PRAGMA foreign_key_check failed). The script has no version check. This is the only silent corruption I found. From the code and the SQL, not run end to end: the decode at insert_optimized_media.py:289, the refusals in populate_db.py and sync_javadoc_json_to_db.py, and the health checker's empty-content query counting every blocked row. None of these files is changed by this PR.

4. CLAUDE.md. Both corrections are right. The "write the master only, which never gets blocks" sentence was added in my last commit and holds only if the master stays 3.x, while the same section says a 4.0.0 database goes to Drive. The 4.x tool list is also incomplete: set_content_type_compression.py and layout_xml_tooltips.py aren't in CLAUDE.md at all, and the others you name aren't listed as failing on 4.x. verify_android_json_db.py crashes on a blocked page (run). rebuild_database.py --pack Templates,Content,ContentBlocks ran clean on the test copy, with foreign_key_check and integrity_check passing. I did not check TooltipManager.py, DocumentationDatabase.py or check-tools myself.

5. Other tools. Run on the test copy: optimize_db_media.py commits and exits 1 after "failed to optimize" for each blocked SVG; minify_text.py and svg_images_to_webp.py write nothing and exit 1; list_database_documents.py prints "Bytes: 0" with exit 0. drop_unread_rows.py printed "Nothing to do" and exited 0 when it had nothing to drop, so the CLAUDE.md claim that it refuses is only true when it has a blocked row to read. bundle_remote_images.py stopped earlier on my test data (missing content types), so I confirmed only the code at :305 and :400-403. I did not check the other tools on your list.

What this means for the PR. Four things are in the text this PR wrote and look mine to fix here: the j/ docstring and one test row, the "master never gets blocks" sentence, the incomplete 4.x tool list, and the drop_unread_rows.py wording. The tool failures themselves are in code this PR doesn't change; the one to do first elsewhere is a version refusal in sync_kdoc_json_to_db.py. Whether the Drive master the documentation team edits stays 3.x or becomes 4.0.0 is the decision every point in 2, 3 and 5 depends on. I haven't made it, and the PR text doesn't settle it.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Vh2DAaJxvRXKRYkgiDqwGQ

…me writers run on blocks The ADFA-6806-tracked tools (sync_kdoc_json_to_db.py, insert_optimized_media.py, migrate_content_to_dictionary_brotli.py) are no longer said to write a master that never gets blocks. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Vh2DAaJxvRXKRYkgiDqwGQ

…hat each does Run on a 4.0.0 test copy or read from the code, as marked; ADFA-6806, ADFA-6807, ADFA-6808. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Vh2DAaJxvRXKRYkgiDqwGQ

…op, and refuses only on a blocked row Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Vh2DAaJxvRXKRYkgiDqwGQ

davidschachterADFA · 2026-10-29T00:18:28Z

Decision on the Drive master: it will become schema 4.0.0 and will not go back to 3.x. That settles the question in my last comment, and it changes two things:

The sentence in CLAUDE.md that these three tools "write the master only, which never gets blocks" is wrong under this decision. They run on a blocked database, and sync_kdoc_json_to_db.py can corrupt one silently. This PR now rewrites that section, completes the 4.x tool list and fixes the drop_unread_rows.py wording, plus the j/ docstring and a templated j/ test row.
Your points on Studio, the workflows and the other tools are no longer fallout to defer: they have to be fixed or guarded before the Drive copy is upgraded. I filed them as follow-ups, and the PR's release-order note now says the Drive copy is upgraded only after all three are done:
- ADFA-6806: the pipeline scripts (sync_kdoc_json_to_db.py first, then insert_optimized_media.py and the partial writers, then the crash-or-refuse tools)
- ADFA-6807: DocDB Studio opens and edits a 4.0.0 database
- ADFA-6808: check-tools (the 3.x tooltip-table failures, and blocked pages flagged as empty)

The CLAUDE.md and test changes are pushed to this branch as aa4f43f8 (four commits: the j/ docstring and a templated j/ test row, the Schema 4.0.0 paragraph rewritten for this decision, the completed 4.x tool list with each claim marked run or code, and the drop_unread_rows.py wording). The Python suites the CI runs pass: 711 passed, 1 skipped, and Studio's own 204 passed. The PR body's release-order note and "Not covered" bullet are updated to match.

hal-eisen-adfa · 2026-10-30T06:37:48Z

Review findings on the 4.0.0 Drive-master plan:

ID	Sev	Where	Finding
F01	High	`CLAUDE.md` "Schema 4.0.0"; `scripts/store_pages_in_blocks.py` `version_refusal`	`CLAUDE.md` says the Drive master becomes 4.0.0 and never goes back to 3.x, and that the blocks step still runs on the copy that ships. But `store_pages_in_blocks.py`, `remint_dictionary.py` and `verify_remint_dictionary.py` all refuse a 4.x file. Once the master is upgraded, every ship-time copy is 4.x, so the documented order (… re-mint, verify, blocks step, pack) can't run: pages added to the master afterwards are never blocked, and the dictionary is never re-minted. ADFA-6806 item 5 covers `populate_db.py` and `sync_javadoc_json_to_db.py`, but not the re-mint pair or the blocks step itself. How does the ship-time pipeline run from a 4.x master?
F02	High	`scripts/sync_kotlin_stdlib_docs/sync_kdoc_json_to_db.py` `write_content`	This script has no version check. On a 4.x file it updates a `k/` page that still fits in one row in place (`UPDATE Content SET content = ?`), leaving a blocked row holding its own content beside its `ContentBlockItems` row. A page that has grown past one row is deleted and re-inserted under a new id, which orphans its `ContentBlockItems` row. Either way it exits 0. This is tracked as ADFA-6806 item 1; that guard has to land before the Drive copy is upgraded.
F03	Med	`ProcessKotlinWebsiteJSON/remint_dictionary.py` `content_blocks_refusal`	The 4.x and block-table refusal is written separately in only two tools (the re-mint and its verify). The other `Content` readers and writers that `CLAUDE.md` lists still crash, commit part of their work, or exit 0 with a wrong answer on a blocked database. A single shared guard, for example in `optimize_db_media.py`, which most of them already import, would cover every caller for ADFA-6806 instead of adding a check to each tool one by one.

davidschachterADFA · 2026-10-30T19:42:36Z

Independent review (`code-review` skill, high) at `aa4f43f`

An independent reviewer read the diff and returned 6 findings. I then checked each against the code at this commit. None blocks the merge. I ran only brotli -d on empty input; I did not run the scripts on a database. The reviewer's line numbers for store_pages_in_blocks.py were wrong (the file has 496 lines); the findings match the code at the lines below.

ID	Where	Finding	Conclusion
F11	`store_pages_in_blocks.py:263`	A zero-length stored Brotli row would abort the run: `brotli -d -c` on empty input exits 1 with "corrupt input"	Not reachable. Brotli encoding an empty row gives one byte, which decodes to empty (checked), so a stored brotli row is never zero-length. Rows already emptied by this step are excluded by the version check
F12	`verify_remint_dictionary.py:150`	`verify()` leaves both connections open when it raises	True, and it never closes them on any path. Only matters when called from a library or test; the CLI exits
F13	`store_pages_in_blocks.py:205`	`declared_major` copies the same query as `remint_dictionary.py:197` and `migrate_tooltip_schema.py:489`	True. A cleanup
F14	`store_pages_in_blocks.py:270`	Decodes queue behind block encodes in one pool	True, but workers stay busy, so no capacity is wasted. No change
F15	`store_pages_in_blocks.py:473`	`VACUUM` after `COMMIT` is not handled if it fails	True, but the database is already committed and valid, and the docstring says "Takes no backup: run it on a copy". Only the space is not reclaimed
F16	several	The ship-time order is written out in about seven places	True: `remint_dictionary.py` (118, 186), `store_pages_in_blocks.py:106`, `CLAUDE.md` (85, 145, 383)

Posted by Claude Code for David Schachter.

davidschachterADFA · 2026-10-30T23:23:12Z

Thanks for these. An independent review at aa4f43f reached F01 on its own and ran it, so I'm answering all three findings.

ID	Response
F01	Agreed, and it goes further than the scripts. On a copy upgraded to 4.0.0, `remint_dictionary.py --seeds 4` exits 1 ("declares documentation major 4"), and `store_pages_in_blocks.py` exits 1 ("already has ContentBlockItems"), writing nothing. `verify_remint_dictionary.py` refuses through the same check (`content_blocks_refusal`, :151). CodeOnTheGo's `compress_docdb.yml` on `stage` would stop first: it refuses anything that is not 3.x (:104-110), and it packs `Templates,Content,TooltipTexts` with no blocks step (:193-196). Neither PR changes that workflow. ADFA-6871 tracks the fix. At ship time, a new step unblocks the copy back into 3.x rows, so the existing order runs unchanged; the Drive master stays 4.0.0. ADFA-6872 upgrades the re-mint, its verify and the blocks step to 4.x, so that step can go. The Drive copy is not upgraded until ADFA-6871 is done, alongside ADFA-6806, ADFA-6807 and ADFA-6808.
F02	Agreed. It is ADFA-6806 item 1, and that ticket already makes the upgrade wait on it.
F03	Agreed. One shared guard beats a check written into each tool, and ADFA-6806 now says so. Ten of the other fifteen scripts there already import `optimize_db_media.py`. The five that do not, including `sync_kdoc_json_to_db.py`, are listed in the ticket.

The same review found two smaller things, now ADFA-6870:

No test packs a database with blocks. After the pack, items stay matched to their pages only because of the two REFERENCES clauses in store_pages_in_blocks.py:163-168. A copy with both clauses removed still passed all 72 tests, but after the pack, every one of the 22,472 block-stored pages failed to read through #2191's queries.
CLAUDE.md:136 is stale. It says the reader PR is "not yet numbered or merged". It is CodeOnTheGo #2191.

Posted by Claude Code for David Schachter.

hal-eisen-adfa · 2026-10-31T00:23:36Z

+> bytes larger), against a 0.33 MB spread between re-mint seeds; `remint_dictionary.py`'s docstring
+> has the figures.
+>
+> **The Google Drive master database the documentation team edits will become 4.0.0 and will never go


This seems to contradict the tools. The copy that ships is made from the Drive master, but store_pages_in_blocks.py accepts only major 3 with no block tables, and remint_dictionary.py / verify_remint_dictionary.py refuse major >= 4 or any block tables. Once the master is 4.0.0, the documented ship-time order (re-mint, verify, blocks, pack) can't run: the dictionary can never be re-minted again, and pages added or edited on the master afterwards stay unblocked rows. It also conflicts with the script docstring ("on the copy that ships only, never the master", store_pages_in_blocks.py:101). Which is the plan: master stays 3.x and only the shipped copy is blocked, or the step learns to re-block a 4.x file?

Neither, quite: the master goes to 4.0.0, but not before ADFA-6871. The order needs a 3.x copy, so until then the master stays 3.x. ADFA-6871 adds a build step that first turns the copy's blocks back into 3.x rows. Re-mint, verify, blocks and pack then run as written, and pages added or edited on the master since get blocked too. ADFA-6872 teaches those three tools 4.x, after which that step can go. CLAUDE.md (bd39a9e) and the docstring (7a3d63b) now say this, and "never the master" is gone.

A correction to my answer above: the plan I gave doesn't work, and 583987f fixes the docs. The order starts from a 2.x copy, not a 3.x one. The Kotlin trim and the a/ pages-to-templates step refuse a file declaring 3 or later (trim_kotlin_json.py:427-432, android_pages_to_templates.py:848-854), and migrate_tooltip_schema.py converts only 2.x. The master is 2.x today, as CLAUDE.md:101 and :287 already said. Nothing converts 3.0 back to 2.x, so turning a 4.x copy's blocks back into 3.x rows (ADFA-6871's approach A, which I also gave in my reply of 10-30) would still stop at the trim.

CLAUDE.md and the docstring now say the master is 2.x and that ADFA-6871 has to settle how a 4.x master is built into a copy that ships. The release order also waits on ADFA-6871. ADFA-6871 now lists the options and leaves the choice open. An independent review of bd39a9e caught this, confirmed with dry runs on a copy of the shipped database.

hal-eisen-adfa · 2026-10-31T00:23:37Z

+> upgrade on every tool below that does not refuse 4.x runs on a blocked database. None of them may be
+> left as it is when the Drive copy is upgraded: each must be fixed or guarded first, as ADFA-6806 (the
+> docs pipeline scripts), ADFA-6807 (DocDB Studio) and ADFA-6808 (check-tools) track.
+>


The list below has about 20 tools that crash, write part of their work, or silently give wrong answers on a 4.0.0 file (sync_kdoc_json_to_db.py corrupts it), and several of the ones that "refuse" do so only because x'' doesn't decode, not because they check. Since most of them already import optimize_db_media, one shared "refuse major >= 4" helper there, called at startup, would guard them all at once instead of tool by tool, and would make ADFA-6806 a lot smaller. Worth doing in this PR, given the master is meant to go to 4.0.0?

I'd keep it in ADFA-6806 rather than this PR. The Drive copy isn't upgraded until ADFA-6806, 6807, 6808 and 6871 are done, so no tool here sees a 4.x file before the guard exists. The guard also changes about 20 tools that this PR otherwise leaves alone. ADFA-6806 already plans one shared check in optimize_db_media.py, credited to you, and it can call the declared_major that 7a3d63b put there.

…pair is refused verify() raised its 4.x refusal with both read-only connections still open, so each refused pair left two handles for the collector (and, on Windows, two files that cannot be deleted). Both now close however it returns. A new test fails without the change. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Vh2DAYjFcUx64N8PMRknzL

…as built, and leave an empty row as it is - The check before the commit counted ContentBlockItems rows but sliced only the blocks it found, so a block lost with its items left behind passed, and those pages would ship empty. It now requires ContentBlocks to hold exactly the blocks written (by sha256) and ContentBlockItems exactly the items written (block, offset, length). That also ends the second q11 decode of every block, and blocks are read from the cursor rather than all at once: build_block has already decoded each block back and matched every item. - An empty Brotli row stopped the whole run ("corrupt input"). It is now left as a row and counted in the summary. - The codec is optimize_db_media.BrotliCodec, which now passes -q 11 -w 24 explicitly (the CLI's defaults for stdin: the same bytes). declared_major and table_exists move to optimize_db_media, and declare_version is migrate_tooltip_schema's, now taking the version. - The docstring says the Drive master stays 3.x until ADFA-6871, as CLAUDE.md now does. Tests: a lost block with its items kept, a changed block, a moved item, an empty row, and each block decoded once; the first four fail without the change. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Vh2DAYjFcUx64N8PMRknzL

…e block reader is CodeOnTheGo #2191 The ship-time order needs a 3.x copy, and the re-mint, its verify and the blocks step each refuse 4.x, so the master is upgraded only once ADFA-6871 turns a 4.x copy's blocks back into rows at build time. ADFA-6872 later teaches those tools 4.x. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Vh2DAYjFcUx64N8PMRknzL

…annot start from a 4.x one bd39a9e said the master is 3.x and that once ADFA-6871 turns a 4.x copy's blocks back into 3.x rows the order runs as written. Both are wrong: the Kotlin trim and the a/ pages-to-templates step refuse 3 or later, migrate_tooltip_schema.py converts only 2.x, and nothing converts 3.0 back to 2.x. So the order starts from a 2.x copy, as the master is today, and ADFA-6871 has to settle how a 4.x master is built into a copy that ships. The release order now waits on ADFA-6871 too. The test docstring for the check before the commit no longer says every damage leaves the blocks decoding. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Vh2DAYjFcUx64N8PMRknzL

davidschachterADFA and others added 9 commits October 26, 2026 19:01

davidschachterADFA mentioned this pull request Oct 27, 2026

ADFA-6683: Read documentation pages stored in compressed blocks (schema 4.0.0) appdevforall/CodeOnTheGo#2191

Open

davidschachterADFA and others added 4 commits October 28, 2026 17:12

ADFA-6683 | Say j/ takes templated javadoc JSON rows too; test one

9313890

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Vh2DAaJxvRXKRYkgiDqwGQ

ADFA-6683 | CLAUDE.md: drop_unread_rows.py exits 0 with nothing to dr…

aa4f43f

…op, and refuses only on a blocked row Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Vh2DAaJxvRXKRYkgiDqwGQ

davidschachterADFA requested a review from hal-eisen-adfa October 29, 2026 04:59

hal-eisen-adfa reviewed Oct 31, 2026

View reviewed changes

davidschachterADFA and others added 4 commits October 30, 2026 22:40

Conversation

davidschachterADFA commented Oct 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Why

Change

Measured

Tests

Not covered

Uh oh!

hal-eisen-adfa commented Oct 28, 2026

Uh oh!

hal-eisen-adfa commented Oct 28, 2026

Uh oh!

hal-eisen-adfa commented Oct 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

hal-eisen-adfa commented Oct 28, 2026

Uh oh!

hal-eisen-adfa commented Oct 28, 2026

Uh oh!

davidschachterADFA commented Oct 29, 2026

Uh oh!

davidschachterADFA commented Oct 29, 2026

Uh oh!

hal-eisen-adfa commented Oct 30, 2026

Uh oh!

davidschachterADFA commented Oct 30, 2026

Independent review (code-review skill, high) at aa4f43f

Uh oh!

davidschachterADFA commented Oct 30, 2026

Uh oh!

Uh oh!

hal-eisen-adfa Oct 31, 2026

Choose a reason for hiding this comment

Uh oh!

davidschachterADFA Oct 31, 2026

Choose a reason for hiding this comment

Uh oh!

davidschachterADFA Oct 31, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

hal-eisen-adfa Oct 31, 2026

Choose a reason for hiding this comment

Uh oh!

davidschachterADFA Oct 31, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

davidschachterADFA commented Oct 27, 2026 •

edited

Loading

hal-eisen-adfa commented Oct 28, 2026 •

edited

Loading

Independent review (`code-review` skill, high) at `aa4f43f`