Repository navigation
Conversation
The file holds the object database's compressed pools in the gaps between blocks, outside the block index. Each pool starts with a 16-byte compression sign, then the declared decompressed total and the block count (both u64), then that many blocks, each a u32 compressed size and a block: a flag byte (bit 0: LZ77-triple coded or memcpy, bit 1: 32-bit or byte sizes), the packed size, the unpacked size and the payload. FindCompressedPools finds every pool whose header and block headers parse and whose declared sizes tile the total inside the file; CompressedPool.Decode runs the blocks (memcpy or the LZ77-triple codec, ported from the decompressor core of ObjectDatabase.dll) and returns the bytes. The layout and codec were recovered from ObjectDatabase.dll: the reader after the compression sign, the block decompressor and its core. Tests cover the synthetic shapes (one big block, small and big blocks together, a zero-block pool, truncated and size-mismatched pools skipped, a non-decoding LZ block reported by Decode, a sign across the scan window boundary) and the sample corpus (every pool of the smaller samples decodes).
…otes Three changes to the pool finder and one to its presentation: - The codec is compressing, so a pool whose declared total is bigger than the container itself is valid: the finder no longer rejects a pool for its declared total being past the container, keeping only the array limit and the per-block bounds. - A compression sign that a raw block's payload happens to carry is data, not a pool start. FindCompressedPools now excludes the byte ranges of the indexed blocks whole, so a sign inside them is never returned as a pool, while a pool in the gap after a block still is. - The class and member documentation now points at this repository's format notes and the sample files instead of naming where the layout was studied; the same applies to the readme of the change. Tests cover a pool whose declared total is past the container (accepted on the shape), a sign inside an indexed block payload (not returned), and a pool in the gap after a block (returned).
A compressed pool holds BL-tagged subgroups, one per object: tag, id, kind (the object's class id), version, flags, an optional GUID, the packed size (a u16 for size classes 0..6, a u32 for 7..14), the unpacked size for the non-raw modes and the payload. A mode 0 payload is stored raw; a mode 1 payload is a zero mask like the file blocks', and its stream may end before the declared unpacked size is full - the tail of the buffer stays zero. Mode 2 (LZ) subgroups have not been met inside the pools examined so far and are reported with an empty payload. CompressedPool.ReadSubgroups decodes the pool, scans it for the BL tag and returns every subgroup whose header parses (a rejected candidate is retried one byte on). PoolSubgroup exposes Offset, Kind, Version, Flags, Mode, ObjectId and Data. Observed payloads store text as a u32 length followed by UTF-16 little-endian data and numbers as little-endian doubles and integers; on the text-annotation sample the text objects carry the strings TEST and РУТЕКСТ.
Four changes to the subgroup parser and its payload decoding: - A candidate whose header runs past the pool (a short false BL header at the end of the pool, a GUID or size field cut short) used to end the whole scan; the scan now retries one byte on, so records that follow are still found. - A zero-mask stream that ends inside a group - a mask byte whose set bit takes a literal that is not there - is a truncated literal, not padding. PayloadDecoder.DecodeZeroMask now reports how it ended (buffer full, stream ended at a group boundary, stream ended inside a group), and the subgroup parser marks such a record with the new PoolSubgroup.Partial flag: the Data keeps the bytes the stream produced, the tail is reconstructed with zeros, and Partial says the tail must not be trusted. A stream that ends exactly at a group boundary stays the valid tail-zero padding it was before. The file blocks' mode 1 keeps its stricter rule: the stream must fill the buffer exactly. - A mode 2 subgroup no longer allocates a buffer from its declared unpacked size before returning an empty payload. - The private-sample test is replaced by a name- and content-free one: the corpus is only required to hold one length-prefixed utf16 string inside a decoded subgroup payload, and the synthetic text is 'Sample text'.
mahach666
force-pushed
the
feat/subgroups
branch
from
November 18, 2026 18:15
d332c12 to
cc29e44
Compare
Contributor
Author
|
All review points are fixed in cc29e44 (rebased on the updated feat/odb-pools):
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
A compressed pool holds BL-tagged subgroups, one per object: tag, id, kind (the object's class id), version, flags,
an optional GUID, the packed size (a u16 for size classes 0..6, a u32 for 7..14), the unpacked size for the
non-raw modes and the payload. A mode 0 payload is raw; a mode 1 payload is a zero mask like the file blocks'. Mode 2
(LZ) subgroups have not been met inside the pools examined so far and are reported with an empty payload.
Clean-room note
The subgroup layout is an independent implementation written against this repository's format notes and confirmed
against the private sample files by the env-gated tests; no vendor-derived code is included.
API
CompressedPool.ReadSubgroups()— decodes the pool, scans it for the BL tag and returns every subgroup whose headerparses. The scan never stops early: a candidate whose header cannot be read (a short false BL header, a cut GUID or
size field) is retried one byte on.
PoolSubgroup—Offset(pool-relative),Kind(class id),Version,Flags,Mode,ObjectId(GUID when flag0x20 is set),
Data(decoded payload; empty for mode 2), andPartial.Padding vs truncation
A zero-mask stream that ends exactly at a group boundary before the declared unpacked size is full leaves the tail
of the buffer zero — valid padding,
Partial = false. A stream that ends inside a group — a mask byte whose setbit takes a literal that is not there — is a truncated literal: the record is damaged,
Datakeeps the bytes thestream produced, the tail is reconstructed with zeros, and
Partial = truesays the tail must not be trusted. Thefile blocks' mode 1 keeps its stricter rule (the stream must fill the buffer exactly).
Fixes in this update
PayloadDecoder.DecodeZeroMasknow reports how it ended (Complete/StreamEnd/MidGroup); the file blocks'overload keeps its exact-fill rule, and the subgroup parser distinguishes padding from truncation.
length-prefixed utf16 string inside a decoded subgroup payload; the synthetic text is a generic one.
Content observed in real payloads
On the text-annotation sample the subgroups of kind
0x009Bcarry the text elements' data: strings as a u32 lengthfollowed by UTF-16 little-endian data, the font name twice, and double/integer parameters in little-endian.
Tests
Partial = false); atruncated literal (
Partial = true, produced bytes kept, zero tail); a length-prefixed utf16 string read back; twosubgroups in order; a mode 2 subgroup reported without data; junk starting with the BL tag skipped;
ReadSubgroupsafter
Disposethrows.string inside a decoded subgroup payload.