Skip to content

Read the object records inside the compressed pools - #20

Closed
mahach666 wants to merge 5 commits into
mainfrom
feat/subgroups
Closed

mahach666 wants to merge 5 commits into
mainfrom
feat/subgroups

Conversation

@mahach666

@mahach666 mahach666 commented Nov 18, 2026 •

Copy link
Copy Markdown
Contributor

What

A compressed pool holds BL-tagged subgroups, one per object: tag, id, kind (the object's class id), version, flags,
an optional GUID, the packed size (a u16 for size classes 0..6, a u32 for 7..14), the unpacked size for the
non-raw modes and the payload. A mode 0 payload is raw; a mode 1 payload is a zero mask like the file blocks'. Mode 2
(LZ) subgroups have not been met inside the pools examined so far and are reported with an empty payload.

Clean-room note

The subgroup layout is an independent implementation written against this repository's format notes and confirmed
against the private sample files by the env-gated tests; no vendor-derived code is included.

API

  • CompressedPool.ReadSubgroups() — decodes the pool, scans it for the BL tag and returns every subgroup whose header
    parses. The scan never stops early: a candidate whose header cannot be read (a short false BL header, a cut GUID or
    size field) is retried one byte on.
  • PoolSubgroup — Offset (pool-relative), Kind (class id), Version, Flags, Mode, ObjectId (GUID when flag
    0x20 is set), Data (decoded payload; empty for mode 2), and Partial.

Padding vs truncation

A zero-mask stream that ends exactly at a group boundary before the declared unpacked size is full leaves the tail
of the buffer zero — valid padding, Partial = false. A stream that ends inside a group — a mask byte whose set
bit takes a literal that is not there — is a truncated literal: the record is damaged, Data keeps the bytes the
stream produced, the tail is reconstructed with zeros, and Partial = true says the tail must not be trusted. The
file blocks' mode 1 keeps its stricter rule (the stream must fill the buffer exactly).

Fixes in this update

  • The scan continues after a rejected header instead of ending at the pool's end.
  • PayloadDecoder.DecodeZeroMask now reports how it ended (Complete / StreamEnd / MidGroup); the file blocks'
    overload keeps its exact-fill rule, and the subgroup parser distinguishes padding from truncation.
  • A mode 2 subgroup no longer allocates a buffer from its declared unpacked size.
  • No private sample names or contents in the tests: the content test only requires the corpus to hold one
    length-prefixed utf16 string inside a decoded subgroup payload; the synthetic text is a generic one.

Content observed in real payloads

On the text-annotation sample the subgroups of kind 0x009B carry the text elements' data: strings as a u32 length
followed by UTF-16 little-endian data, the font name twice, and double/integer parameters in little-endian.

Tests

  • Synthetic: raw subgroup with kind/GUID/data; zero-mask subgroup decoded; tail-zero buffer (Partial = false); a
    truncated literal (Partial = true, produced bytes kept, zero tail); a length-prefixed utf16 string read back; two
    subgroups in order; a mode 2 subgroup reported without data; junk starting with the BL tag skipped; ReadSubgroups
    after Dispose throws.
  • Env-gated corpus: every pool parses to at least one subgroup; the corpus holds at least one length-prefixed utf16
    string inside a decoded subgroup payload.

The file holds the object database's compressed pools in the gaps between
blocks, outside the block index. Each pool starts with a 16-byte compression
sign, then the declared decompressed total and the block count (both u64),
then that many blocks, each a u32 compressed size and a block: a flag byte
(bit 0: LZ77-triple coded or memcpy, bit 1: 32-bit or byte sizes), the packed
size, the unpacked size and the payload.

FindCompressedPools finds every pool whose header and block headers parse and
whose declared sizes tile the total inside the file; CompressedPool.Decode
runs the blocks (memcpy or the LZ77-triple codec, ported from the decompressor
core of ObjectDatabase.dll) and returns the bytes. The layout and codec were
recovered from ObjectDatabase.dll: the reader after the compression sign, the
block decompressor and its core.

Tests cover the synthetic shapes (one big block, small and big blocks
together, a zero-block pool, truncated and size-mismatched pools skipped, a
non-decoding LZ block reported by Decode, a sign across the scan window
boundary) and the sample corpus (every pool of the smaller samples decodes).
…otes

Three changes to the pool finder and one to its presentation:

- The codec is compressing, so a pool whose declared total is bigger than the
  container itself is valid: the finder no longer rejects a pool for its
  declared total being past the container, keeping only the array limit and
  the per-block bounds.
- A compression sign that a raw block's payload happens to carry is data, not
  a pool start. FindCompressedPools now excludes the byte ranges of the
  indexed blocks whole, so a sign inside them is never returned as a pool,
  while a pool in the gap after a block still is.
- The class and member documentation now points at this repository's format
  notes and the sample files instead of naming where the layout was studied;
  the same applies to the readme of the change.

Tests cover a pool whose declared total is past the container (accepted on
the shape), a sign inside an indexed block payload (not returned), and a pool
in the gap after a block (returned).
A compressed pool holds BL-tagged subgroups, one per object: tag, id, kind
(the object's class id), version, flags, an optional GUID, the packed size (a
u16 for size classes 0..6, a u32 for 7..14), the unpacked size for the
non-raw modes and the payload. A mode 0 payload is stored raw; a mode 1
payload is a zero mask like the file blocks', and its stream may end before
the declared unpacked size is full - the tail of the buffer stays zero. Mode
2 (LZ) subgroups have not been met inside the pools examined so far and are
reported with an empty payload.

CompressedPool.ReadSubgroups decodes the pool, scans it for the BL tag and
returns every subgroup whose header parses (a rejected candidate is retried
one byte on). PoolSubgroup exposes Offset, Kind, Version, Flags, Mode,
ObjectId and Data. Observed payloads store text as a u32 length followed by
UTF-16 little-endian data and numbers as little-endian doubles and integers;
on the text-annotation sample the text objects carry the strings TEST and
РУТЕКСТ.
Four changes to the subgroup parser and its payload decoding:

- A candidate whose header runs past the pool (a short false BL header at the
  end of the pool, a GUID or size field cut short) used to end the whole scan;
  the scan now retries one byte on, so records that follow are still found.
- A zero-mask stream that ends inside a group - a mask byte whose set bit
  takes a literal that is not there - is a truncated literal, not padding.
  PayloadDecoder.DecodeZeroMask now reports how it ended (buffer full, stream
  ended at a group boundary, stream ended inside a group), and the subgroup
  parser marks such a record with the new PoolSubgroup.Partial flag: the Data
  keeps the bytes the stream produced, the tail is reconstructed with zeros,
  and Partial says the tail must not be trusted. A stream that ends exactly at
  a group boundary stays the valid tail-zero padding it was before. The file
  blocks' mode 1 keeps its stricter rule: the stream must fill the buffer
  exactly.
- A mode 2 subgroup no longer allocates a buffer from its declared unpacked
  size before returning an empty payload.
- The private-sample test is replaced by a name- and content-free one: the
  corpus is only required to hold one length-prefixed utf16 string inside a
  decoded subgroup payload, and the synthetic text is 'Sample text'.
@mahach666

Copy link
Copy Markdown
Contributor Author

All review points are fixed in cc29e44 (rebased on the updated feat/odb-pools):

  • The scan never ends early: a candidate whose header runs past the pool (short false BL header, cut GUID or size field) is retried one byte on, so records that follow are still found.
  • Padding vs truncation is distinguished: PayloadDecoder.DecodeZeroMask reports how it ended (Complete / StreamEnd / MidGroup). A stream that ends exactly at a group boundary is valid tail-zero padding; a stream that ends inside a group is a truncated literal — the subgroup is marked with the new PoolSubgroup.Partial flag, the Data keeps the bytes the stream produced and the zero tail must not be trusted.
  • A mode 2 subgroup no longer allocates a buffer from its declared unpacked size.
  • No private sample names or contents in the tests: the content test only requires the corpus to hold one length-prefixed utf16 string inside a decoded subgroup payload; the synthetic text is a generic 'Sample text'.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant