Skip to content

Decode mode 2 (LZ) payloads - #17

Merged
mahach666 merged 7 commits into
mainfrom
feat/lz-chunks
Nov 18, 2026
Merged

mahach666 merged 7 commits into
mainfrom
feat/lz-chunks

Conversation

@mahach666

@mahach666 mahach666 commented Nov 16, 2026 •

Copy link
Copy Markdown
Contributor

Closes #7 — continues from the reverted #14 (chunk-chain model was an artifact, not the format).

What changed

The chunk-chain model was wrong. What looked like internal chunk headers in a mode 2 payload is the encoder writing its own sizes as the first literals of a single token stream; the "chunk boundaries" were artifacts of a self-synchronizing walk. ReadChunks, PlaLzChunk and LzChunkWalker are removed.

The real mode 2 format is decoded (LzPayloadDecoder, internal). The payload body after the block header is a chain of sequences producing exactly the declared unpacked size:

  • token byte: high nibble = literal count (15 = extended by 255-chained bytes), literals follow;
  • then a 16-bit big-endian distance and a length byte (0 → 256; otherwise the token's low nibble is the high byte of the match length, up to 4095);
  • distance 0 emits the match length as zero bytes (RLE); a positive distance copies from that many bytes back in the window byte by byte, so overlapping copies replicate a repeating pattern;
  • the body tail after the output is complete is writer slack.

Global window. A mode 2 stream may reference the output of any preceding block of the file, so ReadPayload for a mode 2 block decodes the file from the start into one global window (raw payloads, full zero-mask decodes, mode-2 outputs, in block order) and returns the requested block's output. The cost of the call grows with the size of the file. Preceding blocks that do not decode (false entries of the scan) are skipped and rolled back out of the window.

Validation

  • The format was established on controlled saves (see repo issue Decode mode 2 (LZ) payloads #7 research notes) and cross-checked against the object directories: the decoded global stream's directory records abut with no overlaps, confirming the window matches the real stream.
  • End-to-end run of the shipped code on the six controlled saves: every mode 2 block decodes exactly to its declared size — 2432 of 2432 blocks (m2-a: 401, m2-a2/b/c: 405 each, m2-d/e: 408 each).
  • Synthetic tests (LzPayloadTests): literals, back-references across the raw/mode-2 boundary, overlapping copies, RLE zeros, the length-byte conventions (0 → 256, low nibble as high byte), trailing literals + slack, and the two rejections (back-reference beyond the window, truncated stream).
  • The sample-gated test now asserts every mode 2 block decodes and only false entries are skipped (replaces the old NotSupportedException expectation).

Notes for review

  • PlaDocument.ReadPayload mode-2 branch and the new DecodeGlobalWindow/AppendToWindow/TryAppendToWindow internals in PlaDocument.cs are the core of the change.
  • Known remainder (not in this PR): one ~22 MB RLE-heavy block kind 0x00E4 on a large sample diverges mid-decode under the current literal-extension rule; every other block of that file decodes exactly. Documented in the format spec, tracked for follow-up.
  • docs/public-api.md and CHANGELOG updated; PlaDocument.ReadPayload XML docs now describe all three modes.

ReadChunks walks the chunk chain of an LZ block payload without decompressing it: each chunk reports its offset, header length, body size, output size and compression flag, and the chain is verified against the block's declared size (in real files the chunk output sizes add up to the block's unpacked size; a chain that does not add up is a scanner false positive and throws InvalidDataException). The compressed runs themselves are not decoded yet.
The chunk-chain model was an artifact of the walk: what looked like internal chunk headers is the encoder writing its own sizes as the first literals of one token stream. ReadPayload now decodes the real mode 2 format: a token byte (high nibble = literal count, 15 extended by 255-chained bytes), the literals, a big-endian 16-bit distance and a length byte (0 means 256, otherwise the token's low nibble is the high byte of the length). Distance 0 emits zeros; a positive distance copies from that many bytes back in the global window, byte by byte, so overlapping copies replicate a repeating pattern. The stream produces exactly the block's declared unpacked size; the body tail is writer slack.

A mode 2 token stream may reference the output of any preceding block, so ReadPayload decodes the file from the start into one global window (raw payloads, full zero-mask decodes, mode-2 outputs, in block order) and returns the requested block's output; the cost of the call grows with the size of the file. Blocks that do not decode (false entries of the scan) are skipped and rolled back out of the window; the cross-check against the object directories confirmed the resulting window matches the real stream.

ReadChunks, PlaLzChunk and LzChunkWalker are removed. On the six controlled saves every mode 2 block (2432 total) decodes to its declared size; the sample test now asserts that instead of the old NotSupportedException.
@mahach666 mahach666 changed the title Expose the mode 2 chunk chain Decode mode 2 (LZ) payloads Nov 16, 2026
…buffer blocks

Three refinements established on the big sample corpus (85,429 mode 2 blocks of five files decode with zero failures; the six controlled saves still decode exactly):

- A payload may start with a 9-byte stream header [u8][u32][u32]: present when the u32 at offset 5 equals the block's packed or unpacked size, the token stream starts after it. Without skipping it the parse "self-synchronizes" through the header and occasionally converges on the wrong reading - the old chunk-chain artifact came from exactly this.
- A final match may omit the length byte: the body ends after the distance and the match length is the remainder of the output.
- Some blocks declare an output buffer larger than the stream produces (sizes near whole mebibytes; the stream just ends with the body). The produced bytes fill the head of the buffer, the tail stays zero, and the global window holds the buffer space so directory positions keep addressing it. Detected by retrying the decode with the relaxed ending when the strict decode runs out of body.

A stream that ends with the body is therefore a valid buffer block, not truncation; a back-reference beyond the decoded stream remains the one hard error.
…ssor

Two fixes to the mode 2 reading path:

- The 9-byte stream header was recognised by a single coincidence: the u32 at
  offset 5 matching the block's declared size (or the body length). Four
  literal bytes inside the token stream that happen to equal that size made
  the decoder skip nine real bytes as a header and lose the stream. The
  header is now [u8][u32 body length][u32 unpacked size], and it is only
  recognised when both u32 fields agree with the payload: one equals the
  payload length and the other the declared size (either order). The sample
  files confirm the body length field on every header they carry (1201 of
  1201 blocks).
- A preceding zero-mask block was read into memory whole before its decode,
  while a candidate that does not decode fails on the first chunk. The
  zero-mask payload is now streamed from the file in chunks and decoded as
  the bytes arrive (PayloadDecoder.DecodeZeroMask over the stream), and the
  reading never crosses the payload's packed bounds.

The README, the limitations list and the getting-started guide no longer
suggest that mode 2 payloads are only recognised: all three storage modes
decode, and the NotSupportedException mentions are gone.
@mahach666

Copy link
Copy Markdown
Contributor Author

All review points are fixed in 5c84652:

  • The stream header is now recognized structurally: it is [u8][u32 body length][u32 unpacked size], and both u32 fields must agree with the payload itself — one equals the payload length, the other the block's declared unpacked size (either order). A literal run whose four bytes happen to equal the declared size no longer makes the decoder skip nine real bytes. The sample corpus confirms the body-length field on every header it carries (1201 of 1201 blocks).
  • The zero-mask predecessor is no longer read into memory whole: the payload is streamed from the file in chunks and decoded as the bytes arrive (PayloadDecoder.DecodeZeroMask over the stream), and the reading never crosses the payload's packed bounds — a candidate that fails does so after the first chunk.
  • The README feature list, the narrative, the limitations list and the getting-started guide no longer suggest that mode 2 payloads are only recognized: all three storage modes decode, the synthetic StreamHeader test now builds the header per the confirmed structure.

@mahach666
mahach666 merged commit b3d2ada into main Nov 18, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Decode mode 2 (LZ) payloads

1 participant