Repository navigation
Decode mode 2 (LZ) payloads - #17
Merged
Merged
Conversation
ReadChunks walks the chunk chain of an LZ block payload without decompressing it: each chunk reports its offset, header length, body size, output size and compression flag, and the chain is verified against the block's declared size (in real files the chunk output sizes add up to the block's unpacked size; a chain that does not add up is a scanner false positive and throws InvalidDataException). The compressed runs themselves are not decoded yet.
The chunk-chain model was an artifact of the walk: what looked like internal chunk headers is the encoder writing its own sizes as the first literals of one token stream. ReadPayload now decodes the real mode 2 format: a token byte (high nibble = literal count, 15 extended by 255-chained bytes), the literals, a big-endian 16-bit distance and a length byte (0 means 256, otherwise the token's low nibble is the high byte of the length). Distance 0 emits zeros; a positive distance copies from that many bytes back in the global window, byte by byte, so overlapping copies replicate a repeating pattern. The stream produces exactly the block's declared unpacked size; the body tail is writer slack. A mode 2 token stream may reference the output of any preceding block, so ReadPayload decodes the file from the start into one global window (raw payloads, full zero-mask decodes, mode-2 outputs, in block order) and returns the requested block's output; the cost of the call grows with the size of the file. Blocks that do not decode (false entries of the scan) are skipped and rolled back out of the window; the cross-check against the object directories confirmed the resulting window matches the real stream. ReadChunks, PlaLzChunk and LzChunkWalker are removed. On the six controlled saves every mode 2 block (2432 total) decodes to its declared size; the sample test now asserts that instead of the old NotSupportedException.
…buffer blocks Three refinements established on the big sample corpus (85,429 mode 2 blocks of five files decode with zero failures; the six controlled saves still decode exactly): - A payload may start with a 9-byte stream header [u8][u32][u32]: present when the u32 at offset 5 equals the block's packed or unpacked size, the token stream starts after it. Without skipping it the parse "self-synchronizes" through the header and occasionally converges on the wrong reading - the old chunk-chain artifact came from exactly this. - A final match may omit the length byte: the body ends after the distance and the match length is the remainder of the output. - Some blocks declare an output buffer larger than the stream produces (sizes near whole mebibytes; the stream just ends with the body). The produced bytes fill the head of the buffer, the tail stays zero, and the global window holds the buffer space so directory positions keep addressing it. Detected by retrying the decode with the relaxed ending when the strict decode runs out of body. A stream that ends with the body is therefore a valid buffer block, not truncation; a back-reference beyond the decoded stream remains the one hard error.
…ssor Two fixes to the mode 2 reading path: - The 9-byte stream header was recognised by a single coincidence: the u32 at offset 5 matching the block's declared size (or the body length). Four literal bytes inside the token stream that happen to equal that size made the decoder skip nine real bytes as a header and lose the stream. The header is now [u8][u32 body length][u32 unpacked size], and it is only recognised when both u32 fields agree with the payload: one equals the payload length and the other the declared size (either order). The sample files confirm the body length field on every header they carry (1201 of 1201 blocks). - A preceding zero-mask block was read into memory whole before its decode, while a candidate that does not decode fails on the first chunk. The zero-mask payload is now streamed from the file in chunks and decoded as the bytes arrive (PayloadDecoder.DecodeZeroMask over the stream), and the reading never crosses the payload's packed bounds. The README, the limitations list and the getting-started guide no longer suggest that mode 2 payloads are only recognised: all three storage modes decode, and the NotSupportedException mentions are gone.
Contributor
Author
|
All review points are fixed in 5c84652:
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #7 — continues from the reverted #14 (chunk-chain model was an artifact, not the format).
What changed
The chunk-chain model was wrong. What looked like internal chunk headers in a mode 2 payload is the encoder writing its own sizes as the first literals of a single token stream; the "chunk boundaries" were artifacts of a self-synchronizing walk.
ReadChunks,PlaLzChunkandLzChunkWalkerare removed.The real mode 2 format is decoded (
LzPayloadDecoder, internal). The payload body after the block header is a chain of sequences producing exactly the declared unpacked size:Global window. A mode 2 stream may reference the output of any preceding block of the file, so
ReadPayloadfor a mode 2 block decodes the file from the start into one global window (raw payloads, full zero-mask decodes, mode-2 outputs, in block order) and returns the requested block's output. The cost of the call grows with the size of the file. Preceding blocks that do not decode (false entries of the scan) are skipped and rolled back out of the window.Validation
LzPayloadTests): literals, back-references across the raw/mode-2 boundary, overlapping copies, RLE zeros, the length-byte conventions (0 → 256, low nibble as high byte), trailing literals + slack, and the two rejections (back-reference beyond the window, truncated stream).NotSupportedExceptionexpectation).Notes for review
PlaDocument.ReadPayloadmode-2 branch and the newDecodeGlobalWindow/AppendToWindow/TryAppendToWindowinternals inPlaDocument.csare the core of the change.PlaDocument.ReadPayloadXML docs now describe all three modes.