Skip to content

Expose the decoded global stream and extent reads - #18

Merged
mahach666 merged 5 commits into
mainfrom
feat/object-extents
Nov 18, 2026
Merged

mahach666 merged 5 commits into
mainfrom
feat/object-extents

Conversation

@mahach666

Copy link
Copy Markdown
Contributor

Stacked on #17 (uses the mode-2 decode; review and merge that first). Continues #7.

What this adds

PlaDocument.OpenStream() → PlaStream — the decoded global stream: every block's payload decoded in block order (raw as they are, zero masks as their full decode, mode 2 as their token-stream output). False entries of the scan are skipped. The stream is a snapshot: it holds the whole decoded file in memory and the document may be disposed afterwards.

PlaStream.ReadExtent(...) — the bytes of one extent:

  • by ObjectDirectoryRecord — an object's data. InvalidDataException when the record reaches past the stream;
  • by position, count — arbitrary range; ArgumentOutOfRangeException on bad bounds.

The position/count semantics of ObjectDirectoryRecord are now documented (previously "units not documented"): position and count name a byte extent of the decoded global stream — the object's data.

Evidence

  • Cross-check against the object directories of real files: with the stream built exactly this way (skip false entries, window = raw + mode-1 decodes + mode-2 outputs in block order), directory records abut with no overlaps and minPos = 0 — the addressing matches the real stream.
  • On the six controlled saves, every directory record of every found directory fits the stream (no out-of-bounds records), and spot-check reads return the declared counts. Sample-gated test EveryDirectoryRecordFitsTheGlobalStream asserts this.
  • Recon on the decoded stream: extents contain recognizable object data — UTF-16LE names (e.g. "Flow Exchanger"), GUIDs, counters — which is what the element work will build on.

Notes for review

  • src/PlaIO/PlaStream.cs is the whole new surface; OpenStream on PlaDocument wraps the internal window decode.
  • Memory: the snapshot holds the whole decoded stream (documented in public-api.md under "Resources and threads"). This is the price of random access into a stream that mode-2 blocks reference from anywhere.
  • docs/public-api.md and CHANGELOG updated.

OpenStream decodes the whole file once into a PlaStream snapshot: every block's payload decoded in block order (raw as they are, zero masks as their full decode, mode 2 as their token-stream output), with false entries of the scan skipped. A directory record's position and count name a byte extent of this stream — the object's data — so PlaStream.ReadExtent returns an object's bytes by record or by position and count. The position/count semantics are confirmed against the object directories of real files and documented on the record.
Two fixes to the decoded stream:

- Blocks of the scan that fail to decode are skipped and contribute no bytes;
  every stream offset at or past the first skipped block is shifted by the
  missing bytes. The stream now tracks that boundary and exposes it as
  PlaStream.UnreliableFrom, and both extent reads report it: a read whose
  position is at or past the boundary, or that reaches into it, throws
  InvalidDataException instead of returning bytes of other objects. A
  zero-length extent at the boundary is shifted too. The record fits test of
  the sample corpus applies the exact-fit checks only to the records before
  the boundary and asserts the report for the records past it.
- A record position wider than long (ulong.MaxValue) was cast before the
  bounds check and came out as an ArgumentOutOfRangeException from the extent
  read; the record read now compares against the stream length as unsigned
  and reports the documented InvalidDataException.
@mahach666

Copy link
Copy Markdown
Contributor Author

All review points are fixed in c2ec387 (rebased on the updated feat/lz-chunks):

  • The stream tracks where the offsets stop being exact: blocks of the scan that fail to decode contribute no bytes, and every offset at or past the first skipped block is shifted. PlaStream exposes the boundary as UnreliableFrom and both extent reads report it — a read whose position is at or past the boundary, or that reaches into it, throws InvalidDataException instead of returning bytes of other objects; a zero-length extent at the boundary is shifted too. The sample-corpus test applies the exact-fit checks only to the records before the boundary and asserts the report for the records past it.
  • A record position wider than long (ulong.MaxValue) was cast before the bounds check and surfaced as an ArgumentOutOfRangeException from the extent read; the record read now compares against the stream length as unsigned and reports the documented InvalidDataException before the cast.

@mahach666
mahach666 changed the base branch from feat/lz-chunks to main November 18, 2026 19:34
@mahach666
mahach666 merged commit 6f5a9ef into main Nov 18, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant