Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
23 changes: 23 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,29 @@ All notable changes are documented here. The format follows [Keep a Changelog](h
and the project follows [Semantic Versioning](https://semver.org/spec/v2.0.0.html); versions before 1.0 may change the
API between minor releases.

## [Unreleased]

### Added

- `CompressedPool.ReadSubgroups` and `PoolSubgroup`: the object records the object database keeps inside a pool —
BL-tagged subgroups with the object's class id (`Kind`), `Version`, `Flags`, `ObjectId` (a GUID when the header
carries one), the payload's storage (`Mode`: raw or zero mask) and the decoded payload (`Data`). The headers follow
the file's block layout (a u16 packed size for size classes 0..6, a u32 for 7..14); a zero-mask stream that ends
before its declared unpacked size leaves the tail of the buffer zero. Mode 2 (LZ) subgroups have not been met inside
the pools examined so far and are reported with an empty payload. The records are found by scanning for the BL tag
and accepting the ones whose header parses; a rejected candidate is retried one byte on. Observed payloads store text
as a u32 length followed by UTF-16 little-endian data and numbers as little-endian doubles and integers.

- `PlaDocument.FindCompressedPools` and `CompressedPool`: the compressed storage pools of the object database that the
file holds in the gaps between blocks, each starting with a 16-byte compression sign, a declared decompressed total
and its blocks (each memcpy or LZ77-triple coded, at most one megabyte and change of packed payload per block). The
object database keeps its groups, journals and library data in these pools; a pool is not a block and does not take
part in the block index. `FindCompressedPools` finds every pool whose header and block headers parse and whose
declared sizes tile the total (computed on each call; the scan reads the file in windows and skips the indexed block
payloads, so a sign inside them is never returned as a pool); `Decode` produces the decoded bytes on request. The
layout and the LZ77-triple codec follow the format notes of this repository and are confirmed against the sample
files.

## [0.1.0] - 2026-11-12

### Added
Expand Down
35 changes: 32 additions & 3 deletions docs/public-api.md
Original file line number Diff line number Diff line change
@@ -1,8 +1,9 @@
# Public API

The `PlaIO` package is the supported surface listed here; every public type is in the namespace `PlaIO`. Everything else
(the block scanner, the header parser, the payload decoder, the directory search, the PNG chunk walker) is `internal` in
`PlaIO.Internal`; it can change in any release and is visible only to the test project through `InternalsVisibleTo`.
(the block scanner, the header parser, the payload decoder, the directory search, the pool decoder, the subgroup parser,
the PNG chunk walker) is `internal` in `PlaIO.Internal`; it can change in any release and is visible only to the test
project through `InternalsVisibleTo`.

## File kind

Expand Down Expand Up @@ -30,6 +31,33 @@ The `PlaIO` package is the supported surface listed here; every public type is i
| `ObjectDirectory` | A run of directory records inside a raw block: `Block`, `RecordOffset` (payload offset of the first record), `Phase` (`RecordOffset % 30`), `Records`, `AbuttingPairs`, `OverlappingPairs`, `DuplicatePositions`. The run is an estimate and can be shorter than the real directory: it stops after more than 8 consecutive records without an abutting neighbour. |
| `ObjectDirectoryRecord` | The 30-byte record `[u16 class][u64 pos][u32 count][GUID16]` as `ClassNumber`, `Position`, `Count`, `ObjectId`. The units of position and count are not documented, and the class number is not mapped to a class name. |

## Compressed pools (the object database)

The object database keeps its groups, journals and library data in compressed pools that sit in the gaps between blocks.
A pool starts with a 16-byte compression sign, then the declared decompressed total and the block count (both unsigned
64-bit little endian), then that many blocks, each a u32 compressed size followed by the block: a flag byte (bit 0: the
payload is LZ77-triple coded, otherwise copied; bit 1: the sizes are 32-bit, otherwise single bytes), the packed size
(equal to the compressed size), the unpacked size and the payload. The blocks decode to exactly the declared total. The
layout and the LZ77-triple codec follow the format notes of this repository and are confirmed against the sample files.

| Member | Use |
|---|---|
| `PlaDocument.FindCompressedPools()` | Every pool whose header and block headers parse and whose declared sizes tile the total, in file order. Recomputed on each call; the scan reads the file in windows and skips the indexed block payloads, so a sign inside them is never returned as a pool. |
| `CompressedPool.Offset` | File offset of the pool's compression sign. |
| `CompressedPool.End` | File offset past the last block of the pool. |
| `CompressedPool.TotalDecompressed` | The declared size of the decoded pool in bytes. |
| `CompressedPool.BlockCount` | The number of blocks; at most one megabyte and change of packed payload per block. |
| `CompressedPool.Decode()` | The decoded bytes: every block memcpy or LZ77-triple decoded, concatenated. The call does not cache. Throws `InvalidDataException` when a block does not decode or the blocks do not add up to the total, and `ObjectDisposedException` when the document is disposed. |
| `CompressedPool.ReadSubgroups()` | The object records inside the pool: BL-tagged subgroups whose headers parse, with their payloads decoded. Recomputed on each call. A rejected BL candidate is retried one byte on. |
| `PoolSubgroup.Offset` | Offset of the subgroup header inside the decoded pool. |
| `PoolSubgroup.Kind` | The object's class id (the kind field of the BL header). |
| `PoolSubgroup.Version` | The header's version byte, 0 or 1. |
| `PoolSubgroup.Flags` | The header's flags: the 0x8080 marker, 0x20 marks the GUID, 0x100 and 0x200 select the payload modes, the low 4 bits are the packed size class. |
| `PoolSubgroup.Mode` | The payload's storage: raw (0) or zero mask (1). Mode 2 (LZ) is reported but not decoded; it has not been met inside the pools examined so far. |
| `PoolSubgroup.ObjectId` | The object's GUID, present when flag 0x20 is set. |
| `PoolSubgroup.Partial` | True when decoding stopped at a truncated literal or the payload could not be decoded (unsupported mode or size). A zero-mask stream ending at a group boundary has valid zero padding. |
| `PoolSubgroup.Data` | The decoded payload; empty for mode 2 subgroups. A zero-mask stream that ends before its declared unpacked size leaves the tail of the buffer zero. Observed payloads store text as a u32 length followed by UTF-16 little-endian data and numbers as little-endian doubles and integers. |

## BIMx file (`.bimx`)

| Member | Use |
Expand All @@ -46,7 +74,8 @@ The `PlaIO` package is the supported surface listed here; every public type is i

`PlaDocument` and `BimxDocument` never load the whole file into memory: the block index is built by scanning through the
stream once, at `Open`, and payloads and the cover are read on request. A document is not thread-safe; use one per thread.
After `Dispose`, `ReadPayload`, `FindDirectories` and `ReadCover` throw `ObjectDisposedException`.
After `Dispose`, `ReadPayload`, `FindDirectories`, `FindCompressedPools`, `CompressedPool.Decode`,
`CompressedPool.ReadSubgroups` and `ReadCover` throw `ObjectDisposedException`.

## Diagnostics

Expand Down
47 changes: 47 additions & 0 deletions src/PlaIO/CompressedPool.cs
Original file line number Diff line number Diff line change
@@ -0,0 +1,47 @@
namespace PlaIO;

/// <summary>A compressed storage pool of the ODB object database that the file holds in a gap between blocks: the
/// compression sign, the declared decompressed total and the blocks (each memcpy or LZ77-triple coded). The object
/// database keeps its groups, journals and library data in such pools; a pool is not a block and does not take part in
/// the block index or the payloads. The decoded bytes are produced on request by <see cref="Decode"/>; the cost of a
/// call grows with the pool size. The layout and the codec follow the format notes of this repository and are confirmed
/// against the sample files.</summary>
public sealed class CompressedPool
{
private readonly PlaDocument _document;

internal CompressedPool(PlaDocument document, long offset, long end, long total, int blockCount)
{
_document = document;
Offset = offset;
End = end;
TotalDecompressed = total;
BlockCount = blockCount;
}

/// <summary>File offset of the pool's compression sign.</summary>
public long Offset { get; }

/// <summary>File offset past the last block of the pool.</summary>
public long End { get; }

/// <summary>The declared size of the decoded pool in bytes; the blocks decode to exactly this size.</summary>
public long TotalDecompressed { get; }

/// <summary>The number of blocks; at most one megabyte and change of packed payload per block.</summary>
public int BlockCount { get; }

/// <summary>Decodes the pool: every block memcpy or LZ77-triple decoded, concatenated. The call does not cache; each
/// call decodes again.</summary>
/// <exception cref="InvalidDataException">A block does not decode or the blocks do not add up to
/// <see cref="TotalDecompressed"/>.</exception>
/// <exception cref="ObjectDisposedException">The document is disposed.</exception>
public byte[] Decode() => _document.DecodePool(Offset);

/// <summary>Decodes the pool and reads its object records: the BL-tagged subgroups whose headers parse, with their
/// payloads decoded (raw or zero mask; mode 2 is reported with an empty payload). Recomputed on each call; a
/// rejected BL candidate is retried one byte on, so a short false header can skip over a real record.</summary>
/// <exception cref="InvalidDataException">The pool does not decode.</exception>
/// <exception cref="ObjectDisposedException">The document is disposed.</exception>
public IReadOnlyList<PoolSubgroup> ReadSubgroups() => Internal.SubgroupParser.Parse(Decode());
}
51 changes: 46 additions & 5 deletions src/PlaIO/Internal/PayloadDecoder.cs
Original file line number Diff line number Diff line change
Expand Up @@ -2,16 +2,53 @@ namespace PlaIO.Internal;

internal static class PayloadDecoder
{
/// <summary>How a zero-mask stream ended relative to the destination buffer.</summary>
public enum ZeroMaskEnd
{
/// <summary>The destination buffer is full.</summary>
Complete,
/// <summary>The stream ended exactly at a group boundary before the buffer is full; the tail of the buffer stays
/// zero. Valid for the pool subgroups, whose declared size is the buffer size.</summary>
StreamEnd,
/// <summary>The stream ended inside a group: a mask byte was read whose set bit takes a stream byte that is not
/// there. A truncated literal, never a valid padding.</summary>
MidGroup,
}

/// <summary>Mode 1: a stream of groups, each a mask byte followed by one byte per set bit. Bits are read from the least
/// significant; a set bit takes the next stream byte, a clear bit writes 0x00. Decoding ends when <paramref name="dst"/>
/// is full, so bits of the last mask past the end consume nothing. Returns the number of source bytes consumed; bytes
/// left after that are allowed.</summary>
/// is full, so bits of the last mask past the end consume nothing; the <paramref name="end"/> status tells how the
/// stream ended when <paramref name="dst"/> is not full. Returns the number of source bytes consumed; bytes left after
/// that are allowed.</summary>
public static int DecodeZeroMask(ReadOnlySpan<byte> src, Span<byte> dst, out ZeroMaskEnd end)
{
return DecodeCore(src, dst, out end, out _);
}

/// <summary>Mode 1 for the file blocks: the stream must fill the buffer exactly; a stream that ends before that — at a
/// group boundary or inside a group — does not decode. Returns the number of source bytes consumed; bytes left after
/// that are allowed.</summary>
/// <exception cref="InvalidDataException">The stream ends before the buffer is full.</exception>
public static int DecodeZeroMask(ReadOnlySpan<byte> src, Span<byte> dst)
{
int consumed = 0, written = 0;
int consumed = DecodeCore(src, dst, out _, out var written);
if (written < dst.Length)
throw Short(consumed, src.Length, written, dst.Length);
return consumed;
}

private static int DecodeCore(ReadOnlySpan<byte> src, Span<byte> dst, out ZeroMaskEnd end, out int written)
{
int consumed = 0;
written = 0;
end = ZeroMaskEnd.Complete;
while (written < dst.Length)
{
if (consumed >= src.Length) throw Short(consumed, src.Length, written, dst.Length);
if (consumed >= src.Length)
{
end = ZeroMaskEnd.StreamEnd;
return consumed;
}
int mask = src[consumed++];
for (int bit = 0; bit < 8 && written < dst.Length; bit++)
{
Expand All @@ -20,7 +57,11 @@ public static int DecodeZeroMask(ReadOnlySpan<byte> src, Span<byte> dst)
dst[written++] = 0;
continue;
}
if (consumed >= src.Length) throw Short(consumed, src.Length, written, dst.Length);
if (consumed >= src.Length)
{
end = ZeroMaskEnd.MidGroup;
return consumed;
}
dst[written++] = src[consumed++];
}
}
Expand Down
Loading
Loading