Repository navigation
feat!: harden the readers against corrupted data - #3
Merged
Merged
Conversation
Every read is now checked against the data before anything is read or allocated. Before, corrupted lengths could read past the end of a buffer, return stale pooled memory, or allocate gigabytes before failing, and the readers threw different exceptions for the same input. - Running out of data throws EndOfStreamException, an impossible length in the data InvalidDataException, an invalid argument ArgumentOutOfRangeException. A failed read leaves Position unchanged, apart from a length prefix that was already read. - Positions and seek targets are validated, so they can no longer be negative or wrap around. - ReadArray with a getter is limited by MaxArrayAllocationSize (default 256 MiB), since its length cannot be checked against the data. - GenericFileReader reads through a fixed-size cache; larger reads go straight to the file, and a file that grows or shrinks after opening is handled. - GenericStreamReader puts the stream back where a failed read started. - GenericFileReader and GenericStreamReader take a leaveOpen argument. - After Dispose, the buffer, file and stream readers throw ObjectDisposedException for reads and position access. BREAKING CHANGE: raw reads require T : unmanaged. The readers over memory return ReadOnlySpan<byte>/ReadOnlyMemory<byte> instead of writable views, and FStringMemory holds ReadOnlyMemory<byte>. Corrupted data throws the exceptions listed above instead of IndexOutOfRangeException or OverflowException. Dispose is implemented once in GenericReaderBase; subclasses override Dispose(bool). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
ReaderContractTests runs the same truncated, oversized and invalid inputs against every reader and checks the exception type and where Position ends up. ReaderSpecificTests covers what only one reader has, such as the file reader's cache and the stream reader's position restore. The round-trip tests now include non-ASCII text, and the file reader runs them with several cache sizes. InternalsVisibleTo lets the tests swap the file reader's buffer pool for one that fails on a double return. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The readers parse untrusted, often corrupted archive data, but did not check it consistently:
Read<T>only checked the first byte,ReadArraycopied without a check).ReadArrayallocate gigabytes before failing.position + sizecould overflow in the file reader's cache and read from the wrong offset.This PR checks every read against the data before anything is read or allocated, and makes all readers fail the same way.
Contract
EndOfStreamExceptionInvalidDataExceptionint, or a getter array aboveMaxArrayAllocationSizeArgumentOutOfRangeException[0, max]A failed read leaves
Positionunchanged. A length prefix that was already read stays read, and composite reads (offset overloads, FString and getter arrays) keep the steps that succeeded. The README documents this.Breaking changes (v3)
T : unmanaged.T : structallowed structs with object references, which turned file bytes into GC references.ReadOnlySpan<byte>/ReadOnlyMemory<byte>, andFStringMemoryholdsReadOnlyMemory<byte>. A reader built from aReadOnlySpansuch as"…"u8used to hand out writable spans; writing to them crashed the process.ReadArrayis limited byMaxArrayAllocationSizein bytes (default 256 MiB), because its length cannot be checked against the data.intoverloads ofPosition,LengthandSeekthrow instead of wrapping on file and stream readers beyond 2 GB.Dispose, the buffer, file and stream readers throwObjectDisposedExceptionfor reads and position access.Dispose()lives inGenericReaderBase; subclasses overrideDispose(bool).Other changes
GenericFileReaderuses a fixed-size cache. Reads larger than the cache go straight to the file, so one large read no longer grows the cache for good. The cache is never filled past the length the file had when it was opened.GenericStreamReaderputs the stream back where a failed read started, also when the stream ends before its reportedLengthor decoding fails.leaveOpenforGenericFileReader(SafeFileHandle)andGenericStreamReader(Stream).LoadFromFile(Async)loops over short reads, andLoadFromStream(Async)loads from the stream's current position.Decisions worth a look
Read<T>is about 0.44 ns instead of 0.27 ns (local Stopwatch loop over 2^18 reads). The old check only covered the first byte, so it was not correct._stream.Positioncall per read. The position restore needs it.Validation
dotnet build -c Release: 0 warnings.dotnet test: 300 of 300 passed on net10.0 and net11.0.ReaderContractTestsruns the same truncated, oversized and invalid inputs against every reader, plus the file reader with a 3-byte cache.ReaderSpecificTestscovers per-reader behavior: the file cache, files that shrink or grow after opening, a counting pool that fails on a double return, stream position restore, andleaveOpen.The version stays at 2.3.0, so merging publishes nothing (
--skip-duplicate). The prerelease version is set separately.🤖 Generated by Opus 5.5 with Claude Code in T3 Code