Skip to content

Phase 6 item 6.2: DEFLATE and zlib (clazel-deflate) - #183

Draft
atiernan wants to merge 9 commits into
phase6/6.3-shapefrom
phase6/6.2-deflate
Draft

atiernan wants to merge 9 commits into
phase6/6.3-shapefrom
phase6/6.2-deflate

Conversation

@atiernan

Copy link
Copy Markdown
Member

Phase 6 item 6.2: the new crate clazel-deflate (target tier 2, tier 0 today). Based on phase6/6.1-inventory.

What is there

  • Spec and contracts first (commit 53f42e5): RFC 1951 and RFC 1950 as Verus spec functions of the whole input, with the section and step numbers, ten modelling choices listed at the top, and docs/ERRATA.md (the extra-bit order in RFC 1951 3.2.5 contradicts 3.1.1).
  • Decoder: Inflater::step(src, dst, window), resumable at any byte of input and any size of output room, a 32 KiB window the caller provides, a 15-bit table per code, the output limit enforced before writing. adler, huffman, oneshot beside it.
  • Wuffs' DEFLATE and zlib test data vendored (commit b36a81d, licence and SHA-256 recorded, converter in xtask); miniz_oxide 0.9.1 and adler2 2.0.1 as test dev-dependencies (docs/DEPENDENCIES.md).

Proven (Verus 0.2026.10.04, WSL: 37 verified, 0 errors)

  • Every spec function terminates.
  • adler::update and adler::adler32 equal the spec's Adler-32 for every input.
  • theorem_output_bounded (about the spec): a successful decode never exceeds the limit.
  • The length and distance tables are chains of consecutive ranges (ending 258 and 32,768); the RFC's worked code examples evaluate to its codes; code_usable and the zlib header checks evaluate as they should.
  • 88 mutations of the spec, 88 killed, each certified (cargo xtask mutate clazel-deflate): every length and distance base, every extra-bit count, the incomplete-code rules, the zlib header checks.

Tested (13 tests, all pass)

Wuffs' streams and its six artificial corner cases; a differential against miniz_oxide over 33,979 mutated, truncated and hand-made streams (0 differences); the decoder equal to a naive Rust transcription of the spec on 12,119 streams including every error kind; every input chunk size and output room equal to the whole-input decode; the limit exact; a 64 MiB stream of zeros stopped at 1 MiB; hand-made streams for every length and distance code and each Error.

NOT done (so the crate stays tier 0)

  • Inflater::step, Table::build and Table::lookup are not proven. theorem_chunking_irrelevant, theorem_inflate_total and theorem_huffman_table are statements in contract.rs, not theorems.
  • No independent transcription (independent.rs); no mutation of the block loop, dynamic header or Huffman steps (the compute engine's recursion limit stops a certificate that evaluates a stream through the fixed code).
  • No gzip (6.0(h), later), no Kani.
  • Miri: started, stopped after ~15 minutes; bomb_stops_at_the_limit took 208 s of 300 s, so the strides are unmeasured.
  • Not a fast decoder.

Verify output

  • cargo xtask check: STATIC CHECKS PASSED.
  • cargo xtask verify --part crate clazel-deflate in WSL (Ubuntu): every step PASS (static checks, verus clazel-deflate 37 verified, proof mutation tests 88 killed, conformance clazel-deflate: Wuffs artificial 6/6, Wuffs test/data 5/5, miniz_oxide differential 33979/33979), then FAIL write crate measurements only because git cannot read this Windows-linked worktree from WSL.
  • cargo xtask verify --changed was not run to the end: xtask/ changed, so it checks every crate (hours on the shared WSL). CI's full verification has not run; it waits for the full-verification approval.

🤖 Generated with Claude Code

atiernan and others added 7 commits November 17, 2026 11:35
…any code (item 6.2)

Rule 3: spec.rs transcribes RFC 1951 (section 3.2.2 canonical codes, 3.2.3
blocks, 3.2.4 stored, 3.2.5 length and distance tables, 3.2.6 fixed codes,
3.2.7 dynamic codes) and RFC 1950 (2.2 header, FCHECK, FDICT refused, Adler-32
trailer) as pure functions of the whole input, with the modelling choices
listed at the top (output limit as a stated error, incomplete codes as zlib
treats them, HLIT/HDIST, fuel for termination). contract.rs states what the
executable code is to meet, including the chunking and output-bound theorems.
Verus checks that every spec function terminates (13 verified).

docs/ERRATA.md: RFC 1951 3.2.5's 'most-significant bit first' for extra bits
contradicts 3.1.1; we read them least significant bit first, as every decoder
does. The crate is placed in PLACES; tier stays 0.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
… converter (item 6.2)

Wuffs commit b36a81d (the pin 6.1 recorded): test/data/romeo.txt with its
raw DEFLATE and zlib encodings, midsummer.txt and pi.txt with their zlib
encodings (suite wuffs-deflate), and test/data/artificial-deflate/*.deflate
with their .decompressed files (suite wuffs-deflate-artificial), byte for
byte, with SOURCE.md (upstream, commit, licence, each file's SHA-256) and
LICENSE.md (Wuffs' LICENSE-MIT, our choice of MIT or Apache-2.0). The ICC
profile pair is left out: its licence was not checked.

xtask/src/testdata/wuffs_deflate.rs checks each file's SHA-256 against
SOURCE.md and writes a table embedding it with include_bytes!.

miniz_oxide 0.9.1 (default features off, with-alloc) and its one dependency
adler2 2.0.1 as dev-dependencies for the differential test; both rows in
docs/DEPENDENCIES.md, class test.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…s (item 6.2)

stream.rs: Inflater, a state machine over caller-owned buffers (a 32 KiB
window the caller provides; the decoder allocates only its three code tables
in new()). Every element is read whole or not at all, so the input and output
can be cut anywhere. huffman.rs: the canonical code of RFC 1951 3.2.2 as a
15-bit lookup table, checked for completeness as spec.rs's code_usable says.
adler.rs, oneshot.rs: Adler-32 and whole-input decoding.

Tests (tests/conformance_deflate.rs, 12 tests, all pass): Wuffs' streams and
artificial corner cases; a differential against miniz_oxide over 33,979
streams (mutated, truncated, hand-made error streams), 0 differences; every
chunk and output-room size equal to the whole-input decode; the decoder equal
to a naive Rust transcription of spec.rs (tests/model) on 12,119 streams
including every error kind; the output limit exact and a 64 MiB bomb stopped
at 1 MiB; hand-made streams for every length and distance code, stored
blocks, dynamic blocks, and each Error variant.

Nothing here is proven yet; the Verus parts follow.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…m 6.2)

Verus 0.2026.10.04.426d8b0 in WSL: 36 verified, 0 errors.

- adler::update and adler::adler32 are proven equal to the spec's Adler-32
  (contract::adler_fold, with lemma_sums_fold tying it to spec::adler_sums):
  no overflow, no panic, every input.
- lemmas.rs: the length and distance tables are chains of consecutive ranges
  ending at 258 and 32,768 (lemma_len_chain, lemma_len_ends, lemma_dist_chain,
  lemma_dist_ends), the RFC 1951 3.2.2 examples evaluate to the codes it lists
  (examples.rs, by unfolding), zlib header errors (examples.rs, evaluated),
  code_usable on small sets of lengths (lemmas_usable.rs, evaluated).
- theorem_output_bounded (lemmas_bounded.rs): for every input and limit a
  successful inflate or zlib_decompress produced at most limit bytes.

Spec changes made to let Verus evaluate it: pow2 is written out up to 2^15,
and lengths_ok is a recursive function instead of a quantifier. Neither
changes what the spec computes.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…tem 6.2)

mutations.json: 88 mutations of spec.rs, all killed with a certificate Verus
checks (cargo xtask mutate clazel-deflate: 88 killed, 0 incidental, 0
survived): every length base (29) and distance base (30), the extra-bit count
of every group of length (6) and distance (14) codes, five rules of
code_usable (the incomplete-code rule among them) and four zlib header checks.
certs.rs holds the certificates of the lemmas that evaluate the spec, so that
a lemma whose body fails in a mutant does not stop its certificate from
being checked. Verus: 37 verified, 0 errors.

README.md says what is proven and what is not: the resumable decoder, the
Huffman table build and lookup are tested, not proven, so the plan's
theorem_chunking_irrelevant, theorem_inflate_total and theorem_huffman_table
are still statements in contract.rs. Cargo.toml lists theorem_output_bounded.
docs/phases/phase-6.md: the 6.2 row of the status table.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…status row (item 6.2)

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Under Miri the suite did not finish in six hours. Natively a decode costs
about 1.3 ms however short the stream (the spec model about 6.5 ms), and
Miri runs this code some 1,600 times slower, so under Miri each test now
makes a few dozen decodes: six streams (one of each kind of block, raw
and zlib), no miniz_oxide corpus, mutants of the first stream of each
format only, fewer chunkings and limits, and a 16 KiB bomb. Natively at
these sizes each test takes 0.003 to 0.06 s. The normal test job is
unchanged and runs every stream.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@atiernan
atiernan changed the base branch from phase6/6.1-inventory to phase6/6.3-shape November 17, 2026 11:43
… and changed-crate CI)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
CI ran these tests at 342 to 1825 s each under Miri, about 30,000 times
a native debug run: a decode costs 20 to 40 s there however short the
stream, since each fills two 32,768-entry tables. The decoder is left as
it is (it parses untrusted bytes and is not yet proven, so it is not
redesigned for an interpreter's sake). Under Miri each test now takes
the first one or two streams (`miri_first`), the error test every eighth
error stream, and the model comparison is ignored: one model call takes
minutes there and the model is test code. Timed natively with the Miri
branches forced on, the tests come to 0.05 to 5.5 ms, about 1 to 165 s
of Miri at that rate. The normal test job is unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

This branch is waiting to be deployed

1 waiting deployment
full-verification — 705ff913 Waiting Nov 20, 2026 by atiernan via Approve the proofs #598
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant