Skip to content

Replay the pinned 675-row unicode generation on the installed wheel - #3

Draft
suleman-uzair wants to merge 4 commits into
mainfrom
feat/umath675-replay-20261113
Draft

suleman-uzair wants to merge 4 commits into
mainfrom
feat/umath675-replay-20261113

Conversation

@suleman-uzair

Copy link
Copy Markdown
Member

Replays unicode generation 5 of plurimath-testsuite on the installed platform wheel. That generation holds the 675 UnicodeMath test rows imported from UnicodeMathML (MIT, Noah Doersing 2019).

scripts/corpus_replay.py checks out the commit named in ci/testsuite-pin.yml and selects that ledger entry, not the latest one. It recomputes the snapshot digest and checks every payload's bytes, SHA-256 and row IDs against provenance.yaml. Only after those checks pass does it run any row.

  • Raw-tree rows go through parse_legacy_raw. The preprocessed text and the whole raw tree must be equal, or for a refusal, the grammar byte offset.
  • Typed-model rows go through parse_model_json and are checked for outcome only (accepted or refused). The Ruby LML Document and the Rust model JSON are different shapes, so this PR claims no tree equality for them.
  • Legacy-oracle payloads are counted and not run.

No row is skipped. ci/testsuite-replay.json records the rows executed per boundary: 747 raw-tree and 747 typed-model, which is the 72 generation-4 rows plus the 675 external rows. It also records the SHA-256 of the sorted IDs. The replay fails if any row disagrees or if the executed set differs from that record. Rows nest up to 170 levels, so the replay raises the recursion limit instead of dropping them.

The new replay job runs on Linux, macOS and Windows with Python 3.14, the newest Python with PyYAML libyaml wheels on every platform. After the replay, it runs the testsuite's own validate-stages.rb --pin and validate-unicode-external.rb --generation 5 --executed. The offline tests in scripts/tests/test_corpus_replay.py show that a tampered payload, an unlisted file, the wrong commit, generation or digest, a dropped row and an unsafe path are all refused.

Pinned testsuite commit: acbdf0319dfb9bc49c1726bdbdedac276476e9b5 (merged via plurimath/plurimath-testsuite#50).

suleman-uzair added 4 commits November 13, 2026 14:59
scripts/corpus_replay.py checks out the plurimath-testsuite commit named
by ci/testsuite-pin.yml, selects the pinned ledger entry, recomputes the
snapshot digest, verifies every payload's bytes, SHA-256 and row IDs
against provenance, and only then replays the rows: raw-tree rows through
parse_legacy_raw with full tree equality, typed-model rows through
parse_model_json for their outcome. The executed IDs must match
ci/testsuite-replay.json. Offline tests prove the tamper refusals. A
replay job runs it on Linux, macOS and Windows.
ci/testsuite-pin.yml pins plurimath-testsuite acbdf03 (merged in #50),
unicode generation 5, sha256:282fb3ee... It holds the 72 generation-4
rows and the 675 unicode-external rows at both the raw-tree and the
typed-model boundary. ci/testsuite-replay.json records the 747 + 747 rows
this consumer executes and the SHA-256 of their sorted IDs.

Rows nest up to 170 levels, so the replay raises the recursion limit
instead of dropping deep rows. It writes the executed IDs so the replay
job can run the testsuite's validate-stages.rb --pin and
validate-unicode-external.rb --executed checks.
The typed-model boundary only checked accepted or refused. It now compares
each model row's field names, order and values with the codec JSON the
wheel returns, and reports the first differing path per row.
…6d83b22

Add opt-in parse_model_legacy_json (umath_parse_model_legacy_json_v1) beside the
unchanged Strict parse_model_json. The typed replay compares refusal caller byte
offsets and drops only writer-omitted empty text and empty lists, with negative
controls for the rule. Repin umath-rs, Cargo.lock hash and parsanol SDK.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant