Repository navigation
generate the py ISC parser from the parsanol .parg grammar (WO17 step 4b) - #843
Merged
Merged
Conversation
Replace the hand-written recursive-descent ISC parser (the _Parser half of interscript-py's isc.py) with a grammar-driven one, mirroring interscript-ts's WO17 step-4b milestone (ts PR #128): the canonical interscript-ruby .parg text (vendored verbatim in grammar/grammar_source.py, sha256-pinned b0f7a7caf462a1671a80ef974a59425c636f1d32cb3e405280991b21a6edd712 - the same pin the ts vendored copy carries, synced by scripts/sync_isc_grammar.py) is compiled by a PARG reader (grammar/parg_reader.py), executed by a table-driven PEG interpreter reproducing parsanol's capture/flattening semantics (grammar/peg.py), and lowered to an IscDocument by an action layer (grammar/actions.py) that drives the unchanged W1/W2 expansion (isc._Expander). parse_isc's public API is unchanged; isc.py keeps the document model, the expansion machinery and the compiled-map converter. Evidence (all runs on this branch, maps at origin/main 05f5cc3, 428 maps + 6 libs = 434 files): - documents: the new py-vs-py parity driver (scripts/parse_parity_py.py) parses every corpus file with BOTH parsers and diffs them: 434/434 identical under the maps canonical schema (scripts/parse-parity.mjs's py normalizer) AND 434/434 field-for-field raw equality, including byte-exact from_source override keys and the written constraint order - harness: maps scripts/parse-parity.mjs with this parser on the py side reports 411/434 vs ts - the same score, the same 23 failing files (0 status flips), and the same divergence-kind counts as the hand parser on the same-day ts side (the pre-existing gap is the py normalizer's constraint re-sorting and none-lowering, documented in the harness) - suite: pytest 18 failed / 394 passed / 8 skipped; the failure set is IDENTICAL to a pristine pre-swap run (same 18 test ids: 4 conformance + 14 embedded-map rows, all in parser-independent paths); the golden additivity gate still replays its rows byte-identically - lint: ruff clean over the touched files Four grammar-table extensions (GRAMMAR_EXTENSIONS, one named block), each spelled in the grammar language and corpus-proven dormant: any_arg item fallback, block-rule outside-brace when-guard, hyphenated run dependency names - the same three the ts generated parser ships - plus rule_items accepting `use` inside parallel/sequence blocks (the py suite pins use-in-pass splicing; a corpus-wide scan finds none). Hand-parser behaviors the grammar rejects are documented per site in grammar/actions.py (invalid backslash-u escapes, hyphenated identifiers outside run targets, multi-digit capture refs, stage-level reverse/strip/swapcase, and the metadata raw-form quirks); all corpus-dormant per the parity driver. Known tradeoff, measured on this machine: the table-driven interpreter parses the 434-file corpus in ~86s where the hand parser took ~7s (typical maps still parse in tens of ms; the gap shows only on the megabyte jamo/hanja maps and in the gates that parse the whole corpus - golden 86s -> 187s, full suite 107s -> 306s). Failure sets and outputs are unchanged; closing the gap (closure compilation, first-token prefilters) can proceed later without touching semantics.
Merged
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
WO17 step 4b, python leg: replace the hand-written recursive-descent ISC parser (interscript-py's
isc.py_Parser) with a parser generated from the canonical parsanol grammar, mirroring the merged TypeScript milestone (interscript-ts PR #128).lib/interscript/isc/grammar/isc.pargtext vendored verbatim, sha256-pinned (b0f7a7caf462a1671a80ef974a59425c636f1d32cb3e405280991b21a6edd712, the same pin the ts vendored copy carries); synced byscripts/sync_isc_grammar.py; pin testtests/test_grammar_source.py.parg-reader.ts.isc.py(_Expander, verbatim from the old_Parser) together with the document model and the compiled-map converter.parse_isc's public API is unchanged.rule_itemsacceptinguseinside parallel/sequence blocks (ISC-WF-1 §3 splice-into-pass; the py suite pins it, the corpus never uses it).Evidence (all runs on this branch)
interscript-py/scripts/parse_parity_py.py, maps at origin/main, 434 files): the driver parses every corpus file with BOTH parsers — 434/434 identical under the maps canonical schema and 434/434 field-for-field raw equality, including byte-exactfrom_sourceoverride keys and the written constraint order.scripts/parse-parity.mjs, this branch on the py side): 411/434 — the same score, the same 23 failing files (0 status flips), and the same divergence-kind counts (item-kind 678, constraints 144, item-value 4) as the hand parser on the same-day ts side; the gap is the py normalizer's constraint re-sorting and none-lowering, documented in the harness.pytest tests/— 18 failed / 394 passed / 8 skipped; the failure set is identical to a pristine pre-swap run (same 18 test ids, all pre-existing gaps in parser-independent paths: 4 conformance + 14 embedded-map rows). The golden additivity gate still replays byte-identically.Known tradeoff
The table-driven interpreter parses the 434-file corpus in ~86s where the hand parser took ~7s (typical maps still parse in tens of ms; the gap shows only on the megabyte jamo/hanja maps and in the gates that parse the whole corpus — golden 86s to 187s, full suite 107s to 306s). Outputs and failure sets are unchanged; interpreter performance (closure compilation, first-token prefilters) can be improved later without touching semantics.
Test plan
python -m pytest tests/frominterscript-py(18 pre-existing failures only; set diffed byte-identical against a pristineorigin/mainrun)python scripts/parse_parity_py.py --maps-root <maps-checkout>— 434/434 canonical, 434/434 raw (run pre-swap with--hand-rev HEAD; post-merge, pass--hand-rev <parent-of-swap>)node scripts/parse-parity.mjs --py-root <this-branch>in an interscript/maps checkout — 411/434, 0 status flips vs the hand-py incumbentruff check src scripts tests(touched files)