Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
64 changes: 64 additions & 0 deletions docs/parity/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -581,6 +581,70 @@ engine's corpus parse time (rs 6.5 s each of its 79 s total; ruby

---

## D. WO20 — parse-parity harness residual closure (2026-11-17, #285)

`scripts/parse-parity.mjs` parses the whole corpus (maps/ + libs/)
with the ts parser (`parseIsc` + `expandDocument`) and the py parser
(`parse_isc`), projects both documents onto one canonical JSON schema,
and reports every structural difference with a kind, a path, and both
renderings. The audit baseline was **412/435 files identical** with
826 divergence rows in three kinds: item-kind 678, constraints 144,
item-value 4 — all attributed to the canonical projection, not to
either parser. WO20 closed all three classes by stating the shared
lowering each class collapses to; the harness now reports
**435/435, zero divergence kinds**.

The reconciliation rules (each checkable against the cited code, so a
future reader can re-derive them):

1. **Constraint written order → fixed kind order** (resolves the 144
`constraints` rows plus 138 positional `item-kind` rows and the 4
`item-value` rows — e.g. `alalc-ell-Grek-Latn-2010` writes
`after "Ν" before "Κ"`: ts preserves written order, while the py
SubRule model stores the four kinds as named fields, so the py
normalizer emitted them in fixed `before/after/not_before/
not_after` order; comparing positionally swapped both the kinds
and the values). The four constraint kinds are simultaneous
lookarounds — written order between them carries no semantics —
so the canonical schema now projects BOTH parsers onto the fixed
kind order (stable within a kind). Genuine differences still
diverge: a one-sided kind lands at a different position or changes
the length; a duplicated kind (ts keeps every written copy, py
keeps the last) shows as a length or value mismatch. No corpus
rule writes a duplicated kind, so none of the 286 rows hid one.
2. **`none` → the empty string** (resolves 493 `item-kind` rows —
ts kept parse-level `{type:"none"}` where py lowers at parse).
Both runtimes lower identically at their compile layers: ts
`src/isc/converter.ts` `case "none"` → `StringItem("")` and
`src/runtime/compile-item.ts` `none: { re: "", literal: "" }`; py
`grammar/actions.py` ("converter parity: `none` lowers to the
empty string") → `StringItem("")`. py cannot represent `none`
distinct from `""`, so the canonical form is the shared lowering.
3. **`any_character` → the any_character stdlib alias** (resolves 47
`item-kind` rows — ts `grammar/actions.ts` lowers the `any_char`
literal to a function atom, py to `AliasItem("any_character")`).
Both runtimes compile it through the alias path with the same
body: ts `src/isc/converter.ts` keeps the alias path for
`any_character` ("the primitive spelled as a function atom"),
`src/runtime/compile-item.ts` `any_character: { re: ".",
literal: "" }`; py `items.py` `ALIAS_COMPILED["any_character"]`
`re: "."`. The canonical form is that shared lowering.

The py-vs-py swap driver (`interscript-py/scripts/parse_parity_py.py`)
already projected both of its parsers onto the same schema
(constraints re-sorted into the fixed kind order, `none` lowered);
its canonical judgment is unchanged by this closure, and its raw
projection (which includes the written constraint order) never
compared across grammars.

Gates after the closure: harness 435/435 (zero kinds); py-vs-py
driver canonical 432/435, raw 432/435 (hand rev `7a84260`,
unchanged); py rs-golden zero-delta; ts parser-parity (289 maps)
green. No grammar (`.parg`) edits and no parser/model changes were
required — the residual was entirely in the canonical projection.

---

## Reproducing

```sh
Expand Down
59 changes: 48 additions & 11 deletions scripts/parse-parity.mjs
Original file line number Diff line number Diff line change
Expand Up @@ -14,10 +14,14 @@
// per-parser normalizer (the two IRs render the same source construct
// differently by design — e.g. the Python parser lowers `none` to an empty
// string item and lowers `to`-position function atoms to funcall-inline).
// Where a construct is representable in one parser but NOT the other, or
// renders under a different kind, the harness reports a divergence instead
// of force-passing: the first run is an AUDIT of the four hand-synced
// grammar copies.
// Where the canonical projection cannot equate a construct — a kind or
// value one parser derives and the other does not, a rule shape only one
// can represent — the harness reports a divergence instead of
// force-passing. Constructs BOTH runtimes lower identically at their
// compile layers (`none` -> "", `any_character` -> stdlib alias,
// constraint written order -> fixed kind order) are projected onto that
// shared lowering (WO20 #285: the audit's 826-row residual was exactly
// these three mechanical classes).
//
// Usage:
// node scripts/parse-parity.mjs [--json <report.json>] [--verbose]
Expand Down Expand Up @@ -264,7 +268,7 @@ function parseJsonl(stdout) {
// ---------------------------------------------------------------------------
// Canonical schema — shared target of the two per-parser normalizers
//
// Items: {t:"s",v} | {t:"none"} | {t:"prim",name} | {t:"fn",name}
// Items: {t:"s",v} | {t:"prim",name} | {t:"fn",name}
// | {t:"alias",name} | {t:"ref",id} | {t:"cap",inner}
// | {t:"cat",parts} | {t:"any",of} | {t:"range",lo,hi}
// | {t:"maybe",inner} | {t:"some",inner}
Expand All @@ -283,9 +287,25 @@ function canonTsItem(it) {
if (!it || typeof it !== "object") return { t: "shape", raw: String(it) }
switch (it.type) {
case "string": return { t: "s", v: it.value }
case "none": return { t: "none" }
case "none":
// WO20 (#285): `none` lowers to the empty string on both sides —
// ts at compile (converter.ts `case "none"` -> StringItem("");
// compile-item.ts `none: { re: "", literal: "" }`), py at parse
// (actions.py "converter parity: `none` lowers to the empty
// string"). py cannot represent `none` distinct from "", so the
// canonical form is the shared lowering.
return { t: "s", v: "" }
case "primitive": return { t: "prim", name: it.name }
case "function": return { t: "fn", name: it.name }
case "function":
// WO20 (#285): `any_character` is the any_character primitive
// spelled as a function atom (actions.ts lowers the any_char
// literal this way). Both runtimes compile it through the
// stdlib-alias path — ts converter.ts keeps the alias path
// (compile-item.ts `any_character: { re: ".", literal: "" }`),
// py represents it as AliasItem (items.py ALIAS_COMPILED
// `re: "."`). The canonical form is that shared lowering.
if (it.name === "any_character") return { t: "alias", name: it.name }
return { t: "fn", name: it.name }
case "alias_ref": return { t: "alias", name: it.name }
case "capture": return { t: "ref", id: it.index }
case "capture_group": return { t: "cap", inner: canonTsItem(it.inner) }
Expand Down Expand Up @@ -330,7 +350,9 @@ function canonTsRule(r) {
return {
from: canonTsItem(r.from),
to: canonTsItem(r.to),
constraints: (r.constraints ?? []).map((c) => ({ kind: c.kind, item: canonTsItem(c.item) })),
constraints: orderConstraints(
(r.constraints ?? []).map((c) => ({ kind: c.kind, item: canonTsItem(c.item) })),
),
...(r.when ? { when: [...r.when] } : {}),
}
}
Expand All @@ -339,13 +361,28 @@ function canonTsRule(r) {
// constraint kinds is lost, and a duplicated kind keeps only the last.
const PY_CONSTRAINT_ORDER = ["before", "after", "not_before", "not_after"]

// WO20 (#285): the four constraint kinds are simultaneous lookarounds —
// written order between them carries no semantics, and the py SubRule
// model stores them as four named fields (written order lost, duplicated
// kind keeps the last). The canonical schema therefore projects BOTH
// parsers onto the fixed kind order (stable within a kind). A genuine
// difference still diverges: a kind present on one side only lands at a
// different position or changes the array length, and a duplicated kind
// (ts keeps every written copy, py keeps the last) shows as a length or
// value mismatch.
function orderConstraints(cs) {
return PY_CONSTRAINT_ORDER.flatMap((k) => cs.filter((c) => c.kind === k))
}

function canonPyRule(r) {
return {
from: canonPyItem(r.from_),
to: canonPyItem(r.to),
constraints: PY_CONSTRAINT_ORDER
.filter((k) => r[k] != null)
.map((k) => ({ kind: k, item: canonPyItem(r[k]) })),
constraints: orderConstraints(
PY_CONSTRAINT_ORDER
.filter((k) => r[k] != null)
.map((k) => ({ kind: k, item: canonPyItem(r[k]) })),
),
...(r.when ? { when: [...r.when] } : {}),
}
}
Expand Down
Loading