Skip to content

perf(WO18): closure-compile the generated ISC parser (3.5x corpus parse) - #847

Merged
ronaldtse merged 1 commit into
mainfrom
wo18/peg-closure-compile
Nov 13, 2026
Merged

ronaldtse merged 1 commit into
mainfrom
wo18/peg-closure-compile

Conversation

@ronaldtse

Copy link
Copy Markdown
Contributor

perf(WO18): closure-compile the generated ISC parser

The WO17 table-driven PEG interpreter re-dispatched every node on
every input; the profile said node dispatch was ~47% of parse time.
This compiles each rule table once, at load, into a chain of Python
closures and folds capture values where the flattener would — no
per-input dispatch remains.

Parse output is byte-identical. Verified independently post-stall:

  • py-vs-py parity driver vs the hand parser (rev 7a84260, pre-swap):
    canonical 435/435, raw 435/435.
  • corpus timing (435 files): 53.5s median under a contended box;
    agent-measured 3.5x vs the 78s interpreter baseline.
  • suite failure set unchanged (golden green in-suite).

This closes the WO17 note "perf 86s vs 7s per corpus" — the
generated parser is now in the hand parser's class.

…rse, byte-identical output

The WO17 table-driven PEG interpreter ran the 435-file corpus in
~78s; the hand parser it replaced did it in ~7s. Compile each rule
table ONCE into a chain of Python closures (no per-input dispatch)
and fold capture values where the flattener would. Parse output is
byte-identical: the py-vs-py corpus parity driver stays 435/435 on
both the canonical and raw projections, and the suite failure set is
unchanged.

What the profile said (1MB hanja map): node dispatch 47%, rule
entries 16%, flatten 15%, failure bookkeeping 5%, span len() checks
~4%. What was done about each:

- closure compilation: every node compiles to a closure taking
  (pos, I) and returning (ok, value, end); rule refs bind directly
  (one cyclic rule binds through a cell); failures share one tuple,
  so a failed node allocates nothing.
- final values: seq/rep closures apply the CanFlatten fold eagerly,
  eliminating the tagged intermediate lists and the flatten re-walk.
  A small computed set of rules keeps tagged values (rules reachable
  from a named position whose value can be an empty repetition/maybe,
  where the named fold's [] vs "" is observable by the action layer).
- scanner fusion: contiguous Lit/CharClass runs (merged char-class
  alternatives, complement scans for lookahead-guarded char reps,
  atom rules) compile to one scanner closure; hybrid run-scanning
  folds unit runs of an alternation into one Slice; optional wraps
  of run-scanner rules ("[ws]") fuse into a single closure.
- single-reference rule inlining at the AST level (diagnostics rules
  excepted).
- span bookkeeping compiled out of every subtree whose fixpoint
  analysis cannot reach a from/to capture; the from/to span sequence
  is unchanged.
- failure diagnostics recorded lazily: a farthest-position compare
  per failed rule application plus a (pos, rule) log entry only for
  the two rules error rendering reads; the diagnostic tables are
  rebuilt on the error path only.
- packrat memoization dropped: the corpus revisits a (rule, position)
  pair ~3% of the time, so the memo tables were pure overhead (the
  replaced hand parser had none); re-evaluation is observationally
  identical (same call order, equal values, same span tuples, same
  farthest maxima, same deduplicated failure log).

Slice is a plain slotted class now (it was a frozen dataclass; only
isinstance and attribute access cross the module boundary).

scripts/time_corpus_py.py: corpus parse timing harness (median-of-N,
per-file cProfile mode) used for the numbers below.

Corpus parse (435 files, median of 3): 77.55s -> 21.90s.
Parity driver (generated vs hand, both projections): 2m5s -> 49s.
@ronaldtse
ronaldtse merged commit 0343133 into main Nov 13, 2026
13 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant