Skip to content

WO20: differential fuzz harness across the four ISC parsers #284

Open

Description

@ronaldtse

Goal

Differential fuzzing across all four ISC parsers (rs, ts, py, ruby): throw mutated/generated .isc at every runtime, hunt crashes, hangs, and cross-runtime divergences the curated corpus cannot surface (harness baseline is corpus-only: 412/435, 3 known design kinds).

Approach

  • Generator: mutation fuzzing over the 428-map corpus (truncate, byte-flip, splice rules, inject WF-1 constructs, malformed escapes/editions) + a small grammar-aware generator producing VALID but unusual maps (nested any/capture/maybe, constraint combos, unicode edge literals).
  • Runner: one .isc per case, per runtime, with a per-case timeout (hang = finding). Compare: (a) accept/reject classification, (b) error class, (c) parse output on accepts.
  • Output: scripts/fuzz/ — the generator, runner, a triage report of findings classified crash/hang/divergence, and minimized repros. Known design divergences (documented in docs/parity) are not findings; NEW ones are.
  • Findings become labeled issues per repo (crash/hang = the owning repo; divergence = cross-runtime, filed in maps referencing all four).
  • CI-able smoke mode: a small seeded corpus (~200 cases) that stays green in maps-CI.

Rules: findings get minimized + documented; do NOT fix parser bugs in this issue (fix issues get filed and sequenced). Branch: wo20-fuzz.

Activity

  1. ronaldtse commented on Nov 16, 2026

    @ronaldtse
    ContributorAuthor

    WO20 milestone — harness built and pushed (wo20-fuzz)

    scripts/fuzz/ is on the branch:

    • Generator — corpus mutation (truncate, byte-flip, line surgery, cross-map rule splices, WF-1 construct injection targeting the G5/G7/G6/G8 surfaces, malformed escapes/editions, unicode edge literals) + grammar-aware templates (nested any/capture/maybe, constraint combos, alias chains, duplicate-from blocks, empty-string rules, bounded self-runs).
    • Runner — per-case .isc in its own dir, one driver per runtime, per-case timeout (hang = finding). Drivers: ruby via the DSL load path, ts via the production parse chain, py via interscript.isc (origin/main 0.2.x, PYTHONPATH=src), rs via iscrs parse/transliterate (release 0.3.2).
    • Classifier — crash / hang / divergence, with the docs/parity G1–G12 known-kind filter (signature-based on case content) so documented design divergences don't mask as findings.
    • Minimizer — delta-debugging shrink + repro proof per finding.
    • Smoke mode — committed seeded corpus (~200 cases) + fuzz-smoke.yml; green = zero unexplained findings across available runtimes (ruby+ts in maps-CI, all four locally).

    Runtime builds used for the campaign: fresh clones off origin/main for ruby/ts/py; rs built at the released 0.3.2 commit against crates.io parsanol (origin/main rs is currently coupled to a maintainer-local unreleased parsanol tree and does not build standalone — noting for the record).

    Campaign execution starts next; findings will be triaged, minimized, and filed as labeled issues per repo.

  2. ronaldtse commented on Nov 16, 2026

    @ronaldtse
    ContributorAuthor

    WO20 complete — harness, campaign, findings, smoke

    Delivered on branch wo20-fuzz, PR: #296.

    Campaign (seed 20261117): 1600 cases (1200 corpus-mutation + 400 grammar-aware) × 4 runtimes = 6400 observations, per-case 45 s timeout. Outcomes: 1007 agree / 268 known-kind / 325 findings; the findings cluster into 7 root causes (py dominates: 118 findings vs ruby 29, rs 15, ts 15).

    Issues filed (findings become labeled issues; no parser bugs fixed here):

    Crash/hang (owning repos):

    • interscript-py#43 — hang: parser loops forever on truncated maps (10 cases; 10-line repro, spins in _stage)
    • interscript-py#44 — crash: malformed \u escapes raise raw ValueError instead of IscParseError (107 cases — largest class; fires on note text too)
    • interscript-ruby#864 — crash: NameError: uninitialized constant Node::Item::CaptureRef on ref(N) in from position (14 cases)

    Divergence (cross-runtime, filed in maps referencing all four):

    • maps#292 — ruby silently miscompiles alias = maybe(...): rule never fires (52 cases, abc vs RaRbRcR)
    • maps#293 — py 0.2.x parse boundary differs from ruby/ts/rs in both directions (~130 cases; accepts nested par-in-seq the parity probe pins as rejected 4/4, rejects zero-width primitives in from)
    • maps#294 — self-recursive run stage.<self>: rs SIGABRTs, ruby/ts stack-overflow, only py names the cycle (15 cases)
    • maps#295 — one { inside note text corrupts py's parse into different transliteration output (mutation of din-kaa-Cyrl-Latn-1460-2-2011)

    All repros minimized and committed under scripts/fuzz/findings/; full triage in scripts/fuzz/TRIAGE.md.

    Smoke mode: 199 committed seeded cases, green across all four runtimes locally (155 agree / 44 known / 0 findings); fuzz-smoke.yml runs the ruby+ts legs in maps-CI. Runtime trees via RUBY_LIB/TS_DIST/PY_SRC/ISCRS_BIN; unavailable runtimes are skipped so CI stays green.

    Environment notes: rs built at release 0.3.2 (origin/main currently path-pins an unreleased local parsanol and does not build standalone — recorded in TRIAGE.md); py leg is origin/main 0.2.x (new parser, per the PYTHONPATH=interscript-py/src statement), so some parity-audit known kinds shift under it — classification used content signatures, and known-vs-new calls are documented per cluster.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions