Skip to content

WO20: differential fuzz harness across the four ISC parsers + seeded smoke mode - #296

Merged
ronaldtse merged 3 commits into
mainfrom
wo20-fuzz
Nov 16, 2026
Merged

ronaldtse merged 3 commits into
mainfrom
wo20-fuzz

Conversation

@ronaldtse

Copy link
Copy Markdown
Contributor

Summary

WO20 differential fuzz harness (issue #284): generator, runner, classifier, minimizer, and the CI-able seeded smoke mode.

  • scripts/fuzz/generate.mjs — corpus mutation (truncate, byte-flip, line surgery, cross-map rule splices, WF-1/keyword/escape/unicode injection) + grammar-aware templates (nested any/capture/maybe, constraint combos, alias chains, duplicate froms, empty-string rules, bounded self-runs). Deterministic from a single seed.
  • scripts/fuzz/run.mjs — per-case (isolated dir), per-runtime drivers with per-case timeouts; invalid UTF-8 normalized to U+FFFD identically for all four runtimes.
  • scripts/fuzz/classify.mjs — crash / hang / divergence with the docs/parity G1-G12 known-kind signature filter; hangs and process-level crashes are never explained away.
  • scripts/fuzz/minimize.mjs — delta-debugging shrink with persistence proof.
  • scripts/fuzz/smoke.mjs + committed smoke-cases/ (199 seeded cases, green on all four runtimes locally; ruby+ts in maps-CI via the new fuzz-smoke.yml).
  • scripts/fuzz/TRIAGE.md — campaign report: 1600 cases / 6400 observations, 325 findings -> 7 filed issues (2 interscript-py, 1 interscript-ruby, 4 maps), with minimized repros under findings/.

Runtimes are located via env (RUBY_LIB / TS_DIST / PY_SRC / ISCRS_BIN), defaulting to sibling checkouts; missing runtimes are skipped in smoke mode.

Test plan

  • node scripts/fuzz/smoke.mjs green (all four runtimes locally; ruby+ts legs are what CI runs)
  • Campaign (seed 20261117) classified: 1007 agree / 268 known / 325 findings, all triaged in scripts/fuzz/TRIAGE.md
  • Findings filed: interscript-py#43, interscript-py#44, interscript-ruby#864, maps#292, maps#293, maps#294, maps#295
  • CI fuzz-smoke workflow green on this PR

Generator (corpus mutation + grammar-aware templates), runner (per-case
per-runtime drivers with timeouts), classifier (crash/hang/divergence
with the docs/parity G1-G12 known-kind filter), minimizer, and a
CI-able seeded smoke mode over committed cases. Runtime trees are
located via RUBY_LIB / TS_DIST / PY_SRC / ISCRS_BIN env vars; smoke
skips unavailable runtimes so maps-CI (ruby + ts) stays green.
The committed smoke baseline was rebuilt through all four runtimes via
rebuild-smoke.mjs (seed 20261120, 420-case pool, agree-or-known cases
kept). Classifier fix: hangs and process-level crashes are never
explained by known-kind signatures — only divergences and in-process
crash buckets are filterable by the docs/parity kinds.
Campaign results (seed 20261117): 1007 agree / 268 known-kind / 325
findings over 6400 observations, clustering into 7 root causes:
py parse hang (interscript-py#43), py escape ValueError crash (#44),
ruby CaptureRef NameError crash (interscript-ruby#864), ruby alias-
maybe silent miscompile (maps#292), py parse-boundary divergence
(maps#293), unguarded self-run recursion (maps#294), py note-text
parse corruption (maps#295). Minimized repros committed under
findings/; generator escape literals corrected to the grammar-valid
uXXX/U forms.
Comment on lines +11 to +31
name: Seeded differential-fuzz smoke (ruby + ts)
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v7
- name: Set up Ruby
uses: ruby/setup-ruby@v1
with:
ruby-version: "3.4"
- name: Checkout interscript-ruby
run: git clone --depth=1 https://github.com/interscript/interscript-ruby.git ../interscript-ruby
- name: Install parser dependency
run: gem install parsanol
- uses: actions/setup-node@v7
with:
node-version: "22"
- name: Checkout + build interscript-ts
run: |
git clone --depth=1 --branch main https://github.com/interscript/interscript-ts.git ../interscript-ts
cd ../interscript-ts && npm ci && npm run build
- name: Run seeded smoke corpus
run: node scripts/fuzz/smoke.mjs --runtimes ruby,ts --jobs 4
Comment thread scripts/fuzz/classify.mjs

if (opts.summary) {
const rows = findingList
.map((f) => `| ${f.case} | ${f.kinds.join(",")} | ${f.findings.map((x) => `${x.kind}[${x.runtime}]`).join(", ")} | ${f.findings[0].detail.slice(0, 120).replace(/\|/g, "\\|")} |`)
@ronaldtse
ronaldtse merged commit 189a588 into main Nov 16, 2026
12 of 14 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants