Skip to content

WO20: published benchmarks page + API plane completion status #238

Open

Description

@ronaldtse

Goal

SOTA 2027 needs published, reproducible evidence. Two deliverables:

A. Benchmarks page on the site (interscript.github.io)

  • Reproducible benchmark scripts (checked in): per-runtime corpus parse (median-of-3, the parity-README methodology), per-runtime transliteration throughput on a fixed input set, golden replay time.
  • A /benchmarks page rendering current numbers + machine profile + exact reproduce commands, regenerated by script (never hand-edited numbers).
  • Keep the parity-README table as the audit-day baseline; the page tracks current-HEAD.

B. API plane completion (interscript/interscript-api)

  • Audit the repo against the agreed architecture (code-only npm package, maps as deployment artifacts, no-regex engine directive, bundled maps surface, ml surface). Produce a status table: done / partial / missing.
  • Complete the missing pieces that are code-only (worker code, wrangler config with generated types, package guard tests, package.json exports/files hygiene). Deployment itself (wrangler deploy, secrets) stays owner-held — record the exact deploy commands, do not run them.

Rules: PRs only; the benchmarks PR includes the scripts + page + one CI-green regeneration; the API PR includes the status table in the body. Branches: wo20-benchmarks, wo20-api-status.

Activity

  1. ronaldtse commented on Nov 16, 2026

    @ronaldtse
    ContributorAuthor

    WO20 milestone: deliverable B (API plane) — PR open

    Audit + code-only completion is up as interscript/api#30 (branch wo20-api-status).

    Key findings from the audit:

    • CI on the API plane's main has been red since the WO15 merge — src/ml.ts imports createPlaneModel from interscript/ml, but the pinned engine ^4.0.0 predates the plane module. The PR bumps the engine to ^5.14.0 (parity gate stays byte-identical, 48/48 green) — this was the one worker-code gap that actually broke the plane.
    • The software repo had no wrangler config and no generated types: the PR adds the canonical self-deploy wrangler.jsonc + committed worker-configuration.d.ts (npm run gen:types) with a CI freshness gate; @cloudflare/workers-types is superseded.
    • Package hygiene: CI now builds before vitest, the wrangler dry-run smoke is strict (no more || true), and test/package.test.ts grows phantom-entry / shipped-.d.ts-closure / manifest entry-point guards.
    • Owner-held (recorded in the PR, not run): wrangler deploy paths, ML_TOKEN secret, and the src/generated/models.json snapshot refresh (index v5 → current models.yaml is a release-round action).
    • Note: the issue names interscript/interscript-api, which is the legacy Ruby Lambda (rollback plane). The agreed Cloudflare architecture lives in interscript/api (software) + interscript/api.interscript.org (deployment); the PR body's status table audits the whole plane and leaves the legacy repo unchanged.

    Deliverable A (benchmarks page, branch wo20-benchmarks) is in flight: scripts checked in, clean serial regeneration running on fresh origin/main clones of all four runtimes (the local interscript-ts/interscript-ruby/interscript-rs checkouts sit on diverged branches, so the run uses fresh clones and records every tree's SHA in the generated data). PR to follow.

  2. ronaldtse commented on Nov 16, 2026

    @ronaldtse
    ContributorAuthor

    WO20 complete — both deliverables landed as PRs

    Deliverable A: benchmarks page — #239 (wo20-benchmarks)

    • Checked-in reproducible scripts under scripts/benchmarks/: per-runtime corpus parse (median-of-3, the parity-README methodology), golden replay (all 12,839 rows of the RS golden capture), and transliteration throughput on a committed fixed input set (every 25th golden row, sampled once and frozen). generate.mjs runs every leg strictly serially and writes src/data/benchmarks.json — numbers are script-generated, never hand-edited.
    • /benchmarks page renders the current-HEAD numbers, machine profile, per-tree commit provenance, exact reproduce commands, and keeps the parity-README audit-day table (2026-11-13) frozen as the baseline. Nav link added.
    • The scripts require sibling checkouts of the four runtimes + maps (env-overridable INTERSCRIPT_*/RUBY_BIN); the requirement is stated in scripts/benchmarks/README.md and on the page. The first run used fresh origin/main clones — the long-lived local interscript-ts/interscript-ruby/interscript-rs checkouts sit on diverged branches, so every tree's SHA is recorded in the generated data.
    • One CI-green regeneration included (astro check 0 errors, build green).

    Current-HEAD headline numbers (median of 3, serial, Apple M1 Max, ruby 3.4.8):

    leg rs 0.3.2 ts 5.10.0 ruby 3.7.2 py 0.1.0 (ISC)
    corpus parse, 428 maps 26.4 s 15.3 s 167.4 s 4.7 s
    golden replay, 12,839 rows 66.6 s (capture) 7.1 s, byte-exact 7.6 s, byte-exact 61.8 s, 12,836 exact
    fixed-set throughput 193 rows/s (derived) 2,236 rows/s 2,191 rows/s 74 rows/s

    Deliverable B: API plane — interscript/api#30 (wo20-api-status), CI green

    Status audit table in the PR body; code-only completion landed. The headline: CI on the API plane's main had been red since the WO15 merge — src/ml.ts imports createPlaneModel from interscript/ml while the pinned engine ^4.0.0 has no plane module, and vitest.config.ts aliased that import to a sibling ../interscript-ts checkout that doesn't exist on CI runners. Fixed by bumping the engine to ^5.14.0 and dropping the alias (48/48 tests against the published package, parity gate byte-identical). Also: canonical self-deploy wrangler.jsonc + generated worker-configuration.d.ts with a CI freshness gate, package-guard tests (phantom files, shipped-.d.ts closure, manifest entry points), strict wrangler dry-run smoke, onnxruntime-node devDep for the plane decode tests. Deployment stays owner-held; the exact wrangler deploy/e2e commands are recorded in the PR body — nothing was deployed.

    Findings for the owner (flagged, not fixed)

    1. py runtime divergence: the py ISC engine misses 3 of 12,839 golden rows — the Ў rows of gki-bel/mvd-bel/odni-bel (maps v3.2.3's newest exhaustive per-row tests); ts and ruby replay them byte-exact. Recorded on the benchmarks page.
    2. Ruby bare-parse sweep is toolchain-sensitive: on ruby 4.0.7 with the interscript/interscript-maps gems installed, 120/428 corpus maps fail with MapNotFoundError at parse-time dependency locate (31 remain with an explicit load path); on CI-covered ruby 3.4.8 without those gems, 428/428. Benchmarks pin RUBY_BIN and record the binary; root cause flagged for investigation.
    3. interscript-py main cannot run the ISC corpus at all (legacy .imp engine); all ISC py numbers come from the unmerged wo17/py-override-placement branch of interscript-ruby.
    4. interscript/api's src/generated/models.json snapshot (index v5) is behind the current models registry — refresh is a release-round action alongside the deployment repo pin bump.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions