Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .prettierignore
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,7 @@ test-results/

# Generated files (regenerated by scripts — don't fight their formatting)
src/data/maps-catalogue.json
src/data/benchmarks.json
package-lock.json
public/maps/

Expand Down
5 changes: 3 additions & 2 deletions package.json
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,8 @@
"test:e2e": "playwright test",
"test:e2e:ui": "playwright test --ui",
"generate:catalogue": "tsx scripts/generate-catalogue.mjs",
"generate:models": "tsx scripts/generate-models.mjs"
"generate:models": "tsx scripts/generate-models.mjs",
"generate:benchmarks": "node scripts/benchmarks/generate.mjs"
},
"dependencies": {
"@asciidoctor/core": "^4.0.6",
Expand Down Expand Up @@ -62,4 +63,4 @@
"typescript-eslint": "^8.70.0",
"vitest": "^5.1.0"
}
}
}
90 changes: 90 additions & 0 deletions scripts/benchmarks/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,90 @@
# Benchmarks

Reproducible runtime benchmarks behind the `/benchmarks` page. Every
number on the page comes out of these scripts; the generated data file
(`src/data/benchmarks.json`) is committed but **never hand-edited** —
regenerate it and commit the result.

## Requirements: sibling checkouts

The scripts read the four runtime engines and the map corpus from
sibling checkouts in the monorepo layout (`~/src/interscript/` by
default, one directory above this repo). This is a hard requirement —
the engines under test are the actual runtime trees, not npm installs:

| Tree | Default sibling path | State required |
| ---------------- | ----------------------- | ------------------------------------------------------------------------------------------------------------------------- |
| map corpus | `../maps` | `maps/maps/*.isc` + `maps/libs/*.isc` |
| interscript-rs | `../interscript-rs` | release build: `cargo build --release` |
| interscript-ts | `../interscript-ts` | built dist: `npm ci && npm run build` |
| interscript-ruby | `../interscript-ruby` | lib tree (run with `ruby -I <tree>/lib`) |
| py ISC engine | `../interscript-py-isc` | the `wo17/py-override-placement` branch of the interscript-ruby repo (the py package sits at `interscript-py/` inside it) |

Every location can be overridden with `INTERSCRIPT_MAPS`,
`INTERSCRIPT_RS`, `INTERSCRIPT_TS`, `INTERSCRIPT_RUBY`, and
`INTERSCRIPT_PY` environment variables. The ruby legs use the `ruby`
interpreter by default; override with `RUBY_BIN` when the engine under
test requires a specific toolchain (the interscript-ruby CI matrix
covers 3.3 and 3.4 — run the ruby legs on a covered version; the
machine profile in the generated data records which binary ran).

> The py ISC engine ships on an unmerged branch of the interscript-ruby
> repo; `interscript-py` main still carries the legacy `.imp` engine and
> cannot run the ISC corpus. The generated data records the exact tree
> and commit for every runtime.

Note on rs's engine dependency: the rs checkout pins `parsanol` by an
absolute path to a local checkout. If that checkout does not satisfy the
version requirement, point the dependency at the matching crates.io
release (code-identical) and record the deviation when publishing.

## Methodology

Follows the parity audit's timing protocol
(`maps/docs/parity/README.md` §A.8): median-of-3 runs, **clean serial
execution** — one runtime at a time, nothing else running on the
machine. In-process timing for ruby/ts/py (`Process.clock_gettime` /
`performance.now()` / `perf_counter` around the sweep), wall clock for
the rs CLI legs (`iscrs parse`, `iscrs capture` — adds ~20 ms process
startup). Replay legs do one unmeasured warmup pass before the measured
passes so lazy compilation settles.

Legs:

1. **Corpus parse** — each runtime's native parse entry over every
`maps/maps/*.isc` (py with a dependency resolver over maps+libs, its
bare entry cannot take the corpus alone — parity README G12).
2. **Golden replay** — every row of the RS golden capture
(`interscript-rs/tests/golden/isc-wf0.jsonl`) through each runtime.
rs's number is `iscrs capture` (load + transliterate every corpus
map's embedded tests in one process, includes parse).
3. **Fixed-input throughput** — the committed fixture
(`fixtures/input-set.jsonl`, every 25th golden row, sampled once and
frozen) through each runtime. rs has no batch transliteration CLI, so
its throughput is derived from the capture leg and marked as such.

## Files

- `config.mjs` — sibling-path resolution, machine profile, medians
- `parse-ruby.rb` / `parse-ts.mjs` / `parse-py.py` — corpus parse legs
- `replay-ruby.rb` / `replay-ts.mjs` / `replay-py.py` — jsonl replay
legs (golden + fixture via argument)
- `make-fixture.mjs` — regenerate the fixed input set from the golden
capture (run once; the fixture is committed)
- `generate.mjs` — the orchestrator: runs every leg serially, aggregates
medians, writes `src/data/benchmarks.json`
- `fixtures/input-set.jsonl` — the frozen fixed input set

## Run

```sh
# one-time (or when deliberately re-freezing the input set):
node scripts/benchmarks/make-fixture.mjs

# full regeneration (~30-60 min of clean serial runs):
node scripts/benchmarks/generate.mjs
```

Then commit `src/data/benchmarks.json` together with the runtime
changes it measures. CI runs `astro check` + build, which fail if the
page references data that does not exist.
156 changes: 156 additions & 0 deletions scripts/benchmarks/config.mjs
Original file line number Diff line number Diff line change
@@ -0,0 +1,156 @@
/**
* Shared configuration for the /benchmarks scripts.
*
* All legs read the four runtime engines and the map corpus from
* SIBLING CHECKOUTS of the monorepo layout (see README.md). Paths are
* env-overridable so the scripts stay reproducible on any machine with
* the same layout:
*
* INTERSCRIPT_MAPS maps checkout (maps/maps + maps/libs)
* INTERSCRIPT_RS interscript-rs checkout (release iscrs binary)
* INTERSCRIPT_TS interscript-ts checkout (built dist/)
* INTERSCRIPT_RUBY interscript-ruby checkout
* INTERSCRIPT_PY interscript-py ISC-engine checkout
* (the py/ISC line maintained inside the
* interscript-ruby repo, branch
* wo17/py-override-placement; its Python tree
* sits at <checkout>/interscript-py)
*/

import { execFileSync } from "node:child_process"
import { existsSync, readFileSync } from "node:fs"
import os from "node:os"
import { dirname, resolve } from "node:path"
import { fileURLToPath } from "node:url"

const HERE = dirname(fileURLToPath(import.meta.url))
export const REPO_ROOT = resolve(HERE, "..", "..")

const sibling = (envVar, relative) => {
const fromEnv = process.env[envVar]
if (fromEnv) return resolve(fromEnv)
return resolve(REPO_ROOT, "..", relative)
}

export const SIBLINGS = {
maps: sibling("INTERSCRIPT_MAPS", "maps"),
rs: sibling("INTERSCRIPT_RS", "interscript-rs"),
ts: sibling("INTERSCRIPT_TS", "interscript-ts"),
ruby: sibling("INTERSCRIPT_RUBY", "interscript-ruby"),
py: sibling("INTERSCRIPT_PY", "interscript-py-isc"),
}

export const MAPS_DIR = (s) => resolve(s ?? SIBLINGS.maps, "maps")
export const LIBS_DIR = (s) => resolve(s ?? SIBLINGS.maps, "libs")
export const GOLDEN_JSONL = () => resolve(SIBLINGS.rs, "tests", "golden", "isc-wf0.jsonl")
export const RS_BINARY = () => resolve(SIBLINGS.rs, "target", "release", "iscrs")
export const TS_DIST = () => resolve(SIBLINGS.ts, "dist")
export const RUBY_LIB = () => resolve(SIBLINGS.ruby, "lib")
export const PY_SRC = () => resolve(SIBLINGS.py, "interscript-py", "src")
export const FIXTURE = resolve(HERE, "fixtures", "input-set.jsonl")
/**
* The ruby binary for the ruby legs. Defaults to `ruby`; override when
* the engine under test requires a specific toolchain (the
* interscript-ruby CI matrix covers 3.3/3.4 — run the legs on a
* covered version; the machine profile records which one ran).
*/
export const RUBY_BIN = process.env.RUBY_BIN ?? "ruby"

export function requireSiblings() {
const missing = []
if (!existsSync(MAPS_DIR())) missing.push(`maps corpus: ${MAPS_DIR()}`)
if (!existsSync(RS_BINARY()))
missing.push(`rs release binary (cargo build --release in ${SIBLINGS.rs}): ${RS_BINARY()}`)
if (!existsSync(resolve(TS_DIST(), "isc", "index.js")))
missing.push(`ts built dist (npm ci && npm run build in ${SIBLINGS.ts})`)
if (!existsSync(RUBY_LIB())) missing.push(`ruby lib: ${RUBY_LIB()}`)
if (!existsSync(PY_SRC())) missing.push(`py ISC src: ${PY_SRC()}`)
if (!existsSync(GOLDEN_JSONL())) missing.push(`golden jsonl: ${GOLDEN_JSONL()}`)
if (missing.length > 0) {
console.error(
"Missing sibling checkouts (set INTERSCRIPT_* env vars to point elsewhere):\n " +
missing.join("\n "),
)
process.exit(2)
}
}

/** `git rev-parse HEAD` of a checkout, or null outside a git repo. */
export function headSha(dir) {
try {
return execFileSync("git", ["-C", dir, "rev-parse", "HEAD"], {
encoding: "utf8",
}).trim()
} catch {
return null
}
}

function firstFileLine(path, pattern) {
try {
const text = readFileSync(path, "utf8")
for (const line of text.split("\n")) {
const m = line.match(pattern)
if (m) return m[1]
}
return null
} catch {
return null
}
}

/** Versions carved out of each tree's own metadata (best effort). */
export function runtimeVersions() {
const tsPkg = (() => {
try {
return JSON.parse(readFileSync(resolve(SIBLINGS.ts, "package.json"), "utf8"))
} catch {
return {}
}
})()
return {
rs: firstFileLine(resolve(SIBLINGS.rs, "Cargo.toml"), /^version = "(.+)"$/),
ts: tsPkg.version ?? null,
ruby: firstFileLine(
resolve(SIBLINGS.ruby, "lib", "interscript", "version.rb"),
/VERSION = "(.+)"$/,
),
py: firstFileLine(
resolve(SIBLINGS.py, "interscript-py", "pyproject.toml"),
/^version = "(.+)"$/m,
),
}
}

export function machineProfile() {
const cpu = os.cpus()[0]
return {
os: `${os.type()} ${os.release()}`,
arch: os.arch(),
cpuModel: cpu?.model ?? "unknown",
cores: os.cpus().length,
memoryGb: Math.round(os.totalmem() / 1024 / 1024 / 1024),
node: process.version,
toolchains: {
ruby: toolchainVersion(RUBY_BIN, ["-v"], /ruby (\S+)/),
python: toolchainVersion("python3", ["--version"], /Python (\S+)/),
},
}
}

function toolchainVersion(cmd, args, pattern) {
try {
const out = execFileSync(cmd, args, { encoding: "utf8" })
const m = out.match(pattern)
return m ? m[1] : out.trim()
} catch {
return null
}
}

export function median(values) {
const sorted = [...values].sort((a, b) => a - b)
const mid = Math.floor(sorted.length / 2)
const odd = sorted.length % 2 === 1
return odd ? sorted[mid] : (sorted[mid - 1] + sorted[mid]) / 2
}
Loading
Loading