[not-matthias]

TLDR: Rounds within one process are correlated (JIT tier-up, heap sizing, allocator state), so walltime results depend on the process they ran in. Each walltime benchmark now runs in 3 fresh processes (forks) by default, and their rounds are pooled into one result (tinybench and Vitest 5).

Fork count, highest priority first:

1. CODSPEED_WALLTIME_FORKS=<n>
2. Vitest 5 per-benchmark: bench(name, { forks }, fn) or bench.compare(..., { forks })
3. Integration option: withCodSpeed(bench, { forks }), codspeedPlugin({ forks })
4. 3

CODSPEED_PROCESS_ISOLATION=false runs everything in one process.

How it works

• The parent re-runs the benchmark command (execPath + execArgv + argv, so the runner's V8 flags survive) once per fork, with CODSPEED_ISOLATED_BENCHMARK=<uri>. The fork measures only that benchmark and writes its samples to a temp file. The parent pools them and writes the walltime result.
• Vitest workers spawn the main process's command, and each fork narrows benchmark discovery to the benchmark's file.

Cost (examples/with-typescript-esm tinybench, 9 benchmarks): 77 s with isolation off, 140 s with 1 fork, 412 s with 3 forks. Each fork pays process startup.

Review notes

• Simulation/memory runs, benchmark.js and Vitest 3/4 don't fork.
• Vitest 5: its own report shows zeroed statistics for forked benchmarks; the CodSpeed results hold the pooled rounds. The test timeout is multiplied by the env or plugin fork count. A per-benchmark forks above that count can still time out.
• Vitest's BenchOptions type doesn't include forks, so the per-benchmark option needs a cast in TypeScript.

Closes COD-3960

[codspeed]

Merging this PR will regress 4 benchmarks

3 warnings, 1 note

⚠️ Walltime results may be unreliable

Some benchmarks used the Walltime instrument outside CodSpeed Macro Runners, which can make the results inconsistent.

Learn more about Macro Runners

See affected benchmarks

⚠️ 14 benchmarks are dominated by system calls

System calls cannot be simulated consistently and are left out of the measure. The reported value understates the real cost. Switch to the Walltime instrument to measure them.

Learn more about system calls

See affected benchmarks

⚠️ 8 benchmarks with a significant change ran on different hardware

Their base and head results were measured on machines with a different CPU, core count or memory, so the changes might not come from the code. Run base and head on the same runner type.

Learn more about Macro Runners

See affected benchmarks

ℹ️ 16 benchmarks have no flamegraph

The flamegraph wasn't generated: most of the time was spent in runtime internals, which would leave nothing useful to analyze.

See affected benchmarks

⚡ 23 improved benchmarks
❌ 4 regressed benchmarks
✅ 243 untouched benchmarks
⏩ 56 skipped benchmarks¹

Warning

Please fix the performance issues or acknowledge them on CodSpeed.

Performance Changes

	Mode	Benchmark	BASE	HEAD	Efficiency
❌	CPU Simulation	test_recursive_fibo_10	339.7 µs	690.8 µs	-50.83%
❌	CPU Simulation	wait 500ms	29.7 ms	53 ms	-43.9%
❌	CPU Simulation	wait 1sec	59.1 ms	97.6 ms	-39.48%
❌	Memory	one	1.5 KB	2 KB	-26.19%
⚡	CPU Simulation	switch 1	651.7 µs	305.2 µs	×2.1
⚡	CPU Simulation	recursive fibo 10	677.4 µs	333.1 µs	×2
⚡	Walltime	switch 2	120 ns	72 ns	+66.67%
⚡	Walltime	test_iterative_fibo_10	120 ns	84 ns	+42.86%
⚡	Walltime	test_iterative_fibo_10	120 ns	84 ns	+42.86%
⚡	Walltime	test sync baz 10	120 ns	84 ns	+42.86%
⚡	CPU Simulation	switch 1	586.1 µs	418.7 µs	+39.97%
⚡	Walltime	switch 2	96 ns	72 ns	+33.33%
⚡	Memory	wait 1sec	28.7 KB	22.7 KB	+26.61%
⚡	Walltime	short body	2 µs	1.6 µs	+25.95%
⚡	Walltime	short body	2 µs	1.6 µs	+25.95%
⚡	Walltime	test sync baz 10	120 ns	96 ns	+25%
⚡	Walltime	short body	2 µs	1.6 µs	+23.31%
⚡	Walltime	short body	1.9 µs	1.6 µs	+23.08%
⚡	CPU Simulation	recursive fibo 10	676.4 µs	557.1 µs	+21.41%
⚡	Walltime	short body	1.9 µs	1.6 µs	+18.66%
...	...	...	...	...	...

ℹ️ Only the first 20 benchmarks are displayed. Go to the app to view all benchmarks.

Tip

Investigate this regression by commenting @codspeedbot fix this regression on this PR, or directly use the CodSpeed MCP with your agent.

---

_{Comparing cod-3960-implement-process-isolation-in-codspeed-node (c376bfd) with main (9c93085)}

Footnotes

1. 56 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩

[greptile-apps]

Confidence Score: 5/5

[Medium impact] The stack appears safe to merge, with this assessment depending on PR #111’s changes to walltime execution.

Summary

Adds fresh-process walltime measurements for Tinybench and Vitest 5, with three forks by default. The parent pools their samples into one CodSpeed result.

• Tinybench walltime benchmarks run in fresh processes and pool their rounds.
• Vitest 5 benchmarks run in fresh processes and pool their rounds.

Diagram

sequenceDiagram
  participant Parent as Benchmark parent
  participant Child as Fresh process
  participant File as Temporary samples
  participant Results as CodSpeed results
  loop Each benchmark and fork
    Parent->>Child: Restart command with selected URI
    Child->>Child: Measure selected benchmark
    Child->>File: Write round samples
    Child-->>Parent: Exit
    Parent->>File: Read samples
  end
  Parent->>Results: Write pooled statistics

Loading

_{Reviews (2) · Last reviewed commit: "feat(vitest): run walltime benchmarks in..." · Reviewed by Greptile}

[GuillaumeLagrange]

olgtm

[adriencaccia]

Main question is cost: with 3 forks by default, this repo's walltime benchmark step goes from 351 s on main to 1408 s on this PR. A run with forks should take about as long as one without; see the threads on the budget per fork, the per-fork overhead and the sample count of fast tinybench benchmarks. Vitest hooks must also keep working as they do today.

[adriencaccia]

Each fork runs the full warmup and measurement budget, so 3 forks by default triple the measured time before any fork startup: this repo's walltime benchmark step goes from 351 s on main to 1408 s here, and test_iterative_fibo_10 measures 3.0 s instead of 1.0 s. A run with forks should take about as long as one without: let's split the budget across the forks.

[adriencaccia]

Vitest hooks must keep working as they do today. Here the parent runs the file's beforeAll and beforeEach before spawning the forks and holds what they opened meanwhile, and each fork runs the whole file again: in CI, hooks.bench.ts ran 10 times for its 3 benchmarks. A beforeAll that starts a server on a fixed port would make the forks fail with EADDRINUSE, and costly setup runs 1 + 3 × (benchmarks in the file) times. Let's keep hooks running once around the benchmarks they wrap, and add a bench whose beforeAll holds an exclusive resource so this stays covered.

[adriencaccia]

Why does test_iterative_fibo_10 record 7.2M rounds of 1 iteration for a 1 s budget? tinybench counts only the measured latencies against time, so the clock reads between 120 ns iterations go unbudgeted: on main this benchmark takes 21 s, and the 8 benchmarks under 250 ns of with-typescript-esm and with-typescript-cjs take 135 s of the 351 s walltime step. Forks multiply that (584 s here) and send every sample through JSON, which throws Invalid string length past ~23M samples (tinybench alone records 66.8M for this function in 1 s on my laptop). Let's batch iterations into rounds of at least 1 ms, like pytest-codspeed's min_round_time_ns.

[adriencaccia]

COD-3960 caps the fork overhead at 1.5 s. In CI a fork adds 0.8 to 1.9 s to a tinybench benchmark, 1.3 to 4 s to a Vitest one (parsePr.bench.ts short body: 5.6 s per fork against 1.6 s on main), and 5 to 9 s to the benchmarks with millions of samples. On Vitest it also counts against the test timeout, which is only multiplied by the forks: a group of 3 benchmarks that takes 50 s today needs 3 × (50 + 3 × 4) = 186 s with 3 forks, past the 180 s limit.

[adriencaccia]

CODSPEED_PROCESS_ISOLATION is the variable codspeed-rust reads too, so it stays as the shared convention across integrations. Let's parse it the same way: codspeed-rust panics on anything but 0, false, 1 and true, while here CODSPEED_PROCESS_ISOLATION=off silently keeps the forks on.

[adriencaccia]

The core commit message calls this Isolation.walltime(forks?); it is walltimeIsolation(forks?).