Repository navigation
jitv2: deterministic benchmarking - #438
Open
jamesbraid wants to merge 4 commits into
Open
jamesbraid wants to merge 4 commits into
jamesbraid wants to merge 4 commits into
Conversation
Wire the deterministic mode into the benchmark runner so a result can be reproduced instruction for instruction. In process it builds the machine with synchronous compile and refuses to run unless the build has ci_clock, whose test device clock counts retired instructions; with --iris it passes --jitv2-compile sync, and the emulator binary must be a ci_clock build. The result is labelled <label>+deterministic, since the rates it reports are constant by construction and must not be mistaken for speed. Without the flag nothing changes.
iris counts every guest instruction executed. The count drives the guest's CP0 Count timer and, under ci_clock, guest timing. The interpreter adds 1 per instruction. Compiled code adds the instructions up while compiling and writes the total to the counter only where a block exits or a branch lands. A path that leaves without writing the total loses those instructions. Two paths did that: - A branch-likely that is not taken. It annuls its delay slot, and the code that wrote the total sat in the slot. The not-taken path dropped every instruction since the last write. fpu/scalar_d counted 17 per loop pass instead of 26. - An exception or bail-out (pending interrupt, busy bus) in the middle of a block. It returned without writing the total so far. Write the pending total on both paths. The writes run only on those exits, so no per-instruction work or new branch is added.
iris counts every guest instruction executed, and the count drives the guest's CP0 Count timer. Compiled code adds instructions up while compiling and writes the total to the counter at certain points. For a block's first instruction that write is sometimes part of the instruction's own code: a CP0 instruction, or an instruction that branches back to the top of a loop. When the run loop jumps into a block from outside, it skipped the part of the first instruction that does the counting. A TLB refill handler starts with a CP0 read and is entered this way on every TLB miss. sys/tlb_miss counted 25 instructions per refill instead of 26. Put the counting where both ways into the block run it. Nothing is added per instruction. The same code moves one step later.
…ired instructions Two identical benchmark runs differed by up to 17% in host instructions retired. Most of that was the suite itself: it sizes each kernel's iteration count from the test device's nanosecond registers, so a different run speed picks different counts and the guest executes a different amount of work. The rest was the compile pool: the guest instruction count per loop pass differs between the interpreter and compiled code, so which engine ran a region when the pool finished it changes what the guest sees, and the FP kernels flip between modes. - [jitv2] compile = "sync" (--jitv2-compile sync) leaves the pool stopped and compiles on the CPU thread at the dispatch gate, with the Codegen the executor already keeps for the monitor's `j2 inline on`. The gate branch on jitv2_inline_compile already existed, so the hot path is unchanged. One CPU only, and not in a lightning build; config validation says so. - In a ci_clock build the test device's nanosecond registers count retired instructions at NS_PER_GUEST_CYCLE, as CP0 Count already does, so a guest that sizes its work by time does the same work every run. Other builds are unchanged. The test runs a hand-assembled guest that counts loop passes until 20 us have gone by on that clock, twice, and requires the same count. It runs in a ci_clock build.
Owner
|
we already have --ci-clock ? or i hallucinated it? we should extend --ci-clock mode to make all clock devices based on the atomic instruction count. |
jamesbraid
force-pushed
the
jit-align-determinism
branch
from
November 7, 2026 03:08
b9b3b01 to
3441f50
Compare
jamesbraid
marked this pull request as ready for review
November 7, 2026 05:05
Owner
|
aww needs rebase |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Two identical benchmark runs never did the same work, so a PR's "-1% on kernel X" could not be told from timing noise. This adds an opt-in deterministic mode so runs can be compared instruction for instruction. Two commits, stacked on #439 (exact guest instruction counts), which this needs.
Why runs differ
Two things change the work a run does:
The mode
Opt-in, decided once at machine build, nothing added to a hot path:
[jitv2] compile = "sync"/--jitv2-compile sync: compile code on the CPU thread when it is first run, instead of on background compile threads. One CPU only, not in alightningbuild (validation says so).ci_clockbuild the test device's nanosecond registers count retired instructions (NS_PER_GUEST_CYCLE), as CP0 Count already does, so the suite sizes its work the same way every run. Other builds are unchanged. No new config key or flag.ci_clockonly the test device moved onto the instruction clock. The other guest clocks (timers, RTCs) are separate and not in this PR.iris-bench run --deterministicselects sync compile (in process, or passed through with--iris) and labels the result+deterministic. It needs aci_clockbuild and refuses to run in process without one. Its rates are constant by construction, so it is for comparing instruction counts and host cost, not speed.Whole suite, n=12 each, ship build, on this stack (#439 plus this PR; the deterministic side is a
ci_clockbuild with--jitv2-compile sync):Host cycles still vary 5 to 11% with the same instructions: that is the machine, not the emulator.
Aligning the start of compiled code blocks to 32 or 64 bytes was tried and rejected. Where a loop head lands inside a block decides per-kernel speed, not where the block starts, and some kernels got slower.
Testing
deterministic_mode_repeats_a_time_sized_guest_exactly(ci_clock) runs a hand-assembled guest twice that counts loop passes until 20 us have passed on the guest clock, and requires the same count (and the one the clock implies).--jitv2-compile, the default is background compile, and sync compile is rejected with more than one CPU.cargo test --features jitv2 --lib(release and debug), with and withoutci_clock: all pass.iris-bench run --deterministic --quicktwice: every kernel's iterations, instruction count and checksum identical.