Your own training scripts live here. Cloned upstream repos
(adapters/_repos/cifar10-airbench, cifar10-speedrun, hlb-CIFAR10) are
not in this directory — those are gitignored and live under
adapters/_repos/ instead. This folder is for code you write and want
checked into version control.
-
Drop a Python script into this directory (e.g.
models/my_fast_resnet.py). -
Make sure it prints a line scriptbench can parse for the headline metric:
- Speedrun (target accuracy → wallclock): print something like
final acc=0.945 wall=2.50snear the end. The regexes you supply toscriptbench addextract those numbers. - Accuracy (fixed budget → peak accuracy): print
peak acc=0.873(or however you name it).
- Speedrun (target accuracy → wallclock): print something like
-
Register the script:
uv run scriptbench add \ --name my_fast_resnet \ --script models/my_fast_resnet.py \ --kind speedrun \ --target-acc 0.94 \ --accuracy-regex 'acc=([0-9.]+)' \ --time-regex 'wall=([0-9.]+)s'For an accuracy benchmark instead, use
--kind accuracy --epoch-budget 50and skip--target-acc/--time-regex. -
Run it:
uv run scriptbench run --models my_fast_resnet --runs 3
-
Compare against a baseline:
uv run scriptbench compare \ --baseline airbench --challenger my_fast_resnet \ --metric time_to_target_seconds --runs 3
- The CLI extracts the last match of your regex in stdout, so prints during training (epoch progress) won't override the final reading.
- One capture group per regex.
acc=([0-9.]+)good;acc=[0-9.]+bad. - If your script also prints a wallclock-to-target, pass
--time-regex. scriptbench will use it astime_to_target_secondsinstead of measuring end-to-end wallclock.
scriptbench fingerprints your script's bytes plus the regex/kind config.
Edit the script → cache misses on next run. Edit the regex (via
scriptbench remove then add) → likewise. The version field on the
adapter is derived from a hash of the registration so editing your
regex bumps the fingerprint without you having to think about it.
models/ |
adapters/_repos/ |
|
|---|---|---|
| Source | You — committed to this repo | Upstream third parties — cloned, gitignored |
| How it gets here | You write it | ./scripts/setup_adapters.sh |
| Modify freely? | Yes | No (DESIGN §3.3 — honor the original recipe) |
| Registered with scriptbench? | Manually via scriptbench add |
Automatic (built-in adapters airbench/hiverge/hlb) |