Skip to content

Latest commit

 

History

History

models

Folders and files

NameName
Last commit message
Last commit date

parent directory

..
 
 
 
 

README.md

models/

Your own training scripts live here. Cloned upstream repos (adapters/_repos/cifar10-airbench, cifar10-speedrun, hlb-CIFAR10) are not in this directory — those are gitignored and live under adapters/_repos/ instead. This folder is for code you write and want checked into version control.

Adding a new model

  1. Drop a Python script into this directory (e.g. models/my_fast_resnet.py).

  2. Make sure it prints a line scriptbench can parse for the headline metric:

    • Speedrun (target accuracy → wallclock): print something like final acc=0.945 wall=2.50s near the end. The regexes you supply to scriptbench add extract those numbers.
    • Accuracy (fixed budget → peak accuracy): print peak acc=0.873 (or however you name it).
  3. Register the script:

    uv run scriptbench add \
        --name my_fast_resnet \
        --script models/my_fast_resnet.py \
        --kind speedrun \
        --target-acc 0.94 \
        --accuracy-regex 'acc=([0-9.]+)' \
        --time-regex 'wall=([0-9.]+)s'

    For an accuracy benchmark instead, use --kind accuracy --epoch-budget 50 and skip --target-acc / --time-regex.

  4. Run it:

    uv run scriptbench run --models my_fast_resnet --runs 3
  5. Compare against a baseline:

    uv run scriptbench compare \
        --baseline airbench --challenger my_fast_resnet \
        --metric time_to_target_seconds --runs 3

Regex tips

  • The CLI extracts the last match of your regex in stdout, so prints during training (epoch progress) won't override the final reading.
  • One capture group per regex. acc=([0-9.]+) good; acc=[0-9.]+ bad.
  • If your script also prints a wallclock-to-target, pass --time-regex. scriptbench will use it as time_to_target_seconds instead of measuring end-to-end wallclock.

Cache invalidation

scriptbench fingerprints your script's bytes plus the regex/kind config. Edit the script → cache misses on next run. Edit the regex (via scriptbench remove then add) → likewise. The version field on the adapter is derived from a hash of the registration so editing your regex bumps the fingerprint without you having to think about it.

What goes here vs. adapters/_repos/

models/ adapters/_repos/
Source You — committed to this repo Upstream third parties — cloned, gitignored
How it gets here You write it ./scripts/setup_adapters.sh
Modify freely? Yes No (DESIGN §3.3 — honor the original recipe)
Registered with scriptbench? Manually via scriptbench add Automatic (built-in adapters airbench/hiverge/hlb)