[Adambomb210]

Two cheap ways to find a job's mistakes before paying for the whole run.

krater run: on your own computer, free

Runs a job in Docker, set up as a Krater machine sets it up (cli/src/krater_cli/local.py):

• --entrypoint sh and sh -c <command> in the image's working directory, on --platform linux/amd64;
• the KRATER_* variables, plus KRATER_LOCAL=1;
• krater-local/<name>/outputs and checkpoints mounted at /krater/....

It takes submit's options and job files, ignoring price and GPU options. It also has:

• --index to pick which sweep job runs;
• --input-dir for a local folder that stands in for $KRATER_INPUT_DIR (read-only);
• --resumed, --local-gpu and --docker podman;
• --smoke, which stops the command after N minutes, as on Krater.

Project secrets that are set in the shell are passed by name (-e NAME, so values never appear in argv). Secrets that aren't set are listed. It explains exit codes 125, 126, 127 and 137, and needs no sign-in unless given a project.

krater submit --smoke: on the real machine, for a few cents

The form has the same option as a tick box, "Smoke test first". --smoke-minutes N sets the length (1–30, default 5).

How a smoke test runs:

• The runner gets smoke_seconds from /start and sets KRATER_SMOKE=1.
• It stops the command after N minutes (SIGTERM, then SIGKILL), uploads outputs and the log as usual, then reports /exit with smoke_stopped: true.
• Krater counts that as succeeded: "Smoke test passed: the command ran 5 minutes without failing." Only a smoke job's runner can claim this.
• A command that fails sooner fails as usual, with "Smoke test failed: ...".

What else changes:

• The timer starts with the command, after the pull and restores.
• Backstop: the scheduler ends the machine at N + 10 minutes of running time (krater/services/smoke.py), plus a minute per GB of inputs. The budget hold and the estimate use the same figure.
• The job keeps the member's max runtime, wait and price. krater rerun <job> --no-smoke therefore queues exactly the full run, and --smoke makes any rerun a smoke test.
• A sweep's smoke test queues only its first job. It then says to run the same submit without --smoke.
• Against a Krater from before smoke tests (which would ignore the field), the CLI cancels the job at once and says so.
• The job page shows a "Smoke test" row, krater jobs marks it, and "Run again" keeps the tick.

Migration b7e3f1a9c2d5 adds the nullable column compute_jobs.smoke_minutes; it only adds. CLI is now 0.4.0. Docs are updated in docs/compute/cli.md, docs/compute/runner.md and the members' guide (new section "Trying it before paying", plus KRATER_SMOKE and KRATER_LOCAL in the variables table).

If this is reverted: the old code would run any queued smoke jobs as full-length jobs. They keep the member's max runtime, and the old runner has no timer.

Checked

• Server: full suite, 1933 passed (local Postgres 16). The smoke tests, help, scheduler and cost tests were rerun after the review fixes: 173 passed.
• CLI: 183 passed.
• Runner: Go vet and test -race pass, gofmt is clean, and there are 3 new tests (the timer stops and reports, an earlier failure is the command's own, and KRATER_SMOKE is unset otherwise).
• alembic upgrade head and alembic check are clean.
• krater run was tried live with Docker Desktop:
  • python:3.12-slim: inputs read and output written, exit 0;
  • a typo'd command: exit 127, with the hint;
  • alpine with --smoke-minutes 1: SIGTERM at 60 s, "Smoke test passed", exit 0, no container left.
• Not checked yet: a smoke test on a real Vast machine, where the runner's timer reports smoke_stopped to a real portal. That needs one paid staging run (about a cent: sleep 600 with --smoke-minutes 1).

🤖 Generated with Claude Code