[GavinDPinto]

Closes #30. Closes #31. Part of #22.

watcloud slurm run builds the Slurm request from plain flags, prints it, and runs it.

watcloud slurm run --gpu-mem 8G --disk 40G --mem 16G --time 4h --docker
watcloud slurm run --batch --gpu-mem 8G --mem 16G --time 12h -- python train.py

What it does

• Flags to srun/sbatch: --cpus, --mem, --disk (tmpdisk), --gpu-mem (shard), --gpus/--gpu-type, --time, --partition. --dry-run prints the command instead.
• --docker starts dockerd in the job and sets DOCKER_HOST. It defaults --disk to 20 GiB instead of the silent 100 MiB.
• --batch submits with sbatch: a command through --wrap, or a script file. It allows compute_dense, needs a time limit (from --time or the script), and prints the job ID, output file and a watcloud subscription hint.
• A script's #SBATCH lines still apply and count in the checks. Flags override them one at a time: --disk replaces only the script's tmpdisk and keeps its GPU. When sbatch won't read the script (bash train.sh, or inside the Docker wrapper), its #SBATCH lines are passed on the command line.
• Checks before submitting:
  • Sizes and times need a unit.
  • Requests no node could ever run are rejected, naming the limit hit. Node sizes come from scontrol and count only what a job can get.
  • A missing script path, such as a typo, fails at submit instead of after the queue wait.
  • A job without --mem is told it gets 512 MiB per CPU; GPU and Docker jobs get a warning.
  • The rules in job_submit.lua are applied up front: no interactive jobs on compute_dense, a warning past 6 h.
  • If Slurm doesn't answer within 5 s, it warns that the checks were skipped.
• watcloud docker start -h now points to slurm run --docker.

Testing

• Unit tests in internal/slurm: parsing, #SBATCH lines, the srun/sbatch lines, node checks against the cluster's real node sizes, and the Slurm timeout. go test ./... passes on Linux.
• Use cases run on the cluster are written up in this comment: 35 on 9 Nov, and the review's batch-script cases again on 11 Nov. Highlights:
  • A CUDA image pull fails with raw srun's 100 MiB default and works with --docker.
  • A 6.14 GB PyTorch image fits in the 20 GiB default.
  • 13 common mistakes are stopped before submitting.
  • Script jobs get what the script and flags ask for together, per sacct. For example, --disk 40G on a script with a GPU share gives gres/shard=8192,gres/tmpdisk=40960.
  • Failed, timed-out and queued batch jobs are reported the same way as with plain sbatch.

🤖 Generated with Claude Code

[GavinDPinto]

EDIT: This doc is not up to date, see latest comment on this PR for updated doc
slurm-run-use-case-testing.md

The above doc has some information on the common use-cases for the command as well as some incorrect usage and the associated error messages outputted.

[feinburger244]

Interactive mode is fine. Batch mode with a script file has four problems that block merge. The use-case doc doesn't hit them, because its script case only sets --job-name and --output.

1. --disk, --gpu-mem and --gpus wipe the script's #SBATCH --gres line.
A command-line --gres replaces the script's whole value; Slurm doesn't merge them. The README and --help say flags override #SBATCH, with no exception.

• With #SBATCH --gres shard:8192,tmpdisk:20480 in the script, --batch --disk 40G --time 12h -- train.sh submits --gres tmpdisk:40960. The job gets no GPU and no warning.
• --gpu-mem 8G on a script with #SBATCH --gres tmpdisk:1024 drops /tmp to 100 MiB.

2. The checks ignore the script's directives.

• The docs' compute_dense example sets --partition and --time. --batch -- slurm_compute_dense_partition.sh fails with "--batch needs --time", and adding --time 2d fails with "the longest job on compute is 1 d".
• A script that sets tmpdisk still gets the "/tmp gets the 100 MiB default" note.

3. Some mistakes fail only after the queue wait.

• -- ./trian.sh (a typo) goes through --wrap and exits 127 when the job starts. sbatch ./trian.sh fails at submit.
• -- bash train.sh goes through --wrap, so the script's #SBATCH lines are silently ignored, with the same no-GPU result as 1.

4. Without --mem, a job gets 512 MiB.
DefMemPerCPU=512, so the README's --batch --gpu-mem 8G --time 12h -- python train.py and --gpus 1 --time 1h -- python train.py each get 1 CPU and 512 MiB, with no warning.

Minor

• sinfo's %c and %m are totals, and CoreSpecCount and MemSpecLimit=2048 reserve part of them. The 2080 Ti nodes give jobs 5 CPUs, yet --cpus 6 --gpus 1 --gpu-type rtx_2080_ti passes the check.
• With --docker, the script's --output is ignored, but no output path is printed. With --array, the printed path is wrong; the files are slurm-<id>_<index>.out.
• Shebang scripts run from Slurm's spool copy, so import utils and $(dirname "$0") don't find the script's directory. This isn't documented.
• --gpu-type rtx_4090 --gpu-mem 24G is rejected with "the largest has 24.0 GiB", though the card has 24564 MiB. The "one 4090 share" test passes because of the rtx_3090 nodes, not the 4090.

No CI runs the tests, so confirm go test ./... passes.

[GavinDPinto]

@feinburger244 thanks, all four are fixed in 4ff1f72, plus the minor ones:

1. Flags now merge into the script's --gres instead of replacing it: --disk 40G on #SBATCH --gres shard:8192,tmpdisk:20480 submits shard:8192,tmpdisk:40960.
2. Checks use the script's --partition/--time/--mem/--cpus-per-task/--gres, so the compute_dense script needs no flags.
3. A missing script path fails at submit. For bash train.sh (and scripts inside the Docker wrapper), the script's #SBATCH lines are passed on the command line, since sbatch never reads them.
4. Jobs without --mem are told the default (DefMemPerCPU × CPUs), as a warning for GPU/Docker jobs; README examples pass --mem.

Minor: node sizes now come from scontrol (CPUEfctv, RealMemory − MemSpecLimit, partition membership), array output paths are right, sizes print exactly (24564 MiB), and the spool-copy behaviour is documented. go test ./... passes on Linux.

Retested on the cluster, per sacct:

• --batch --disk 40G -- ./train.sh (script: shard:8192,tmpdisk:20480) → gres/shard=8192,gres/tmpdisk=40960; the job had a GPU and a 40G /tmp
• --batch -- ./dense.sh (script sets compute_dense, 2 days) → ran on compute_dense with a 2-00:00:00 limit
• --batch --time 10m -- bash train.sh → gres/shard=8192,gres/tmpdisk=20480,mem=8G
• -- ./trian.sh → ./trian.sh: no such file, at submit
• --cpus 6 --gpus 1 --gpu-type rtx_2080_ti → rejected; --gpu-mem 24G --gpu-type rtx_4090 → "the largest has 24564 MiB"
• --gpu-mem 8G without --mem → warns "the job gets 512 MiB (512 MiB per CPU)"

Updated use-case write-up, with a new "Batch scripts after the review" section:

slurm-run-use-case-testing.md