The target machine's card is busy with someone else's job, so a single
end-to-end script stalls on work that does not actually need a GPU.
Compiling the CUDA extensions needs nvcc, not a device, and downloading 18 GB
of data needs neither. Those are the slow parts (~50 min + ~30 min), so phase A
now runs entirely without the card:
run_setup.sh bootstrap, conda, extensions, patches, data no GPU
run_train.sh voxel_max measurement, training, evaluation GPU
run_setup reports the GPU but never fails on it, and verify_env.py gained
SKIP_CUDA_CHECK so import coverage still runs when no device is visible.
TORCH_CUDA_ARCH_LIST is stated rather than probed, since the card may be
unavailable at build time.
run_train waits for the GPU instead of failing when it is busy: it polls until
enough VRAM frees up (12h default), so it can be queued ahead of time. Past the
deadline it proceeds anyway and lets the measured voxel_max adapt to whatever
is actually free.
keepalive.sh now takes the phase to supervise. Replaces run_all.sh and RUN.md
with SETUP.md and TRAIN.md. Adds selfcheck.sh, which syntax-checks every script
and flags CRLF endings - a shell script with either fails at its first line,
which for an unattended weekend run means losing the weekend.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Built for running over a weekend with nobody at the keyboard.
run_all.sh chains every phase from bare machine to trained model and retries
each one with exponential backoff. Preflight is deliberately not retried: a
missing HF token or absent GPU will not fix itself, and burning the weekend on
a doomed retry loop is worse than failing in the first minute.
keepalive.sh sits above it and relaunches run_all if the process disappears
entirely (VM restart, OOM kill). Safe because every phase is idempotent -- a
restart re-checks what is already done and continues, and do_train hands the
newest checkpoint to the watchdog so training resumes instead of starting over.
voxel_max is measured rather than assumed: candidates are tried high to low and
the first that actually fits in VRAM wins. On WSL2 an oversized value does not
OOM, it silently spills to host RAM at 25-100x the cost, so peak allocation is
checked instead of trusting that the run worked.
RUN_ALL_DRYRUN runs preflight alone, to prove the checks pass before leaving.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
