nbrightandClaude Opus 5
7a6b850ae6
Built for running over a weekend with nobody at the keyboard. run_all.sh chains every phase from bare machine to trained model and retries each one with exponential backoff. Preflight is deliberately not retried: a missing HF token or absent GPU will not fix itself, and burning the weekend on a doomed retry loop is worse than failing in the first minute. keepalive.sh sits above it and relaunches run_all if the process disappears entirely (VM restart, OOM kill). Safe because every phase is idempotent -- a restart re-checks what is already done and continues, and do_train hands the newest checkpoint to the watchdog so training resumes instead of starting over. voxel_max is measured rather than assumed: candidates are tried high to low and the first that actually fits in VRAM wins. On WSL2 an oversized value does not OOM, it silently spills to host RAM at 25-100x the cost, so peak allocation is checked instead of trusting that the run worked. RUN_ALL_DRYRUN runs preflight alone, to prove the checks pass before leaving. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
8 lines
321 B
Bash
8 lines
321 B
Bash
#!/usr/bin/env bash
|
|
# SUM Parts - run only run_all.sh's preflight, to prove it before leaving
|
|
#
|
|
# Sources run_all.sh with RUN_ALL_DRYRUN set so the phase list is skipped and
|
|
# only the checks execute. Cheap, no side effects, no GPU work.
|
|
set -uo pipefail
|
|
RUN_ALL_DRYRUN=1 bash "$(dirname "${BASH_SOURCE[0]}")/run_all.sh"
|