nbrightandClaude Opus 5
7a6b850ae6
Built for running over a weekend with nobody at the keyboard. run_all.sh chains every phase from bare machine to trained model and retries each one with exponential backoff. Preflight is deliberately not retried: a missing HF token or absent GPU will not fix itself, and burning the weekend on a doomed retry loop is worse than failing in the first minute. keepalive.sh sits above it and relaunches run_all if the process disappears entirely (VM restart, OOM kill). Safe because every phase is idempotent -- a restart re-checks what is already done and continues, and do_train hands the newest checkpoint to the watchdog so training resumes instead of starting over. voxel_max is measured rather than assumed: candidates are tried high to low and the first that actually fits in VRAM wins. On WSL2 an oversized value does not OOM, it silently spills to host RAM at 25-100x the cost, so peak allocation is checked instead of trusting that the run worked. RUN_ALL_DRYRUN runs preflight alone, to prove the checks pass before leaving. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
84 lines
2.4 KiB
Bash
84 lines
2.4 KiB
Bash
#!/usr/bin/env bash
|
|
# SUM Parts - keep run_all.sh alive across anything that kills it
|
|
#
|
|
# run_all.sh already retries individual phases, and train_watchdog resumes
|
|
# training from its checkpoint. This is the layer above both: it restarts
|
|
# run_all itself if the whole process disappears -- a WSL VM restart, an OOM
|
|
# kill, a stray pkill.
|
|
#
|
|
# That is safe because every phase is idempotent. A restart re-checks what is
|
|
# already done (conda env, patches, downloaded archives, training checkpoint)
|
|
# and continues from there rather than redoing it.
|
|
#
|
|
# Stops when:
|
|
# - STATUS says DONE -> success, exits 0
|
|
# - STATUS says FAILED -> a hard error like a missing HF token;
|
|
# retrying cannot fix it, exits 1
|
|
# - MAX_RESTARTS reached -> exits 1
|
|
#
|
|
# Usage (this is the one command to run before leaving):
|
|
# setsid nohup bash scripts/keepalive.sh > ~/keepalive.out 2>&1 &
|
|
#
|
|
# Check on it:
|
|
# cat ~/sum-parts/runs/run_all/STATUS
|
|
# tail -f ~/sum-parts/runs/run_all/run.log
|
|
set -uo pipefail
|
|
|
|
SCRIPTS="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
|
OUT="$HOME/sum-parts/runs/run_all"
|
|
STATUS="$OUT/STATUS"
|
|
KLOG="$OUT/keepalive.log"
|
|
|
|
MAX_RESTARTS="${MAX_RESTARTS:-40}"
|
|
COOLDOWN="${COOLDOWN:-90}"
|
|
|
|
mkdir -p "$OUT"
|
|
|
|
klog() { echo "[$(date '+%F %T')] $*" | tee -a "$KLOG"; }
|
|
|
|
state_of() {
|
|
[ -f "$STATUS" ] || { echo "none"; return; }
|
|
grep -E '^state' "$STATUS" | head -1 | cut -d: -f2- | tr -d ' '
|
|
}
|
|
|
|
klog "keepalive starting (max $MAX_RESTARTS restarts, ${COOLDOWN}s cooldown)"
|
|
|
|
restarts=0
|
|
while :; do
|
|
s=$(state_of)
|
|
case "$s" in
|
|
DONE)
|
|
klog "run_all reports DONE -- finished"
|
|
exit 0
|
|
;;
|
|
FAILED)
|
|
klog "run_all reports FAILED -- a hard error that restarting will not fix:"
|
|
sed 's/^/ /' "$STATUS" | tee -a "$KLOG"
|
|
exit 1
|
|
;;
|
|
esac
|
|
|
|
if pgrep -f "run_all.sh" | grep -qv "$$"; then
|
|
sleep 30
|
|
continue
|
|
fi
|
|
|
|
if [ "$restarts" -ge "$MAX_RESTARTS" ]; then
|
|
klog "hit MAX_RESTARTS=$MAX_RESTARTS -- stopping"
|
|
exit 1
|
|
fi
|
|
|
|
if [ "$restarts" -gt 0 ]; then
|
|
klog "run_all is not running (state='$s') -- restart #$restarts"
|
|
else
|
|
klog "launching run_all"
|
|
fi
|
|
|
|
bash "$SCRIPTS/run_all.sh"
|
|
rc=$?
|
|
klog "run_all exited rc=$rc, state='$(state_of)'"
|
|
|
|
restarts=$((restarts + 1))
|
|
sleep "$COOLDOWN"
|
|
done
|