Split the unattended run into a GPU-free phase and a GPU phase
The target machine's card is busy with someone else's job, so a single end-to-end script stalls on work that does not actually need a GPU. Compiling the CUDA extensions needs nvcc, not a device, and downloading 18 GB of data needs neither. Those are the slow parts (~50 min + ~30 min), so phase A now runs entirely without the card: run_setup.sh bootstrap, conda, extensions, patches, data no GPU run_train.sh voxel_max measurement, training, evaluation GPU run_setup reports the GPU but never fails on it, and verify_env.py gained SKIP_CUDA_CHECK so import coverage still runs when no device is visible. TORCH_CUDA_ARCH_LIST is stated rather than probed, since the card may be unavailable at build time. run_train waits for the GPU instead of failing when it is busy: it polls until enough VRAM frees up (12h default), so it can be queued ahead of time. Past the deadline it proceeds anyway and lets the measured voxel_max adapt to whatever is actually free. keepalive.sh now takes the phase to supervise. Replaces run_all.sh and RUN.md with SETUP.md and TRAIN.md. Adds selfcheck.sh, which syntax-checks every script and flags CRLF endings - a shell script with either fails at its first line, which for an unattended weekend run means losing the weekend. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -0,0 +1,37 @@
|
||||
#!/usr/bin/env bash
|
||||
# SUM Parts - syntax-check every script in this directory
|
||||
#
|
||||
# Cheap guard: a shell script with a syntax error fails at the first line it
|
||||
# reaches, which for an unattended weekend run means losing the weekend.
|
||||
set -uo pipefail
|
||||
|
||||
cd "$(dirname "${BASH_SOURCE[0]}")"
|
||||
|
||||
fail=0
|
||||
|
||||
echo "=== bash ==="
|
||||
for f in *.sh; do
|
||||
printf ' %-26s' "$f"
|
||||
if bash -n "$f" 2>/dev/null; then echo "OK"; else echo "SYNTAX ERROR"; fail=1; fi
|
||||
done
|
||||
|
||||
echo
|
||||
echo "=== python ==="
|
||||
for f in *.py; do
|
||||
printf ' %-26s' "$f"
|
||||
if python3 -c "import ast,sys; ast.parse(open(sys.argv[1],encoding='utf-8').read())" "$f" 2>/dev/null
|
||||
then echo "OK"; else echo "SYNTAX ERROR"; fail=1; fi
|
||||
done
|
||||
|
||||
echo
|
||||
echo "=== CRLF check (breaks bash on Linux) ==="
|
||||
if grep -rlU $'\r' ./*.sh ./*.py 2>/dev/null; then
|
||||
echo " ^ these have CRLF line endings and will fail with: bash: \$'\\r': command not found"
|
||||
fail=1
|
||||
else
|
||||
echo " clean"
|
||||
fi
|
||||
|
||||
echo
|
||||
[ "$fail" -eq 0 ] && echo "ALL OK" || echo "PROBLEMS FOUND"
|
||||
exit "$fail"
|
||||
Reference in New Issue
Block a user