nbrightandClaude Opus 5
3fdd7ab3f0
The target machine's card is busy with someone else's job, so a single end-to-end script stalls on work that does not actually need a GPU. Compiling the CUDA extensions needs nvcc, not a device, and downloading 18 GB of data needs neither. Those are the slow parts (~50 min + ~30 min), so phase A now runs entirely without the card: run_setup.sh bootstrap, conda, extensions, patches, data no GPU run_train.sh voxel_max measurement, training, evaluation GPU run_setup reports the GPU but never fails on it, and verify_env.py gained SKIP_CUDA_CHECK so import coverage still runs when no device is visible. TORCH_CUDA_ARCH_LIST is stated rather than probed, since the card may be unavailable at build time. run_train waits for the GPU instead of failing when it is busy: it polls until enough VRAM frees up (12h default), so it can be queued ahead of time. Past the deadline it proceeds anyway and lets the measured voxel_max adapt to whatever is actually free. keepalive.sh now takes the phase to supervise. Replaces run_all.sh and RUN.md with SETUP.md and TRAIN.md. Adds selfcheck.sh, which syntax-checks every script and flags CRLF endings - a shell script with either fails at its first line, which for an unattended weekend run means losing the weekend. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
38 lines
1008 B
Bash
38 lines
1008 B
Bash
#!/usr/bin/env bash
|
|
# SUM Parts - syntax-check every script in this directory
|
|
#
|
|
# Cheap guard: a shell script with a syntax error fails at the first line it
|
|
# reaches, which for an unattended weekend run means losing the weekend.
|
|
set -uo pipefail
|
|
|
|
cd "$(dirname "${BASH_SOURCE[0]}")"
|
|
|
|
fail=0
|
|
|
|
echo "=== bash ==="
|
|
for f in *.sh; do
|
|
printf ' %-26s' "$f"
|
|
if bash -n "$f" 2>/dev/null; then echo "OK"; else echo "SYNTAX ERROR"; fail=1; fi
|
|
done
|
|
|
|
echo
|
|
echo "=== python ==="
|
|
for f in *.py; do
|
|
printf ' %-26s' "$f"
|
|
if python3 -c "import ast,sys; ast.parse(open(sys.argv[1],encoding='utf-8').read())" "$f" 2>/dev/null
|
|
then echo "OK"; else echo "SYNTAX ERROR"; fail=1; fi
|
|
done
|
|
|
|
echo
|
|
echo "=== CRLF check (breaks bash on Linux) ==="
|
|
if grep -rlU $'\r' ./*.sh ./*.py 2>/dev/null; then
|
|
echo " ^ these have CRLF line endings and will fail with: bash: \$'\\r': command not found"
|
|
fail=1
|
|
else
|
|
echo " clean"
|
|
fi
|
|
|
|
echo
|
|
[ "$fail" -eq 0 ] && echo "ALL OK" || echo "PROBLEMS FOUND"
|
|
exit "$fail"
|