The target machine's card is busy with someone else's job, so a single
end-to-end script stalls on work that does not actually need a GPU.
Compiling the CUDA extensions needs nvcc, not a device, and downloading 18 GB
of data needs neither. Those are the slow parts (~50 min + ~30 min), so phase A
now runs entirely without the card:
run_setup.sh bootstrap, conda, extensions, patches, data no GPU
run_train.sh voxel_max measurement, training, evaluation GPU
run_setup reports the GPU but never fails on it, and verify_env.py gained
SKIP_CUDA_CHECK so import coverage still runs when no device is visible.
TORCH_CUDA_ARCH_LIST is stated rather than probed, since the card may be
unavailable at build time.
run_train waits for the GPU instead of failing when it is busy: it polls until
enough VRAM frees up (12h default), so it can be queued ahead of time. Past the
deadline it proceeds anyway and lets the measured voxel_max adapt to whatever
is actually free.
keepalive.sh now takes the phase to supervise. Replaces run_all.sh and RUN.md
with SETUP.md and TRAIN.md. Adds selfcheck.sh, which syntax-checks every script
and flags CRLF endings - a shell script with either fails at its first line,
which for an unattended weekend run means losing the weekend.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Built for running over a weekend with nobody at the keyboard.
run_all.sh chains every phase from bare machine to trained model and retries
each one with exponential backoff. Preflight is deliberately not retried: a
missing HF token or absent GPU will not fix itself, and burning the weekend on
a doomed retry loop is worse than failing in the first minute.
keepalive.sh sits above it and relaunches run_all if the process disappears
entirely (VM restart, OOM kill). Safe because every phase is idempotent -- a
restart re-checks what is already done and continues, and do_train hands the
newest checkpoint to the watchdog so training resumes instead of starting over.
voxel_max is measured rather than assumed: candidates are tried high to low and
the first that actually fits in VRAM wins. On WSL2 an oversized value does not
OOM, it silently spills to host RAM at 25-100x the cost, so peak allocation is
checked instead of trusting that the run worked.
RUN_ALL_DRYRUN runs preflight alone, to prove the checks pass before leaving.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
SETUP.md sent a fresh machine straight to setup_env.sh, which calls conda and
builds inside ~/sum-parts. Neither exists on a clean box, so the first command
failed. This repo only carries our scripts; the benchmark itself is a separate
clone.
bootstrap.sh installs miniconda into $HOME (no sudo), clones
tudelft3d/SUM-Parts-Benchmarks, and reports whether the GPU has room for the
paper's voxel_max=64000 (~16.5 GB). Idempotent - an existing install or clone
is reported and left alone.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Reproduces the SUM Parts (CVPR 2025) face-labeling benchmark on a single
consumer GPU, then applies it to drone-photogrammetry road survey meshes.
Verified on RTX 3060 12GB / WSL2 Ubuntu 22.04 / CUDA 11.8 / torch 2.0.1:
- CUDA extensions build (pointnet2_batch, pointops, chamfer_dist, emd,
subsampling)
- PointNet 100 epochs reaches mIoU 17.19, matching the paper's reported 15.1
- OBJ -> PLY conversion round-trips through the model and yields per-point
predictions
Four upstream source patches, all idempotent, originals preserved:
- numpy aliases removed in 1.24 (np.long etc.) and collections ABCs moved in
python 3.10
- the blind test split ships label = -1, which crashed ConfusionMatrix
- mode=val referenced `epoch` before assignment
Documents the traps that cost the most time, including VRAM overflow silently
falling back to host RAM on WSL2 (25-100x slowdown, no OOM) and the colour
scale mismatch between r/g/b float32 and red/green/blue uint8.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
