The target machine's card is busy with someone else's job, so a single
end-to-end script stalls on work that does not actually need a GPU.
Compiling the CUDA extensions needs nvcc, not a device, and downloading 18 GB
of data needs neither. Those are the slow parts (~50 min + ~30 min), so phase A
now runs entirely without the card:
run_setup.sh bootstrap, conda, extensions, patches, data no GPU
run_train.sh voxel_max measurement, training, evaluation GPU
run_setup reports the GPU but never fails on it, and verify_env.py gained
SKIP_CUDA_CHECK so import coverage still runs when no device is visible.
TORCH_CUDA_ARCH_LIST is stated rather than probed, since the card may be
unavailable at build time.
run_train waits for the GPU instead of failing when it is busy: it polls until
enough VRAM frees up (12h default), so it can be queued ahead of time. Past the
deadline it proceeds anyway and lets the measured voxel_max adapt to whatever
is actually free.
keepalive.sh now takes the phase to supervise. Replaces run_all.sh and RUN.md
with SETUP.md and TRAIN.md. Adds selfcheck.sh, which syntax-checks every script
and flags CRLF endings - a shell script with either fails at its first line,
which for an unattended weekend run means losing the weekend.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
SETUP.md walks the phases one at a time, which is what you want when something
broke. It is the wrong document to hand someone leaving for the weekend.
RUN.md is the three commands to run before walking away, what each layer of
retry covers, and what to read on Monday. It states plainly which single step
cannot be automated (the HuggingFace gate needs a browser) and that the gate is
per-account, so copying the token is enough.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Reproduces the SUM Parts (CVPR 2025) face-labeling benchmark on a single
consumer GPU, then applies it to drone-photogrammetry road survey meshes.
Verified on RTX 3060 12GB / WSL2 Ubuntu 22.04 / CUDA 11.8 / torch 2.0.1:
- CUDA extensions build (pointnet2_batch, pointops, chamfer_dist, emd,
subsampling)
- PointNet 100 epochs reaches mIoU 17.19, matching the paper's reported 15.1
- OBJ -> PLY conversion round-trips through the model and yields per-point
predictions
Four upstream source patches, all idempotent, originals preserved:
- numpy aliases removed in 1.24 (np.long etc.) and collections ABCs moved in
python 3.10
- the blind test split ships label = -1, which crashed ConfusionMatrix
- mode=val referenced `epoch` before assignment
Documents the traps that cost the most time, including VRAM overflow silently
falling back to host RAM on WSL2 (25-100x slowdown, no OOM) and the colour
scale mismatch between r/g/b float32 and red/green/blue uint8.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
