nbrightandClaude Opus 5
f39b093106
Adds STATUS.md as the handoff document: benchmark numbers, the bare-earth
metrics that actually matter for this project, the Korean-data domain gap that
retraining will not fix, and what to do on the 24 GB machine.
The memory problem was in how a prediction's colours were turned back into
class indices. Every consumer built an (N, 13, 3) float64 temporary:
d = ((rgb[:, None, :] - COLOR_MAP[None, :, :]) ** 2).sum(axis=2)
That is ~250 MB of intermediates per 800k-point tile, several live at once, and
a full 4.7M-point block pushes it into gigabytes. main.py writes exact palette
entries, so an exact hash lookup resolves nearly every point with no large
temporary; only leftovers fall back to a chunked distance search. Peak RSS on a
470k-point tile drops to 61 MB. Extracted to sumparts_palette.py and shared by
coarse_eval.py and split_by_class.py.
Also from this round:
- patch_cm_mutation.sh: ConfusionMatrix.update() rewrote the caller's pred
tensor in place, folding every ignore_index point into class num_classes-1.
test() saves its visualization from that same tensor afterwards, so an
unlabelled tile came out 100% wall and the model looked degenerate when it
was not.
- patch_class_mask.sh: SUMPARTS_MASK_CLASSES drops known-absent classes from
the argmax. Measured on Seosan and it does not help - the runner-up for
"water" is "wall", not "terrain" - but the experiment is worth keeping.
- split_by_class.py now writes .ply alongside .obj. A vertex-only OBJ has zero
faces and most viewers render nothing, which is why the first export looked
broken.
- verify_outputs.sh reads exported files back with a parser, so "here are your
files" can be checked rather than asserted.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
52 lines
1.8 KiB
Bash
52 lines
1.8 KiB
Bash
#!/usr/bin/env bash
|
|
# SUM Parts - wait for training to finish, then run every evaluation we care about
|
|
#
|
|
# Runs unattended so nobody has to sit watching for the last epoch:
|
|
# 1. wait for the trainer to exit
|
|
# 2. coarse evaluation on SUM val -> building / vegetation / vehicle / ground
|
|
# 3. inference on the Seosan tile -> does our own data work with this model
|
|
set -uo pipefail
|
|
|
|
SCRIPTS="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
|
OUT="$HOME/sum-parts/runs/after_train"
|
|
LOG="$OUT/after_train.log"
|
|
|
|
mkdir -p "$OUT"
|
|
exec > >(tee -a "$LOG") 2>&1
|
|
|
|
say() { echo "[$(date '+%F %T')] $*"; }
|
|
|
|
say "waiting for training to finish"
|
|
waited=0
|
|
while pgrep -f 'main.py --cfg' > /dev/null; do
|
|
sleep 30
|
|
waited=$((waited + 30))
|
|
[ $((waited % 300)) -eq 0 ] && say " still running (${waited}s)"
|
|
[ "$waited" -gt 7200 ] && { say " timed out after 2h"; break; }
|
|
done
|
|
say "trainer no longer running"
|
|
|
|
# also wait out the watchdog, so it does not relaunch under us
|
|
pkill -f train_watchdog.sh 2>/dev/null && say "stopped the watchdog"
|
|
sleep 5
|
|
|
|
echo
|
|
say "=== best checkpoint ==="
|
|
CKPT=$(find "$HOME/sum-parts/semantic_segmentation/PointNeXt_bundle/examples/segmentation/log/sumv2_triangle" \
|
|
-name '*pointvector*_ckpt_best.pth' -printf '%T@ %p\n' 2>/dev/null \
|
|
| sort -rn | head -1 | cut -d' ' -f2-)
|
|
say "$CKPT"
|
|
grep -ahE 'Best ckpt @E' "$HOME/sum-parts/runs/pointvector-xl_"*/train.log 2>/dev/null | tail -2
|
|
|
|
echo
|
|
say "=== 1/2 coarse evaluation on SUM val (4 classes) ==="
|
|
TEST_VOXEL_MAX=24000 bash "$SCRIPTS/eval_coarse.sh" || say "eval_coarse returned non-zero"
|
|
|
|
echo
|
|
say "=== 2/2 inference on the Seosan tile ==="
|
|
bash "$SCRIPTS/poc_infer.sh" || say "poc_infer returned non-zero"
|
|
bash "$SCRIPTS/poc_check_pred.sh" || say "poc_check_pred returned non-zero"
|
|
|
|
echo
|
|
say "AFTER TRAIN DONE -- log: $LOG"
|