nbrightandClaude Opus 5
f39b093106
Adds STATUS.md as the handoff document: benchmark numbers, the bare-earth
metrics that actually matter for this project, the Korean-data domain gap that
retraining will not fix, and what to do on the 24 GB machine.
The memory problem was in how a prediction's colours were turned back into
class indices. Every consumer built an (N, 13, 3) float64 temporary:
d = ((rgb[:, None, :] - COLOR_MAP[None, :, :]) ** 2).sum(axis=2)
That is ~250 MB of intermediates per 800k-point tile, several live at once, and
a full 4.7M-point block pushes it into gigabytes. main.py writes exact palette
entries, so an exact hash lookup resolves nearly every point with no large
temporary; only leftovers fall back to a chunked distance search. Peak RSS on a
470k-point tile drops to 61 MB. Extracted to sumparts_palette.py and shared by
coarse_eval.py and split_by_class.py.
Also from this round:
- patch_cm_mutation.sh: ConfusionMatrix.update() rewrote the caller's pred
tensor in place, folding every ignore_index point into class num_classes-1.
test() saves its visualization from that same tensor afterwards, so an
unlabelled tile came out 100% wall and the model looked degenerate when it
was not.
- patch_class_mask.sh: SUMPARTS_MASK_CLASSES drops known-absent classes from
the argmax. Measured on Seosan and it does not help - the runner-up for
"water" is "wall", not "terrain" - but the experiment is worth keeping.
- split_by_class.py now writes .ply alongside .obj. A vertex-only OBJ has zero
faces and most viewers render nothing, which is why the first export looked
broken.
- verify_outputs.sh reads exported files back with a parser, so "here are your
files" can be checked rather than asserted.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
46 lines
1.6 KiB
Bash
46 lines
1.6 KiB
Bash
#!/usr/bin/env bash
|
|
# SUM Parts - prove the exported files are actually readable
|
|
#
|
|
# "I made you some files" is worth nothing if they do not open. This reads every
|
|
# one back with a parser and reports what a viewer will find inside.
|
|
set -uo pipefail
|
|
|
|
ROOT="${1:-/mnt/d/AI_Test/sum-part}"
|
|
|
|
source "$HOME/miniconda3/etc/profile.d/conda.sh"
|
|
conda activate sumparts
|
|
|
|
python - "$ROOT" <<'PY'
|
|
import sys
|
|
from pathlib import Path
|
|
|
|
import numpy as np
|
|
from plyfile import PlyData
|
|
|
|
root = Path(sys.argv[1])
|
|
print(f"root: {root}\n")
|
|
|
|
print(f" {'folder':<10} {'file':<6} {'points':>10} {'properties':<34} {'extent (m)'}")
|
|
ok = True
|
|
for d in sorted(p for p in root.iterdir() if p.is_dir()):
|
|
for f in sorted(d.glob("*.ply")):
|
|
try:
|
|
v = PlyData.read(str(f))["vertex"]
|
|
props = ",".join(p.name for p in v.properties)
|
|
xyz = np.stack([v["x"], v["y"], v["z"]], axis=1)
|
|
ext = np.round(xyz.max(0) - xyz.min(0), 1)
|
|
print(f" {d.name:<10} {'ply':<6} {len(v):>10,} {props:<34} {ext.tolist()}")
|
|
except Exception as e:
|
|
ok = False
|
|
print(f" {d.name:<10} {'ply':<6} UNREADABLE: {type(e).__name__}: {e}")
|
|
|
|
for f in sorted(d.glob("*.obj")):
|
|
n = sum(1 for line in f.open(encoding="utf-8") if line.startswith("v "))
|
|
faces = sum(1 for line in f.open(encoding="utf-8") if line.startswith("f "))
|
|
note = "" if faces else " <- no faces; most viewers show nothing"
|
|
print(f" {d.name:<10} {'obj':<6} {n:>10,} {'v x y z r g b':<34} {faces} faces{note}")
|
|
|
|
print()
|
|
print("ALL PLY READABLE" if ok else "SOME FILES FAILED TO PARSE")
|
|
PY
|