commit 609d9a69721bd111807d42592f4313e82c648033 Author: nbright Date: Fri Aug 21 10:29:25 2026 +0900 Add SUM Parts reproduction and Seosan Myeongcheon application pipeline Reproduces the SUM Parts (CVPR 2025) face-labeling benchmark on a single consumer GPU, then applies it to drone-photogrammetry road survey meshes. Verified on RTX 3060 12GB / WSL2 Ubuntu 22.04 / CUDA 11.8 / torch 2.0.1: - CUDA extensions build (pointnet2_batch, pointops, chamfer_dist, emd, subsampling) - PointNet 100 epochs reaches mIoU 17.19, matching the paper's reported 15.1 - OBJ -> PLY conversion round-trips through the model and yields per-point predictions Four upstream source patches, all idempotent, originals preserved: - numpy aliases removed in 1.24 (np.long etc.) and collections ABCs moved in python 3.10 - the blind test split ships label = -1, which crashed ConfusionMatrix - mode=val referenced `epoch` before assignment Documents the traps that cost the most time, including VRAM overflow silently falling back to host RAM on WSL2 (25-100x slowdown, no OOM) and the colour scale mismatch between r/g/b float32 and red/green/blue uint8. Co-Authored-By: Claude Opus 5 (1M context) diff --git a/.gitignore b/.gitignore new file mode 100644 index 0000000..395c7f9 --- /dev/null +++ b/.gitignore @@ -0,0 +1,15 @@ +# Point clouds and meshes — regenerate with scripts/mesh_to_ply.py instead +output/ +*.ply +*.obj +*.pth +*.zip + +# Source-tree backups left by the patch scripts +*.orig +*.bak +*.py.bak + +__pycache__/ +*.pyc +.venv/ diff --git a/README.md b/README.md new file mode 100644 index 0000000..2d41090 --- /dev/null +++ b/README.md @@ -0,0 +1,48 @@ +# sum-parts-test + +드론 사진측량 메시를 **건물 / 수목 / 차량 / 지면**으로 분할하기 위한 +[SUM Parts](https://github.com/tudelft3d/SUM-Parts-Benchmarks) (CVPR 2025) 재현 및 적용 작업. + +대상 데이터: 서산 명천 도로 프로젝트 — ContextCapture OBJ 6블록, EPSG:5186. + +## 이 레포에 있는 것 + +| 경로 | 내용 | +|---|---| +| [SETUP.md](SETUP.md) | **새 머신에서 시작하는 절차** — 여기부터 | +| [docs/pipeline.html](docs/pipeline.html) | 6단계 공정 정의 (브라우저로 열 것) | +| [docs/SUM-Parts-검토노트.md](docs/SUM-Parts-검토노트.md) | 트러블슈팅 16건, 데이터 스키마 실측, 라이선스 | +| [scripts/](scripts/) | 환경 구축 · 학습 · 평가 · 변환 스크립트 47개 | + +데이터와 체크포인트는 커밋하지 않는다(`.gitignore`). 스크립트로 재생성한다. + +## 요약 + +- **모델**: PointVector (논문 mIoU 70.0%, 번들 최고이자 최속) +- **학습 자산**: SUM Parts face 트랙 13클래스 → 우리 4클래스로 통합 +- **제약**: 저자가 학습 가중치를 공개하지 않아 직접 학습이 유일한 경로 +- **VRAM**: 논문 설정 `voxel_max 64000`은 16.5GB 필요 → 12GB 카드 불가, 24GB 필요 + +## 빠른 시작 + +```bash +bash scripts/setup_env.sh +bash scripts/setup_pointnext.sh +bash scripts/patch_numpy_aliases.sh +bash scripts/patch_unlabeled_test.sh +bash scripts/patch_val_mode.sh +python scripts/verify_env.py # ALL OK 확인 + +bash scripts/download_data.sh all # HF 게이트 수동 수락 선행 +bash scripts/prepare_full_split.sh +bash scripts/link_data.sh + +bash scripts/launch_overnight.sh pointvector-xl +``` + +자세한 것은 [SETUP.md](SETUP.md). + +## 라이선스 + +스크립트와 문서는 자유롭게 쓰되, 참조하는 SUM Parts는 +**데이터 CC BY-NC 4.0 / 코드 GPL-3.0**이다. 상업 이용은 원저자 허락이 필요하다. diff --git a/SETUP.md b/SETUP.md new file mode 100644 index 0000000..b64ff63 --- /dev/null +++ b/SETUP.md @@ -0,0 +1,196 @@ +# 새 머신에서 시작하기 + +RTX 3090(24GB) 머신에서 SUM Parts + PointVector 학습을 재현하는 절차. + +3060(12GB)에서는 VRAM이 모자라 `voxel_max`를 24000으로 낮춰야 했다. +**24GB에서는 논문 설정 64000을 그대로 쓴다** — 이게 이 머신으로 옮기는 유일한 이유다. + +--- + +## 0. 전제 + +| 항목 | 필요 | +|---|---| +| OS | Windows + WSL2 (Ubuntu 22.04) 또는 네이티브 Linux | +| GPU | RTX 3090 24GB, 드라이버가 WSL에서 인식될 것 | +| 디스크 | 40GB 이상 여유 (데이터 13GB + 환경 + 체크포인트) | +| 계정 | HuggingFace 계정 (데이터 게이트 수락에 필요) | + +WSL이면 `nvidia-smi`가 WSL 안에서 GPU를 보여야 한다. 안 보이면 여기서 멈추고 드라이버부터. + +```bash +nvidia-smi # RTX 3090 24576MiB 가 보여야 함 +``` + +--- + +## 1. 클론 + +```bash +git clone /sum-parts-test.git +cd sum-parts-test +``` + +스크립트는 경로를 `/mnt/d/MYCLAUDE_PROJECT/sum-parts-test/scripts`로 하드코딩한 곳이 있다. +다른 경로에 두면 아래 한 줄로 일괄 치환한다. + +```bash +NEW=$(pwd)/scripts +grep -rl '/mnt/d/MYCLAUDE_PROJECT/sum-parts-test/scripts' scripts/ \ + | xargs sed -i "s|/mnt/d/MYCLAUDE_PROJECT/sum-parts-test/scripts|$NEW|g" +``` + +--- + +## 2. 환경 구축 (약 40분) + +```bash +bash scripts/setup_env.sh # conda + CUDA 11.8 + torch 2.0.1 +bash scripts/setup_pointnext.sh # 의존성 + CUDA 확장 5종 빌드 +``` + +`setup_env.sh` 안의 `TORCH_CUDA_ARCH_LIST`를 3090에 맞춰야 한다. + +```bash +# RTX 3060 = 8.6, RTX 3090 = 8.6 (둘 다 Ampere라 동일, 수정 불필요) +# RTX 4090이면 8.9로 변경 +``` + +이어서 업스트림 소스 패치 4건. **전부 멱등이라 여러 번 실행해도 안전하다.** + +```bash +bash scripts/patch_numpy_aliases.sh # np.long 등 제거된 별칭 + collections ABC +bash scripts/patch_unlabeled_test.sh # 블라인드 test셋의 label=-1 처리 +bash scripts/patch_val_mode.sh # mode=val 의 UnboundLocalError +``` + +**게이트 — 통과 못 하면 다음으로 가지 말 것:** + +```bash +python scripts/verify_env.py +# 기대: ALL OK (확장 6종 import + openpoints 체인 전부) +``` + +--- + +## 3. 데이터 (약 30분) + +HuggingFace 게이트를 **브라우저에서 1회 수동 수락**해야 한다. 자동화 불가. + +1. https://huggingface.co/datasets/gwxgrxhyz/SUM-Parts +2. 로그인 → CC BY-NC 4.0 수락 +3. 토큰을 `~/.cache/huggingface/token`에 두거나 `huggingface-cli login` + +```bash +bash scripts/download_data.sh all # 5.3GB 받아 13GB로 전개 +bash scripts/prepare_full_split.sh # validate/ -> val/ 심볼릭 링크 +bash scripts/link_data.sh # PointNeXt_bundle/data 연결 +``` + +**게이트:** 세 split이 `24 / 8 / 8`로 집계될 것. + +--- + +## 4. VRAM 확인 (5분) — 24GB에서 반드시 먼저 + +논문 설정이 실제로 들어가는지 확인한다. 3060에서는 여기서 실패했다. + +```bash +CFG=pointvector-xl ITERS=5 bash scripts/sweep_voxel_max.sh 40000 48000 64000 +``` + +기대 결과 (3090 24GB): + +| voxel_max | peak VRAM | 판정 | +|---|---|---| +| 64000 | 약 16.5G | ✅ 들어감 | + +**`fits? = yes`가 나와야 한다.** `NO (spilling)`이면 그 값은 쓰면 안 된다. + +> ⚠️ **WSL2에서 VRAM 초과는 OOM을 내지 않는다.** 드라이버가 호스트 RAM으로 흘려서 +> 학습이 **조용히 완주한다 — 25~100배 느리게.** 반드시 peak VRAM 수치로 판정할 것. +> 판별 보조 지표는 **전력**: 사용률 100%인데 전력이 낮으면 연산이 아니라 PCIe 전송 대기다. + +--- + +## 5. 학습 (논문 설정, 약 3~5시간) + +```bash +CFG_VOXEL_MAX=64000 VAL_VOXEL_MAX=64000 \ + bash scripts/launch_overnight.sh pointvector-xl +``` + +`setsid nohup`으로 분리 실행되므로 터미널·세션을 닫아도 살아남는다. +크래시하면 최신 체크포인트에서 자동 재개한다(최대 8회). + +진행 확인: + +```bash +bash scripts/check_training.sh # epoch, GPU, best miou +bash scripts/verify_speed.sh # HEALTHY / DEGRADED 판정 +bash scripts/epoch_timing.sh # epoch별 소요시간, 감속 지점 특정 +``` + +중단: + +```bash +bash scripts/stop_training.sh # 워치독 먼저 죽여서 자동 재시작 방지 +``` + +**게이트:** `verify_speed.sh`가 `HEALTHY`, peak VRAM이 카드 용량의 95% 미만. + +--- + +## 6. 평가 + +```bash +bash scripts/final_eval.sh # val(라벨 있음) + test(블라인드, 예측만) +bash scripts/eval_coarse.sh # 4클래스 통합 성적 ← 우리 과제 기준 +``` + +`eval_coarse.sh`가 실제로 중요한 수치를 낸다. SUM 13클래스를 +건물 / 수목 / 차량 / 지면으로 합쳐서 채점한다. + +**게이트:** 통합 mIoU가 **"전부 건물" 무지성 분류기(IoU 약 67%)를 이길 것.** +절대값이 아니라 baseline 대비로 판정한다. + +--- + +## 참고 — 3060 실측값 (비교 기준) + +| 모델 | voxel_max | peak VRAM | s/iter | 100 epoch | +|---|---|---|---|---| +| pointnet | 64000 | 6.01G | 0.291 | 2.9h | +| pointnet++msg | 64000 | 4.16G | 0.675 | 6.8h | +| pointvector-xl | 24000 | 6.46G | 0.402 | 2.9h | +| pointvector-xl | 64000 | **16.49G** | 13.113 | ❌ 12GB 불가 | +| pointnext-xl | 32000 | 8.03G | 0.635 | 6.4h | +| pointnext-xl | 64000 | **15.47G** | 46.980 | ❌ 12GB 불가 | + +논문 보고치 (face 트랙, 12클래스): + +| 모델 | mIoU | +|---|---| +| PointNet | 15.1% | +| PointNet++ | 33.1% | +| PointNeXt | 65.3% | +| **PointVector** | **70.0%** | + +3060에서 pointnet 100 epoch 실측 = **17.19%** (논문 15.1%와 근사, 재현 확인됨). + +--- + +## 알려진 제약 + +- **test 세트는 블라인드다.** 라벨이 전부 `-1`이라 로컬 채점이 불가능하다. + 논문 수치와 직접 대조하려면 예측을 저자(gaoweixiaocuhk@gmail.com)에게 보내야 한다. +- **`voxel_max`는 중립적 손잡이가 아니다.** 모델의 동작점 일부다. + 같은 체크포인트가 검증 프로토콜에 따라 mIoU 17.19 / 4.20으로 갈렸다. +- **라이선스**: 데이터 CC BY-NC 4.0, 코드 GPL-3.0. 상업 이용은 저자 허락 필요. + +--- + +## 상세 기록 + +- [docs/SUM-Parts-검토노트.md](docs/SUM-Parts-검토노트.md) — 트러블슈팅 16건, 실측 데이터 스키마, 라이선스 검토 +- [docs/pipeline.html](docs/pipeline.html) — 전체 공정 정의 (브라우저로 열 것) diff --git a/docs/SUM-Parts-검토노트.md b/docs/SUM-Parts-검토노트.md new file mode 100644 index 0000000..8ec10c2 --- /dev/null +++ b/docs/SUM-Parts-검토노트.md @@ -0,0 +1,1185 @@ +# SUM Parts 검토 노트 + +> 대상: [tudelft3d/SUM-Parts-Benchmarks](https://github.com/tudelft3d/SUM-Parts-Benchmarks) +> 목적: 로컬 재현 테스트 + 한국 샘플 데이터 적용 가능성 확인 +> 작성일: 2026-08-20 + +--- + +## 1. 프로젝트 개요 + +**SUM Parts** — TU Delft 3D geoinformation, CVPR 2025. +도시 **텍스처 3D 메시**의 **part-level(부품 단위) 시맨틱 세그멘테이션** 벤치마크. + +| 항목 | 내용 | +|---|---| +| 규모 | 2.5 km² 도시 메시 | +| 어노테이션 | 이중 트랙 — face 단위 + texture 픽셀 단위 | +| 클래스 | 21종 | +| 텍스처 메시 포맷 | ASCII PLY | +| 시맨틱 포인트클라우드 포맷 | binary PLY | +| 데이터 배포 | Hugging Face `gwxgrxhyz/SUM-Parts` (**게이트 있음**, 아래 참조) | +| 논문 | [arXiv:2503.15300](https://arxiv.org/abs/2503.15300) | +| 프로젝트 페이지 | https://tudelft3d.github.io/SUMParts/ | +| 연락처 | gaoweixiaocuhk@gmail.com (Weixiao Gao) | + +### 클래스 — README는 21이라 하지만, 코드상 트랙별로 다르다 + +README 전체 목록 (21종): + +``` +unclassified, terrain, high vegetation, water, car, boat, +wall, roof surface, facade surface, chimney, dormer, balcony, +roof installation, window, door, low vegetation, +impervious surface, road, road marking, cycle lane, sidewalk +``` + +**실제 학습 코드에서는 트랙별로 분리된다** (PointNeXt `openpoints/dataset/` 확인 결과): + +| 트랙 | 클래스 수 | 목록 | +|---|---|---| +| `SUMV2_Triangle` (face/메시) | **13** | unclassified, terrain, high_vegetation, facade_surface, water, car, boat, roof_surface, chimney, dormer, balcony, roof_installation, wall | +| `SUMV2_Texture` (텍스처 픽셀) | **20** | 위 + window, door, low_vegetation, road, road_marking 등 세부 클래스 | + +→ 벤치마크 재현 시 **cfg의 `num_classes`와 데이터 트랙을 반드시 일치**시켜야 한다. +`ignore_index: 0` (unclassified는 손실 계산에서 제외). + +### PLY 필드 규약 + +| 필드 | 의미 | +|---|---| +| `f:color` | face 단위 색상 | +| `v:color` | vertex(point) 단위 색상 | +| `f:label` / `v:label` | 시맨틱 라벨 | +| `h:texcoord` | 텍스처 좌표 (halfedge) | + +### 리포지토리 구조 + +- `semantic_segmentation/` — 딥러닝 베이스라인 + - KPConv (`train_UrbanMesh.py`, `UrbanMesh.py`) + - PointNeXt_bundle (PointNet, PointNet++, PointNeXt, PointVector) + - Open3D_ML (SparseConvUNet, RandLA-Net) + - SPG + - RF_MRF / SUM_RF / PSSNet → 별도 PSSNet repo의 `sumv2` 브랜치 +- `interactive_annotation/` — SAM, SimpleClick 기반 텍스처 어노테이션 도구 +- `assets/` — 문서 이미지, 설정 파일 + +### 데이터 접근 — 게이트 걸려 있음 ⚠️ + +HF 데이터셋이 `gated: auto`. 토큰만으로는 안 되고 **계정별 약관 수락이 1회 필요**하다. +수락 전에는 모든 다운로드가 `HTTP 403`. + +**수동 절차 (브라우저 필요, 자동화 불가)** + +1. https://huggingface.co/datasets/gwxgrxhyz/SUM-Parts 접속 +2. 로그인 후 게이트 폼에서 CC BY-NC 4.0 수락 +3. 이후 캐시된 토큰으로 자동 다운로드 가능 + +**배포 파일** + +| 파일 | 내용 | +|---|---| +| `demo.zip` | 소량 샘플 — 스모크 테스트용. **여기부터 시작** | +| `mesh.zip` | 텍스처 메시 (ASCII PLY) | +| `pcl.zip` | 시맨틱 포인트클라우드 (binary PLY) | + +### 데이터 레이아웃 (PointNeXt 기준) + +`cfgs/sumv2_triangle/default.yaml` → `data_root: ../../data/sumv2_tri_texpcl/` + +⚠️ **이 상대경로는 `main.py`의 작업 디렉토리 기준**이다. `main.py`는 +`examples/segmentation/`에서 실행되므로 `../../` = **`PointNeXt_bundle/`**이지 +repo 루트가 아니다. 착각하면 `Totally 0 samples in train set`으로 조용히 실패한다. + +``` +PointNeXt_bundle/data/sumv2_tri_texpcl/ +├── train/*.ply +├── val/*.ply +├── test/*.ply +└── processed/ # presample 캐시, 자동 생성 +``` + +여기서는 데이터를 `/data`에 두고 심볼릭 링크로 연결했다 +([scripts/link_data.sh](../scripts/link_data.sh)). + +**demo.zip 실제 구조** (train/val/test 아님 — 타일 1장짜리 쇼케이스): + +``` +data/ +├── mesh/ +│ ├── textured_mesh/ demo.ply + demo.jpg +│ ├── face_label/ demo_face_label.ply + .jpg +│ ├── pixel_label/ demo_pixel_label.ply + .png +│ └── full_pixel_label/ demo_full_pixel_label.ply + .png +└── pcl/ + ├── face_labeling_pcl/ demo_{texsp,fdcen,poisson,rand}_pcl.ply + └── texture_labeling_pcl/ demo_{texsp,poisson,rand}_pcl.ply +``` + +cfg 디렉토리명 `sumv2_tri_texpcl` = **triangle 트랙 + texsp 샘플러** +→ `pcl/face_labeling_pcl/demo_texsp_pcl.ply` + +**PLY vertex 속성** (실측 + `read_ply_with_plyfilelib` 확인): + +| 속성 | 실제 배포본 | 필수 | 비고 | +|---|---|---|---| +| `x`, `y`, `z` | ✅ | ✅ | float32 | +| `nx`, `ny`, `nz` | ✅ | ✗ | 로더가 무시 | +| `r`, `g`, `b` | ✅ | ✅ | **`red,green,blue` 아님.** 로더는 둘 다 받는다 | +| `label` | ✅ | ✅ | int32 | +| `sp_id` | texsp 파일만 | ✗ | 슈퍼픽셀 ID | +| `object_index` | ✗ | ✗ | 로더는 지원하나 demo엔 없음 | + +**실측 라벨 분포** + +| 트랙 | 라벨 값 | 클래스 수 | +|---|---|---| +| `face_labeling_pcl` | `{0..10, 12}` | 13 ✅ `SUMV2_Triangle` 일치 | +| `texture_labeling_pcl` | `{0..9, 11..19}` | 20 ✅ `SUMV2_Texture` 일치 | + +**타일 규모**: 252 × 252 × 40 m, 47만 포인트 (fdcen만 21만). +좌표 원점 `(7749, 5499, -1.6)` — **저자도 전체 CRS 좌표가 아닌 로컬 미터계로 배포**한다. + +→ **한국 데이터 변환 시 이 스키마만 맞추면 된다.** 텍스처 불필요. + +### 주요 학습 하이퍼파라미터 (sumv2_triangle 기본값) + +| 항목 | 값 | +|---|---| +| `voxel_size` | 0.02 | +| `voxel_max` (train) | 64000 | +| `loop` | 30 (학습셋 30회 반복 → epoch 수는 적어도 됨) | +| `epochs` | 100 | +| `batch_size` | 2 (PointNeXt-XL 기준. PointNet: 6, PointNet++: 10) | +| `ignore_index` | 0 (unclassified 제외) | +| optimizer | adamw, lr 0.01, cosine | +| `cls_weighed_loss` | True (클래스 불균형 보정) | + +제공 모델 cfg: `pointnet.yaml`, `pointnet++msg.yaml`, `pointnext-xl.yaml`, `pointvector-xl.yaml` + +### 평가 + +현재는 각 데이터에 내장된 GT 라벨로 자체 평가. 논문과 동일한 fine-grained 테스트셋 평가는 +**예측 결과를 저자 이메일로 보내면 로컬에서 채점**해 줌. 자동 평가 코드는 HF에 추가 예정. + +### 시각화 + +- **Mapple** (권장) — `f:color`, `v:color`, 라벨 범례 표시 가능 +- **MeshLab** — face color/texture는 보이나 스칼라(라벨)는 처리 못 함 + +--- + +## 2. 라이선스 검토 + +**두 개가 따로 논다. 분리해서 봐야 한다.** + +| 대상 | 라이선스 | 배포 조건 | +|---|---|---| +| 코드 (GitHub repo) | **GPL-3.0** | 파생 코드 배포 시 소스 공개 + GPL-3.0 승계. 비공개 상용 제품에 혼합 불가 | +| 데이터셋 (HF SUM-Parts) | **CC BY-NC 4.0** | **비상업 한정**. 출처 표기 필수. 재배포/개변 허용 | + +### 결론 + +- **로컬 실험/학습 목적** → 제약 없음. 그냥 쓰면 된다. +- **논문 / 오픈소스 릴리즈** → 가능. 단 코드는 GPL-3.0으로 내고, SUM Parts(2025) + SUM(2021) 둘 다 인용. +- **상업 이용** → 저자에게 별도 허락 필요 (gaoweixiaocuhk@gmail.com). + +### 주의 3가지 + +1. **학습 가중치** — NC 데이터로 뽑은 weight는 NC로 취급하는 게 안전. 상업 배포 금지 쪽으로 본다. +2. **베이스라인별 개별 라이선스** — KPConv/PointNeXt/Open3D-ML은 대개 MIT, SPG는 별도 확인 필요. vendoring 시 각 LICENSE 동봉. +3. **데이터 자체를 repo에 커밋 금지** — HF 링크 + 다운로드 스크립트만 둔다. + +> 법률 자문 아님. 최종 확인은 repo `LICENSE` 파일 + HF 데이터셋 카드 원문. + +### 인용 + +```bibtex +@InProceedings{Gao_2025_CVPR, + author = {Gao, Weixiao and Nan, Liangliang and Ledoux, Hugo}, + title = {SUM Parts: Benchmarking Part-Level Semantic Segmentation of Urban Meshes}, + booktitle = {Proceedings of CVPR}, + month = {June}, + year = {2025}, + pages = {24474-24484} +} +``` + +--- + +## 3. 실행 환경 판단 + +### 왜 Windows 네이티브로 안 되나 + +`railway-client` 프로젝트의 SAM 3.1은 Windows에서 잘 돈다. 이유는 **순수 PyTorch, 컴파일 없음**. +SUM 베이스라인은 성격이 다르다 — `nvcc + 컴파일러`로 **커스텀 CUDA 커널을 직접 빌드**해야 한다. + +| 하려는 것 | Windows 네이티브 | +|---|---| +| `interactive_annotation/` (SAM, SimpleClick) | ✅ 가능 | +| PLY 로드 / 변환 / 시각화 | ✅ 가능 | +| KPConv / PointNeXt / SPG 학습 | ❌ `pointnet2_ops`, cut-pursuit 등 Windows 패치 필요 | + +**결론**: WSL2가 이미 설치돼 있으므로 그대로 사용. 학습만 WSL, 데이터 변환/어노테이션은 Windows. +`/mnt/d/`로 파일 공유되니 복사 불필요. + +### 실측 환경 + +**Windows 호스트** + +| 항목 | 값 | +|---|---| +| OS | Windows 10 Enterprise 19045 | +| GPU | RTX 3060 12GB, 드라이버 610.47 | +| Python | 3.11.9 | +| MSVC | Visual Studio Professional 2022 ✅ | +| CUDA Toolkit (nvcc) | ❌ 없음 | +| WSL | Ubuntu-22.04 (v2), docker-desktop | + +**WSL2 / Ubuntu-22.04** + +| 항목 | 값 | 상태 | +|---|---|---| +| GPU 패스스루 | RTX 3060 12GB 인식됨 | ✅ | +| gcc / g++ | 11.4.0 | ✅ | +| git | 2.34.1 | ✅ | +| Python (시스템) | 3.10.12 | ✅ | +| 디스크 `/` | 251G 중 192G 여유 | ✅ | +| 디스크 `/mnt/d` | 7.3T 중 2.9T 여유 | ✅ | +| RAM | 31G | ✅ | +| nvcc | 없음 | ❌ 설치 필요 | +| conda | 없음 | ❌ 설치 필요 | +| **sudo** | **암호 필요 (NOPASSWD 아님)** | ⚠️ apt 사용 불가 | + +### 최종 구성 (검증 완료) + +``` +nvcc : 11.8.89 (~/miniconda3/envs/sumparts/bin/nvcc) +torch : 2.0.1+cu118 +cuda : available → NVIDIA GeForce RTX 3060 +numpy : 1.26.4 +python : 3.10 +``` + +**업스트림 `install.sh`와 다르게 간 이유** + +원본은 python 3.7 / torch 1.12.1 / cu113 / `cudatoolkit=11.3` 전제. +py3.7은 EOL이고 `requirements.txt` 핀(`setuptools==59.5.0`, `protobuf==3.19.4`, +`tensorboard==2.8.0` 등)이 py3.10에서 해결되지 않는다. 그래서: + +| 항목 | 원본 | 여기 | 이유 | +|---|---|---|---| +| python | 3.7 | 3.10 | 3.7 EOL | +| torch | 1.12.1+cu113 | 2.0.1+cu118 | 드라이버 610.47 / sm_86 대응 | +| 채널 | anaconda defaults | conda-forge + nvidia | ToS/상용 라이선스 회피 | +| `deepspeed` | 설치 | 제외 | sumv2 학습 경로에서 미사용, 빌드 무거움 | +| `mkdocs-*` | 설치 | 제외 | 문서 전용 | +| `chamfer_dist`, `emd` 확장 | 빌드 | 제외 | reconstruction 태스크 전용 | +| `plyfile` | **누락됨** | 추가 | 데이터 로더가 import 하는데 requirements.txt에 없음 | +| `setuptools` | 59.5.0 | <80 | 80+에서 `python setup.py install` 제거됨 | + +빌드 대상 CUDA 확장 3종: `pointnet2_batch`, `subsampling`, `pointops`. +`TORCH_CUDA_ARCH_LIST=8.6` 고정 (RTX 3060) — 전 아키텍처 빌드 방지. + +### Anaconda 채널 회피 (중요) + +기본 miniconda는 `repo.anaconda.com/pkgs/main`, `pkgs/r`을 쓰는데, +이 채널은 **ToS 수락이 필요하고 일정 규모 이상 조직에는 상용 라이선스 의무**가 붙는다. +`conda create` 시 `CondaToSNonInteractiveError`로 막힌다. + +→ 수락하지 않고 우회. `--override-channels -c conda-forge` + CUDA는 `nvidia` 채널. +`conda config --remove channels defaults` + `channel_priority strict`. + +### sudo 제약 대응 + +`sudo` 암호가 필요해 `apt install cuda-toolkit`을 자동화할 수 없다. +→ **conda 경로로 우회**. 둘 다 sudo 없이 설치된다. + +- miniconda → `~/miniconda3` (홈 디렉토리, sudo 불필요) +- CUDA Toolkit → `conda install -c nvidia cuda-toolkit` (env 내부 설치, sudo 불필요) + +### 배치 원칙 + +| 대상 | 위치 | 이유 | +|---|---|---| +| 코드 / conda env | WSL ext4 (`~`) | `/mnt/d`는 9p 파일시스템이라 CUDA 확장 빌드가 매우 느림 | +| 데이터셋 | `/mnt/d` | 용량 크고 Windows 도구와 공유 필요 | + +--- + +## 3-1. 빌드 트러블슈팅 기록 + +CUDA 확장 3종(`pointnet2_batch`, `subsampling`, `pointops`) 빌드에서 걸린 것들. +전부 **최신 툴체인 × 2022년 코드베이스** 충돌이다. + +### ① Anaconda ToS 차단 + +``` +CondaToSNonInteractiveError: Terms of Service have not been accepted for the +following channels: https://repo.anaconda.com/pkgs/main, .../pkgs/r +``` + +수락하면 넘어가지만 상용 라이선스 의무가 붙는다. +→ `--override-channels -c conda-forge` + `nvidia` 채널로 회피. defaults 미사용. + +### ② ninja SIGPIPE + +``` +subprocess.CalledProcessError: Command '['ninja', '-v']' died with . +RuntimeError: Error compiling objects for extension +``` + +컴파일 에러가 아니다. torch 2.0의 `cpp_extension`이 `ninja -v` 출력을 파이프로 읽는데 +**ninja ≥1.12가 거기서 SIGPIPE로 죽는다**. torch가 원인을 뭉개고 +"Error compiling objects"로만 표시해 오해하기 쉽다. + +→ `pip install ninja==1.11.1.1` + +### ③ setuptools에서 distutils.msvccompiler 제거 + +``` +ModuleNotFoundError: No module named 'distutils.msvccompiler' +``` + +`openpoints/cpp/subsampling/setup.py:2` 가 `numpy.distutils.misc_util`를 import 하는데, +`numpy.distutils`는 내부적으로 `distutils.msvccompiler`를 찾는다. +이 모듈은 **setuptools 74.0에서 제거**됨. (초기 핀 `<80`은 79.0.1을 뽑아서 여전히 실패) + +→ `pip install setuptools==69.5.1` +(단, `numpy<2` 도 함께 필요. numpy 2에는 `numpy.distutils` 자체가 없다.) + +### ④ WSL 인스턴스 사망 + +``` +Wsl/Service/CreateInstance/E_FAIL +``` + +컴파일 도중 WSL VM이 통째로 내려갔다. 컴파일 에러와 무관. + +→ `wsl --shutdown` 후 재접속으로 복구. 재발 시 `MAX_JOBS` 낮춰서 병렬 컴파일 부하 감소. + +### ⑤ 확장 모듈 이름 (함정) + +`pointnet2_batch/setup.py`가 만드는 모듈명은 **`pointnet2_batch_cuda`**다. +업스트림 PointNet++ 포크들의 `pointnet2_cuda`가 아니다. +검증 스크립트에서 이걸 틀리면 빌드는 다 성공했는데 `ModuleNotFoundError`로 실패한 것처럼 보인다. + +| 확장 | import 이름 | +|---|---| +| pointnet2_batch | `pointnet2_batch_cuda` | +| pointops | `pointops_cuda` | +| subsampling | `from openpoints.cpp.subsampling import grid_subsampling` | + +### ⑥ `unzip` 미설치 + +exit 127. sudo 암호가 필요해 `apt install unzip`을 못 한다. +→ python `zipfile`로 압축 해제. + +### ⑦ `wandb`가 모듈 스코프에서 import 됨 + +``` +main.py:8: import argparse, yaml, os, logging, numpy as np, csv, wandb, glob +ModuleNotFoundError: No module named 'wandb' +``` + +`wandb.use_wandb=False`로 꺼도 소용없다. **import 자체가 조건 없이 실행**되므로 설치는 필수. +→ `pip install wandb` + `WANDB_MODE=disabled` (로그인 요구 차단). + +### ⑧ chamfer_dist / emd는 "선택"이 아니다 + +``` +openpoints/cpp/chamfer_dist/__init__.py:10: import chamfer +ModuleNotFoundError: No module named 'chamfer' +``` + +기능상 reconstruction 전용이 맞다. 하지만 `openpoints/models/__init__.py:9`가 +`.reconstruction`을 **조건 없이 import** → `maskedpointvit.py:10` → `chamfer_dist`. +세그멘테이션만 돌려도 import 체인에 걸린다. 빌드 필수. + +(`deepspeed` 제외는 유효 — import 체인에 없음.) + +### ⑨ emd 모듈명도 다르다 + +`emd/setup.py`: `setup(name='emd_ext', ext_modules=[CUDAExtension(name='emd_cuda', ...)])` +→ import 이름은 **`emd_cuda`**. `emd`는 `openpoints/cpp/emd/__init__.py`의 파이썬 별칭일 뿐. + +### ⑩ `data_root` 상대경로 기준점 + +``` +[SUMV2_Triangle]: Totally 0 samples in train set. +``` + +예외 없이 조용히 0개. `data_root: ../../data/...`는 **`main.py`의 cwd 기준**이고, +`main.py`는 `examples/segmentation/`에서 돈다 → `../../` = `PointNeXt_bundle/`. +repo 루트가 아니다. + +→ [scripts/link_data.sh](../scripts/link_data.sh)로 `PointNeXt_bundle/data` → `/data` 심볼릭 링크. + +### ⑪ 샘플 수 < batch_size → 엉뚱한 예외 + +``` +0it [00:00, ?it/s] +AttributeError: 'int' object has no attribute 'diag' + at openpoints/utils/metrics.py:84 -> return self.value.diag() +``` + +메시지가 원인을 전혀 안 가리킨다. 실제 원인은 **dataloader의 `drop_last`**: +타일 1장 × `loop=1` = 샘플 1개인데 pointnet `batch_size=6` → **배치 0개** → +혼동행렬이 `int 0`으로 남아 `.diag()` 호출에서 터진다. + +→ `dataset.train.loop`를 키우고 `batch_size=1`. 데이터가 적을 때 반드시 걸리는 함정. + +### ⑫ Git Bash에서 WSL 호출 시 경로 변환 + +``` +bash: C:/Program Files/Git/mnt/d/.../smoke_train.sh: No such file or directory +``` + +Git Bash(MSYS)가 `/mnt/d/...`를 Windows 경로로 자동 변환한다. +→ WSL 호출은 PowerShell에서 하거나 `MSYS_NO_PATHCONV=1`. + +### ⑬ numpy 제거 별칭 + +``` +sumv2_triangle.py:134: label = label.astype(np.long) +AttributeError: module 'numpy' has no attribute 'long' +``` + +`np.long` / `np.int` / `np.float` / `np.bool` / `np.object` / `np.str`는 +**numpy 1.24에서 제거**됐다. 원본은 numpy 1.20 전제. + +numpy를 내리는 건 불가 — torch 2.0.1은 `numpy<2`를 요구하고, numpy 1.20에는 py3.10 휠이 없다. +→ 소스 패치. [scripts/patch_numpy_aliases.sh](../scripts/patch_numpy_aliases.sh) (8개 파일) + +| 별칭 | 치환 | +|---|---| +| `np.long` | `np.int64` | +| `np.int` | `int` | +| `np.float` | `float` | +| `np.bool` | `bool` | +| `np.object` | `object` | +| `np.str` | `str` | + +`\b` 경계 필수 — 안 그러면 `np.int32`, `np.float32`까지 망가진다. + +### ⑭ collections ABC 이동 + +``` +point_transformer_gpu.py:282: isinstance(self.angle, collections.Iterable) +AttributeError: module 'collections' has no attribute 'Iterable' +``` + +Python 3.10에서 `collections.abc`로 이동. 같은 패치 스크립트가 처리한다. + +### ⑮ BatchNorm은 배치 1을 거부 + +``` +ValueError: Expected more than 1 value per channel when training, + got input size torch.Size([1, 512]) +``` + +⑪ 해결하려고 `batch_size=1`로 내렸더니 이번엔 BN이 막는다. +**양쪽에서 조인다**: 너무 크면 `drop_last`로 배치 0개, 너무 작으면 BN 거부. +→ `batch_size=2` (BN 하한) + `loop`로 타일 반복해 온전한 배치 확보. + +### ⑯ WSL 크래시가 남긴 0바이트 패키지 ★ + +가장 찾기 어려웠던 것. + +``` +main.py:666: all_logits = scatter(all_logits, idx_points, dim=0, reduce='mean') +TypeError: 'module' object is not callable +``` + +`from torch_scatter import scatter`가 함수가 아니라 서브모듈을 반환. 이유: + +``` +torch_scatter/__init__.py 0 bytes +torch_scatter/scatter.py 0 bytes +torch_scatter/_scatter_cuda.so 0 bytes +``` + +④의 WSL VM 사망 당시 pip이 열고 있던 파일이 **전부 빈 껍데기로 디스크에 남았다**. +`gdown/*.py` 전부, `vtkmodules/*.so` 다수도 동일. + +**고약한 이유**: import가 성공한다. 모듈이 비어있을 뿐이라 에러가 한참 뒤 호출 지점에서 +엉뚱한 메시지로 터진다. 설치 로그에도 `Successfully installed`로 남아 있다. + +**탐지**: +```bash +find -type f \( -name '*.py' -o -name '*.so' \) -size 0 ! -name '__init__.py' +``` +`__init__.py`는 원래 빈 경우가 많으니 제외. 잔여 정상 케이스: +`torch/cuda/error.py`, `torch/ao/.../observation_type.py`, `sklearn/_built_with_meson.py`. + +→ [scripts/repair_env.sh](../scripts/repair_env.sh)로 크래시 당시 배치 전체 강제 재설치. + +**교훈**: VM/컨테이너가 설치 도중 죽으면 그 시점 패키지들을 무조건 재설치해라. +pip은 손상을 감지하지 못한다. + +### 무시해도 되는 경고 + +``` +warning: 'T* at::Tensor::data() const' is deprecated: + Tensor.data() is deprecated. Please use Tensor.data_ptr() instead. +warning: There are no g++ version bounds defined for CUDA version 11.8 +``` + +전부 경고. 빌드 실패 원인 아니다. + +### 재실행 전략 + +빌드 단계만 [scripts/build_ext.sh](../scripts/build_ext.sh)로 분리했다. +의존성 재설치 없이 컴파일만 재시도 가능하고, 이미 import 되는 확장은 건너뛴다 +(`FORCE=1`이면 전부 재빌드). + +--- + +## 3-2. 실학습 가능성 — RTX 3060 12GB 실측 (2026-08-20) + +### 결론: 가능하다. 단 XL 계열은 `voxel_max`를 낮춰야 한다. + +### 데이터 규모 + +| 항목 | 값 | +|---|---| +| `pcl.zip` | 4.66 GB | +| `mesh.zip` | 0.52 GB | +| `demo.zip` | 0.12 GB | +| 타일 수 | **train 24 / val 8 / test 8** (총 40) | + +디스크 192GB 여유 → 저장은 문제 없음. + +### ★ WSL2의 함정 — VRAM 초과가 OOM을 내지 않는다 + +WSL2의 NVIDIA 드라이버는 VRAM을 넘으면 **호스트 RAM으로 흘린다**(system memory fallback). +네이티브 Linux라면 `torch.cuda.OutOfMemoryError`로 즉시 죽을 설정이 여기서는 +**조용히 완주한다 — 25~100배 느리게.** + +``` +peak VRAM 15.47 GB on a 12 GB card <- 물리적으로 불가능한 수치가 보고된다 +``` + +"돌아간다"는 사실만 보고 설정을 확정하면 20일짜리 학습을 시작하게 된다. +**반드시 peak VRAM을 카드 용량과 대조해야 한다.** + +### `voxel_max` 스윕 (pointnext-xl, batch_size 2) + +| voxel_max | pts/batch | peak VRAM | s/iter | 판정 | +|---|---|---|---|---| +| 24000 | 48,000 | 6.17 G | **0.434** | ✅ | +| 32000 | 64,000 | 8.03 G | **0.635** | ✅ | +| 40000 | 80,000 | 9.88 G | **1.169** | ✅ | +| 48000 | 96,000 | 11.75 G | **29.169** | ❌ 폴백 | +| **64000** (cfg 기본값) | 128,000 | 15.47 G | **46.980** | ❌ 폴백 | + +경계가 선명하다. 40000 → 48000 사이에서 **25배** 절벽. +정상 구간 안에서도 s/iter가 초선형으로 증가한다(0.434 → 0.635 → 1.169). + +### 모델별 실측 + +| cfg | params | bs | 설정 | peak VRAM | s/iter | 적합 | +|---|---|---|---|---|---|---| +| pointnet | 3.6M | 2 | voxel_max 64000 (원본) | 6.01 G | 0.291 | ✅ | +| pointnet++msg | 3.0M | 2 | voxel_max 64000 (원본) | 4.16 G | 0.675 | ✅ | +| pointnext-xl | 41.6M | 2 | voxel_max 32000 | 8.03 G | 0.635 | ✅ | +| pointvector-xl | 24.1M | 2 | voxel_max 24000 | 6.46 G | 0.402 | ✅ | +| pointnext-xl | 41.6M | 2 | voxel_max 64000 | 15.47 G | 46.980 | ❌ | +| pointvector-xl | 24.1M | 2 | voxel_max 64000 | 16.49 G | 13.113 | ❌ | + +> `batch_size`는 네 cfg 모두 `default.yaml`의 **2**를 그대로 쓴다. +> default.yaml 주석의 "PointNet: 6, PointNet++: 10"은 실제 설정에 반영돼 있지 않다. + +### 학습 시간 추정 + +반복 횟수 = train 24타일 × `loop` 30 ÷ `batch_size` 2 = **360 iter/epoch** +× `epochs` 100 = **36,000 iter** + +| 모델 | 설정 | 학습 시간 (val 제외) | +|---|---|---| +| pointnet | 원본 | **2.9 h** | +| pointnet++msg | 원본 | **6.8 h** | +| pointnext-xl | voxel_max 32000 | **6.4 h** | +| pointvector-xl | voxel_max 24000 | **4.0 h** | +| ~~pointnext-xl~~ | ~~원본 64000~~ | ~~**470 h ≈ 20일**~~ | + +`val_freq: 1`이라 매 epoch 8타일 검증이 붙는다 → **1~3시간 추가**. +`val_freq: 5`로 올리면 대부분 회수된다. + +### 남는 제약 + +- **XL 계열은 논문 설정 재현 불가.** `voxel_max` 64000 → 32000은 샘플당 포인트를 절반으로 + 줄이는 것이고, 컨텍스트 윈도우가 좁아진다. 논문 수치와 직접 비교하면 안 된다. +- **작은 모델 2개(pointnet, pointnet++msg)는 원본 설정 그대로 학습 가능하다.** + 재현성이 중요하면 이쪽부터. +- 측정은 5~8 iteration 평균이며 warm-up 2회는 제외했다. 장시간 학습의 열/클럭 변동은 반영 안 됨. + +### 스크립트 + +| 파일 | 역할 | +|---|---| +| [scripts/bench_models.py](../scripts/bench_models.py) | 모델별 peak VRAM + s/iter 측정, VRAM 초과 여부 판정 | +| [scripts/run_bench.sh](../scripts/run_bench.sh) | 벤치 실행 래퍼 | +| [scripts/sweep_voxel_max.sh](../scripts/sweep_voxel_max.sh) | `voxel_max` 스윕으로 VRAM 한계 탐색 | +| [scripts/check_hf_sizes.py](../scripts/check_hf_sizes.py) | HF 아카이브 크기 조회 | +| [scripts/list_archive.sh](../scripts/list_archive.sh) | 아카이브 내용·타일 수 확인 (압축 해제 없이) | +| [scripts/verify_archives.sh](../scripts/verify_archives.sh) | 다운로드 아카이브 무결성 검사 | +| [scripts/check_alloc_conf.sh](../scripts/check_alloc_conf.sh) | `PYTORCH_CUDA_ALLOC_CONF` 유효성 사전 검증 | +| [scripts/epoch_timing.sh](../scripts/epoch_timing.sh) | epoch별 소요 시간 → 감속 시작 지점 특정 | +| [scripts/iter_rate.sh](../scripts/iter_rate.sh) | 초기 vs 최근 반복 속도 비교 | +| [scripts/verify_speed.sh](../scripts/verify_speed.sh) | 현재 속도가 정상 범위인지 판정 | +| [scripts/diag_run.sh](../scripts/diag_run.sh) | 실행 실패 시 로그 일괄 덤프 | + +### 무인 실행 스크립트 + +| 파일 | 역할 | +|---|---| +| [scripts/launch_overnight.sh](../scripts/launch_overnight.sh) | `setsid nohup` 분리 실행. `RESUME_CKPT`로 이어하기 | +| [scripts/train_watchdog.sh](../scripts/train_watchdog.sh) | 크래시 시 최신 체크포인트에서 자동 재개 (최대 8회) | +| [scripts/check_training.sh](../scripts/check_training.sh) | 진행 epoch·GPU·best miou·워치독 이벤트 | +| [scripts/stop_training.sh](../scripts/stop_training.sh) | 워치독 먼저 종료 후 학습 중단 | +| [scripts/monitor_training.sh](../scripts/monitor_training.sh) | 진행·감속·크래시·완료 이벤트 스트림 | +| [scripts/train_full.sh](../scripts/train_full.sh) | 단발 실학습 (워치독 없이) | +| [scripts/prepare_full_split.sh](../scripts/prepare_full_split.sh) | `val` → `validate` 심볼릭 링크 | +| [scripts/show_layout.sh](../scripts/show_layout.sh) | 전개된 데이터 레이아웃 확인 | + +--- + +## 3-2-1. 실학습 중 발생한 무증상 10배 감속 ★★ + +벤치마크에서 발견한 "VRAM 초과가 OOM을 안 낸다"가 **실제 학습에서 재현됐다.** +다만 원인이 예상과 달랐다. + +### 증상 + +``` +epoch 1-10 ~103 s/epoch (벤치 예측 105 s와 일치) +epoch 11 +1011 s ← 20:45부터 급변 +epoch 12-20 ~1100 s/epoch +``` + +반복당 속도: **0.29 s/it → 3.14 s/it (10.7배)**. 남은 80 epoch에 24시간. + +에러 없음. 경고 없음. GPU 사용률 100%, 클럭 1942MHz 만빵, 온도 51°C, 스로틀 전무. +**로그만 보면 완벽하게 정상이다.** + +### 오진과 정정 + +처음엔 데스크톱 앱(브라우저·Electron)이 VRAM을 잠식했다고 봤다. +`nvidia-smi`에 Brave, Edge, WebView2, VS Code 등이 잔뜩 잡혀 있었기 때문이다. + +**학습을 죽이고 재측정해서 뒤집혔다:** + +| 상태 | Dedicated (VRAM) | Shared (호스트 RAM) | +|---|---|---| +| 학습 중 | 11,715 MB | **18,967 MB** | +| 학습 종료 후 | **1,245 MB** | 468 MB | + +데스크톱 앱은 1.2GB뿐. **11.7GB + 19GB를 전부 학습 프로세스가 잡고 있었다.** +벤치 측정값 6.01GB의 약 2배로 불어난 것. + +> 교훈: 남 탓하기 전에 **의심 대상을 죽여보고 기준선을 재라.** 30초면 된다. + +### 진단 경로 + +| 단계 | 관측 | 배제/확정 | +|---|---|---| +| GPU 스로틀 | 51°C, `HW Slowdown: Not Active` | 열/전력 배제 | +| WSL 자원 | swap 0, IO wait 0, load 1.16 | 메모리·디스크 배제 | +| 프로세스 | main python **CPU 99.8% 단일 스레드** | CUDA 동기화 스핀 = GPU 대기 | +| **전력** | **61.7W** (3060 TDP 170W) | 연산 아님 → **전송 대기** | +| 반복 속도 | 초기 0.29 s/it → 3.14 s/it | epoch 간 오버헤드 아닌 **반복 자체** 감속 | +| **Perf 카운터** | **Shared Usage 18,967 MB** | **시스템 메모리 폴백 확정** | + +**전력 소비가 가장 빠른 판별 지표였다.** 사용률 100%인데 전력이 낮으면 연산이 아니라 대기다. + +```powershell +# 폴백 여부 직접 확인 (nvidia-smi로는 안 보인다) +(Get-Counter '\GPU Adapter Memory(*)\Shared Usage','\GPU Adapter Memory(*)\Dedicated Usage').CounterSamples +``` + +### 기여 요인 — cfg의 검증 설정 + +```yaml +train: { voxel_max: 64000 } +val: { voxel_max: null } # ← 타일 전체(47만 포인트)를 통째로 +``` + +학습 스텝은 상한이 있는데 **검증만 무제한**이다. 이 거대한 일시 할당이 캐싱 할당자 풀을 +부풀린다. 감속 시작이 epoch 10 검증 직후인 것과 일치한다. + +**핵심**: PyTorch 캐싱 할당자는 한 번 shared로 넘어간 풀을 되돌리지 않는다. +VRAM이 비어도 자동 회복 없음 → **프로세스 재시작이 유일한 복구 경로.** + +### 조치 + +| 항목 | 값 | +|---|---| +| 할당자 | `PYTORCH_CUDA_ALLOC_CONF=garbage_collection_threshold:0.7,max_split_size_mb:128` | +| 검증 상한 | `dataset.val.voxel_max=64000` (학습과 동일) | +| 재개 | epoch 20 체크포인트 (진행분 손실 없음) | + +**복구 확인** + +``` +반복 속도 : 3.56 it/s = 0.28 s/it (벤치 0.291과 일치) +GPU 전력 : 61.7W → 140.08W +``` + +### ⚠️ `expandable_segments`는 torch 2.0에서 못 쓴다 + +첫 시도에서 `PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True`를 썼다가 즉시 크래시. + +``` +RuntimeError: Unrecognized CachingAllocator option: expandable_segments +``` + +**torch 2.1부터 지원**한다. 이 환경은 2.0.1. +게다가 traceback이 `model.to(device)`를 가리켜 모델 문제처럼 보인다. + +→ [scripts/check_alloc_conf.sh](../scripts/check_alloc_conf.sh)로 **실행 전 2초 만에 검증**하도록 만들었다. + +워치독은 이때 "재시도해도 새 체크포인트가 안 생긴다 = 학습 진입 전 실패"를 감지하고 +무한 재시도 없이 정상 중단했다. 설계 의도대로 동작. + +### 남는 편차 — 명시 + +`dataset.val.voxel_max`를 64000으로 제한했으므로 **val_miou는 서브샘플된 타일 기준**이다. +체크포인트 선택 신호로만 유효하다. 최종 test는 `test()`가 전체 타일을 슬라이딩 윈도우로 +처리하므로 **보고 수치에는 영향이 없다.** + +### 감시에 반영 + +침묵이 성공으로 오인되지 않도록 [scripts/monitor_training.sh](../scripts/monitor_training.sh)에 +**속도 회귀 감지**를 추가했다. 1.5 s/it을 넘으면 GPU 메모리·전력과 함께 즉시 알린다. +크래시만 감시하면 이런 무증상 감속은 영원히 못 잡는다. + +--- + +## 3-4. 실학습 결과 — pointnet 100 epoch (2026-08-21) + +### 완주 + +``` +epoch 100/100 완료 01:56:58 +Best ckpt @E90 + val_oa 49.12 + val_macc 25.47 + val_miou 17.19 +``` + +| 항목 | 값 | +|---|---| +| 모델 | pointnet (3.6M params) | +| 데이터 | `face_labeling/texsp_pcl`, train 24 / val 8 / test 8 | +| 설정 | `voxel_max 64000`(원본), `batch_size 2`, `val_freq 5`, cosine LR 0.01→1.2e-5 | +| 소요 | 20:08 시작 → 01:57 완료. 순수 학습 시간 약 3.4h (감속 사고 3h 별도) | +| 속도 | 102초/epoch, 0.28 s/it (벤치 0.291과 일치) | + +### 수렴 추이 + +| epoch | train_miou | best val_miou | +|---|---|---| +| 10 | 12.80 | 2.77 | +| 20 | 15.79 | 8.04 | +| 31 | 17.56 | 8.49 | +| 40 | 17.86 | 10.40 | +| 50 | 18.89 | 15.62 | +| 60 | 19.22 | 16.04 | +| 70 | 19.09 | 16.93 | +| 80 | 19.62 | 16.93 | +| **90** | — | **17.19** ← best | +| 100 | 19.95 | 17.19 | + +train 19.95 / val 17.19 — 격차가 작아 과적합 징후 없다. + +### ★ test 세트는 라벨이 없다 (블라인드) + +학습 직후 test 단계에서 크래시: + +``` +main.py:672 cm.update(pred, label) +metrics.py:74 RuntimeError: bincount only supports 1-d non-negative integral inputs +``` + +원인 조사 결과 **우리 설정 문제가 아니었다**: + +| split | 라벨 값 | +|---|---| +| train 24타일 | `0 ~ 12` 정상 | +| val 8타일 | `0 ~ 12` 정상 | +| **test 8타일** | **전부 `-1`** (dtype int32, uniq=1) | + +README와 일치한다: + +> "For fine-grained test set evaluation consistent with the paper, +> **send predictions to our email for local assessment.**" + +의도적인 블라인드 테스트셋이고 `-1`은 자리표시자다. +업스트림 코드가 `if label is not None`만 검사하고 `-1`을 안 걸러서, +`true × num_classes + pred`가 음수가 되어 `torch.bincount`가 거부한 것. + +> **로컬에서 test_miou는 원천적으로 계산 불가.** 논문 수치와 직접 대조하려면 +> 예측 결과를 저자에게 보내야 한다 (gaoweixiaocuhk@gmail.com). + +### 업스트림 버그 패치 2건 + +| # | 증상 | 원인 | 패치 | +|---|---|---|---| +| A | `bincount ... non-negative` | 블라인드 test의 `-1` 라벨 미처리 | [patch_unlabeled_test.sh](../scripts/patch_unlabeled_test.sh) — 전부 음수면 "정답 없음"으로 간주, 예측만 생성 | +| B | `UnboundLocalError: 'epoch'` (`mode=val`) | `epoch`이 학습 루프 안에서만 바인딩되는데 val 경로가 참조 | [patch_val_mode.sh](../scripts/patch_val_mode.sh) — `epoch = best_epoch` 선바인딩 | + +둘 다 멱등이고 원본은 `main.py.orig`로 보존한다. 라벨 있는 데이터의 채점 동작은 바뀌지 않는다. + +### 워치독 동작 검증 + +학습 완주 후 test 크래시로 rc=1이 되자 워치독이 재개를 시도했고, +2회째에 **"새 체크포인트가 생기지 않음 = 학습 진입 전 실패"**를 감지해 중단했다. +무한 재시도 없이 설계대로 동작했다. + +### 남는 편차 + +학습 중 `dataset.val.voxel_max`를 64000으로 제한했다(§3-2-1). 따라서 위 `val_miou 17.19`는 +**서브샘플된 검증셋 기준**이다. `voxel_max: null`(전체 타일)로 다시 평가한 수치는 +[scripts/final_eval.sh](../scripts/final_eval.sh) 결과로 별도 기록한다. + +--- + +## 3-3. WSL2 안정화 + +VM이 **2회** 사망했다. + +| # | 시점 | 상황 | +|---|---|---| +| 1 | CUDA 확장 컴파일 중 | `Wsl/Service/CreateInstance/E_FAIL`. pip 진행 중 파일들이 **0바이트**로 남음 (§3-1 ⑯) | +| 2 | GPU 벤치(VRAM 만재) + 4.7GB 다운로드 동시 | 두 작업 동시 exit 253 | + +`.wslconfig`가 없어 WSL2가 호스트 64GB의 50%(≈32GB)를 동적으로 잡고 있었다. + +**적용한 설정** (`C:\Users\nbright\.wslconfig`) + +```ini +[wsl2] +memory=24GB # 동적 50% → 고정 상한 +swap=16GB # 스파이크 시 VM 사망 대신 페이지아웃 +processors=8 # cfg의 num_workers: 6 감안 +vmIdleTimeout=60000 # 연속 실행 시 VM 부팅 비용 회피 +``` + +`pageReporting`은 뺐다 — 기본값이 `true`이고, 명시하면 WSL 2.7.12가 파싱 경고를 낸다. + +적용 후 확인: `Mem 23GB / Swap 16GB / 8 cpus`, GPU 정상 인식, 경고 없음. + +**운용 규칙**: GPU 만재 작업과 대용량 다운로드를 **동시에 돌리지 않는다.** + +> `wsl --shutdown`은 WSL 배포판만 종료한다. Windows 네이티브 프로세스는 영향 없다. + +--- + +## 4. 한국 데이터 적용 계획 + +### 두 갈래 경로 + +**A. 포인트 클라우드 경로 (권장, 스모크 테스트용)** + +포인트 기반 방법(KPConv/PointNeXt/RandLA-Net)은 `xyz + rgb + label`만 필요. 텍스처 불필요. + +``` +한국 메시(OBJ+texture) → face 샘플링 → binary PLY (v:color, v:label) +``` + +- SUM으로 학습한 모델 → 한국 타일 zero-shot 추론 +- 라벨 없으면 전부 `0`(unclassified)로 채워도 추론은 동작 + +**B. 메시 경로 (텍스처 기반 21클래스 풀 파이프라인)** + +`h:texcoord` + `f:label` 포함 ASCII PLY 필요. 변환 공수 큼. 후순위. + +### 확보된 실데이터 — 서산 명천 (2026-08) + +`D:\MyProject_대용량샘플\02. 데이터\01. 서산 명천_08월` + +``` +├── 01. 드론원본이미지 JPG 8,120장 (132 GB) + PPK(.obs/.nav/.MRK) +│ 고고도 120m / 저고도 70m × 수직 / 경사, 1구역(미션1)·2구역(미션7) +├── 02. 본태모델 ★ OBJ 6개 (3.0 GB) + JPG 텍스처 144장 (908 MB) +├── 03. 지오이드모델 .tif 2.8 GB +├── 04. 카메라파라미터 .txt 6개 +└── 05. 정사영상 .tif 25.8 GB + .tfw + .dwg +``` + +**본태모델 = ContextCapture 계열 OBJ, 블록 6개** + +| 구역 | 블록 | OBJ 크기 | +|---|---|---| +| 구역1(미션1) | BlockYBY / BlockYYA / BlockYYX | 645 / 433 / 464 MB | +| 구역2(미션7) | BlockBXY / BlockYBA / BlockYBX | 497 / 355 / 634 MB | + +**`metadata.xml` — 좌표계 (결정적)** + +```xml +EPSG:5186+9999 +154113.37472199186,466558.57182090241,151.18628846539363 +155184.79128718161,469705.02542312664,149.57080945797654 +``` + +→ **좌표가 이미 로컬 미터계 + 원점 오프셋.** SUM Parts 배포 관례와 동일. +재투영 작업 불필요. (단 원본 easting/northing이나 경위도를 그대로 넣으면 cfg의 +반경·voxel 설정이 전부 미터 기준이라 조용히 망가진다.) + +**메시 실측** (`scripts/inspect_obj.py`) + +| | BlockYBA | BlockYYA | SUM Parts demo | +|---|---|---|---| +| vertex | 2,287,325 | 2,699,696 | — | +| face | 4,553,583 | 5,377,777 | — | +| UV | ✅ 3,121,900 | ✅ 3,886,962 | ✅ | +| vertex color | ✗ | ✗ | — | +| 머티리얼 | **17** | **21** | — | +| extent | 465×487×28 m | 439×404×46 m | 252×252×40 m | +| 면적 | 0.227 km² | 0.177 km² | — | +| 밀도 | 20.1 face/m² | 30.3 face/m² | — | + +**적합 판정 근거** +- 텍스처 메시 + UV 완비 → SUM Parts와 동일 모달리티 +- 좌표계 관례 일치 +- 블록이 SUM 타일(252m)의 약 4배 면적 → 블록당 4타일, 6블록 = **약 24타일** 확보 가능 + +**주의 2가지** +1. **멀티머티리얼** — 블록당 텍스처 아틀라스 17~21장. `trimesh.load(force='mesh')`로 + 합치면 텍스처가 전부 날아가 결과가 회색이 된다. Scene으로 받아 머티리얼별로 샘플링해야 함. +2. **vertex color 없음** — 텍스처가 유일한 색상 소스. 폴백 경로가 없다. + +### 왜 PLY로 변환하나 (OBJ/GLB 아니고) + +포맷 선택이 아니라 **모델 요구사항**이다. + +```python +# openpoints/dataset/sumv2_triangle/sumv2_triangle.py +self.data_list = glob.glob(os.path.join(data_root, split, "*.ply")) +plydata = PlyData.read(filename) +labels = plydata['vertex']['label'] +``` + +- 데이터로더가 `*.ply`만 glob한다. OBJ/GLB는 스캔조차 안 함 +- 네트워크 입력은 **포인트 클라우드**지 메시가 아니다 +- OBJ/GLB에는 포인트별 `label` 스칼라를 담을 표준 필드가 없다. + SUM Parts 저자가 텍스처 메시조차 PLY로 배포하는 이유도 같다 (`f:label`, `v:label`) + +원본 OBJ는 읽기만 하고 손대지 않는다. PLY는 모델 입력용 중간 산출물이다. + +``` +BlockYBA.obj (원본 불변) → 면적가중 샘플링 → tile.ply → 모델 → 예측 +``` + +### 다른 한국 데이터 후보 (참고) + +- **V-World 3D** (국토지리정보원) — 3D 건물 / 지형 +- **서울 S-Map** — 3D 서울, 텍스처 메시 + +--- + +## 4-1. POC 결과 (2026-08-20) — 통과 + +범위를 의도적으로 **타일 1장**으로 제한. "우리 데이터가 이 파이프라인을 통과하는가"만 확인. +성능 측정 아님. + +**입력**: 구역2(미션7)/BlockYBA, 중앙 252m bbox `(300, 720) ~ (552, 972)` + +``` +[17:10:12] loaded 17 geometry group(s), 4,553,583 faces total +[17:10:13] cropped surface area: 14,609 m^2 across 283,099 faces + geom 0..16 : 머티리얼별 면적 비례 샘플링 +[17:10:19] extent: [137.84 178.07 19.48] m origin: [299.6 794.32 -67.58] +[17:10:19] wrote seosan_BlockYBA_tile0.ply (470,000 points, label=0, colour=sum) +``` + +**스키마 대조** (`scripts/check_ply.py`) + +| | 변환 결과 | SUM demo | 판정 | +|---|---|---|---| +| properties | `x,y,z,r,g,b,label` | `x,y,z,nx,ny,nz,r,g,b,label,sp_id` | OK (법선·sp_id는 로더가 무시) | +| colour | float32 [0,1] | float32 [0,1] | ✅ 일치 | +| label dtype | int32 | int32 | ✅ | +| extent | 137.8 × 178.1 × 19.5 m | 252.2 × 252.2 × 39.7 m | 노선형 vs 정사각 타일 | +| density | 19.1 pts/m² | 7.4 pts/m² | 2.6× 조밀 | + +**추론** (`scripts/poc_infer.sh`) + +``` +Successful Loading the ckpt from ..._ckpt_best.pth +length of test dataset: 1 +Test on 0-th cloud [2]/[3]: 100% +test_oa nan, test_macc 0.00, test_miou 0.00 +POC INFER DONE (exit=0) +``` + +`test_oa = nan`은 **정상**이다. 전 포인트 라벨이 0이고 `ignore_index: 0`이라 채점 대상이 없다. + +**예측 산출물** (`scripts/check_pred.py`) + +``` +visualization/seosan_BlockYBA_tile0_pred.ply 13 MB +points : 470,000 +properties : ['x','y','z','red','green','blue','label'] +palette fit: exact + +class points share +wall 470,000 100.00% +``` + +470,000 포인트 전부에 클래스 예측이 붙었다. 전부 `wall`인 건 1 epoch 모델(val_miou 0.21)이 +사실상 단일 클래스만 뱉기 때문이며, 예상된 결과다. + +> **결론: 배관은 뚫렸다.** OBJ → 샘플링 → PLY → 모델 → 포인트별 예측 → 시각화 PLY. +> 모델 품질에 대해서는 이 POC가 아무것도 말해주지 않는다. + +### POC에서 드러난 것 3가지 + +**① 색상 스케일 — 조용히 망가지는 함정 ★** + +첫 변환에서 `red,green,blue` uint8 [0,255]로 썼다. SUM 배포본은 `r,g,b` **float32 [0,1]**이다. + +로더는 정규화하지 않는다: +```python +rgb = np.stack([...'r','g','b'...]).astype(np.float32) +if np.max(rgb) > 1: + rgb = rgb # ← no-op. 읽으면 정규화할 것 같지만 아무것도 안 한다 +``` +cfg의 활성 transform도 `[PointsToTensor, PointCloudScaling, PointCloudRotation, +PointCloudJitter]` 뿐 — `NumpyChromaticNormalize`는 주석 처리돼 있다. + +→ **파일에 든 값이 그대로 네트워크에 들어간다.** 255배 큰 색상 특징을 주게 되고, +에러 없이 결과만 무의미해진다. `--colour-style sum`이 기본값인 이유. + +**② 블록은 정사각형이 아니다 — 노선을 따라 뻗은 형태** + +252×252m bbox로 잘랐는데 실제 표면적은 14,609 m² (bbox 면적 63,504 m²의 **23%**). +결과 extent도 137×178m. 도로 프로젝트라 노선을 따라 길게 뻗은 형태다. 도시 전역을 덮는 SUM Parts와 형상이 다르다. + +→ 균등 격자로 타일링하면 빈 타일이 대량 발생한다. **노선 축을 따라 자르는 전략**이 필요하다. +앞서 "블록당 4타일 × 6블록 = 24타일" 추정은 낙관적이며, 실제로는 더 적다. + +**③ 멀티머티리얼 처리 필수** + +`trimesh.load(..., force='mesh')`로 합치면 머티리얼별 텍스처가 전부 유실되어 결과가 +회색으로 나온다. Scene으로 받아 geometry(=머티리얼)별로 UV 샘플링해야 한다. +BlockYBA는 17개, BlockYYA는 21개 그룹. + +### 다음 단계에서 필요한 것 + +POC는 배관만 확인했다. 의미 있는 결과를 내려면: + +1. **제대로 학습된 체크포인트** — SUM Parts 전체 데이터(`mesh.zip`/`pcl.zip`)로 학습. + 현재는 demo 타일 1장 1 epoch짜리뿐이다. +2. **노선 축 기반 타일링** — 빈 타일 회피 +3. **한국 타일 라벨링** — zero-shot 성능을 측정하려면 GT가 필요하다. 최소 2~3타일. + SAM 기반 어노테이션 도구(`interactive_annotation/`)가 repo에 있다. +4. **클래스 정의 대조** — SUM 13클래스(terrain/roof/facade/chimney/dormer...)는 도시 건물 중심. + 도로 프로젝트에서 필요한 건물/수목/차량/지면은 SUM 13클래스를 합치면 그대로 나온다 (§4-2). + +### 예상 결과 + +SUM Parts는 헬싱키 항공 메시 기반. 한국 도시는 지붕 형태(평지붕 + 옥탑), 건물 밀도, 도로 표시가 다르다. +→ **zero-shot mIoU 큰 폭 하락 예상**. 그게 정상이며, 그 수치 자체가 저자에게 보낼 메일의 근거가 된다. +타일 2~3장 라벨링해 fine-tune 하면 크게 개선될 것. + +--- + +## 5. 진행 순서 + +1. ~~환경 조사~~ ✅ +2. ~~WSL2 환경 셋업~~ ✅ (miniconda, conda env py3.10, CUDA 11.8, torch 2.0.1+cu118) +3. ~~repo clone~~ ✅ `~/sum-parts` +4. ~~PointNeXt CUDA 확장 빌드~~ ✅ (3종 전부 import 확인) +5. ~~HF 게이트 수락~~ ✅ (브라우저 수동) +6. ~~`demo.zip` 다운로드 + 압축 해제 + 스키마 검증~~ ✅ +7. ~~1 epoch 스모크 테스트~~ ✅ **파이프라인 생존 확인** +8. ~~한국 실데이터 조사~~ ✅ 서산 명천, ContextCapture OBJ 6블록 +9. ~~한국 타일 → PLY 변환~~ ✅ BlockYBA 타일 1장, 470k 포인트 +10. ~~POC 추론~~ ✅ **배관 관통 확인** (§4-1) +11. **제대로 된 학습** ← 다음. `pcl.zip` 전체 받아 실학습 +12. 노선 축 기반 타일링 + 한국 타일 라벨링 +13. Zero-shot 평가 + Mapple 시각화 +14. 결과 정리 후 저자에게 메일 + +### 스모크 테스트 결과 (2026-08-20) + +``` +cfg=pointnet epochs=1 loop=4 bs=2 + +Train Epoch [1/1] Loss 3.474 Acc 0.08 +Best ckpt @E1 val_oa 2.48 val_macc 8.33 val_miou 0.21 +Test [0]/[1] cloud +Best ckpt @E1 test_oa 2.48 test_macc 8.33 test_miou 0.21 +save results in log/.../ALo5iXF6AAHrWkAghKA8nv.csv +exit=0 +``` + +**체인 전 구간 통과**: PLY 로드 → grid subsample → CUDA ops → forward/backward +→ 검증 → 체크포인트 저장 → 재로드 → 테스트 추론 → CSV 저장. + +⚠️ **수치는 무의미하다.** 1 epoch, 타일 1장, train/val/test 모두 같은 타일. +파이프라인 생존만 확인한 것이고, 어떤 성능 주장에도 쓸 수 없다. + +### 검증된 스택 + +``` +python 3.10 +nvcc 11.8.89 +torch 2.0.1+cu118 cuda available: True (RTX 3060) +numpy 1.26.4 +ninja 1.11.1.1 (1.12+ 금지) +setuptools 69.5.1 (74+ 금지) +torch-scatter 2.1.2+pt20cu118 + +pointnet2_batch_cuda : OK +pointops_cuda : OK +grid_subsampling : OK +plyfile / torch_scatter : OK +``` + +### 스크립트 + +| 파일 | 역할 | +|---|---| +| [scripts/setup_env.sh](../scripts/setup_env.sh) | conda env + CUDA + torch | +| [scripts/setup_pointnext.sh](../scripts/setup_pointnext.sh) | python 의존성 → build_ext.sh 호출 | +| [scripts/build_ext.sh](../scripts/build_ext.sh) | CUDA 확장 빌드만 (재실행 가능, `FORCE=1`로 전체 재빌드) | +| [scripts/repair_env.sh](../scripts/repair_env.sh) | WSL 크래시로 0바이트가 된 패키지 재설치 | +| [scripts/patch_numpy_aliases.sh](../scripts/patch_numpy_aliases.sh) | 제거된 numpy 별칭 / collections ABC 소스 패치 | +| [scripts/verify_env.py](../scripts/verify_env.py) | 컴파일 확장 + import 체인 일괄 점검 | +| [scripts/download_data.sh](../scripts/download_data.sh) | HF 데이터 다운로드 + 압축 해제 | +| [scripts/prepare_demo_split.sh](../scripts/prepare_demo_split.sh) | demo 타일을 train/val/test로 배치 | +| [scripts/link_data.sh](../scripts/link_data.sh) | `PointNeXt_bundle/data` 심볼릭 링크 | +| [scripts/smoke_train.sh](../scripts/smoke_train.sh) | 1 epoch 파이프라인 생존 테스트 | +| [scripts/inspect_obj.py](../scripts/inspect_obj.py) | 대용량 OBJ 스트리밍 통계 (vertex/face/UV/bbox/머티리얼) | +| [scripts/mesh_to_ply.py](../scripts/mesh_to_ply.py) | 멀티머티리얼 텍스처 메시 → SUM 스키마 PLY (bbox 크롭 지원) | +| [scripts/check_ply.py](../scripts/check_ply.py) | PLY를 SUM 스키마와 대조 검증 | +| [scripts/poc_korea.sh](../scripts/poc_korea.sh) | 서산 명천 타일 1장 변환 + 검증 | +| [scripts/poc_infer.sh](../scripts/poc_infer.sh) | 변환된 타일로 추론 | +| [scripts/check_pred.py](../scripts/check_pred.py) | 예측 PLY의 클래스 분포 복원 (팔레트 역매핑) | +| [scripts/poc_check_pred.sh](../scripts/poc_check_pred.sh) | 최신 예측 PLY 자동 탐색 후 검사 | + +### 실행 순서 (처음부터 재현 시) + +```bash +bash scripts/setup_env.sh # conda env + CUDA 11.8 + torch 2.0.1 +bash scripts/setup_pointnext.sh # 의존성 + CUDA 확장 빌드 +bash scripts/patch_numpy_aliases.sh # 소스 현대화 패치 +bash scripts/download_data.sh # HF 게이트 수락 후 +bash scripts/prepare_demo_split.sh +bash scripts/link_data.sh +python scripts/verify_env.py # 전부 OK 확인 +bash scripts/smoke_train.sh +``` + +WSL이 중간에 죽으면 `bash scripts/repair_env.sh` 먼저 돌리고 재개. + +--- + +## 6. 메모 + +- VSCode에서 WSL 전환: 좌하단 `><` 버튼 → "Connect to WSL" +- 다만 Windows 셸에서 `wsl -d Ubuntu-22.04 -- bash -lc ""` 로 직접 실행도 가능 (전환 불필요) +- `Downloads`에 `교각.hmeg`, `교량.hmeg`, `sss.obj` 등 자체 3D 데이터 존재 → 한국 샘플 후보로 검토 가치 있음 diff --git a/docs/pipeline.html b/docs/pipeline.html new file mode 100644 index 0000000..a44ab3a --- /dev/null +++ b/docs/pipeline.html @@ -0,0 +1,646 @@ +서산 명천 메시 분할 파이프라인 + + + + + + +
+ +
+
공정 정의 · v1 · 2026-08-21
+

서산 명천 메시 분할 파이프라인

+

+ 드론 사진측량 메시를 건물·수목·차량·지면으로 분할하고, + 원본 면(face)을 클래스별로 쪼개 내보내기까지의 전 공정. +

+ +
+
대상
서산 명천 도로
+
원본
OBJ 6블록 · 3.0 GB
+
좌표계
EPSG:5186
+
학습 자산
SUM Parts
+
모델
PointVector
+
연산
RTX 3060 12 GB
+
+
+ +

+ 공정은 6단계다. 앞 3단계는 1회성 자산 구축이고, + 뒤 3단계는 타일마다 반복된다. + 각 단계에는 통과 조건(게이트)이 있다. 게이트를 건너뛰면 + 다음 단계에서 원인을 알 수 없는 형태로 실패한다 — 실제로 그렇게 여러 번 잃었다. +

+ +

공정

+ +
+ + +
+
+ 단계 0 + 실행 환경 + 1회 · 반나절 +
+

+ 커스텀 CUDA 커널을 직접 빌드해야 하므로 Windows 네이티브로는 안 된다. WSL2에서만 성립한다. +

+
+

입력

없음

+

처리

conda 환경 · CUDA 11.8 · torch 2.0.1 · CUDA 확장 5종 빌드

+

출력

동작하는 sumparts 환경

+
+
# 환경 + 확장 빌드 + 소스 현대화 패치 +bash scripts/setup_env.sh +bash scripts/setup_pointnext.sh +bash scripts/patch_numpy_aliases.sh +bash scripts/patch_unlabeled_test.sh +bash scripts/patch_val_mode.sh
+
+ 게이트 + python scripts/verify_env.py 가 ALL OK 를 낼 것. + 확장 6종 import + openpoints 체인 전부 통과해야 한다. +
+
+ + +
+
+ 단계 1 + 학습 데이터 확보 + 1회 · 30분 +
+

+ 한국 데이터에 라벨이 없으므로 SUM Parts로 대신 학습한다. + 저자가 학습 가중치를 공개하지 않아 직접 학습이 유일한 경로다. +

+
+

입력

HuggingFace 게이트 수락 (브라우저 수동)

+

처리

다운로드 5.3 GB → 전개 13 GB → split 정리

+

출력

train 24 / val 8 / test 8 타일

+
+
bash scripts/download_data.sh all +bash scripts/prepare_full_split.sh +bash scripts/link_data.sh
+
+

배포본 split 이름이 로더와 다르다

+

디렉토리는 validate/인데 로더는 val/을 찾는다. + 그대로 두면 예외 없이 검증 0개로 학습이 돌아간다. + prepare_full_split.sh가 심볼릭 링크로 해결한다.

+
+
+ 게이트 + 세 split 모두 24 / 8 / 8로 집계될 것. +
+
+ + +
+
+ 단계 2 + 모델 학습 + 1회 · 4시간 +
+

+ PointVector를 쓴다. 논문 실측 mIoU 70.0 %로 번들 최고이고, PointNeXt보다 빠르다. + PointNet은 15.1 %로 실무용이 아니다. +

+
+

입력

SUM Parts face 트랙 · 13클래스

+

처리

100 epoch · voxel_max 24000 · 워치독 자동 재개

+

출력

ckpt_best.pth 41 MB

+
+
bash scripts/launch_overnight.sh pointvector-xl +bash scripts/check_training.sh # 진행 확인
+
+

VRAM 초과가 OOM을 내지 않는다

+

WSL2 드라이버는 VRAM을 넘으면 호스트 RAM으로 흘린다. + 학습은 조용히 완주한다 — 25~100배 느리게. + 12 GB 카드에서 peak 15.47 GB가 찍힌다.

+

판별법: 전력. 사용률 100 %인데 전력이 낮으면(3060 기준 60 W대) + 연산이 아니라 PCIe 전송 대기다. 정상이면 140 W대.

+
+
+ 게이트 + verify_speed.sh 가 HEALTHY 를 낼 것. + peak VRAM < 11 GB 이고 전력이 120 W를 넘어야 한다. +
+
+ + +
+
+ 단계 3 + 한국 데이터 변환 + 타일마다 · 5분 +
+

+ 원본 OBJ는 읽기만 한다. 모델 입력은 포인트 클라우드 PLY여야 한다 — + 데이터로더가 *.ply만 스캔하고, OBJ에는 포인트별 라벨 필드가 없다. +

+
+

입력

ContextCapture OBJ + 텍스처 아틀라스 17~21장

+

처리

노선 축 타일링 → 면적가중 샘플링 → 텍스처 UV 조회

+

출력

타일당 47만 포인트 PLY

+
+
python scripts/inspect_obj.py Block.obj # 범위·머티리얼 파악 +python scripts/mesh_to_ply.py Block.obj tile.ply \ + --bbox X0 Y0 X1 Y1 --points 470000 +python scripts/check_ply.py tile.ply # 스키마 대조
+
+

색상 스케일이 조용히 망가진다

+

SUM 배포본은 r,g,b float32 [0,1]이다. + red,green,blue uint8로 쓰면 로더가 정규화를 하지 않아 + 255배 큰 특징값이 네트워크에 들어간다. 에러 없이 결과만 무의미해진다.

+

멀티머티리얼도 함정이다. trimesh.load(force='mesh')로 합치면 + 텍스처가 전부 유실되어 결과가 회색이 된다. Scene으로 받아 머티리얼별로 샘플링해야 한다.

+
+
+ 게이트 + check_ply.py가 SUM 배포본과 나란히 OK를 낼 것. + 회색 폴백 경고가 없어야 한다. +
+
+ + +
+
+ 단계 4 + 추론 및 클래스 통합 + 타일마다 · 1분 +
+

+ SUM 13클래스를 우리가 필요한 4클래스로 합친다. + 창·문 수준 세부는 필요 없으므로 세부 클래스의 개별 성적은 비용이 아니다. +

+
+

입력

변환된 PLY + 학습 체크포인트

+

처리

슬라이딩 윈도우 추론 → 13→4 매핑

+

출력

포인트별 클래스 + 예측 PLY

+
+
+ + + + + + + + +
우리 클래스SUM 원본 클래스논문 IoU
건물facade · roof · chimney · dormer · balcony · roof_installation · wall85.9–91.7 %
수목high_vegetation96.8 %
차량car · boat95.2 %
지면 (도로 포함)terrain92.3 %
+
+

+ 도로면은 face 트랙에서 terrain에 포함된다. 별도 road 클래스는 + texture 트랙(19클래스)에만 있고 우리에겐 필요 없다. +

+
+ 게이트 + Mapple로 육안 확인 + 통합 4클래스 mIoU 측정. + "전부 건물" 무지성 분류기(IoU 67 %)를 반드시 이길 것. + 못 이기면 모델이 퇴화한 상태다. +
+
+ + +
+
+ 단계 5 + 메시 분할 + 타일마다 · 미구현 +
+

+ 최종 산출물. 포인트 예측을 원본 면으로 되돌려 클래스별 OBJ로 쪼갠다. +

+
+

입력

포인트별 예측 + 원본 OBJ

+

처리

포인트→면 역매핑 → 면 다수결 → 클래스별 분리

+

출력

building.obj · vegetation.obj · vehicle.obj · ground.obj

+
+
+

원본 vertex는 불가침

+

분할은 재생성이 아니라 면(face) 분할이다. + 구멍 메우기는 새 vertex 덧대기만 허용된다. 원본 지면 Z를 DTM으로 덮어쓰면 실패다.

+

현재 변환기는 샘플링 시 face index를 남기지 않는다. + 역매핑을 위해 face index 보존 기능 추가가 필요하다.

+
+
+ 게이트 + 분할된 면 수의 합 = 원본 면 수. 누락도 중복도 없을 것. +
+
+ +
+ +

정확도가 부족할 때

+ +

+ 단계 4의 게이트를 통과 못 하면 도메인 갭 때문이다. + 헬싱키 도시로 학습한 모델을 한국 도로 현장에 적용하는 구조적 한계다. + 그때 붙이는 분기가 아래다. +

+ +
+
+ 분기 + 자동 라벨링 + 파인튜닝 + 조건부 +
+
+

입력

드론 원본 8,120장 + 카메라 포즈

+

처리

SAM 3.1 텍스트 프롬프트 → 2D 마스크 → 포즈로 3D 투영 → 면 투표

+

출력

한국 데이터 자동 라벨 → 파인튜닝

+
+
    +
  • SAM 3의 텍스트 프롬프트가 전제다. SAM 2는 덩어리만 나누고 이름을 못 붙인다.
    • +
  • 카메라 포즈 없으면 2D→3D 전이 자체가 불가능하다. 서산 명천은 PPK와 ContextCapture 메타가 있어 성립한다.
    • +
  • 가림 처리 필요 — 메시로 depth test 해서 안 보이는 면에 투표하면 안 된다.
    • +
  • 사람 몫은 전수 라벨링이 아니라 소량 검증이다. 자동 라벨로 학습하고 자동 라벨로 평가하면 같은 오류를 서로 확인해주는 꼴이라 정확도를 알 수 없다. 최소 1~2타일은 사람이 확인한 정답이 있어야 한다.
    • +
+
+ +

모델 선택 근거

+ +
+ + + + + + + + + + +
모델발표핵심논문 mIoU우리 실측 s/iter
PointNet2017전체 한 번에 max-pool. 이웃 개념 없음15.1 %0.291
PointNet++2017이웃끼리 묶어 계층 처리33.1 %0.675
PointNeXt2022++ 구조 유지, 학습법·크기 개선65.3 %0.635
PointVector2023이웃 특징을 고차원 벡터로 합침70.0 %0.402
+
+ +

+ s/iter는 RTX 3060 12 GB 실측, XL 모델은 VRAM에 맞춰 voxel_max를 낮춘 값이다. + PointVector가 가장 정확하고 동시에 가장 빠르다. +

+ +

현재 상태

+ +
+ + + + + + + + + + +
단계상태비고
0 · 실행 환경완료확장 5종 빌드, 패치 4건
1 · 학습 데이터완료13 GB 전개, 24/8/8
2 · 모델 학습진행 중PointVector 100 epoch
3 · 데이터 변환검증됨BlockYBA 1타일 POC 통과
4 · 추론·통합대기학습 완료 후
5 · 메시 분할미구현face index 보존 필요
+
+ +

제약

+ +
    +
  • test 세트는 블라인드다. 라벨이 전부 -1이라 로컬 채점이 원천 불가하다. 논문 수치와 직접 대조하려면 예측을 저자에게 보내야 한다.
    • +
  • 어노테이션 도구는 비공개다. 저자가 유료 서비스로 판매 중이라 한국 데이터 라벨링에 쓸 수 없다.
    • +
  • XL 모델은 논문 설정 재현이 불가하다. voxel_max를 64000에서 낮춰야 12 GB에 들어간다. 이 값은 중립적 손잡이가 아니라 모델의 동작점 일부다 — 같은 체크포인트가 프로토콜에 따라 mIoU 17.19 / 4.20으로 갈렸다.
    • +
  • 데이터는 CC BY-NC 4.0, 코드는 GPL-3.0이다. 상업 이용은 저자 허락이 필요하고, NC 데이터로 뽑은 가중치도 NC로 취급하는 게 안전하다.
    • +
+ +
+ 실측 환경 — RTX 3060 12 GB · WSL2 Ubuntu 22.04 · CUDA 11.8 · torch 2.0.1 · Python 3.10
+ 상세 기록은 docs/SUM-Parts-검토노트.md, 스크립트는 scripts/. +
+ +
diff --git a/scripts/bench_models.py b/scripts/bench_models.py new file mode 100644 index 0000000..94f6856 --- /dev/null +++ b/scripts/bench_models.py @@ -0,0 +1,157 @@ +#!/usr/bin/env python3 +"""Measure peak VRAM and per-iteration time for each sumv2_triangle model. + +Answers "will full training fit on this GPU, and how long would it take" with +numbers off the actual card rather than a guess. Builds the model and the real +train loader from each cfg, runs a handful of train steps at the cfg's own +batch_size, and reports peak allocated memory plus iterations/second. + +Run from PointNeXt_bundle/examples/segmentation. + +Usage: + python bench_models.py [--iters 8] [--cfgs pointnet pointnext-xl ...] +""" +from __future__ import annotations + +import argparse +import sys +import time +from pathlib import Path + +import torch + +sys.path.append(str(Path(__file__).resolve().parent)) +sys.path.append("../../") + +from openpoints.utils import EasyConfig # noqa: E402 +from openpoints.dataset import build_dataloader_from_cfg, get_features_by_keys # noqa: E402 +from openpoints.models import build_model_from_cfg # noqa: E402 +from openpoints.loss import build_criterion_from_cfg # noqa: E402 +from openpoints.optim import build_optimizer_from_cfg # noqa: E402 + +ALL_CFGS = ["pointnet", "pointnet++msg", "pointnext-xl", "pointvector-xl"] + + +def bench(cfg_name: str, iters: int, voxel_max: int | None = None, + batch_size: int | None = None) -> dict: + cfg = EasyConfig() + cfg.load(f"../../cfgs/sumv2_triangle/{cfg_name}.yaml", recursive=True) + cfg.rank, cfg.distributed, cfg.mp = 0, False, False + if voxel_max is not None: + cfg.dataset.train.voxel_max = voxel_max + if batch_size is not None: + cfg.batch_size = batch_size + + model = build_model_from_cfg(cfg.model).cuda() + n_params = sum(p.numel() for p in model.parameters()) + + cfg.criterion_args.weight = None + criterion = build_criterion_from_cfg(cfg.criterion_args).cuda() + optimizer = build_optimizer_from_cfg(model, lr=cfg.lr, **cfg.optimizer) + + train_loader = build_dataloader_from_cfg( + cfg.batch_size, cfg.dataset, cfg.dataloader, + datatransforms_cfg=cfg.datatransforms, split="train", distributed=False, + ) + + torch.cuda.empty_cache() + torch.cuda.reset_peak_memory_stats() + + model.train() + times: list[float] = [] + it = iter(train_loader) + n_points = None + + for i in range(iters): + try: + data = next(it) + except StopIteration: + it = iter(train_loader) + data = next(it) + + for k in data: + data[k] = data[k].cuda(non_blocking=True) + target = data["y"].squeeze(-1) + data["x"] = get_features_by_keys(data, cfg.feature_keys) + if n_points is None: + n_points = int(data["pos"].shape[0] * data["pos"].shape[1]) \ + if data["pos"].dim() == 3 else int(data["pos"].shape[0]) + + torch.cuda.synchronize() + t0 = time.perf_counter() + + logits = model(data) + loss = criterion(logits, target) + loss.backward() + optimizer.step() + optimizer.zero_grad() + + torch.cuda.synchronize() + dt = time.perf_counter() - t0 + if i >= 2: # skip warm-up iterations + times.append(dt) + + peak = torch.cuda.max_memory_allocated() / 1024**3 + reserved = torch.cuda.max_memory_reserved() / 1024**3 + avg = sum(times) / len(times) if times else float("nan") + + del model, optimizer, criterion, train_loader + torch.cuda.empty_cache() + + return { + "cfg": cfg_name, "params_m": n_params / 1e6, "batch_size": cfg.batch_size, + "voxel_max": cfg.dataset.train.voxel_max, "points_per_batch": n_points, + "peak_gb": peak, "reserved_gb": reserved, "sec_per_iter": avg, + } + + +def main() -> None: + ap = argparse.ArgumentParser() + ap.add_argument("--iters", type=int, default=8) + ap.add_argument("--cfgs", nargs="*", default=ALL_CFGS) + ap.add_argument("--voxel-max", type=int, default=None, + help="override dataset.train.voxel_max (cfg default: 64000)") + ap.add_argument("--batch-size", type=int, default=None) + args = ap.parse_args() + + total_gb = torch.cuda.get_device_properties(0).total_memory / 1024**3 + print(f"GPU: {torch.cuda.get_device_name(0)} {total_gb:.1f} GB") + if args.voxel_max or args.batch_size: + print(f"overrides: voxel_max={args.voxel_max} batch_size={args.batch_size}") + print("NOTE: on WSL2 the NVIDIA driver spills past VRAM into host RAM instead") + print(" of raising OOM. A peak above the card's capacity means the run") + print(" was paging over PCIe -- it completes, but uselessly slowly.\n") + + rows = [] + for name in args.cfgs: + print(f"--- benchmarking {name} ---", flush=True) + try: + rows.append(bench(name, args.iters, args.voxel_max, args.batch_size)) + print(f" ok\n", flush=True) + except torch.cuda.OutOfMemoryError as e: + print(f" OOM: {str(e)[:120]}\n", flush=True) + rows.append({"cfg": name, "oom": True}) + torch.cuda.empty_cache() + except Exception as e: # noqa: BLE001 + print(f" FAILED {type(e).__name__}: {str(e)[:200]}\n", flush=True) + rows.append({"cfg": name, "error": f"{type(e).__name__}: {e}"}) + torch.cuda.empty_cache() + + print() + print(f"{'cfg':<16}{'params':>9}{'bs':>4}{'pts/batch':>12}" + f"{'peak VRAM':>11}{'s/iter':>9} fits?") + print("-" * 70) + for r in rows: + if r.get("oom"): + print(f"{r['cfg']:<16}{'':>9}{'':>4}{'':>12}{'OOM':>11}{'':>9}") + elif r.get("error"): + print(f"{r['cfg']:<16} {r['error'][:44]}") + else: + fits = "yes" if r["peak_gb"] < total_gb * 0.95 else "NO (spilling)" + print(f"{r['cfg']:<16}{r['params_m']:>8.1f}M{r['batch_size']:>4}" + f"{r['points_per_batch']:>12,}{r['peak_gb']:>10.2f}G" + f"{r['sec_per_iter']:>9.3f} {fits}") + + +if __name__ == "__main__": + main() diff --git a/scripts/build_ext.sh b/scripts/build_ext.sh new file mode 100644 index 0000000..30cb99d --- /dev/null +++ b/scripts/build_ext.sh @@ -0,0 +1,113 @@ +#!/usr/bin/env bash +# SUM Parts - build only the CUDA extensions (rerunnable) +# +# Split out of setup_pointnext.sh so a failed compile can be retried without +# reinstalling every python dependency. +# +# ninja note: torch 2.0's cpp_extension shells out to `ninja -v` and reads its +# output through a pipe. ninja >= 1.12 dies with SIGPIPE there, which torch +# surfaces only as the useless "Error compiling objects for extension". +# Pinning ninja to 1.11.x avoids it. +set -euo pipefail + +CONDA_ROOT="$HOME/miniconda3" +ENV_NAME="sumparts" +REPO="$HOME/sum-parts/semantic_segmentation/PointNeXt_bundle" + +source "$CONDA_ROOT/etc/profile.d/conda.sh" +conda activate "$ENV_NAME" + +export TORCH_CUDA_ARCH_LIST="8.6" # RTX 3060 +export CUDA_HOME="$CONDA_PREFIX" +export PATH="$CUDA_HOME/bin:$PATH" +export MAX_JOBS="${MAX_JOBS:-4}" + +echo "=== pin ninja to 1.11.x ===" +pip install --no-cache-dir "ninja==1.11.1.1" +ninja --version + +# subsampling/setup.py imports numpy.distutils, which imports +# distutils.msvccompiler. setuptools removed that module in 74.0, so anything +# newer dies with "No module named 'distutils.msvccompiler'". +echo "=== pin setuptools to 69.5.1 (numpy.distutils needs it) ===" +pip install --no-cache-dir "setuptools==69.5.1" +python -c "import setuptools; print('setuptools', setuptools.__version__)" + +# Skip extensions that already import cleanly, so a retry does not redo a +# 5-minute nvcc pass that already succeeded. FORCE=1 rebuilds everything. +have() { python -c "import $1" 2>/dev/null; } + +# NOTE: pointnet2_batch's setup.py names the extension pointnet2_batch_cuda, +# not pointnet2_cuda as in upstream PointNet++ forks. +if [ "${FORCE:-0}" = "1" ] || ! have pointnet2_batch_cuda; then + echo "=== build pointnet2_batch ===" + cd "$REPO/openpoints/cpp/pointnet2_batch" + rm -rf build ./*.egg-info dist + python setup.py install +else + echo "=== skip pointnet2_batch (already importable) ===" +fi + +echo "=== build subsampling ===" +cd "$REPO/openpoints/cpp/subsampling" +rm -rf build ./*.so +python setup.py build_ext --inplace + +if [ "${FORCE:-0}" = "1" ] || ! have pointops_cuda; then + echo "=== build pointops ===" + cd "$REPO/openpoints/cpp/pointops" + rm -rf build ./*.egg-info dist + python setup.py install +else + echo "=== skip pointops (already importable) ===" +fi + +# chamfer_dist and emd only matter for reconstruction tasks, but +# openpoints/models/__init__.py imports .reconstruction unconditionally, which +# imports chamfer_dist -- so even a pure segmentation run fails at import time +# without them. They are not optional in practice. +if [ "${FORCE:-0}" = "1" ] || ! have chamfer; then + echo "=== build chamfer_dist ===" + cd "$REPO/openpoints/cpp/chamfer_dist" + rm -rf build ./*.egg-info dist + python setup.py install +else + echo "=== skip chamfer_dist (already importable) ===" +fi + +# emd/setup.py names the package emd_ext and the extension emd_cuda; plain +# `emd` is only the python-level alias in openpoints/cpp/emd/__init__.py. +if [ "${FORCE:-0}" = "1" ] || ! have emd_cuda; then + echo "=== build emd ===" + cd "$REPO/openpoints/cpp/emd" + rm -rf build ./*.egg-info dist + python setup.py install +else + echo "=== skip emd (already importable) ===" +fi + +echo "=== verify ===" +cd "$REPO" +python - <<'PY' +import sys +print("torch :", __import__("torch").__version__, + "| cuda", __import__("torch").cuda.is_available()) +ok = True +for m in ["pointnet2_batch_cuda", "pointops_cuda", "chamfer", "emd_cuda", + "torch_scatter", "plyfile"]: + try: + __import__(m) + print(f"{m:22s}: OK") + except Exception as e: + ok = False + print(f"{m:22s}: FAIL {type(e).__name__}: {e}") +try: + from openpoints.cpp.subsampling import grid_subsampling # noqa: F401 + print(f"{'grid_subsampling':22s}: OK") +except Exception as e: + ok = False + print(f"{'grid_subsampling':22s}: FAIL {type(e).__name__}: {e}") +sys.exit(0 if ok else 1) +PY + +echo "BUILD DONE" diff --git a/scripts/check_alloc_conf.sh b/scripts/check_alloc_conf.sh new file mode 100644 index 0000000..910a75f --- /dev/null +++ b/scripts/check_alloc_conf.sh @@ -0,0 +1,34 @@ +#!/usr/bin/env bash +# SUM Parts - verify PYTORCH_CUDA_ALLOC_CONF is accepted before committing a run +# +# torch 2.0.1 aborts at CUDA init on an unknown key, and the traceback points at +# model.to(device) rather than at the env var, so an invalid setting looks like +# a model problem. Check it in two seconds instead. +set -uo pipefail + +source "$HOME/miniconda3/etc/profile.d/conda.sh" +conda activate sumparts + +CONF="${1:-garbage_collection_threshold:0.7,max_split_size_mb:128}" + +echo "torch : $(python -c 'import torch; print(torch.__version__)')" +echo "testing: PYTORCH_CUDA_ALLOC_CONF=$CONF" + +PYTORCH_CUDA_ALLOC_CONF="$CONF" python - <<'PY' +import os +import torch + +print("env :", os.environ.get("PYTORCH_CUDA_ALLOC_CONF")) +x = torch.zeros(1024, 1024, device="cuda") +y = (x + 1).sum().item() +print("cuda : OK", torch.cuda.get_device_name(0), "| smoke sum =", y) +print("alloc : %.1f MB" % (torch.cuda.memory_allocated() / 1024**2)) +PY +rc=$? + +if [ $rc -eq 0 ]; then + echo "ACCEPTED" +else + echo "REJECTED (rc=$rc) -- do not launch with this setting" +fi +exit $rc diff --git a/scripts/check_hf_sizes.py b/scripts/check_hf_sizes.py new file mode 100644 index 0000000..fbcd438 --- /dev/null +++ b/scripts/check_hf_sizes.py @@ -0,0 +1,22 @@ +#!/usr/bin/env python3 +"""Report the download sizes of the SUM Parts archives on Hugging Face.""" +from huggingface_hub import HfApi + +REPO = "gwxgrxhyz/SUM-Parts" + +api = HfApi() +info = api.repo_info(REPO, repo_type="dataset", files_metadata=True) + +rows = [] +for s in info.siblings: + size = s.size or (s.lfs.size if getattr(s, "lfs", None) else None) + if size and size > 1_000_000: + rows.append((s.rfilename, size)) + +rows.sort(key=lambda r: -r[1]) +total = 0 +for name, size in rows: + print(f"{size / 1e9:>8.2f} GB {name}") + total += size +print(f"{'-' * 30}") +print(f"{total / 1e9:>8.2f} GB total (files > 1 MB)") diff --git a/scripts/check_infer_cfg.sh b/scripts/check_infer_cfg.sh new file mode 100644 index 0000000..318ee70 --- /dev/null +++ b/scripts/check_infer_cfg.sh @@ -0,0 +1,25 @@ +#!/usr/bin/env bash +# SUM Parts - did the dataset.test.voxel_max override actually land? +set -uo pipefail + +LOG="${1:-$HOME/sum-parts/runs/coarse_eval/infer.log}" + +echo "log: $LOG" +[ -f "$LOG" ] || { echo " missing"; exit 1; } + +echo +echo "=== opts line (what was overridden) ===" +grep -a '^opts:' "$LOG" || echo " (none)" + +echo +echo "=== dataset block from the cfg dump ===" +sed -n '/^dataset:/,/^datatransforms:/p' "$LOG" | head -25 + +echo +echo "=== how many sub-clouds per tile ===" +grep -ao 'Test on [0-9]*-th cloud \[[0-9]*\]/\[[0-9]*\]' "$LOG" \ + | sed 's/.*\///' | sort -u | head + +echo +echo "=== reported metrics ===" +grep -aE 'test_oa|iou per cls' "$LOG" | tail -4 diff --git a/scripts/check_ply.py b/scripts/check_ply.py new file mode 100644 index 0000000..c71e835 --- /dev/null +++ b/scripts/check_ply.py @@ -0,0 +1,98 @@ +#!/usr/bin/env python3 +"""Compare PLY point clouds against what the SUM Parts loader needs. + +Pass the converted file first and a reference SUM Parts tile second to see the +two side by side. + +Usage: + python check_ply.py mine.ply [reference.ply ...] +""" +from __future__ import annotations + +import sys +from pathlib import Path + +import numpy as np +from plyfile import PlyData + +# read_ply_with_plyfilelib tries red/green/blue first, then r/g/b +RGB_SETS = (("red", "green", "blue"), ("r", "g", "b")) + + +def report(path: Path) -> bool: + ply = PlyData.read(str(path)) + v = ply["vertex"] + props = [p.name for p in v.properties] + + print(f"=== {path.name} ===") + print(f" format : {'binary' if not ply.text else 'ascii'}") + print(f" points : {len(v):,}") + print(f" properties: {props}") + + ok = True + + missing_xyz = [c for c in ("x", "y", "z") if c not in props] + if missing_xyz: + print(f" MISSING : {missing_xyz}") + ok = False + else: + xyz = np.stack([v["x"], v["y"], v["z"]], axis=1) + lo, hi = xyz.min(axis=0), xyz.max(axis=0) + ext = hi - lo + print(f" extent : {np.round(ext, 2).tolist()} m") + print(f" origin : {np.round(lo, 2).tolist()}") + if ext.max() < 0.5: + print(" WARNING : extent < 0.5 -- degrees, not metres?") + ok = False + area = ext[0] * ext[1] + if area > 0: + print(f" density : {len(v) / area:,.1f} pts/m^2") + + rgb_set = next((s for s in RGB_SETS if all(c in props for c in s)), None) + if rgb_set is None: + print(f" MISSING : colour -- need {RGB_SETS[0]} or {RGB_SETS[1]}") + ok = False + else: + rgb = np.stack([v[c] for c in rgb_set], axis=1) + print(f" colour : {rgb_set} dtype={rgb.dtype} " + f"range=[{rgb.min()}, {rgb.max()}] mean={rgb.mean(axis=0).round(1).tolist()}") + grey = int((rgb == 128).all(axis=1).sum()) + if grey: + print(f" WARNING : {grey:,} points are exactly grey (texture miss?)") + + if "label" not in props: + print(" MISSING : label") + ok = False + else: + lab = np.asarray(v["label"]) + u, c = np.unique(lab, return_counts=True) + shown = list(zip(u.tolist(), c.tolist()))[:14] + print(f" label : dtype={lab.dtype} uniq={u.tolist()[:20]}") + print(f" label hist: {shown}") + if u.tolist() == [0]: + print(" note : all unclassified -- inference only, cannot train/score") + + print(f" verdict : {'OK' if ok else 'INCOMPATIBLE'}") + print() + return ok + + +def main() -> None: + if len(sys.argv) < 2: + print(__doc__) + raise SystemExit(2) + + all_ok = True + for a in sys.argv[1:]: + p = Path(a) + if not p.exists(): + print(f"{a}: not found\n") + all_ok = False + continue + all_ok &= report(p) + + raise SystemExit(0 if all_ok else 1) + + +if __name__ == "__main__": + main() diff --git a/scripts/check_pred.py b/scripts/check_pred.py new file mode 100644 index 0000000..29c3d95 --- /dev/null +++ b/scripts/check_pred.py @@ -0,0 +1,81 @@ +#!/usr/bin/env python3 +"""Summarise a prediction PLY written by main.py's visualization step. + +main.py colours predictions through SUMV2_Triangle_COLOR_MAP rather than +writing a label field, so recover the class by matching each point's RGB back +to that palette. + +Usage: + python check_pred.py .../seosan_BlockYBA_tile0_pred.ply +""" +from __future__ import annotations + +import sys +from pathlib import Path + +import numpy as np +from plyfile import PlyData + +# openpoints/dataset/sumv2_triangle/sumv2_triangle.py +CLASSES = ['unclassified', 'terrain', 'high_vegetation', 'facade_surface', + 'water', 'car', 'boat', 'roof_surface', 'chimney', 'dormer', + 'balcony', 'roof_installation', 'wall'] + +COLOR_MAP = np.array([ + (0., 0., 0.), (170., 85., 0.), (0., 255., 0.), (255., 255., 0.), + (0., 255., 255.), (255., 0., 255.), (0., 0., 153.), (85., 85., 127.), + (255., 50., 50.), (85., 0., 127.), (50., 125., 150.), (50., 0., 50.), + (215., 160., 140.), +]) + + +def main() -> None: + if len(sys.argv) != 2: + print(__doc__) + raise SystemExit(2) + + path = Path(sys.argv[1]) + if not path.exists(): + print(f"{path}: not found") + raise SystemExit(1) + + v = PlyData.read(str(path))["vertex"] + props = [p.name for p in v.properties] + print(f"=== {path.name} ===") + print(f" points : {len(v):,}") + print(f" properties : {props}") + + rgb_set = next((s for s in (("red", "green", "blue"), ("r", "g", "b")) + if all(c in props for c in s)), None) + if rgb_set is None: + print(" no colour channels -- cannot recover predicted class") + raise SystemExit(1) + + rgb = np.stack([np.asarray(v[c], dtype=np.float64) for c in rgb_set], axis=1) + if rgb.max() <= 1.0: + rgb = rgb * 255.0 + + # nearest palette entry per point + d = ((rgb[:, None, :] - COLOR_MAP[None, :, :]) ** 2).sum(axis=2) + cls = d.argmin(axis=1) + resid = np.sqrt(d.min(axis=1)) + + print(f" palette fit: max residual {resid.max():.1f} " + f"({'exact' if resid.max() < 1 else 'approximate'})") + print() + print(f" {'class':<20} {'points':>10} {'share':>7}") + u, c = np.unique(cls, return_counts=True) + order = np.argsort(-c) + for i in order: + k, n = int(u[i]), int(c[i]) + name = CLASSES[k] if k < len(CLASSES) else f"?{k}" + print(f" {name:<20} {n:>10,} {100 * n / len(cls):>6.2f}%") + + print() + print(f" distinct classes predicted: {len(u)}") + if len(u) == 1: + print(" note: single class everywhere -- expected from a 1-epoch model") + + +if __name__ == "__main__": + main() diff --git a/scripts/check_test_labels.sh b/scripts/check_test_labels.sh new file mode 100644 index 0000000..7b3a074 --- /dev/null +++ b/scripts/check_test_labels.sh @@ -0,0 +1,47 @@ +#!/usr/bin/env bash +# SUM Parts - inspect label values in the test split +# +# test() crashed in ConfusionMatrix.update with +# RuntimeError: bincount only supports 1-d non-negative integral inputs +# which means true*num_classes + pred went negative, non-integral, or 2-D. +# Check what the files actually contain. +set -uo pipefail + +DATA="${1:-$HOME/sum-parts/data/face_labeling/texsp_pcl}" + +source "$HOME/miniconda3/etc/profile.d/conda.sh" +conda activate sumparts + +python - "$DATA" <<'PY' +import sys +from pathlib import Path + +import numpy as np +from plyfile import PlyData + +root = Path(sys.argv[1]) +print(f"root: {root}\n") + +for split in ("test", "train"): + files = sorted((root / split).glob("*.ply")) + print(f"=== {split}: {len(files)} files ===") + for f in files: + v = PlyData.read(str(f))["vertex"] + props = [p.name for p in v.properties] + if "label" not in props: + print(f" {f.name:<34} NO LABEL FIELD props={props}") + continue + lab = np.asarray(v["label"]) + u = np.unique(lab) + neg = int((lab < 0).sum()) + flag = "" + if neg: + flag += f" NEGATIVE x{neg}" + if u.max() > 12: + flag += f" OUT-OF-RANGE max={u.max()}" + if not np.issubdtype(lab.dtype, np.integer): + flag += f" NON-INTEGER dtype={lab.dtype}" + print(f" {f.name:<34} n={len(lab):>8,} dtype={str(lab.dtype):<8} " + f"range=[{u.min()},{u.max()}] uniq={len(u)}{flag}") + print() +PY diff --git a/scripts/check_training.sh b/scripts/check_training.sh new file mode 100644 index 0000000..5ad86dc --- /dev/null +++ b/scripts/check_training.sh @@ -0,0 +1,58 @@ +#!/usr/bin/env bash +# SUM Parts - status of the unattended training run +set -uo pipefail + +RUNROOT="$HOME/sum-parts/runs" + +WORKDIR="${1:-}" +if [ -z "$WORKDIR" ]; then + if [ -f "$RUNROOT/.latest" ]; then + WORKDIR=$(cat "$RUNROOT/.latest") + else + WORKDIR=$(find "$RUNROOT" -maxdepth 1 -mindepth 1 -type d -printf '%T@ %p\n' 2>/dev/null \ + | sort -rn | head -1 | cut -d' ' -f2-) + fi +fi + +if [ -z "$WORKDIR" ] || [ ! -d "$WORKDIR" ]; then + echo "no run directory found under $RUNROOT" + exit 1 +fi + +echo "=== run: $(basename "$WORKDIR") ===" +[ -f "$WORKDIR/status.txt" ] && cat "$WORKDIR/status.txt" + +echo +echo "=== process ===" +if pgrep -f train_watchdog.sh > /dev/null; then + pgrep -af "train_watchdog.sh" + pgrep -af "main.py" | head -3 +else + echo "watchdog NOT running" +fi + +echo +echo "=== GPU ===" +nvidia-smi --query-gpu=memory.used,memory.total,utilization.gpu,temperature.gpu \ + --format=csv,noheader 2>/dev/null || echo "nvidia-smi unavailable" + +echo +echo "=== epochs completed ===" +if [ -f "$WORKDIR/train.log" ]; then + grep -oE 'Epoch [0-9]+ LR [0-9.]+ train_miou [0-9.]+, val_miou [0-9.]+, best val miou [0-9.]+' \ + "$WORKDIR/train.log" 2>/dev/null | tail -8 || echo " (no epoch lines yet)" + echo + echo " total epoch lines: $(grep -cE 'Epoch [0-9]+ LR' "$WORKDIR/train.log" 2>/dev/null || echo 0)" + echo " log size : $(du -h "$WORKDIR/train.log" | cut -f1)" + echo " last write : $(date -r "$WORKDIR/train.log" '+%F %T')" +else + echo " no train.log yet" +fi + +echo +echo "=== watchdog events ===" +[ -f "$WORKDIR/watchdog.log" ] && tail -12 "$WORKDIR/watchdog.log" + +echo +echo "=== best so far ===" +grep -E 'Find a better ckpt' "$WORKDIR/train.log" 2>/dev/null | tail -3 || echo " none yet" diff --git a/scripts/coarse_eval.py b/scripts/coarse_eval.py new file mode 100644 index 0000000..a8ee0d0 --- /dev/null +++ b/scripts/coarse_eval.py @@ -0,0 +1,179 @@ +#!/usr/bin/env python3 +"""Score SUM Parts predictions at the granularity this project actually needs. + +The benchmark reports 13 fine classes. The task here is coarser: separate a +drone-photogrammetry mesh into building / vegetation / vehicle / ground. Window +and single-wall detail is explicitly out of scope. + +That difference matters for how the numbers read. A fine-grained mIoU averages +in classes like dormer, balcony and roof_installation, which a weak baseline +scores 0 on -- but every one of those collapses into "building" here, so their +individual failure costs nothing. Merging first, then scoring, measures the +thing that is actually wanted. + +Mapping (SUM index -> coarse): + 1 terrain -> ground + 2 high_vegetation -> vegetation + 3 facade_surface -> building + 5 car -> vehicle + 6 boat -> vehicle + 7 roof_surface -> building + 8 chimney -> building + 9 dormer -> building + 10 balcony -> building + 11 roof_installation -> building + 12 wall -> building + 0 unclassified -> ignored + 4 water -> ignored (not a target class here) + +Usage: + python coarse_eval.py pred.ply gt.ply [more pairs ...] + python coarse_eval.py --pred-dir DIR --gt-dir DIR +""" +from __future__ import annotations + +import argparse +import sys +from pathlib import Path + +import numpy as np +from plyfile import PlyData + +FINE = ['unclassified', 'terrain', 'high_vegetation', 'facade_surface', + 'water', 'car', 'boat', 'roof_surface', 'chimney', 'dormer', + 'balcony', 'roof_installation', 'wall'] + +COARSE = ['ignored', 'ground', 'vegetation', 'building', 'vehicle'] + +# fine index -> coarse index (0 = ignored) +FINE_TO_COARSE = np.array([ + 0, # unclassified + 1, # terrain -> ground + 2, # high_vegetation -> vegetation + 3, # facade_surface -> building + 0, # water -> ignored + 4, # car -> vehicle + 4, # boat -> vehicle + 3, # roof_surface -> building + 3, # chimney -> building + 3, # dormer -> building + 3, # balcony -> building + 3, # roof_installation -> building + 3, # wall -> building +], dtype=np.int64) + +# main.py's visualization writes colours, not labels; recover the class by +# matching against the palette it used +COLOR_MAP = np.array([ + (0., 0., 0.), (170., 85., 0.), (0., 255., 0.), (255., 255., 0.), + (0., 255., 255.), (255., 0., 255.), (0., 0., 153.), (85., 85., 127.), + (255., 50., 50.), (85., 0., 127.), (50., 125., 150.), (50., 0., 50.), + (215., 160., 140.), +]) + + +def load_labels(path: Path) -> np.ndarray: + """Fine class per point: from a `label` field, else decoded from colour.""" + v = PlyData.read(str(path))["vertex"] + props = [p.name for p in v.properties] + + if "label" in props: + lab = np.asarray(v["label"]).astype(np.int64).ravel() + # a prediction ply may carry a placeholder label; fall through if so + if lab.max() >= 0 and not (lab == lab[0]).all(): + return lab + if lab.max() >= 0 and lab[0] >= 0: + return lab + + rgb_set = next((s for s in (("red", "green", "blue"), ("r", "g", "b")) + if all(c in props for c in s)), None) + if rgb_set is None: + raise SystemExit(f"{path}: no label field and no colour to decode") + + rgb = np.stack([np.asarray(v[c], dtype=np.float64) for c in rgb_set], axis=1) + if rgb.max() <= 1.0: + rgb *= 255.0 + d = ((rgb[:, None, :] - COLOR_MAP[None, :, :]) ** 2).sum(axis=2) + return d.argmin(axis=1).astype(np.int64) + + +def confusion(pred: np.ndarray, true: np.ndarray, n: int) -> np.ndarray: + k = (true >= 0) & (true < n) & (pred >= 0) & (pred < n) + return np.bincount(true[k] * n + pred[k], minlength=n * n).reshape(n, n) + + +def report(cm: np.ndarray, names: list[str], skip: set[int]) -> None: + tp = np.diag(cm).astype(np.float64) + actual = cm.sum(axis=1).astype(np.float64) + predicted = cm.sum(axis=0).astype(np.float64) + union = actual + predicted - tp + + print(f" {'class':<12} {'IoU':>7} {'recall':>8} {'points':>12}") + ious = [] + for i, name in enumerate(names): + if i in skip: + continue + iou = 100.0 * tp[i] / union[i] if union[i] > 0 else 0.0 + rec = 100.0 * tp[i] / actual[i] if actual[i] > 0 else 0.0 + ious.append(iou) + print(f" {name:<12} {iou:>6.2f}% {rec:>7.2f}% {int(actual[i]):>12,}") + + scored = [i for i in range(len(names)) if i not in skip] + oa_tp = tp[scored].sum() + oa_n = actual[scored].sum() + print() + print(f" mIoU : {np.mean(ious):.2f}% (over {len(ious)} classes)") + print(f" OA : {100.0 * oa_tp / oa_n if oa_n else 0:.2f}%") + + +def main() -> None: + ap = argparse.ArgumentParser(description=__doc__, + formatter_class=argparse.RawDescriptionHelpFormatter) + ap.add_argument("pairs", nargs="*", help="pred.ply gt.ply [pred gt ...]") + ap.add_argument("--pred-dir", type=Path) + ap.add_argument("--gt-dir", type=Path) + args = ap.parse_args() + + pairs: list[tuple[Path, Path]] = [] + if args.pred_dir and args.gt_dir: + for p in sorted(args.pred_dir.glob("*_pred.ply")): + stem = p.name.replace("_pred.ply", ".ply") + g = args.gt_dir / stem + if g.exists(): + pairs.append((p, g)) + else: + print(f"warn: no ground truth for {p.name}", file=sys.stderr) + else: + if len(args.pairs) % 2: + raise SystemExit("pairs must come as pred gt pred gt ...") + pairs = [(Path(args.pairs[i]), Path(args.pairs[i + 1])) + for i in range(0, len(args.pairs), 2)] + + if not pairs: + raise SystemExit("nothing to evaluate") + + cm_fine = np.zeros((13, 13), dtype=np.int64) + cm_coarse = np.zeros((5, 5), dtype=np.int64) + + for pred_p, gt_p in pairs: + pred = load_labels(pred_p) + true = load_labels(gt_p) + if len(pred) != len(true): + print(f"warn: {pred_p.name} has {len(pred):,} points but " + f"{gt_p.name} has {len(true):,} -- skipped", file=sys.stderr) + continue + print(f" + {gt_p.name} ({len(true):,} pts)") + cm_fine += confusion(pred, true, 13) + cm_coarse += confusion(FINE_TO_COARSE[pred], FINE_TO_COARSE[true], 5) + + print() + print("=== fine (13 SUM classes, benchmark granularity) ===") + report(cm_fine, FINE, skip={0}) + + print() + print("=== coarse (what this project needs) ===") + report(cm_coarse, COARSE, skip={0}) + + +if __name__ == "__main__": + main() diff --git a/scripts/diag_run.sh b/scripts/diag_run.sh new file mode 100644 index 0000000..e7e9177 --- /dev/null +++ b/scripts/diag_run.sh @@ -0,0 +1,28 @@ +#!/usr/bin/env bash +# SUM Parts - dump everything about the most recent run attempt +set -uo pipefail + +RUNROOT="$HOME/sum-parts/runs" +D="${1:-$(cat "$RUNROOT/.latest" 2>/dev/null)}" + +echo "run dir: $D" +[ -d "$D" ] || { echo " (missing)"; exit 1; } +ls -la "$D" + +for f in watchdog.log nohup.out status.txt train.log; do + echo + echo "=== $f ===" + if [ -f "$D/$f" ]; then + tail -30 "$D/$f" + else + echo " (absent)" + fi +done + +echo +echo "=== processes ===" +pgrep -af 'train_watchdog|main.py' || echo " none running" + +echo +echo "=== syntax check of watchdog ===" +bash -n /mnt/d/MYCLAUDE_PROJECT/sum-parts-test/scripts/train_watchdog.sh && echo " OK" diff --git a/scripts/download_data.sh b/scripts/download_data.sh new file mode 100644 index 0000000..03a157b --- /dev/null +++ b/scripts/download_data.sh @@ -0,0 +1,89 @@ +#!/usr/bin/env bash +# SUM Parts - dataset download from Hugging Face +# +# PREREQUISITE: the dataset is GATED (gated: auto). Before this works you must, +# once, in a browser: +# 1. open https://huggingface.co/datasets/gwxgrxhyz/SUM-Parts +# 2. log in and accept the CC BY-NC 4.0 terms on the gate form +# Without that step every download returns HTTP 403. +# +# Usage: +# bash download_data.sh # demo only (smoke test, small) +# bash download_data.sh all # demo + mesh + pcl (large) +set -euo pipefail + +CONDA_ROOT="$HOME/miniconda3" +ENV_NAME="sumparts" +REPO_ID="gwxgrxhyz/SUM-Parts" +# cfgs/sumv2_triangle/default.yaml uses data_root: ../../data/... relative to +# PointNeXt_bundle, which resolves to /data/ +DATA_DIR="$HOME/sum-parts/data" +DL_DIR="$DATA_DIR/_archives" + +source "$CONDA_ROOT/etc/profile.d/conda.sh" +conda activate "$ENV_NAME" + +python -c "import huggingface_hub" 2>/dev/null || pip install --no-cache-dir "huggingface_hub[cli]" + +# Reuse the token already cached on the Windows side if WSL has none. +if [ ! -f "$HOME/.cache/huggingface/token" ] && [ -f /mnt/c/Users/"$USER"/.cache/huggingface/token ]; then + mkdir -p "$HOME/.cache/huggingface" + cp /mnt/c/Users/"$USER"/.cache/huggingface/token "$HOME/.cache/huggingface/token" + chmod 600 "$HOME/.cache/huggingface/token" + echo "copied HF token from Windows profile" +fi + +if [ "${1:-demo}" = "all" ]; then + FILES=(demo.zip mesh.zip pcl.zip) +else + FILES=(demo.zip) +fi + +mkdir -p "$DL_DIR" + +for f in "${FILES[@]}"; do + echo "=== downloading $f ===" + python - "$REPO_ID" "$f" "$DL_DIR" <<'PY' +import sys +from huggingface_hub import hf_hub_download +repo_id, filename, out_dir = sys.argv[1:4] +p = hf_hub_download( + repo_id=repo_id, + filename=filename, + repo_type="dataset", + local_dir=out_dir, +) +print("saved:", p) +PY +done + +# Extract with python's zipfile rather than `unzip`: the distro has no unzip +# installed and sudo needs a password here, so apt is not an option. +for f in "${FILES[@]}"; do + echo "=== extracting $f ===" + python - "$DL_DIR/$f" "$DATA_DIR" <<'PY' +import sys, zipfile +src, dest = sys.argv[1:3] +with zipfile.ZipFile(src) as z: + names = z.namelist() + print(f" {len(names)} entries") + z.extractall(dest) +print(" ->", dest) +PY +done + +# NOTE: no `find ... | head` here. Under `set -euo pipefail`, head closing the +# pipe sends SIGPIPE to find and the script exits 141 -- reported as a failed +# download even though everything extracted fine. +echo "=== resulting layout (depth 2) ===" +find "$DATA_DIR" -maxdepth 2 -not -path '*/_archives/*' -type d | sort + +echo +echo "=== ply counts ===" +for d in "$DATA_DIR"/*/; do + [ "$(basename "$d")" = "_archives" ] && continue + n=$(find "$d" -name '*.ply' 2>/dev/null | wc -l) + [ "$n" -gt 0 ] && printf '%-28s %5s ply\n' "$(basename "$d")/" "$n" +done + +echo "DATA DONE" diff --git a/scripts/epoch_timing.sh b/scripts/epoch_timing.sh new file mode 100644 index 0000000..9707420 --- /dev/null +++ b/scripts/epoch_timing.sh @@ -0,0 +1,35 @@ +#!/usr/bin/env bash +# SUM Parts - per-epoch wall time, to locate where a run slowed down +set -uo pipefail + +RUNROOT="$HOME/sum-parts/runs" +WORKDIR="${1:-$(cat "$RUNROOT/.latest" 2>/dev/null)}" +LOG="$WORKDIR/train.log" + +[ -f "$LOG" ] || { echo "no train.log at $WORKDIR"; exit 1; } + +grep -E 'Epoch [0-9]+ LR' "$LOG" | awk ' +{ + # [08/20 20:28:44 SUMV2_Triangle]: Epoch 10 LR ... + # $1 $2 $3 $4 $5 + split($2, t, ":") + sec = t[1]*3600 + t[2]*60 + t[3] + for (i = 1; i <= NF; i++) if ($i == "Epoch") { ep = $(i+1); break } + if (prev_sec != "") { + d = sec - prev_sec + if (d < 0) d += 86400 # crossed midnight + printf "epoch %3d %s +%6.1f s", ep, t[1]":"t[2]":"t[3], d + if (d > slow_thresh) printf " <-- SLOW" + printf "\n" + } else { + printf "epoch %3d %s\n", ep, t[1]":"t[2]":"t[3] + } + prev_sec = sec +} +BEGIN { slow_thresh = 300 } +' + +echo +echo "--- summary ---" +grep -E 'Epoch [0-9]+ LR' "$LOG" | tail -1 +echo "epochs done: $(grep -cE 'Epoch [0-9]+ LR' "$LOG")" diff --git a/scripts/eval_coarse.sh b/scripts/eval_coarse.sh new file mode 100644 index 0000000..7e3f84d --- /dev/null +++ b/scripts/eval_coarse.sh @@ -0,0 +1,70 @@ +#!/usr/bin/env bash +# SUM Parts - score the trained model on the classes this project actually needs +# +# Runs the sliding-window test() path over the VAL split (which has real labels, +# unlike the blind test split), then re-scores the predictions after merging +# SUM's 13 fine classes down to building / vegetation / vehicle / ground. +set -uo pipefail + +CONDA_ROOT="$HOME/miniconda3" +SEG="$HOME/sum-parts/semantic_segmentation/PointNeXt_bundle/examples/segmentation" +DATA="$HOME/sum-parts/data/face_labeling/texsp_pcl" +SCRIPTS="/mnt/d/MYCLAUDE_PROJECT/sum-parts-test/scripts" +OUT="$HOME/sum-parts/runs/coarse_eval" + +source "$CONDA_ROOT/etc/profile.d/conda.sh" +conda activate sumparts +export WANDB_MODE=disabled WANDB_SILENT=true CUDA_HOME="$CONDA_PREFIX" +export PYTORCH_CUDA_ALLOC_CONF="garbage_collection_threshold:0.7,max_split_size_mb:128" + +mkdir -p "$OUT" +cd "$SEG" + +CKPT=$(find "$SEG/log/sumv2_triangle" -name '*_ckpt_best.pth' -printf '%T@ %p\n' \ + | sort -rn | head -1 | cut -d' ' -f2-) +echo "checkpoint: $(basename "$CKPT")" + +# test() writes prediction plys and slides over whole tiles; point it at val so +# there is ground truth to score against. +# +# TEST_VOXEL_MAX matters more than it looks. PointNet max-pools ONE global +# feature over whatever it is handed, so the number of points per forward pass +# is part of the model's operating point, not a neutral batching knob. Training +# used 64000-point crops; the cfg's test default (null) feeds ~350k-point chunks +# and the predictions collapse toward the majority class. Set it to 64000 to +# make inference match training. +TEST_VOXEL_MAX="${TEST_VOXEL_MAX:-}" +VM_ARG=() +[ -n "$TEST_VOXEL_MAX" ] && VM_ARG=(dataset.test.voxel_max="$TEST_VOXEL_MAX") + +echo "=== inference over val split (sliding window) ===" +echo " test voxel_max: ${TEST_VOXEL_MAX:-null (cfg default)}" +python -u main.py \ + --cfg ../../cfgs/sumv2_triangle/pointnet.yaml \ + mode=test \ + --pretrained_path "$CKPT" \ + dataset.common.data_root="$DATA" \ + dataset.test.split=val \ + "${VM_ARG[@]}" \ + wandb.use_wandb=False \ + > "$OUT/infer.log" 2>&1 +rc=$? +if [ $rc -ne 0 ]; then + echo " FAILED rc=$rc" + tail -15 "$OUT/infer.log" + exit $rc +fi +grep -aE 'test_oa|iou per cls' "$OUT/infer.log" | tail -4 + +VIS=$(dirname "$(dirname "$CKPT")")/visualization +echo +echo "=== predictions in $VIS ===" +ls -1 "$VIS" | grep -c '_pred.ply' || true + +echo +echo "=== re-scoring at project granularity ===" +python "$SCRIPTS/coarse_eval.py" --pred-dir "$VIS" --gt-dir "$DATA/val" \ + | tee "$OUT/coarse.txt" + +echo +echo "logs in $OUT" diff --git a/scripts/final_eval.sh b/scripts/final_eval.sh new file mode 100644 index 0000000..012f451 --- /dev/null +++ b/scripts/final_eval.sh @@ -0,0 +1,84 @@ +#!/usr/bin/env bash +# SUM Parts - final evaluation of the trained checkpoint +# +# Two passes, because the splits differ in what they can tell you: +# +# val : labeled (0..12) -> produces real numbers you can quote locally +# test : label = -1 everywhere (blind set) -> produces predictions only. +# The authors score it; see the README's "send predictions to our +# email for local assessment". +# +# Requires patch_unlabeled_test.sh and patch_val_mode.sh to have run, otherwise +# the test pass dies in ConfusionMatrix on the -1 placeholders and mode=val dies +# with UnboundLocalError on `epoch`. +# +# No voxel_max override here on purpose. The cfg already validates with +# voxel_max: null (whole tiles), which is what we want for the final number -- +# training capped it only to keep the allocator inside VRAM. Passing +# `dataset.val.voxel_max=null` on the command line does NOT work: it arrives as +# the string "null" and crop_pc then compares int >= str. +set -uo pipefail + +CONDA_ROOT="$HOME/miniconda3" +SEG="$HOME/sum-parts/semantic_segmentation/PointNeXt_bundle/examples/segmentation" +DATA="$HOME/sum-parts/data/face_labeling/texsp_pcl" +OUT="$HOME/sum-parts/runs/final_eval" + +source "$CONDA_ROOT/etc/profile.d/conda.sh" +conda activate sumparts +export WANDB_MODE=disabled WANDB_SILENT=true CUDA_HOME="$CONDA_PREFIX" +export PYTORCH_CUDA_ALLOC_CONF="garbage_collection_threshold:0.7,max_split_size_mb:128" + +mkdir -p "$OUT" + +CKPT=$(find "$SEG/log/sumv2_triangle" -name '*_ckpt_best.pth' -printf '%T@ %p\n' \ + | sort -rn | head -1 | cut -d' ' -f2-) +[ -n "$CKPT" ] || { echo "error: no checkpoint found" >&2; exit 1; } + +echo "checkpoint: $CKPT" +echo "data : $DATA" +echo + +cd "$SEG" + +run_mode() { + local mode="$1" log="$OUT/${1}.log" + echo "=== mode=$mode ===" + set +e + python -u main.py \ + --cfg ../../cfgs/sumv2_triangle/pointnet.yaml \ + mode="$mode" \ + --pretrained_path "$CKPT" \ + dataset.common.data_root="$DATA" \ + wandb.use_wandb=False \ + val_batch_size=1 \ + > "$log" 2>&1 + local rc=$? + set -e + if [ $rc -eq 0 ]; then + echo " ok" + else + echo " FAILED rc=$rc" + tail -12 "$log" + fi + grep -aE 'val_oa|test_oa|iou per cls|Best ckpt' "$log" | tail -6 + echo + return $rc +} + +# val first: this is the number we can actually stand behind locally +run_mode val +val_rc=$? + +# test: predictions only, no score possible +run_mode test +test_rc=$? + +echo "=== prediction files ===" +find "$SEG/log/sumv2_triangle" -name '*_pred.ply' -newermt '-30 minutes' \ + -printf '%p (%s bytes)\n' 2>/dev/null | tail -12 + +echo +echo "logs in $OUT" +[ $val_rc -eq 0 ] && echo "FINAL EVAL: val OK" || echo "FINAL EVAL: val FAILED" +[ $test_rc -eq 0 ] && echo "FINAL EVAL: test OK" || echo "FINAL EVAL: test FAILED" diff --git a/scripts/inspect_obj.py b/scripts/inspect_obj.py new file mode 100644 index 0000000..c7a93e0 --- /dev/null +++ b/scripts/inspect_obj.py @@ -0,0 +1,97 @@ +#!/usr/bin/env python3 +"""Stream an OBJ and report the numbers that decide how to tile it for SUM Parts. + +Reads line by line so a multi-GB mesh does not have to fit in RAM. Reports +vertex/face counts, the bounding box, whether UVs and vertex colours are +present, and the material/texture references. + +Usage: + python inspect_obj.py path/to/Block.obj [more.obj ...] +""" +from __future__ import annotations + +import sys +from pathlib import Path + + +def inspect(path: Path) -> dict: + n_v = n_vt = n_vn = n_f = 0 + has_vertex_color = False + mtllib: list[str] = [] + usemtl: set[str] = set() + lo = [float("inf")] * 3 + hi = [float("-inf")] * 3 + + with path.open("r", encoding="utf-8", errors="replace") as f: + for line in f: + if line.startswith("v "): + n_v += 1 + parts = line.split() + x, y, z = float(parts[1]), float(parts[2]), float(parts[3]) + if x < lo[0]: lo[0] = x + if y < lo[1]: lo[1] = y + if z < lo[2]: lo[2] = z + if x > hi[0]: hi[0] = x + if y > hi[1]: hi[1] = y + if z > hi[2]: hi[2] = z + # "v x y z r g b" is how some exporters carry per-vertex colour + if len(parts) >= 7: + has_vertex_color = True + elif line.startswith("vt "): + n_vt += 1 + elif line.startswith("vn "): + n_vn += 1 + elif line.startswith("f "): + n_f += 1 + elif line.startswith("mtllib"): + mtllib.append(line.split(maxsplit=1)[1].strip()) + elif line.startswith("usemtl"): + usemtl.add(line.split(maxsplit=1)[1].strip()) + + return { + "path": path, + "size_mb": path.stat().st_size / 1024 / 1024, + "n_v": n_v, "n_vt": n_vt, "n_vn": n_vn, "n_f": n_f, + "vertex_color": has_vertex_color, + "mtllib": mtllib, "usemtl": sorted(usemtl), + "lo": lo, "hi": hi, + } + + +def main() -> None: + if len(sys.argv) < 2: + print(__doc__) + raise SystemExit(2) + + for arg in sys.argv[1:]: + p = Path(arg) + if not p.exists(): + print(f"{arg}: not found") + continue + + r = inspect(p) + ext = [r["hi"][i] - r["lo"][i] for i in range(3)] + area = ext[0] * ext[1] + + print(f"=== {p.name} ({r['size_mb']:,.0f} MB) ===") + print(f" vertices : {r['n_v']:>12,}") + print(f" faces : {r['n_f']:>12,}") + print(f" texcoords : {r['n_vt']:>12,} {'(UV present)' if r['n_vt'] else '(NO UV)'}") + print(f" normals : {r['n_vn']:>12,}") + print(f" vtx color : {r['vertex_color']}") + print(f" mtllib : {r['mtllib']}") + print(f" materials : {len(r['usemtl'])} -> {r['usemtl'][:5]}" + f"{' ...' if len(r['usemtl']) > 5 else ''}") + print(f" bbox min : {[round(v, 2) for v in r['lo']]}") + print(f" bbox max : {[round(v, 2) for v in r['hi']]}") + print(f" extent : {[round(v, 2) for v in ext]} (metres, local)") + print(f" footprint : {area / 1e6:.3f} km^2") + if area: + print(f" density : {r['n_f'] / area:,.1f} faces/m^2") + # SUM Parts tiles are ~252 m square; this is how many that footprint is + print(f" ~252m tiles: {max(1, round(ext[0] / 252)) * max(1, round(ext[1] / 252))}") + print() + + +if __name__ == "__main__": + main() diff --git a/scripts/inspect_progress.sh b/scripts/inspect_progress.sh new file mode 100644 index 0000000..f650c5b --- /dev/null +++ b/scripts/inspect_progress.sh @@ -0,0 +1,34 @@ +#!/usr/bin/env bash +# SUM Parts - look at the actual epoch lines, to check for interleaving or +# a non-monotonic "best val miou" +set -uo pipefail + +RUNROOT="$HOME/sum-parts/runs" +D="${1:-$(cat "$RUNROOT/.latest" 2>/dev/null)}" +LOG="$D/train.log" + +echo "run: $D" +[ -f "$LOG" ] || { echo " no train.log"; exit 1; } + +echo +echo "=== last 14 epoch lines ===" +grep -aE 'Epoch [0-9]+ LR' "$LOG" | tail -14 + +echo +echo "=== 'Find a better ckpt' events (last 8) ===" +grep -aE 'Find a better ckpt' "$LOG" | tail -8 + +echo +echo "=== how many distinct run_names wrote to this log ===" +grep -aoE 'sumv2_triangle-train-pointnet-ngpus1-[0-9]{8}-[0-9]{6}-[A-Za-z0-9]+' "$LOG" \ + | sort -u | sed 's/^/ /' + +echo +echo "=== effective val_freq (Val: bars per epoch) ===" +echo " epoch lines : $(grep -acE 'Epoch [0-9]+ LR' "$LOG")" +echo " val passes : $(grep -aoc 'Val: 100%' "$LOG" 2>/dev/null || echo '?')" + +echo +echo "=== processes ===" +pgrep -af 'main.py --cfg' | awk '{print " pid " $1}' | head +echo " count: $(pgrep -cf 'main.py --cfg')" diff --git a/scripts/iter_rate.sh b/scripts/iter_rate.sh new file mode 100644 index 0000000..1b945a0 --- /dev/null +++ b/scripts/iter_rate.sh @@ -0,0 +1,35 @@ +#!/usr/bin/env bash +# SUM Parts - compare early vs recent per-iteration speed in a running train log +# +# Separates "each iteration got slower" from "something between epochs got +# slower" (validation, checkpointing, dataloader restart). +set -uo pipefail + +RUNROOT="$HOME/sum-parts/runs" +WORKDIR="${1:-$(cat "$RUNROOT/.latest" 2>/dev/null)}" +LOG="$WORKDIR/train.log" + +[ -f "$LOG" ] || { echo "no train.log at ${WORKDIR:-}"; exit 1; } + +echo "log: $LOG" +echo + +echo "=== earliest 15 rate readings ===" +grep -oE '[0-9.]+(s/it|it/s)' "$LOG" | head -15 | tr '\n' ' ' +echo +echo + +echo "=== most recent 15 rate readings ===" +grep -oE '[0-9.]+(s/it|it/s)' "$LOG" | tail -15 | tr '\n' ' ' +echo +echo + +echo "=== Train Epoch progress lines (first / last) ===" +grep -oE 'Train Epoch \[[0-9]+/[0-9]+\] Loss [0-9.]+ Acc [0-9.]+' "$LOG" | head -3 +echo " ..." +grep -oE 'Train Epoch \[[0-9]+/[0-9]+\] Loss [0-9.]+ Acc [0-9.]+' "$LOG" | tail -3 +echo + +echo "=== log growth ===" +echo " size : $(du -h "$LOG" | cut -f1)" +echo " last write: $(date -r "$LOG" '+%F %T') (now $(date '+%F %T'))" diff --git a/scripts/launch_overnight.sh b/scripts/launch_overnight.sh new file mode 100644 index 0000000..678cec6 --- /dev/null +++ b/scripts/launch_overnight.sh @@ -0,0 +1,48 @@ +#!/usr/bin/env bash +# SUM Parts - start an unattended training run and return immediately +# +# setsid detaches the watchdog into its own session, so it keeps running after +# the launching shell, the Claude Code session, or the Windows terminal goes +# away. It does NOT survive `wsl --shutdown` or a Windows reboot -- nothing run +# inside WSL does. +# +# Usage: +# bash launch_overnight.sh # pointnet, 100 epochs +# EPOCHS=100 bash launch_overnight.sh pointnet +# +# Then check on it any time with: +# bash check_training.sh +set -euo pipefail + +SCRIPTS="/mnt/d/MYCLAUDE_PROJECT/sum-parts-test/scripts" +CFG="${1:-pointnet}" +STAMP=$(date +%Y%m%d-%H%M%S) +WORKDIR="$HOME/sum-parts/runs/${CFG}_${STAMP}" + +mkdir -p "$WORKDIR" +echo "$WORKDIR" > "$HOME/sum-parts/runs/.latest" + +# already running? +if pgrep -f "train_watchdog.sh" > /dev/null; then + echo "a watchdog is already running:" + pgrep -af "train_watchdog.sh" + echo + echo "stop it first with: bash $SCRIPTS/stop_training.sh" + exit 1 +fi + +STAMP="$STAMP" EPOCHS="${EPOCHS:-100}" VAL_FREQ="${VAL_FREQ:-5}" \ +MAX_RETRIES="${MAX_RETRIES:-8}" CFG_VOXEL_MAX="${CFG_VOXEL_MAX:-}" \ +RESUME_CKPT="${RESUME_CKPT:-}" \ + setsid nohup bash "$SCRIPTS/train_watchdog.sh" "$CFG" \ + > "$WORKDIR/nohup.out" 2>&1 < /dev/null & + +sleep 3 + +echo "launched: $CFG" +echo "workdir : $WORKDIR" +echo +pgrep -af "train_watchdog.sh" || echo "WARNING: watchdog not visible in process list" +echo +echo "check progress: bash $SCRIPTS/check_training.sh" +echo "stop : bash $SCRIPTS/stop_training.sh" diff --git a/scripts/link_data.sh b/scripts/link_data.sh new file mode 100644 index 0000000..7b6565f --- /dev/null +++ b/scripts/link_data.sh @@ -0,0 +1,40 @@ +#!/usr/bin/env bash +# SUM Parts - point the cfg's data_root at the downloaded dataset +# +# cfgs/sumv2_triangle/default.yaml sets: +# data_root: ../../data/sumv2_tri_texpcl/ +# That path is resolved from the *working directory* of main.py, which is +# examples/segmentation. So ../../ is PointNeXt_bundle, NOT the repo root -- +# the dataset has to be reachable at PointNeXt_bundle/data/. +# +# download_data.sh puts the data in /data, so link the two. +set -euo pipefail + +REPO_DATA="$HOME/sum-parts/data" +BUNDLE="$HOME/sum-parts/semantic_segmentation/PointNeXt_bundle" + +if [ ! -d "$REPO_DATA" ]; then + echo "error: $REPO_DATA missing -- run download_data.sh first" >&2 + exit 1 +fi + +if [ -e "$BUNDLE/data" ] && [ ! -L "$BUNDLE/data" ]; then + echo "error: $BUNDLE/data exists and is not a symlink; refusing to replace" >&2 + ls -la "$BUNDLE/data" >&2 + exit 1 +fi + +ln -sfn "$REPO_DATA" "$BUNDLE/data" +echo "linked: $BUNDLE/data -> $REPO_DATA" + +echo "=== resolution check (as main.py sees it) ===" +cd "$BUNDLE/examples/segmentation" +for track in sumv2_tri_texpcl sumv2_tex_texpcl; do + for split in train val test; do + d="../../data/$track/$split" + n=$(find "$d" -name '*.ply' 2>/dev/null | wc -l) + printf '%-18s %-6s %s ply\n' "$track" "$split" "$n" + done +done + +echo "LINK DONE" diff --git a/scripts/list_archive.sh b/scripts/list_archive.sh new file mode 100644 index 0000000..e698241 --- /dev/null +++ b/scripts/list_archive.sh @@ -0,0 +1,54 @@ +#!/usr/bin/env bash +# SUM Parts - list an archive's contents without extracting it +# +# Used to count tiles per split, which is what turns a per-iteration benchmark +# into a wall-clock training estimate. +set -euo pipefail + +ARCHIVE="${1:-$HOME/sum-parts/data/_archives/mesh.zip}" + +source "$HOME/miniconda3/etc/profile.d/conda.sh" +conda activate sumparts + +python - "$ARCHIVE" <<'PY' +import re +import sys +import zipfile +from collections import Counter +from pathlib import Path + +path = Path(sys.argv[1]) +z = zipfile.ZipFile(path) +names = z.namelist() + +print(f"=== {path.name} : {len(names)} entries ===\n") + +dirs = Counter() +for n in names: + p = "/".join(n.split("/")[:-1]) or "." + dirs[p] += 1 +for d, c in sorted(dirs.items()): + print(f" {c:>5} {d}/") + +print() +splits = Counter() +for n in names: + m = re.search(r"(?:^|/)(train|val|test|training|validation)(?:/|_)", n, re.I) + if m: + splits[m.group(1).lower()] += 1 +if splits: + print("split hints:", dict(splits)) + +ply = [n for n in names if n.lower().endswith(".ply")] +print(f"\n.ply entries: {len(ply)}") + +per_split = Counter(n.split("/")[0] for n in ply) +for s, c in sorted(per_split.items()): + print(f" {c:>4} ply in {s}/") + +for n in ply[:5]: + print(f" e.g. {n}") + +total = sum(i.file_size for i in z.infolist()) +print(f"\nuncompressed total: {total / 1e9:.2f} GB") +PY diff --git a/scripts/mesh_to_ply.py b/scripts/mesh_to_ply.py new file mode 100644 index 0000000..f251a48 --- /dev/null +++ b/scripts/mesh_to_ply.py @@ -0,0 +1,279 @@ +#!/usr/bin/env python3 +"""Convert a textured mesh into the PLY point cloud SUM Parts expects. + +Target schema, from openpoints/dataset/sumv2_triangle/sumv2_triangle.py +(read_ply_with_plyfilelib) and confirmed against the shipped demo tile: + + element vertex N + property float x / y / z + property uchar red / green / blue (the demo uses r/g/b; the loader + accepts either spelling) + property uchar label 0 == unclassified, which cfg ignores + property int object_index optional + +Written for ContextCapture-style aerial OBJ exports, which are the shape the +Seosan Myeongcheon blocks come in: + + * multi-material -- one texture atlas per material (17-21 per block), so the + mesh has to be sampled per material, not as one merged blob. Merging with + force='mesh' silently drops every texture and the output comes out grey. + * no vertex colours -- the texture is the ONLY colour source. There is no + fallback to degrade to. + * coordinates already local and metric, with the true origin recorded in + metadata.xml as SRSOrigin (EPSG:5186+9999). Same convention SUM Parts + itself ships, so no reprojection is needed -- but never feed raw easting/ + northing or degrees, since every cfg radius and voxel size is in metres. + +--bbox crops before sampling, which is how you get one ~252 m SUM-sized tile +out of a 465 x 487 m block. + +Usage: + python mesh_to_ply.py Block.obj out.ply --points 500000 + python mesh_to_ply.py Block.obj tile.ply --bbox 300 700 552 952 +""" +from __future__ import annotations + +import argparse +import sys +import time +from pathlib import Path + +import numpy as np + + +def die(msg: str) -> None: + print(f"error: {msg}", file=sys.stderr) + raise SystemExit(1) + + +def log(msg: str) -> None: + print(f"[{time.strftime('%H:%M:%S')}] {msg}", flush=True) + + +def load_scene(path: Path): + """Load as a Scene so each material keeps its own texture.""" + try: + import trimesh + except ImportError: + die("trimesh not installed. run: pip install trimesh pillow") + + log(f"loading {path.name} ({path.stat().st_size / 1e6:,.0f} MB) ...") + scene = trimesh.load(path, process=False) + + if isinstance(scene, trimesh.Trimesh): + geoms = [scene] + else: + geoms = list(scene.geometry.values()) + + geoms = [g for g in geoms if getattr(g, "faces", None) is not None and len(g.faces)] + if not geoms: + die(f"{path} has no faces") + + log(f"loaded {len(geoms)} geometry group(s), " + f"{sum(len(g.faces) for g in geoms):,} faces total") + return geoms + + +def crop(geom, bbox): + """Keep faces whose centroid falls inside the XY bbox. Returns a mask.""" + if bbox is None: + return np.ones(len(geom.faces), dtype=bool) + x0, y0, x1, y1 = bbox + c = geom.vertices[geom.faces].mean(axis=1) + return (c[:, 0] >= x0) & (c[:, 0] < x1) & (c[:, 1] >= y0) & (c[:, 1] < y1) + + +def sample_geom(geom, face_mask, n_points, seed): + """Area-weighted sampling restricted to the masked faces, with colour.""" + import trimesh + + idx = np.flatnonzero(face_mask) + if idx.size == 0 or n_points <= 0: + return np.empty((0, 3)), np.empty((0, 3), dtype=np.uint8) + + tri = geom.vertices[geom.faces[idx]] # (m, 3, 3) + area = trimesh.triangles.area(tri) + total = area.sum() + if total <= 0: + return np.empty((0, 3)), np.empty((0, 3), dtype=np.uint8) + + rng = np.random.default_rng(seed) + pick = rng.choice(idx.size, size=n_points, p=area / total) + + # uniform barycentric coordinates over each chosen triangle + r1, r2 = rng.random(n_points), rng.random(n_points) + s = np.sqrt(r1) + bary = np.stack([1 - s, s * (1 - r2), s * r2], axis=1) # (n, 3) + + xyz = (tri[pick] * bary[:, :, None]).sum(axis=1) + rgb = colour(geom, idx[pick], bary) + return xyz, rgb + + +def colour(geom, face_idx, bary) -> np.ndarray: + """Per-point RGB from the geometry's texture; grey if it has none.""" + n = len(face_idx) + visual = getattr(geom, "visual", None) + + uv = getattr(visual, "uv", None) + material = getattr(visual, "material", None) + image = getattr(material, "image", None) if material is not None else None + + if uv is not None and image is not None: + try: + tri_uv = np.asarray(uv)[geom.faces[face_idx]] # (n, 3, 2) + point_uv = (tri_uv * bary[:, :, None]).sum(axis=1) + img = np.asarray(image.convert("RGB")) + h, w = img.shape[:2] + # OBJ UV origin is bottom-left; image rows run top-down + px = np.clip((point_uv[:, 0] % 1.0) * (w - 1), 0, w - 1).astype(np.int32) + py = np.clip((1.0 - point_uv[:, 1] % 1.0) * (h - 1), 0, h - 1).astype(np.int32) + return img[py, px].astype(np.uint8) + except Exception as e: # noqa: BLE001 - exporters vary wildly + print(f"warn: texture sampling failed ({e})", file=sys.stderr) + + vc = getattr(visual, "vertex_colors", None) + if vc is not None and len(vc) == len(geom.vertices): + vc = np.asarray(vc)[:, :3].astype(np.float64) + return (vc[geom.faces[face_idx]] * bary[:, :, None]).sum(axis=1).astype(np.uint8) + + return np.full((n, 3), 128, dtype=np.uint8) + + +def write_ply(path: Path, xyz, rgb, label: int, object_index: int | None, + colour_style: str = "sum") -> None: + """Write the point cloud. + + colour_style='sum' reproduces exactly what SUM Parts ships: fields named + r/g/b holding float32 in [0, 1]. That match is not cosmetic. The loader + normalises nothing -- + + rgb = np.stack([...'red','green','blue'...]).astype(np.uint8) + except ValueError: + rgb = np.stack([...'r','g','b'...]).astype(np.float32) + if np.max(rgb) > 1: + rgb = rgb # <- no-op, despite how it reads + + -- and the active datatransforms in cfgs/sumv2_triangle/default.yaml are + [PointsToTensor, PointCloudScaling, PointCloudRotation, PointCloudJitter]; + NumpyChromaticNormalize is commented out. So whatever is in the file reaches + the network unscaled. Writing 0-255 where the training data was 0-1 hands + the model colour features 255x too large. + + colour_style='uint8' writes red/green/blue as uint8 instead, for viewers + that expect the conventional PLY spelling. + """ + try: + from plyfile import PlyData, PlyElement + except ImportError: + die("plyfile not installed. run: pip install plyfile") + + if colour_style == "sum": + cols = [("r", "f4"), ("g", "f4"), ("b", "f4")] + vals = (rgb.astype(np.float32) / 255.0) + else: + cols = [("red", "u1"), ("green", "u1"), ("blue", "u1")] + vals = rgb + + # demo tiles store label as int32 + dtype = [("x", "f4"), ("y", "f4"), ("z", "f4"), *cols, ("label", "i4")] + if object_index is not None: + dtype.append(("object_index", "i4")) + + arr = np.empty(len(xyz), dtype=dtype) + arr["x"], arr["y"], arr["z"] = xyz[:, 0], xyz[:, 1], xyz[:, 2] + for i, (name, _) in enumerate(cols): + arr[name] = vals[:, i] + arr["label"] = label + if object_index is not None: + arr["object_index"] = object_index + + path.parent.mkdir(parents=True, exist_ok=True) + PlyData([PlyElement.describe(arr, "vertex")], text=False).write(str(path)) + + +def main() -> None: + ap = argparse.ArgumentParser( + description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter) + ap.add_argument("input", type=Path) + ap.add_argument("output", type=Path) + ap.add_argument("--points", type=int, default=500_000, + help="total points to sample (default: 500000)") + ap.add_argument("--bbox", type=float, nargs=4, metavar=("X0", "Y0", "X1", "Y1"), + help="crop to this XY box in the mesh's own local coords") + ap.add_argument("--label", type=int, default=0, + help="constant label to stamp; 0 = unclassified (default: 0)") + ap.add_argument("--object-index", type=int, default=None) + ap.add_argument("--colour-style", choices=("sum", "uint8"), default="sum", + help="'sum' (default): r/g/b float32 0-1, byte-for-byte what " + "SUM Parts ships. 'uint8': red/green/blue 0-255 for viewers") + ap.add_argument("--keep-origin", action="store_true", + help="do NOT translate the result to a local origin") + ap.add_argument("--seed", type=int, default=0) + args = ap.parse_args() + + if not args.input.exists(): + die(f"{args.input} not found") + if not 0 <= args.label <= 255: + die("--label must fit in a uint8 (0-255)") + + geoms = load_scene(args.input) + + # Budget points per geometry by cropped surface area, so a material that + # covers half the tile gets half the points. + import trimesh + masks, areas = [], [] + for g in geoms: + m = crop(g, args.bbox) + masks.append(m) + a = trimesh.triangles.area(g.vertices[g.faces[m]]).sum() if m.any() else 0.0 + areas.append(a) + + total_area = float(sum(areas)) + if total_area <= 0: + die("nothing left after cropping -- check --bbox against the mesh bbox") + log(f"cropped surface area: {total_area:,.0f} m^2 across " + f"{sum(int(m.sum()) for m in masks):,} faces") + + parts_xyz, parts_rgb = [], [] + for i, (g, m, a) in enumerate(zip(geoms, masks, areas)): + if a <= 0: + continue + n = int(round(args.points * a / total_area)) + if n <= 0: + continue + xyz, rgb = sample_geom(g, m, n, args.seed + i) + if len(xyz): + parts_xyz.append(xyz) + parts_rgb.append(rgb) + log(f" geom {i:>3}: {int(m.sum()):>8,} faces {a:>12,.0f} m^2 -> {len(xyz):>8,} pts") + + if not parts_xyz: + die("no points sampled") + + xyz = np.vstack(parts_xyz) + rgb = np.vstack(parts_rgb) + + lo, hi = xyz.min(axis=0), xyz.max(axis=0) + log(f"extent: {np.round(hi - lo, 2)} m origin: {np.round(lo, 2)}") + + if not args.keep_origin: + xyz = xyz - lo + log("translated to local origin (min corner -> 0,0,0)") + + if (hi - lo).max() < 0.5: + print("warn: extent under 0.5 units -- looks like degrees, not metres. " + "Reproject to a metric CRS (e.g. EPSG:5186) first.", file=sys.stderr) + + grey = int((rgb == 128).all(axis=1).sum()) + if grey: + print(f"warn: {grey:,}/{len(rgb):,} points fell back to grey " + "(missing texture)", file=sys.stderr) + + write_ply(args.output, xyz, rgb, args.label, args.object_index, args.colour_style) + log(f"wrote {args.output} ({len(xyz):,} points, label={args.label}, " + f"colour={args.colour_style})") + + +if __name__ == "__main__": + main() diff --git a/scripts/monitor_training.sh b/scripts/monitor_training.sh new file mode 100644 index 0000000..e40fbe6 --- /dev/null +++ b/scripts/monitor_training.sh @@ -0,0 +1,113 @@ +#!/usr/bin/env bash +# SUM Parts - emit training progress as discrete events, exit when the run ends +# +# Designed to be selective: a 100-epoch run writes ~100 epoch lines, which would +# be 100 notifications. So this emits every Nth epoch, plus anything that would +# change what to do next -- watchdog retries, tracebacks, OOM, GPU trouble, and +# the final outcome. +# +# Silence must not be mistakable for success, so the terminal check keys off the +# watchdog process disappearing, not off a success marker appearing. +set -uo pipefail + +RUNROOT="$HOME/sum-parts/runs" +EVERY="${EVERY:-10}" # emit one progress line per this many epochs +POLL="${POLL:-120}" +TOTAL="${TOTAL:-100}" # target epoch count, for the "n/TOTAL" display + +WORKDIR="${1:-}" +if [ -z "$WORKDIR" ]; then + WORKDIR=$(cat "$RUNROOT/.latest" 2>/dev/null || true) +fi +if [ -z "$WORKDIR" ] || [ ! -d "$WORKDIR" ]; then + echo "MONITOR-ERROR: no run directory (looked at $RUNROOT/.latest)" + exit 1 +fi + +TRAINLOG="$WORKDIR/train.log" +WDLOG="$WORKDIR/watchdog.log" + +echo "MONITOR-START $(basename "$WORKDIR") every=${EVERY}ep poll=${POLL}s" + +last_epoch_reported=0 +wd_lines=0 +err_lines=0 + +while true; do + # --- progress ------------------------------------------------------ + if [ -f "$TRAINLOG" ]; then + line=$(grep -aE 'Epoch [0-9]+ LR' "$TRAINLOG" 2>/dev/null | tail -1) + if [ -n "$line" ]; then + # Report the epoch number the trainer itself prints. Counting log + # lines is wrong after a resume: the new log starts mid-run, so a + # run continuing at epoch 31 would be announced as epoch 11. + ep=$(echo "$line" | grep -oE 'Epoch [0-9]+' | grep -oE '[0-9]+') + if [ -n "$ep" ] && [ $((ep / EVERY)) -gt $((last_epoch_reported / EVERY)) ]; then + best=$(grep -aE 'Find a better ckpt' "$TRAINLOG" | tail -1 \ + | grep -oE 'val_miou [0-9.]+' | tail -1) + echo "EPOCH $ep/$TOTAL | ${line#*] } | best: ${best:-none}" + last_epoch_reported=$ep + fi + fi + fi + + # --- watchdog events (retries, resumes, give-ups) ------------------ + if [ -f "$WDLOG" ]; then + c=$(wc -l < "$WDLOG") + if [ "$c" -gt "$wd_lines" ]; then + tail -n +$((wd_lines + 1)) "$WDLOG" \ + | grep -E 'attempt|exited rc=|giving up|exhausted|finished cleanly|FATAL' || true + wd_lines=$c + fi + fi + + # --- speed regression ----------------------------------------------- + # The first run silently fell from 0.29 s/it to 3.1 s/it when the caching + # allocator pool overflowed VRAM into host RAM. No error, no OOM -- just a + # 24-hour ETA. Watch the rate, not only the exit code. + if [ -f "$TRAINLOG" ]; then + r=$(grep -aoE '[0-9.]+(s/it|it/s)' "$TRAINLOG" | tail -1) + if [ -n "$r" ]; then + sec=$(awk -v x="$r" 'BEGIN { + if (x ~ /it\/s$/) { sub(/it\/s$/,"",x); printf "%.3f", (x>0 ? 1.0/x : 0) } + else { sub(/s\/it$/,"",x); printf "%.3f", x } + }') + slow=$(awk -v v="$sec" 'BEGIN { print (v > 1.5) ? 1 : 0 }') + if [ "$slow" = "1" ] && [ "${slow_reported:-0}" = "0" ]; then + mem=$(nvidia-smi --query-gpu=memory.used,power.draw --format=csv,noheader 2>/dev/null) + echo "SLOWDOWN: ${sec} s/it (healthy is ~0.29) | GPU $mem" + slow_reported=1 + elif [ "$slow" = "0" ]; then + slow_reported=0 + fi + fi + fi + + # --- failure signatures in the trainer itself ---------------------- + if [ -f "$TRAINLOG" ]; then + e=$(grep -cE 'Traceback|CUDA error|out of memory|OutOfMemory|Killed|AssertionError' \ + "$TRAINLOG" 2>/dev/null || echo 0) + if [ "$e" -gt "$err_lines" ]; then + echo "ERROR-SIGNAL x$((e - err_lines)):" + grep -E 'Traceback|CUDA error|out of memory|OutOfMemory|Killed|AssertionError' \ + "$TRAINLOG" | tail -3 + err_lines=$e + fi + fi + + # --- terminal check: watchdog gone means the run is over, either way + if ! pgrep -f train_watchdog.sh > /dev/null 2>&1; then + sleep 5 + state=$(grep -E '^state' "$WORKDIR/status.txt" 2>/dev/null | cut -d: -f2- | xargs) + final=$(grep -aE 'Epoch [0-9]+ LR' "$TRAINLOG" 2>/dev/null | tail -1 \ + | grep -oE 'Epoch [0-9]+' | grep -oE '[0-9]+') + echo "MONITOR-END state='${state:-unknown}' last_epoch=${final:-0}/$TOTAL" + if [ -f "$TRAINLOG" ]; then + echo "--- tail ---" + tail -8 "$TRAINLOG" + fi + exit 0 + fi + + sleep "$POLL" +done diff --git a/scripts/parse_ious.sh b/scripts/parse_ious.sh new file mode 100644 index 0000000..f05fa9a --- /dev/null +++ b/scripts/parse_ious.sh @@ -0,0 +1,56 @@ +#!/usr/bin/env bash +# SUM Parts - map the per-class IoU array onto class names +# +# The log prints a bare numpy array. Guessing whether it starts at class 0 or +# class 1 by eye is how you end up reporting "car IoU 95.6". Count it. +set -uo pipefail + +LOG="${1:-$HOME/sum-parts/runs/val_ab/val_capped.log}" + +source "$HOME/miniconda3/etc/profile.d/conda.sh" +conda activate sumparts + +python - "$LOG" <<'PY' +import re +import sys +from pathlib import Path + +CLASSES = ['unclassified', 'terrain', 'high_vegetation', 'facade_surface', + 'water', 'car', 'boat', 'roof_surface', 'chimney', 'dormer', + 'balcony', 'roof_installation', 'wall'] + +text = Path(sys.argv[1]).read_text(encoding="utf-8", errors="replace") + +m = re.findall(r"iou per cls is:\s*\[([^\]]*)\]", text) +if not m: + print("no 'iou per cls' line found in", sys.argv[1]) + raise SystemExit(1) + +vals = [float(x) for x in m[-1].split()] +print(f"file : {Path(sys.argv[1]).name}") +print(f"entries: {len(vals)} (num_classes = {len(CLASSES)})") +print() + +if len(vals) == len(CLASSES): + names = CLASSES + note = "array covers classes 0..12" +elif len(vals) == len(CLASSES) - 1: + names = CLASSES[:-1] + note = ("array is one short of num_classes. ConfusionMatrix remaps the " + "ignore_index into slot num_classes-1, so the last class shares a " + "bucket with ignored points and is dropped from the report.") +else: + names = [f"class_{i}" for i in range(len(vals))] + note = "unexpected length -- names are positional only" + +print(f"note : {note}\n") +print(f" {'#':>2} {'class':<20} {'IoU':>7}") +for i, (n, v) in enumerate(zip(names, vals)): + mark = " <-- 0" if v == 0 else "" + print(f" {i:>2} {n:<20} {v:>7.2f}{mark}") + +nz = [v for v in vals if v > 0] +print() +print(f" non-zero classes : {len(nz)}/{len(vals)}") +print(f" mean over all : {sum(vals)/len(vals):.2f}") +PY diff --git a/scripts/patch_numpy_aliases.sh b/scripts/patch_numpy_aliases.sh new file mode 100644 index 0000000..6e9f08f --- /dev/null +++ b/scripts/patch_numpy_aliases.sh @@ -0,0 +1,82 @@ +#!/usr/bin/env bash +# SUM Parts - modernise APIs the upstream code uses that newer runtimes removed +# +# Two families, same root cause (2022 code on a 2025 toolchain): +# 1. numpy aliases removed in numpy 1.24 +# 2. collections ABCs moved to collections.abc in python 3.10 +# +# The upstream env pins numpy 1.20, where np.long / np.int / np.float / np.bool +# / np.object / np.str still existed as aliases. numpy 1.24 removed them, so on +# any modern numpy the dataset loaders die inside a DataLoader worker with +# +# AttributeError: module 'numpy' has no attribute 'long' +# +# and the traceback points at the worker, not at the version mismatch. +# +# Downgrading numpy is not an option here: torch 2.0.1 needs numpy<2 but +# numpy 1.20 has no python 3.10 wheels. So patch the source instead. +# +# Mapping follows what the aliases actually were: +# np.long -> np.int64 (alias of python int, i.e. C long) +# np.int -> int +# np.float -> float +# np.bool -> bool +# np.object -> object +# np.str -> str +# +# Idempotent: re-running finds nothing to change. Writes .bak files on first +# touch only. +set -euo pipefail + +REPO="${1:-$HOME/sum-parts/semantic_segmentation/PointNeXt_bundle}" + +if [ ! -d "$REPO/openpoints" ]; then + echo "error: $REPO does not look like PointNeXt_bundle" >&2 + exit 1 +fi + +cd "$REPO" + +ABCS='Iterable\|Mapping\|MutableMapping\|Sequence\|Callable\|Hashable\|Iterator\|Container\|Sized' +PATTERN="np\.long\b\|np\.int\b\|np\.float\b\|np\.bool\b\|np\.object\b\|np\.str\b\|collections\.\($ABCS\)\b" + +echo "=== before ===" +grep -rn "$PATTERN" --include='*.py' . || echo " (none)" + +# \b keeps np.int from matching np.int32/np.int64, np.float from np.float32, etc. +# The collections rule skips anything already written as collections.abc.X. +find . -name '*.py' -print0 | xargs -0 sed -i.bak \ + -e 's/\bnp\.long\b/np.int64/g' \ + -e 's/\bnp\.int\b/int/g' \ + -e 's/\bnp\.float\b/float/g' \ + -e 's/\bnp\.bool\b/bool/g' \ + -e 's/\bnp\.object\b/object/g' \ + -e 's/\bnp\.str\b/str/g' \ + -e "s/\bcollections\.\($ABCS\)\b/collections.abc.\1/g" + +# sed -i.bak writes a .bak for every file it opens, not just changed ones +find . -name '*.py.bak' -print0 | while IFS= read -r -d '' b; do + if cmp -s "$b" "${b%.bak}"; then rm -f "$b"; fi +done + +# `import collections` alone does not pull in collections.abc on every python, +# so make sure any file that now references collections.abc imports it. +grep -rl 'collections\.abc\.' --include='*.py' . | while IFS= read -r f; do + if ! grep -q '^import collections.abc' "$f"; then + sed -i 's/^import collections$/import collections\nimport collections.abc/' "$f" + fi +done + +echo +echo "=== after ===" +if grep -rn "$PATTERN" --include='*.py' .; then + echo "WARNING: some occurrences remain" >&2 +else + echo " clean" +fi + +echo +echo "=== files changed (.bak kept) ===" +find . -name '*.py.bak' | sed 's/\.bak$//' || true + +echo "PATCH DONE" diff --git a/scripts/patch_unlabeled_test.sh b/scripts/patch_unlabeled_test.sh new file mode 100644 index 0000000..271f8e3 --- /dev/null +++ b/scripts/patch_unlabeled_test.sh @@ -0,0 +1,73 @@ +#!/usr/bin/env bash +# SUM Parts - let test() run on the withheld (unlabeled) test split +# +# The 8 test tiles ship with label = -1 for every point. That is deliberate: +# the test set is blind, and the README says to email predictions to the authors +# for scoring. Training and validation labels are normal (0..12). +# +# main.py only checks `if label is not None`, so the -1 placeholder flows into +# ConfusionMatrix.update, where +# unique_mapping = true * num_classes + pred +# goes negative and torch.bincount rejects it: +# RuntimeError: bincount only supports 1-d non-negative integral inputs +# +# The fix treats an all-negative label array as "no ground truth": predictions +# are still produced and written, metrics are simply skipped for that tile. +# This changes no scoring behaviour on labeled data. +# +# Idempotent -- re-running detects the patch is already applied. +set -euo pipefail + +source "$HOME/miniconda3/etc/profile.d/conda.sh" +conda activate sumparts + +REPO="${1:-$HOME/sum-parts/semantic_segmentation/PointNeXt_bundle}" +MAIN="$REPO/examples/segmentation/main.py" + +[ -f "$MAIN" ] || { echo "error: $MAIN not found" >&2; exit 1; } + +if grep -q 'SUMPARTS-UNLABELED-TEST' "$MAIN"; then + echo "already patched" + exit 0 +fi + +cp -n "$MAIN" "$MAIN.orig" 2>/dev/null || true + +python - "$MAIN" <<'PY' +import sys +from pathlib import Path + +p = Path(sys.argv[1]) +src = p.read_text(encoding="utf-8") + +old = """ coord, feat, label, idx_points, voxel_idx, reverse_idx_part, reverse_idx = load_data(data_path, cfg) + if label is not None: + label = torch.from_numpy(label.astype(int).squeeze()).cuda(non_blocking=True) +""" + +new = """ coord, feat, label, idx_points, voxel_idx, reverse_idx_part, reverse_idx = load_data(data_path, cfg) + # SUMPARTS-UNLABELED-TEST: the shipped test split carries label = -1 for + # every point (blind test set; predictions are emailed to the authors for + # scoring). Passing that to ConfusionMatrix makes true*num_classes+pred + # negative and torch.bincount raises. Treat it as "no ground truth" so + # predictions are still produced. Labeled splits are unaffected. + if label is not None and (label < 0).all(): + logging.info(f' no ground truth in {os.path.basename(data_path)} ' + f'(all labels are -1) -- predicting without scoring') + label = None + if label is not None: + label = torch.from_numpy(label.astype(int).squeeze()).cuda(non_blocking=True) +""" + +if old not in src: + print("PATTERN NOT FOUND -- main.py differs from what this patch expects", file=sys.stderr) + raise SystemExit(1) + +p.write_text(src.replace(old, new), encoding="utf-8") +print("patched:", p) +PY + +python -c "import ast,sys; ast.parse(open(sys.argv[1], encoding='utf-8').read())" "$MAIN" \ + && echo "syntax OK" + +echo "PATCH DONE (original kept at $MAIN.orig)" diff --git a/scripts/patch_val_mode.sh b/scripts/patch_val_mode.sh new file mode 100644 index 0000000..97fc041 --- /dev/null +++ b/scripts/patch_val_mode.sh @@ -0,0 +1,58 @@ +#!/usr/bin/env bash +# SUM Parts - fix mode=val crashing before it starts +# +# main.py:227 validate_fn(model, val_loader, cfg, num_votes=1, epoch=epoch) +# UnboundLocalError: local variable 'epoch' referenced before assignment +# +# `epoch` is only bound inside the training loop, so the standalone validation +# path references it before it exists. Bind it to -1 (the same sentinel +# validate() already documents in its signature) right before the call. +# +# Idempotent. +set -euo pipefail + +source "$HOME/miniconda3/etc/profile.d/conda.sh" +conda activate sumparts + +REPO="${1:-$HOME/sum-parts/semantic_segmentation/PointNeXt_bundle}" +MAIN="$REPO/examples/segmentation/main.py" + +[ -f "$MAIN" ] || { echo "error: $MAIN not found" >&2; exit 1; } + +if grep -q 'SUMPARTS-VAL-EPOCH' "$MAIN"; then + echo "already patched" + exit 0 +fi + +python - "$MAIN" <<'PY' +import sys +from pathlib import Path + +p = Path(sys.argv[1]) +src = p.read_text(encoding="utf-8") + +old = """ if cfg.mode == 'val': + best_epoch, best_val = load_checkpoint(model, pretrained_path=cfg.pretrained_path) + val_miou, val_macc, val_oa, val_ious, val_accs = validate_fn(model, val_loader, cfg, num_votes=1, epoch=epoch)""" + +new = """ if cfg.mode == 'val': + best_epoch, best_val = load_checkpoint(model, pretrained_path=cfg.pretrained_path) + # SUMPARTS-VAL-EPOCH: `epoch` is only bound inside the training + # loop below, so mode=val referenced it before assignment and + # died with UnboundLocalError. -1 is the sentinel validate() + # already defaults to. + epoch = best_epoch if best_epoch is not None else -1 + val_miou, val_macc, val_oa, val_ious, val_accs = validate_fn(model, val_loader, cfg, num_votes=1, epoch=epoch)""" + +if old not in src: + print("PATTERN NOT FOUND -- main.py differs from what this patch expects", file=sys.stderr) + raise SystemExit(1) + +p.write_text(src.replace(old, new), encoding="utf-8") +print("patched:", p) +PY + +python -c "import ast,sys; ast.parse(open(sys.argv[1], encoding='utf-8').read())" "$MAIN" \ + && echo "syntax OK" + +echo "PATCH DONE" diff --git a/scripts/ply_for_viewer.py b/scripts/ply_for_viewer.py new file mode 100644 index 0000000..de97bc6 --- /dev/null +++ b/scripts/ply_for_viewer.py @@ -0,0 +1,68 @@ +#!/usr/bin/env python3 +"""Rewrite a SUM-schema PLY into the spelling desktop viewers expect. + +The training files carry colour as r/g/b float32 in [0,1] — that is what the +SUM Parts loader reads and what the network was trained on. Most mesh/point +viewers (CloudCompare, MeshLab, Mapple) look for red/green/blue uint8 instead, +and when they don't find it they either render the cloud flat grey or list the +floats as scalar fields. + +This converts colour spelling only. Coordinates and labels pass through +untouched, so the geometry you inspect is exactly the geometry the model saw. + +Usage: + python ply_for_viewer.py in.ply out.ply +""" +from __future__ import annotations + +import sys +from pathlib import Path + +import numpy as np +from plyfile import PlyData, PlyElement + + +def main() -> None: + if len(sys.argv) != 3: + print(__doc__) + raise SystemExit(2) + + src, dst = Path(sys.argv[1]), Path(sys.argv[2]) + if not src.exists(): + raise SystemExit(f"{src}: not found") + + v = PlyData.read(str(src))["vertex"] + props = [p.name for p in v.properties] + print(f"in : {src.name} {len(v):,} points props={props}") + + rgb_set = next((s for s in (("r", "g", "b"), ("red", "green", "blue")) + if all(c in props for c in s)), None) + if rgb_set is None: + raise SystemExit("no colour channels found") + + rgb = np.stack([np.asarray(v[c], dtype=np.float64) for c in rgb_set], axis=1) + if rgb.max() <= 1.0: + rgb = rgb * 255.0 + print(" colour was float 0-1 -> scaling to 0-255") + rgb = np.clip(rgb, 0, 255).astype(np.uint8) + + dtype = [("x", "f4"), ("y", "f4"), ("z", "f4"), + ("red", "u1"), ("green", "u1"), ("blue", "u1")] + if "label" in props: + dtype.append(("label", "i4")) + + arr = np.empty(len(v), dtype=dtype) + arr["x"], arr["y"], arr["z"] = v["x"], v["y"], v["z"] + arr["red"], arr["green"], arr["blue"] = rgb[:, 0], rgb[:, 1], rgb[:, 2] + if "label" in props: + arr["label"] = np.asarray(v["label"]).astype(np.int32) + + dst.parent.mkdir(parents=True, exist_ok=True) + PlyData([PlyElement.describe(arr, "vertex")], text=False).write(str(dst)) + + print(f"out : {dst} ({dst.stat().st_size / 1e6:.1f} MB)") + print(f" mean RGB {rgb.mean(axis=0).round(1).tolist()}") + + +if __name__ == "__main__": + main() diff --git a/scripts/poc_check_pred.sh b/scripts/poc_check_pred.sh new file mode 100644 index 0000000..e2cb216 --- /dev/null +++ b/scripts/poc_check_pred.sh @@ -0,0 +1,22 @@ +#!/usr/bin/env bash +# SUM Parts - inspect the prediction PLY the POC inference run produced +set -euo pipefail + +CONDA_ROOT="$HOME/miniconda3" +ENV_NAME="sumparts" +LOGROOT="$HOME/sum-parts/semantic_segmentation/PointNeXt_bundle/examples/segmentation/log" +SCRIPTS="/mnt/d/MYCLAUDE_PROJECT/sum-parts-test/scripts" + +source "$CONDA_ROOT/etc/profile.d/conda.sh" +conda activate "$ENV_NAME" + +PRED=$(find "$LOGROOT" -name '*_pred.ply' -printf '%T@ %p\n' | sort -rn | head -1 | cut -d' ' -f2-) +if [ -z "$PRED" ]; then + echo "error: no *_pred.ply found under $LOGROOT" >&2 + exit 1 +fi + +echo "prediction: $PRED" +ls -lh "$PRED" +echo +python "$SCRIPTS/check_pred.py" "$PRED" diff --git a/scripts/poc_infer.sh b/scripts/poc_infer.sh new file mode 100644 index 0000000..d7a5d61 --- /dev/null +++ b/scripts/poc_infer.sh @@ -0,0 +1,58 @@ +#!/usr/bin/env bash +# SUM Parts - run inference on the converted Korean tile +# +# This answers one question: does a SUM Parts model accept our data and emit +# per-point predictions? It does NOT measure anything. The checkpoint is the +# 1-epoch smoke-test model and the tile carries label=0 everywhere, so the +# reported mIoU/OA are meaningless by construction. Look at whether it runs and +# what the predicted class histogram looks like, nothing else. +set -euo pipefail + +CONDA_ROOT="$HOME/miniconda3" +ENV_NAME="sumparts" +REPO="$HOME/sum-parts/semantic_segmentation/PointNeXt_bundle" +SEG="$REPO/examples/segmentation" + +TILE="$HOME/sum-parts/data/korea_poc/seosan_BlockYBA_tile0.ply" +TRACK="$HOME/sum-parts/data/korea_poc_track" +LOG="${LOG:-/tmp/sumparts_poc_infer.log}" + +source "$CONDA_ROOT/etc/profile.d/conda.sh" +conda activate "$ENV_NAME" +export WANDB_MODE=disabled WANDB_SILENT=true +export CUDA_HOME="$CONDA_PREFIX" + +[ -f "$TILE" ] || { echo "error: $TILE missing -- run poc_korea.sh first" >&2; exit 1; } + +# The dataset class globs {train,val,test}/*.ply and errors on a missing split, +# so all three have to exist even for a test-only run. +for split in train val test; do + mkdir -p "$TRACK/$split" + ln -f "$TILE" "$TRACK/$split/$(basename "$TILE")" +done +rm -rf "$TRACK/processed" + +CKPT=$(find "$SEG/log/sumv2_triangle" -name '*_ckpt_best.pth' -printf '%T@ %p\n' \ + | sort -rn | head -1 | cut -d' ' -f2-) +[ -n "$CKPT" ] || { echo "error: no checkpoint found -- run smoke_train.sh first" >&2; exit 1; } +echo "checkpoint: $CKPT" + +cd "$SEG" + +set +e +python -u main.py \ + --cfg ../../cfgs/sumv2_triangle/pointnet.yaml \ + mode=test \ + --pretrained_path "$CKPT" \ + dataset.common.data_root="$TRACK" \ + wandb.use_wandb=False \ + batch_size=2 \ + val_batch_size=1 \ + > "$LOG" 2>&1 +rc=$? +set -e + +echo "=== exit=$rc | last 25 lines ===" +tail -25 "$LOG" +[ "$rc" -eq 0 ] && echo "POC INFER DONE" || echo "POC INFER FAILED (rc=$rc)" +exit "$rc" diff --git a/scripts/poc_korea.sh b/scripts/poc_korea.sh new file mode 100644 index 0000000..67651e1 --- /dev/null +++ b/scripts/poc_korea.sh @@ -0,0 +1,55 @@ +#!/usr/bin/env bash +# SUM Parts - Korean data proof of concept +# +# Scope is deliberately one tile. The point is to answer "does our data go +# through this pipeline at all", not to measure anything. Nothing here produces +# a number worth quoting. +# +# Source: Seosan Myeongcheon, 구역2(미션7)/BlockYBA +# ContextCapture OBJ, 4,553,583 faces, 17 texture atlases +# local metric coords, EPSG:5186+9999, SRSOrigin 155184.79/469705.03/149.57 +# block extent 465.36 x 487.46 x 27.55 m +# +# We cut one 252 m tile out of the middle -- 252 m is the size of the tile +# SUM Parts ships, so the point density and neighbourhood radii line up with +# what the cfgs assume. +set -euo pipefail + +CONDA_ROOT="$HOME/miniconda3" +ENV_NAME="sumparts" +SCRIPTS="/mnt/d/MYCLAUDE_PROJECT/sum-parts-test/scripts" + +SRC="/mnt/d/MyProject_대용량샘플/02. 데이터/01. 서산 명천_08월/02. 본태모델/구역2(미션7)/BlockYBA/BlockYBA.obj" +OUT_DIR="$HOME/sum-parts/data/korea_poc" +OUT="$OUT_DIR/seosan_BlockYBA_tile0.ply" + +# centre 252 m tile of the block's 465 x 487 m footprint +X0=300; Y0=720; X1=552; Y1=972 + +# demo_texsp_pcl.ply carries 471,726 points over a 252 m tile; match it so the +# voxel_size 0.02 / voxel_max 64000 settings behave the same way +POINTS=470000 + +source "$CONDA_ROOT/etc/profile.d/conda.sh" +conda activate "$ENV_NAME" + +if [ ! -f "$SRC" ]; then + echo "error: source mesh not found:" >&2 + echo " $SRC" >&2 + exit 1 +fi + +mkdir -p "$OUT_DIR" + +echo "=== converting one 252m tile ===" +python "$SCRIPTS/mesh_to_ply.py" "$SRC" "$OUT" \ + --points "$POINTS" \ + --bbox "$X0" "$Y0" "$X1" "$Y1" \ + --label 0 + +echo +echo "=== verifying against the SUM Parts schema ===" +python "$SCRIPTS/check_ply.py" "$OUT" \ + "$HOME/sum-parts/data/pcl/face_labeling_pcl/demo_texsp_pcl.ply" + +echo "POC CONVERT DONE" diff --git a/scripts/prepare_demo_split.sh b/scripts/prepare_demo_split.sh new file mode 100644 index 0000000..6bdda93 --- /dev/null +++ b/scripts/prepare_demo_split.sh @@ -0,0 +1,52 @@ +#!/usr/bin/env bash +# SUM Parts - lay the demo tile out as a train/val/test split +# +# demo.zip ships a single showcase tile, not a split. The dataset class globs +# /{train,val,test}/*.ply, so nothing runs until that structure +# exists. +# +# WARNING: this puts the SAME tile in all three splits. That is fine for a +# pipeline smoke test (does the chain survive?) and meaningless as a +# measurement -- every val/test number it produces is train-set memorisation. +# Do not quote any mIoU from this layout. +# +# Track selection follows the cfg directory name: cfgs/sumv2_triangle uses +# data_root ../../data/sumv2_tri_texpcl, i.e. the triangle (face-label) track +# sampled with the texture-superpixel sampler -> face_labeling_pcl/*_texsp_pcl.ply +set -euo pipefail + +DATA="$HOME/sum-parts/data" + +# track name -> source ply +declare -A SRC=( + [sumv2_tri_texpcl]="$DATA/pcl/face_labeling_pcl/demo_texsp_pcl.ply" + [sumv2_tex_texpcl]="$DATA/pcl/texture_labeling_pcl/demo_texsp_pcl.ply" +) + +for track in "${!SRC[@]}"; do + src="${SRC[$track]}" + if [ ! -f "$src" ]; then + echo "skip $track: $src not found" + continue + fi + echo "=== $track <- $(basename "$src") ===" + for split in train val test; do + mkdir -p "$DATA/$track/$split" + # hardlink: same tile in three splits, one copy on disk + ln -f "$src" "$DATA/$track/$split/$(basename "$src")" + done + # a stale presample cache silently overrides the ply files + rm -rf "$DATA/$track/processed" +done + +echo +echo "=== layout ===" +for track in "${!SRC[@]}"; do + [ -d "$DATA/$track" ] || continue + for split in train val test; do + n=$(find "$DATA/$track/$split" -name '*.ply' 2>/dev/null | wc -l) + printf '%-20s %-6s %s ply\n' "$track" "$split" "$n" + done +done + +echo "SPLIT DONE" diff --git a/scripts/prepare_full_split.sh b/scripts/prepare_full_split.sh new file mode 100644 index 0000000..81af2c0 --- /dev/null +++ b/scripts/prepare_full_split.sh @@ -0,0 +1,52 @@ +#!/usr/bin/env bash +# SUM Parts - make the extracted dataset match what the loader globs for +# +# pcl.zip ships splits named train / validate / test. The dataset class asks for +# split='val': +# +# if split == "train" or split == 'val': +# self.data_list = glob.glob(os.path.join(data_root, split, "*.ply")) +# +# so `validate/` is never found and validation silently runs on 0 samples -- +# the same quiet failure mode as a wrong data_root. Symlink val -> validate. +# +# Track layout after extraction: +# face_labeling/{face_cen,possion,random,texsp}_pcl/ -> 13-class triangle +# pixel_labeling/{possion,random,texsp}_pcl/ -> 20-class texture +# +# cfgs/sumv2_triangle uses data_root ../../data/sumv2_tri_texpcl, i.e. the +# triangle track sampled with the texture-superpixel sampler: +# face_labeling/texsp_pcl +set -euo pipefail + +DATA="$HOME/sum-parts/data" + +echo "=== linking val -> validate in every track ===" +for track in "$DATA"/face_labeling/*_pcl "$DATA"/pixel_labeling/*_pcl; do + [ -d "$track" ] || continue + if [ -d "$track/validate" ] && [ ! -e "$track/val" ]; then + ln -sfn validate "$track/val" + echo " linked $(basename "$(dirname "$track")")/$(basename "$track")/val -> validate" + fi +done + +echo +echo "=== split sizes ===" +printf '%-40s %6s %6s %6s\n' "track" "train" "val" "test" +for track in "$DATA"/face_labeling/*_pcl "$DATA"/pixel_labeling/*_pcl; do + [ -d "$track" ] || continue + rel="$(basename "$(dirname "$track")")/$(basename "$track")" + # -L so the val symlink is followed; plain find would report 0 and look + # like the link did not work + tr=$(find -L "$track/train" -name '*.ply' 2>/dev/null | wc -l) + va=$(find -L "$track/val" -name '*.ply' 2>/dev/null | wc -l) + te=$(find -L "$track/test" -name '*.ply' 2>/dev/null | wc -l) + printf '%-40s %6s %6s %6s\n' "$rel" "$tr" "$va" "$te" +done + +echo +echo "=== recommended data_root ===" +echo " triangle (13 classes) : $DATA/face_labeling/texsp_pcl" +echo " texture (20 classes) : $DATA/pixel_labeling/texsp_pcl" + +echo "SPLIT DONE" diff --git a/scripts/repair_env.sh b/scripts/repair_env.sh new file mode 100644 index 0000000..04ab98c --- /dev/null +++ b/scripts/repair_env.sh @@ -0,0 +1,79 @@ +#!/usr/bin/env bash +# SUM Parts - repair packages truncated by the WSL VM crash +# +# The WSL instance died (Wsl/Service/CreateInstance/E_FAIL) partway through the +# dependency install. Everything pip had open at that moment landed on disk as +# 0-byte files. The failure mode is nasty because import still succeeds -- the +# module is simply empty, so you get things like +# +# from torch_scatter import scatter +# TypeError: 'module' object is not callable +# +# pointing at the call site rather than at the broken install. +# +# This reinstalls the batch that was in flight. Zero-byte __init__.py files are +# normal (namespace markers), so the scan below ignores those. +set -euo pipefail + +CONDA_ROOT="$HOME/miniconda3" +ENV_NAME="sumparts" + +source "$CONDA_ROOT/etc/profile.d/conda.sh" +conda activate "$ENV_NAME" + +SITE=$(python -c "import site; print(site.getsitepackages()[0])") + +scan() { + find "$SITE" -type f \( -name '*.py' -o -name '*.so' \) -size 0 \ + ! -name '__init__.py' | wc -l +} + +echo "=== suspicious 0-byte files before: $(scan) ===" + +# The dependency batch that was installing when the VM went down. +PKGS=( + plyfile scikit-learn easydict PyYAML tensorboard termcolor tqdm + multimethod h5py matplotlib pandas shortuuid gdown Cython pyvista + wandb trimesh pillow +) + +echo "=== force-reinstalling ${#PKGS[@]} packages ===" +pip install --no-cache-dir --force-reinstall "${PKGS[@]}" + +# Pins that other steps depend on; --force-reinstall above can pull them up. +echo "=== restoring pins ===" +pip install --no-cache-dir "numpy<2" "setuptools==69.5.1" "ninja==1.11.1.1" + +# torch-scatter must match the exact torch build, so it needs its own index. +python -c "import torch_scatter, inspect; assert callable(torch_scatter.scatter)" 2>/dev/null || { + echo "=== reinstalling torch-scatter ===" + pip install --no-cache-dir --force-reinstall torch-scatter \ + -f https://data.pyg.org/whl/torch-2.0.1+cu118.html +} + +echo +echo "=== suspicious 0-byte files after: $(scan) ===" +find "$SITE" -type f \( -name '*.py' -o -name '*.so' \) -size 0 \ + ! -name '__init__.py' | head -20 + +echo +echo "=== import check ===" +python - <<'PY' +import importlib +mods = ["numpy", "torch", "torch_scatter", "plyfile", "sklearn", "matplotlib", + "pandas", "h5py", "yaml", "easydict", "tqdm", "termcolor", "multimethod", + "wandb", "trimesh", "PIL", "gdown"] +bad = [] +for m in mods: + try: + importlib.import_module(m) + print(f"{m:14s} OK") + except Exception as e: + bad.append(m) + print(f"{m:14s} FAIL {type(e).__name__}: {e}") +import torch_scatter +print("torch_scatter.scatter callable:", callable(torch_scatter.scatter)) +raise SystemExit(1 if bad or not callable(torch_scatter.scatter) else 0) +PY + +echo "REPAIR DONE" diff --git a/scripts/run_bench.sh b/scripts/run_bench.sh new file mode 100644 index 0000000..3fafdfd --- /dev/null +++ b/scripts/run_bench.sh @@ -0,0 +1,29 @@ +#!/usr/bin/env bash +# SUM Parts - benchmark VRAM and speed of each sumv2_triangle model +# +# Uses whatever tiles are in data/sumv2_tri_texpcl. With only the demo tile +# present the per-iteration cost is still representative: voxel_max caps how +# many points reach the network per sample, so timing does not depend on how +# many tiles exist. +set -euo pipefail + +CONDA_ROOT="$HOME/miniconda3" +ENV_NAME="sumparts" +SEG="$HOME/sum-parts/semantic_segmentation/PointNeXt_bundle/examples/segmentation" +SCRIPTS="/mnt/d/MYCLAUDE_PROJECT/sum-parts-test/scripts" +LOG="${LOG:-/tmp/sumparts_bench.log}" + +source "$CONDA_ROOT/etc/profile.d/conda.sh" +conda activate "$ENV_NAME" +export WANDB_MODE=disabled WANDB_SILENT=true CUDA_HOME="$CONDA_PREFIX" + +cd "$SEG" + +set +e +python -u "$SCRIPTS/bench_models.py" "$@" > "$LOG" 2>&1 +rc=$? +set -e + +grep -v "it/s\]" "$LOG" | tail -40 +echo "(full log: $LOG)" +exit "$rc" diff --git a/scripts/setup_env.sh b/scripts/setup_env.sh new file mode 100644 index 0000000..df0b858 --- /dev/null +++ b/scripts/setup_env.sh @@ -0,0 +1,49 @@ +#!/usr/bin/env bash +# SUM Parts - WSL2 conda env setup (no sudo required) +# +# NOTE: Anaconda's "defaults" channels (repo.anaconda.com/pkgs/*) require accepting +# a Terms of Service that carries commercial-license obligations for larger orgs. +# This script deliberately avoids them entirely: conda-forge for packages, +# nvidia channel for the CUDA toolkit, both with --override-channels. +set -euo pipefail + +CONDA_ROOT="$HOME/miniconda3" +ENV_NAME="sumparts" + +export PATH="$CONDA_ROOT/bin:$PATH" +source "$CONDA_ROOT/etc/profile.d/conda.sh" + +echo "=== [0/5] pin channels to conda-forge (drop anaconda defaults) ===" +conda config --remove channels defaults 2>/dev/null || true +conda config --add channels conda-forge +conda config --set channel_priority strict + +echo "=== [1/5] create env: $ENV_NAME (python 3.10) ===" +conda create -n "$ENV_NAME" -y --override-channels -c conda-forge python=3.10 + +conda activate "$ENV_NAME" + +echo "=== [2/5] CUDA Toolkit 11.8 (nvidia channel, no sudo) ===" +conda install -y --override-channels -c "nvidia/label/cuda-11.8.0" cuda-toolkit + +echo "=== [3/5] PyTorch 2.0.1 + cu118 (pip wheels) ===" +pip install --no-cache-dir \ + torch==2.0.1+cu118 torchvision==0.15.2+cu118 \ + --index-url https://download.pytorch.org/whl/cu118 + +echo "=== [4/5] pin numpy<2 (torch 2.0.x is ABI-incompatible with numpy 2) ===" +pip install --no-cache-dir "numpy<2" + +echo "=== [5/5] verify ===" +which nvcc +nvcc --version | tail -2 +python - <<'PY' +import torch, numpy +print("torch :", torch.__version__) +print("cuda :", torch.version.cuda) +print("avail :", torch.cuda.is_available()) +print("device :", torch.cuda.get_device_name(0) if torch.cuda.is_available() else None) +print("numpy :", numpy.__version__) +PY + +echo "ENV DONE" diff --git a/scripts/setup_pointnext.sh b/scripts/setup_pointnext.sh new file mode 100644 index 0000000..5e1cc9a --- /dev/null +++ b/scripts/setup_pointnext.sh @@ -0,0 +1,68 @@ +#!/usr/bin/env bash +# SUM Parts - PointNeXt_bundle dependencies + CUDA extension build +# +# The upstream install.sh assumes python 3.7 / torch 1.12.1 / cu113 and pins +# requirements.txt to versions that no longer resolve on python 3.10. +# This script keeps the same package set but relaxes the pins, and drops +# packages that are not needed for the sumv2 segmentation task: +# - deepspeed (not used by the sumv2 training path, heavy build) +# - mkdocs-* (docs only) +# It also adds two packages the code needs but requirements.txt omits: +# - plyfile : imported by both sumv2 dataset loaders +# - wandb : main.py imports it at module scope, so it must exist even when +# wandb.use_wandb=False. Run with WANDB_MODE=disabled. +# NOTE: the chamfer_dist / emd extensions look reconstruction-only, but +# openpoints/models/__init__.py imports .reconstruction unconditionally, so +# segmentation runs need them too. build_ext.sh builds them. +set -euo pipefail + +CONDA_ROOT="$HOME/miniconda3" +ENV_NAME="sumparts" +REPO="$HOME/sum-parts/semantic_segmentation/PointNeXt_bundle" + +source "$CONDA_ROOT/etc/profile.d/conda.sh" +conda activate "$ENV_NAME" + +# RTX 3060 == compute capability 8.6. Pin it so nvcc does not build every arch. +export TORCH_CUDA_ARCH_LIST="8.6" +export CUDA_HOME="$CONDA_PREFIX" +export PATH="$CUDA_HOME/bin:$PATH" + +echo "=== [1/4] python deps ===" +# setuptools 69.5.1 : subsampling/setup.py imports numpy.distutils, which needs +# distutils.msvccompiler -- removed in setuptools 74.0. The extensions also +# still use `python setup.py install`, dropped in setuptools 80. +pip install --no-cache-dir "setuptools==69.5.1" wheel + +pip install --no-cache-dir \ + plyfile \ + scikit-learn \ + ninja \ + easydict \ + PyYAML \ + tensorboard \ + termcolor \ + tqdm \ + multimethod \ + h5py \ + matplotlib \ + pandas \ + shortuuid \ + gdown \ + Cython \ + pyvista \ + wandb \ + trimesh \ + pillow \ + "numpy<2" + +echo "=== [2/4] torch-scatter (matched to torch 2.0.1+cu118) ===" +pip install --no-cache-dir torch-scatter \ + -f https://data.pyg.org/whl/torch-2.0.1+cu118.html + +echo "=== [3/4] build CUDA extensions ===" +# Delegated to build_ext.sh so a failed compile can be retried on its own. +# That script also pins ninja==1.11.1.1 (torch 2.0 + ninja>=1.12 dies on SIGPIPE). +bash "$(dirname "$0")/build_ext.sh" + +echo "POINTNEXT DONE" diff --git a/scripts/show_layout.sh b/scripts/show_layout.sh new file mode 100644 index 0000000..e022088 --- /dev/null +++ b/scripts/show_layout.sh @@ -0,0 +1,24 @@ +#!/usr/bin/env bash +# SUM Parts - what actually landed on disk after extraction +set -euo pipefail + +DATA="$HOME/sum-parts/data" +cd "$DATA" + +printf '%-26s %8s %10s\n' "directory" "ply" "size" +printf -- '--------------------------------------------------\n' +for d in */; do + n=$(find "$d" -name '*.ply' 2>/dev/null | wc -l) + s=$(du -sh "$d" 2>/dev/null | cut -f1) + printf '%-26s %8s %10s\n' "$d" "$n" "$s" +done + +echo +echo "=== ply per split, two levels down ===" +find . -mindepth 2 -maxdepth 3 -type d 2>/dev/null | sort | while IFS= read -r d; do + n=$(find "$d" -maxdepth 1 -name '*.ply' 2>/dev/null | wc -l) + [ "$n" -gt 0 ] && printf '%-52s %4s ply\n' "$d" "$n" +done + +echo +echo "total on disk: $(du -sh "$DATA" | cut -f1)" diff --git a/scripts/smoke_train.sh b/scripts/smoke_train.sh new file mode 100644 index 0000000..88986c6 --- /dev/null +++ b/scripts/smoke_train.sh @@ -0,0 +1,74 @@ +#!/usr/bin/env bash +# SUM Parts - pipeline smoke test +# +# Goal is NOT accuracy. Goal is proving the whole chain survives: +# ply load -> grid subsample -> cuda ops -> forward -> backward -> val +# Uses the lightest model (pointnet) and 1 epoch. Expect garbage mIoU. +# +# Usage: +# bash smoke_train.sh # pointnet, 1 epoch +# bash smoke_train.sh pointnext-xl 2 # other cfg / epoch count +set -euo pipefail + +CONDA_ROOT="$HOME/miniconda3" +ENV_NAME="sumparts" +REPO="$HOME/sum-parts/semantic_segmentation/PointNeXt_bundle" + +CFG="${1:-pointnet}" +EPOCHS="${2:-1}" + +source "$CONDA_ROOT/etc/profile.d/conda.sh" +conda activate "$ENV_NAME" + +export CUDA_HOME="$CONDA_PREFIX" +export PATH="$CUDA_HOME/bin:$PATH" + +# main.py imports wandb at module scope, so the package must exist even with +# wandb.use_wandb=False. Disabled mode keeps it from asking for a login. +export WANDB_MODE=disabled +export WANDB_SILENT=true + +cd "$REPO/examples/segmentation" + +# Two constraints squeeze BS from both sides, and both fail confusingly: +# +# too large -> the loader drops the last partial batch, so a single tile with +# the cfg's batch_size (6 for pointnet) yields ZERO batches. The confusion +# matrix is then never populated and cm.all_metrics() dies on +# AttributeError: 'int' object has no attribute 'diag' +# +# too small -> BatchNorm1d refuses a batch of 1 while training: +# ValueError: Expected more than 1 value per channel when training, +# got input size torch.Size([1, 512]) +# +# So: BS=2 (the BatchNorm floor), and LOOP repeats the tile enough times to fill +# whole batches. +LOOP="${3:-4}" +BS="${BS:-2}" + +LOG="${LOG:-/tmp/sumparts_smoke.log}" + +echo "=== smoke train: cfg=$CFG epochs=$EPOCHS loop=$LOOP bs=$BS ===" +echo "log: $LOG" + +# Capture both streams to a file. Tracebacks were getting lost when this ran +# under a wrapper that only kept stdout, which made crashes look silent. +set +e +python -u main.py \ + --cfg "../../cfgs/sumv2_triangle/${CFG}.yaml" \ + mode=train \ + epochs="$EPOCHS" \ + wandb.use_wandb=False \ + dataset.train.loop="$LOOP" \ + batch_size="$BS" \ + val_batch_size=1 \ + val_freq=1 \ + > "$LOG" 2>&1 +rc=$? +set -e + +echo "=== exit=$rc | last 30 lines ===" +tail -30 "$LOG" + +[ "$rc" -eq 0 ] && echo "SMOKE DONE" || echo "SMOKE FAILED (rc=$rc)" +exit "$rc" diff --git a/scripts/stop_training.sh b/scripts/stop_training.sh new file mode 100644 index 0000000..155d151 --- /dev/null +++ b/scripts/stop_training.sh @@ -0,0 +1,32 @@ +#!/usr/bin/env bash +# SUM Parts - stop the unattended training run +# +# Kills the watchdog first so it does not treat the dying trainer as a crash +# and immediately resume it. +set -uo pipefail + +if ! pgrep -f train_watchdog.sh > /dev/null && ! pgrep -f "main.py" > /dev/null; then + echo "nothing running" + exit 0 +fi + +echo "stopping watchdog:" +pgrep -af train_watchdog.sh || true +pkill -f train_watchdog.sh || true +sleep 2 + +echo "stopping trainer:" +pgrep -af "main.py" || true +pkill -f "examples/segmentation/main.py" || true +sleep 3 + +if pgrep -f "main.py" > /dev/null; then + echo "still alive, sending SIGKILL" + pkill -9 -f "examples/segmentation/main.py" || true +fi + +echo +echo "remaining:" +pgrep -af "train_watchdog.sh|main.py" || echo " clean" +echo +echo "checkpoints are kept -- relaunch resumes from the latest one." diff --git a/scripts/sweep_voxel_max.sh b/scripts/sweep_voxel_max.sh new file mode 100644 index 0000000..cdf81f5 --- /dev/null +++ b/scripts/sweep_voxel_max.sh @@ -0,0 +1,54 @@ +#!/usr/bin/env bash +# SUM Parts - find the largest voxel_max that still fits in VRAM +# +# Why this matters: on WSL2 the NVIDIA driver spills past VRAM into host RAM +# instead of raising OOM, so an oversized config still "works" -- at 47 s/iter +# instead of 0.43. Dropping voxel_max until the peak fits gave a 108x speedup +# on pointnext-xl. But voxel_max is also the cfg value the paper trained with +# (64000), so anything lower is a deviation worth naming. This sweep finds how +# close to 64000 the card can actually get. +set -uo pipefail + +CONDA_ROOT="$HOME/miniconda3" +SEG="$HOME/sum-parts/semantic_segmentation/PointNeXt_bundle/examples/segmentation" +SCRIPTS="/mnt/d/MYCLAUDE_PROJECT/sum-parts-test/scripts" + +CFG="${CFG:-pointnext-xl}" +ITERS="${ITERS:-5}" +VALUES="${*:-24000 32000 40000 48000 64000}" + +source "$CONDA_ROOT/etc/profile.d/conda.sh" +conda activate sumparts +export WANDB_MODE=disabled WANDB_SILENT=true CUDA_HOME="$CONDA_PREFIX" + +cd "$SEG" + +echo "sweeping $CFG over voxel_max: $VALUES" +echo +printf '%-10s %-12s %-11s %-10s %s\n' "voxel_max" "pts/batch" "peak VRAM" "s/iter" "fits?" +printf -- '---------------------------------------------------------\n' + +for vm in $VALUES; do + log="/tmp/sweep_${CFG}_${vm}.log" + python -u "$SCRIPTS/bench_models.py" \ + --iters "$ITERS" --voxel-max "$vm" --cfgs "$CFG" > "$log" 2>&1 + rc=$? + if [ $rc -ne 0 ]; then + printf '%-10s %s\n' "$vm" "FAILED (rc=$rc, see $log)" + continue + fi + line=$(grep -E "^${CFG} +[0-9]" "$log" | tail -1) + if [ -z "$line" ]; then + printf '%-10s %s\n' "$vm" "no result (see $log)" + continue + fi + # cfg params bs pts peak s/iter fits... + pts=$(echo "$line" | awk '{print $4}') + peak=$(echo "$line" | awk '{print $5}') + sec=$(echo "$line" | awk '{print $6}') + fits=$(echo "$line" | cut -d' ' -f7- | sed 's/^ *//') + printf '%-10s %-12s %-11s %-10s %s\n' "$vm" "$pts" "$peak" "$sec" "$fits" +done + +echo +echo "SWEEP DONE" diff --git a/scripts/train_full.sh b/scripts/train_full.sh new file mode 100644 index 0000000..c091485 --- /dev/null +++ b/scripts/train_full.sh @@ -0,0 +1,79 @@ +#!/usr/bin/env bash +# SUM Parts - full training run on the real dataset +# +# Usage: +# bash train_full.sh # pointnet, paper settings +# bash train_full.sh pointnext-xl # auto-picks a voxel_max that fits +# CFG_VOXEL_MAX=40000 bash train_full.sh pointnext-xl +# EPOCHS=20 bash train_full.sh pointnet +# +# Measured on this box (RTX 3060 12GB), 24 train tiles, loop 30, batch_size 2 +# => 360 iter/epoch: +# +# pointnet voxel_max 64000 (paper) 6.01G 0.291 s/iter ~2.9 h/100ep +# pointnet++msg voxel_max 64000 (paper) 4.16G 0.675 s/iter ~6.8 h/100ep +# pointvector-xl voxel_max 24000 6.46G 0.402 s/iter ~4.0 h/100ep +# pointnext-xl voxel_max 32000 8.03G 0.635 s/iter ~6.4 h/100ep +# +# The XL rows are NOT the paper configuration. At voxel_max 64000 they need +# ~15-16 GB, and on WSL2 the driver spills past VRAM into host RAM instead of +# raising OOM -- the run completes at 47 s/iter, i.e. ~20 days for 100 epochs. +# Numbers from a reduced voxel_max are not comparable to the published ones. +set -euo pipefail + +CONDA_ROOT="$HOME/miniconda3" +ENV_NAME="sumparts" +SEG="$HOME/sum-parts/semantic_segmentation/PointNeXt_bundle/examples/segmentation" +DATA="$HOME/sum-parts/data/face_labeling/texsp_pcl" # triangle track, 13 classes + +CFG="${1:-pointnet}" +EPOCHS="${EPOCHS:-100}" +VAL_FREQ="${VAL_FREQ:-5}" # cfg default is 1; every epoch costs 8 val tiles +LOG="${LOG:-/tmp/sumparts_train_${CFG}.log}" + +# largest voxel_max measured to stay inside 12 GB for each model +if [ -n "${CFG_VOXEL_MAX:-}" ]; then + VOXEL_MAX="$CFG_VOXEL_MAX" +else + case "$CFG" in + pointnext-xl) VOXEL_MAX=32000 ;; + pointvector-xl) VOXEL_MAX=24000 ;; + *) VOXEL_MAX=64000 ;; # paper setting; small models fit + esac +fi + +source "$CONDA_ROOT/etc/profile.d/conda.sh" +conda activate "$ENV_NAME" +export WANDB_MODE=disabled WANDB_SILENT=true CUDA_HOME="$CONDA_PREFIX" + +[ -d "$DATA/train" ] || { echo "error: $DATA/train missing -- run download_data.sh and prepare_full_split.sh" >&2; exit 1; } + +echo "=== full training ===" +echo " cfg : $CFG" +echo " data_root : $DATA" +echo " train/val/test : $(find -L "$DATA/train" -name '*.ply' | wc -l)/$(find -L "$DATA/val" -name '*.ply' | wc -l)/$(find -L "$DATA/test" -name '*.ply' | wc -l)" +echo " epochs : $EPOCHS" +echo " val_freq : $VAL_FREQ" +echo " voxel_max : $VOXEL_MAX$([ "$VOXEL_MAX" -lt 64000 ] && echo ' (reduced from paper 64000 to fit VRAM)')" +echo " log : $LOG" +echo + +cd "$SEG" + +set +e +python -u main.py \ + --cfg "../../cfgs/sumv2_triangle/${CFG}.yaml" \ + mode=train \ + dataset.common.data_root="$DATA" \ + dataset.train.voxel_max="$VOXEL_MAX" \ + epochs="$EPOCHS" \ + val_freq="$VAL_FREQ" \ + wandb.use_wandb=False \ + > "$LOG" 2>&1 +rc=$? +set -e + +echo "=== exit=$rc | last 25 lines ===" +tail -25 "$LOG" +[ "$rc" -eq 0 ] && echo "TRAIN DONE" || echo "TRAIN FAILED (rc=$rc)" +exit "$rc" diff --git a/scripts/train_watchdog.sh b/scripts/train_watchdog.sh new file mode 100644 index 0000000..8449064 --- /dev/null +++ b/scripts/train_watchdog.sh @@ -0,0 +1,215 @@ +#!/usr/bin/env bash +# SUM Parts - unattended training with automatic resume +# +# The WSL VM has died twice under load on this box, so an overnight run needs to +# survive the process disappearing. main.py checkpoints every epoch and supports +# +# mode=resume --pretrained_path /checkpoint/_ckpt_latest.pth +# +# When pretrained_path sits inside a checkpoint/ directory, resume_exp_directory +# reuses the SAME run folder and resume_checkpoint restores epoch, optimizer and +# scheduler -- so a restart continues rather than starting over. +# +# This wrapper runs the training, and if it exits non-zero it waits, finds the +# newest checkpoint, and resumes. Up to MAX_RETRIES times. It stops retrying if +# the run reached the final epoch, or if two consecutive attempts fail without +# a new checkpoint being written (that means it is failing before it can train, +# so retrying is pointless). +# +# Usage: +# bash train_watchdog.sh [cfg] +# MAX_RETRIES=10 EPOCHS=100 bash train_watchdog.sh pointnet +# +# For an unattended launch use launch_overnight.sh, which detaches this from +# the terminal. +set -uo pipefail + +CONDA_ROOT="$HOME/miniconda3" +ENV_NAME="sumparts" +SEG="$HOME/sum-parts/semantic_segmentation/PointNeXt_bundle/examples/segmentation" +DATA="$HOME/sum-parts/data/face_labeling/texsp_pcl" + +CFG="${1:-pointnet}" +EPOCHS="${EPOCHS:-100}" +VAL_FREQ="${VAL_FREQ:-5}" +MAX_RETRIES="${MAX_RETRIES:-8}" +RETRY_WAIT="${RETRY_WAIT:-60}" + +RUNROOT="$HOME/sum-parts/runs" +STAMP="${STAMP:-$(date +%Y%m%d-%H%M%S)}" +WORKDIR="$RUNROOT/${CFG}_${STAMP}" +LOG="$WORKDIR/watchdog.log" +TRAINLOG="$WORKDIR/train.log" +STATUS="$WORKDIR/status.txt" + +if [ -n "${CFG_VOXEL_MAX:-}" ]; then + VOXEL_MAX="$CFG_VOXEL_MAX" +else + case "$CFG" in + pointnext-xl) VOXEL_MAX=32000 ;; + pointvector-xl) VOXEL_MAX=24000 ;; + *) VOXEL_MAX=64000 ;; + esac +fi + +mkdir -p "$WORKDIR" + +say() { echo "[$(date '+%F %T')] $*" | tee -a "$LOG"; } + +write_status() { + { + echo "cfg : $CFG" + echo "workdir : $WORKDIR" + echo "state : $1" + echo "attempt : ${attempt:-0}/$MAX_RETRIES" + echo "epochs : $EPOCHS (val_freq $VAL_FREQ)" + echo "voxel_max : $VOXEL_MAX" + echo "updated : $(date '+%F %T')" + echo "last epoch : $(last_epoch)" + echo "train log : $TRAINLOG" + } > "$STATUS" +} + +# newest *_ckpt_latest.pth under the segmentation log tree +find_ckpt() { + find "$SEG/log/sumv2_triangle" -name '*_ckpt_latest.pth' -newermt "@$START_EPOCH_TS" \ + -printf '%T@ %p\n' 2>/dev/null | sort -rn | head -1 | cut -d' ' -f2- +} + +last_epoch() { + grep -oE 'Epoch [0-9]+ LR' "$TRAINLOG" 2>/dev/null | tail -1 | grep -oE '[0-9]+' || echo "-" +} + +source "$CONDA_ROOT/etc/profile.d/conda.sh" +conda activate "$ENV_NAME" +export WANDB_MODE=disabled WANDB_SILENT=true CUDA_HOME="$CONDA_PREFIX" + +# Why expandable_segments: cfgs/sumv2_triangle/default.yaml validates with +# val: { voxel_max: null } +# i.e. the whole tile (~470k points) in one go, while training is capped at +# voxel_max. Those oversized transient allocations fragment the caching +# allocator, its pool grows past 12 GB, and on WSL2 the driver quietly spills +# the excess into host RAM instead of raising OOM. +# +# Observed on the first run: epochs 1-10 took ~103 s each, then from epoch 11 +# (right after the epoch-10 validation) every epoch took ~1100 s -- a 10x +# slowdown with the GPU at 100% util, full clocks, no thermal throttle, and +# 11.7 GB dedicated + 19.0 GB shared. Killing the process dropped the card back +# to 1.2 GB, so it was the trainer's own pool, not other apps. +# +# NOTE: expandable_segments is NOT available here -- it landed in torch 2.1 and +# this env is on 2.0.1, where it aborts at startup with +# RuntimeError: Unrecognized CachingAllocator option: expandable_segments +# garbage_collection_threshold and max_split_size_mb do exist in 2.0. +export PYTORCH_CUDA_ALLOC_CONF="${PYTORCH_CUDA_ALLOC_CONF:-garbage_collection_threshold:0.7,max_split_size_mb:128}" + +# Validation runs with voxel_max: null, i.e. a whole ~470k-point tile in one +# forward pass, which peaks far above the capped training step. Bounding it to +# the training value keeps the allocator pool inside VRAM. +# +# This is a real deviation, stated plainly: val_miou is then computed on +# subsampled tiles, so it is only a model-selection signal. It does not affect +# the reported test numbers -- test() slides over the full tile regardless. +VAL_VOXEL_MAX="${VAL_VOXEL_MAX:-64000}" + +if [ ! -d "$DATA/train" ]; then + say "FATAL: $DATA/train missing. run download_data.sh + prepare_full_split.sh" + write_status "failed-no-data" + exit 1 +fi + +# RESUME_CKPT lets a relaunch continue an earlier run instead of starting over. +# Needed because find_ckpt only considers checkpoints written after this +# watchdog started, so a fresh watchdog would otherwise ignore existing ones. +RESUME_CKPT="${RESUME_CKPT:-}" +if [ -n "$RESUME_CKPT" ] && [ ! -f "$RESUME_CKPT" ]; then + say "FATAL: RESUME_CKPT does not exist: $RESUME_CKPT" + write_status "failed-bad-resume-ckpt" + exit 1 +fi + +START_EPOCH_TS=$(date +%s) +cd "$SEG" + +say "cfg=$CFG epochs=$EPOCHS val_freq=$VAL_FREQ voxel_max=$VOXEL_MAX" +say "data=$DATA train/val/test = $(find -L "$DATA/train" -name '*.ply' | wc -l)/$(find -L "$DATA/val" -name '*.ply' | wc -l)/$(find -L "$DATA/test" -name '*.ply' | wc -l)" +say "workdir=$WORKDIR" + +attempt=0 +prev_ckpt="" + +while [ "$attempt" -le "$MAX_RETRIES" ]; do + ckpt=$(find_ckpt) + # on the very first attempt, an explicitly supplied checkpoint wins + if [ "$attempt" -eq 0 ] && [ -n "$RESUME_CKPT" ]; then + ckpt="$RESUME_CKPT" + fi + + if [ -z "$ckpt" ]; then + say "attempt $attempt: fresh start" + write_status "running (fresh)" + python -u main.py \ + --cfg "../../cfgs/sumv2_triangle/${CFG}.yaml" \ + mode=train \ + dataset.common.data_root="$DATA" \ + dataset.train.voxel_max="$VOXEL_MAX" \ + dataset.val.voxel_max="$VAL_VOXEL_MAX" \ + epochs="$EPOCHS" \ + val_freq="$VAL_FREQ" \ + wandb.use_wandb=False \ + >> "$TRAINLOG" 2>&1 + rc=$? + else + say "attempt $attempt: resuming from $(basename "$ckpt")" + write_status "running (resumed)" + python -u main.py \ + --cfg "../../cfgs/sumv2_triangle/${CFG}.yaml" \ + mode=resume \ + --pretrained_path "$ckpt" \ + dataset.common.data_root="$DATA" \ + dataset.train.voxel_max="$VOXEL_MAX" \ + dataset.val.voxel_max="$VAL_VOXEL_MAX" \ + epochs="$EPOCHS" \ + val_freq="$VAL_FREQ" \ + wandb.use_wandb=False \ + >> "$TRAINLOG" 2>&1 + rc=$? + fi + + if [ "$rc" -eq 0 ]; then + say "training finished cleanly (rc=0) at epoch $(last_epoch)" + write_status "done" + say "last 20 lines:" + tail -20 "$TRAINLOG" | tee -a "$LOG" + exit 0 + fi + + say "attempt $attempt exited rc=$rc at epoch $(last_epoch)" + tail -15 "$TRAINLOG" | tee -a "$LOG" + + new_ckpt=$(find_ckpt) + if [ "$attempt" -gt 0 ] && [ "$new_ckpt" = "$prev_ckpt" ]; then + say "no new checkpoint since the last attempt -- failing before training, giving up" + write_status "failed-no-progress" + exit "$rc" + fi + prev_ckpt="$new_ckpt" + + attempt=$((attempt + 1)) + if [ "$attempt" -gt "$MAX_RETRIES" ]; then + say "exhausted $MAX_RETRIES retries" + write_status "failed-retries-exhausted" + exit "$rc" + fi + + write_status "waiting ${RETRY_WAIT}s before retry" + say "waiting ${RETRY_WAIT}s, then retrying" + sleep "$RETRY_WAIT" + + # if the VM itself restarted, the GPU may take a moment to be usable again + for i in $(seq 1 10); do + if nvidia-smi -L >/dev/null 2>&1; then break; fi + say " GPU not ready yet ($i/10)" + sleep 15 + done +done diff --git a/scripts/val_protocol_ab.sh b/scripts/val_protocol_ab.sh new file mode 100644 index 0000000..b4a9841 --- /dev/null +++ b/scripts/val_protocol_ab.sh @@ -0,0 +1,64 @@ +#!/usr/bin/env bash +# SUM Parts - A/B the validation protocol on the same checkpoint +# +# Training reported val_miou 17.19 @E90, but a standalone mode=val gave 4.20. +# The only difference is dataset.val.voxel_max: +# +# A) 64000 what training used after the memory mitigation (§3-2-1), +# i.e. validation on a 64k-point crop, matching the training +# input size +# B) null the cfg's own default: the whole ~700k-point tile in one +# forward pass +# +# PointNet pools a single global feature over whatever it is given, so input +# size is not a neutral knob -- this measures how much it moved the number. +set -uo pipefail + +CONDA_ROOT="$HOME/miniconda3" +SEG="$HOME/sum-parts/semantic_segmentation/PointNeXt_bundle/examples/segmentation" +DATA="$HOME/sum-parts/data/face_labeling/texsp_pcl" +OUT="$HOME/sum-parts/runs/val_ab" + +source "$CONDA_ROOT/etc/profile.d/conda.sh" +conda activate sumparts +export WANDB_MODE=disabled WANDB_SILENT=true CUDA_HOME="$CONDA_PREFIX" +export PYTORCH_CUDA_ALLOC_CONF="garbage_collection_threshold:0.7,max_split_size_mb:128" + +mkdir -p "$OUT" +cd "$SEG" + +CKPT="${CKPT:-$(find "$SEG/log/sumv2_triangle" -name '*_ckpt_best.pth' -printf '%T@ %p\n' \ + | sort -rn | head -1 | cut -d' ' -f2-)}" +echo "checkpoint: $(basename "$CKPT")" +echo + +run_val() { + local tag="$1"; shift + local log="$OUT/val_${tag}.log" + python -u main.py \ + --cfg ../../cfgs/sumv2_triangle/pointnet.yaml \ + mode=val \ + --pretrained_path "$CKPT" \ + dataset.common.data_root="$DATA" \ + wandb.use_wandb=False \ + val_batch_size=1 \ + "$@" \ + > "$log" 2>&1 + local rc=$? + if [ $rc -ne 0 ]; then + echo " [$tag] FAILED rc=$rc" + tail -6 "$log" + return 1 + fi + grep -aA1 'Best ckpt' "$log" | tail -2 | sed "s/^/ [$tag] /" +} + +echo "=== A: voxel_max 64000 (what training measured) ===" +run_val capped dataset.val.voxel_max=64000 +echo + +echo "=== B: voxel_max null (cfg default, whole tile) ===" +run_val full +echo + +echo "logs in $OUT" diff --git a/scripts/verify_archives.sh b/scripts/verify_archives.sh new file mode 100644 index 0000000..c79cbe5 --- /dev/null +++ b/scripts/verify_archives.sh @@ -0,0 +1,55 @@ +#!/usr/bin/env bash +# SUM Parts - check the downloaded archives survived +# +# The WSL VM has now died twice under load, and the first time it left pip's +# in-flight files as 0-byte stubs. A download interrupted the same way leaves a +# truncated zip, so verify before trusting anything that was in flight. +set -euo pipefail + +source "$HOME/miniconda3/etc/profile.d/conda.sh" +conda activate sumparts + +python - <<'PY' +import zipfile +from pathlib import Path + +d = Path.home() / "sum-parts/data/_archives" +expected = {"demo.zip": 0.12, "mesh.zip": 0.52, "pcl.zip": 4.66} # GB, from the HF listing + +if not d.exists(): + print(f"{d} missing") + raise SystemExit(1) + +bad = [] +for f in sorted(d.glob("*.zip")): + gb = f.stat().st_size / 1e9 + want = expected.get(f.name) + line = f"{f.name:<12} {gb:>6.2f} GB" + if want: + line += f" (expected ~{want:.2f} GB)" + if gb < want * 0.97: + line += " TRUNCATED" + bad.append(f.name) + print(line) + continue + try: + z = zipfile.ZipFile(f) + broken = z.testzip() + if broken: + line += f" CORRUPT at {broken}" + bad.append(f.name) + else: + line += f" OK, {len(z.namelist())} entries" + except Exception as e: + line += f" UNREADABLE {type(e).__name__}: {e}" + bad.append(f.name) + print(line) + +for name in expected: + if not (d / name).exists(): + print(f"{name:<12} MISSING") + +print() +print("all good" if not bad else f"re-download: {', '.join(bad)}") +raise SystemExit(1 if bad else 0) +PY diff --git a/scripts/verify_env.py b/scripts/verify_env.py new file mode 100644 index 0000000..1044862 --- /dev/null +++ b/scripts/verify_env.py @@ -0,0 +1,100 @@ +#!/usr/bin/env python3 +"""Check that every compiled extension and import the sumv2 training path needs +is actually present. + +Extension import names do not match their directory names, which is an easy way +to waste an hour chasing a build that already succeeded: + + openpoints/cpp/pointnet2_batch -> pointnet2_batch_cuda + openpoints/cpp/pointops -> pointops_cuda + openpoints/cpp/chamfer_dist -> chamfer + openpoints/cpp/emd -> emd_cuda (package name: emd_ext) + openpoints/cpp/subsampling -> openpoints.cpp.subsampling.grid_subsampling + +Run from PointNeXt_bundle/ (or anywhere, if openpoints is importable). +""" +import os +import sys +from pathlib import Path + + +def add_openpoints_to_path() -> Path | None: + """Put PointNeXt_bundle on sys.path. + + main.py does this itself with a hardcoded '../../', but this script lives + outside the repo, so walk up from cwd (then from the default clone path) + until a directory containing openpoints/ turns up. + """ + candidates = [Path.cwd(), *Path.cwd().parents, + Path.home() / "sum-parts/semantic_segmentation/PointNeXt_bundle"] + for c in candidates: + if (c / "openpoints" / "__init__.py").exists(): + sys.path.insert(0, str(c)) + return c + return None + + +MODULES = [ + "torch", + "numpy", + "pointnet2_batch_cuda", + "pointops_cuda", + "chamfer", + "emd_cuda", + "torch_scatter", + "plyfile", + "wandb", + "trimesh", +] + +FROM_IMPORTS = [ + ("openpoints.cpp.subsampling", "grid_subsampling"), + ("openpoints.models", "build_model_from_cfg"), + ("openpoints.dataset", "build_dataloader_from_cfg"), +] + + +def main() -> int: + ok = True + + root = add_openpoints_to_path() + print(f"openpoints root: {root or 'NOT FOUND'}") + if root is None: + ok = False + + try: + import torch + print(f"torch {torch.__version__} | cuda {torch.version.cuda} | " + f"available {torch.cuda.is_available()}") + if torch.cuda.is_available(): + print(f"device: {torch.cuda.get_device_name(0)}") + except Exception as e: # noqa: BLE001 + print(f"torch import failed: {e}") + return 1 + + print() + for name in MODULES: + try: + __import__(name) + print(f"{name:32s} OK") + except Exception as e: # noqa: BLE001 + ok = False + print(f"{name:32s} FAIL {type(e).__name__}: {e}") + + print() + for mod, attr in FROM_IMPORTS: + try: + m = __import__(mod, fromlist=[attr]) + getattr(m, attr) + print(f"{mod + '.' + attr:32s} OK") + except Exception as e: # noqa: BLE001 + ok = False + print(f"{mod + '.' + attr:32s} FAIL {type(e).__name__}: {e}") + + print() + print("ALL OK" if ok else "SOME CHECKS FAILED") + return 0 if ok else 1 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/scripts/verify_speed.sh b/scripts/verify_speed.sh new file mode 100644 index 0000000..3daea23 --- /dev/null +++ b/scripts/verify_speed.sh @@ -0,0 +1,55 @@ +#!/usr/bin/env bash +# SUM Parts - did the run come back to healthy speed? +# +# Healthy for pointnet at voxel_max 64000 on this box is ~0.29 s/it (benchmark) +# and ~103 s/epoch. The degraded state was ~3.1 s/it and ~1100 s/epoch, caused +# by the caching allocator pool overflowing VRAM into host RAM. +set -uo pipefail + +RUNROOT="$HOME/sum-parts/runs" +D="${1:-$(cat "$RUNROOT/.latest" 2>/dev/null)}" +LOG="$D/train.log" + +echo "run: $D" +[ -f "$LOG" ] || { echo " no train.log yet"; exit 1; } + +echo +echo "=== resume confirmation ===" +grep -aE 'Resume|Successful Loading|start_epoch' "$LOG" | tail -3 || echo " (none found)" + +echo +echo "=== current epoch ===" +grep -aoE 'Train Epoch \[[0-9]+/[0-9]+\]' "$LOG" | tail -1 || echo " not started" + +echo +echo "=== recent iteration rates ===" +grep -aoE '[0-9.]+(s/it|it/s)' "$LOG" | tail -12 | tr '\n' ' ' +echo + +echo +echo "=== GPU ===" +nvidia-smi --query-gpu=memory.used,memory.total,utilization.gpu,power.draw --format=csv,noheader + +echo +echo "=== verdict ===" +# tqdm prints either "N.NNs/it" or "N.NNit/s" depending on which side of 1 the +# rate falls, so take whichever form appeared LAST and normalise to s/it. +# (Grepping only for 's/it' picks up a stale warm-up reading and misjudges a +# run that has since recovered.) +last=$(grep -aoE '[0-9.]+(s/it|it/s)' "$LOG" | tail -1) +if [ -z "$last" ]; then + echo " no rate readings yet" + exit 0 +fi + +sec=$(awk -v r="$last" 'BEGIN { + if (r ~ /it\/s$/) { sub(/it\/s$/, "", r); printf "%.4f", (r > 0 ? 1.0 / r : 0) } + else { sub(/s\/it$/, "", r); printf "%.4f", r } +}') + +echo " last reading: $last -> ${sec} s/it" +awk -v v="$sec" 'BEGIN { + if (v < 0.8) print " HEALTHY (benchmark is 0.291 s/it)"; + else if (v < 1.5) print " MARGINAL"; + else print " DEGRADED -- allocator likely spilling to host RAM"; +}'