diff --git a/.gitignore b/.gitignore index 395c7f9..008684b 100644 --- a/.gitignore +++ b/.gitignore @@ -5,6 +5,13 @@ output/ *.pth *.zip +# Paper PDFs — 88 MB, and every one is a public arXiv download. +# scripts/convert_papers.sh regenerates the markdown from them. +# The converted .md and extracted figures ARE tracked: they carry the Vision +# annotations, which took a GPU pass to produce and are not reproducible +# byte-for-byte. +docs/papers/*.pdf + # Source-tree backups left by the patch scripts *.orig *.bak diff --git a/NEXT.md b/NEXT.md new file mode 100644 index 0000000..d238b0a --- /dev/null +++ b/NEXT.md @@ -0,0 +1,159 @@ +# 다음 세션 시작 프롬프트 + +목표가 정해졌다. 이 문서를 읽고 아래 프롬프트로 이어간다. + +--- + +## 단기 목표 + +> **bare earth 모델과 나머지를 분리한다. OBJ(mesh) 형태로 분리한 뒤, +> 그 나머지 모델을 다시 분류한다. 지면이 없어지면 분류가 더 쉬울 것이다.** + +이 순서는 타당하다. 근거: + +- 지면이 전체 포인트의 24~40%를 차지한다 — 제거하면 남은 문제가 그만큼 작아진다 +- 지면을 걷어내면 건물·수목·차량이 **공간적으로 분리된 덩어리**가 된다. + 지면이 있으면 전부 하나로 이어져 있어 연결 성분 분석이 안 된다 +- 2단계 분류는 실패가 국소화된다. 1단계에서 지면을 놓쳐도 2단계가 무너지지 않는다 + +--- + +## 프롬프트 + +```` +docs/sum-parts-explained.html 의 "Bare Earth" 탭과 STATUS.md 를 먼저 읽어라. +그 다음 이어서 작업한다. + +## 목표 + +bare earth(지면)와 나머지를 분리해서 각각 OBJ 메시로 내보낸다. +그 다음 나머지를 건물/수목/차량으로 다시 분류한다. + +## 현재 가진 것 + +- PointVector 학습 완료 (3060, voxel_max 24000, 100 epoch) + SUM val 기준 지면 precision 95.24% / recall 66.10% +- 서산 명천 BlockYBA 타일 1장 변환·추론 완료 +- 클래스별 분리 산출물 D:\AI_Test\sum-part\{ground,building,tree,vehicle,unseen}\ + 단 포인트 클라우드(.ply)이고 메시가 아니다 + +## 해야 할 일 — 순서대로 + +### 1. 이진 학습이 실제로 이득인지 확인 + +가설: 학습부터 2클래스(지면/비지면)로 하면 경계 정확도가 오른다. +근거: 13클래스 51.83% → 4클래스 통합 72.14%. 합칠수록 올랐다. + +구현: 데이터로더에서 라벨 remap. + terrain(1) → 1, 나머지 → 2, unclassified(0) → 0(ignore) + num_classes: 3, 나머지 cfg 그대로 + +측정: 지면 precision / recall을 현재 95.24 / 66.10과 비교. +약 4시간. 3060으로 가능하다. + +### 2. CSF를 서산 타일에 단독 적용 + +Cloth Simulation Filter. 학습 불필요, 파라미터 몇 개. +비탈면(절토·성토)에 강한 것이 알려져 있고, SUM(헬싱키 평지)으로는 +비탈면을 배울 수 없으므로 이 부분은 기하 기법이 필요하다. + +파이썬 구현: CSF (pip install cloth-simulation-filter) 또는 PDAL filters.csf + +측정: 신경망 결과와 어디가 다른지. 특히 경사면에서. + +### 3. 조합 + +신경망(의미) → 나무·차량·건물 제거 → CSF(기하) → 비탈면 보존 +둘의 합의/불일치 지점을 본다. + +### 4. 포인트 → 메시 복원 ← 여기가 미구현이고 핵심 + +현재 변환기(scripts/mesh_to_ply.py)는 면적가중 샘플링만 하고 +**각 포인트가 어느 face에서 왔는지 기록하지 않는다.** +포인트 예측을 원본 메시로 되돌리려면 이 역매핑이 필요하다. + +해야 할 것: + a. mesh_to_ply.py에 face index 보존 추가 (PLY에 face_idx 속성) + b. 포인트 예측 → face 다수결 → face별 클래스 + c. 클래스별로 face를 분리해 OBJ 저장 + +원칙: 원본 vertex는 불가침. 분리는 재생성이 아니라 face 분할이다. +구멍 메우기는 새 vertex 덧대기만 허용된다. + +검증: 분할된 face 수의 합 == 원본 face 수. 누락도 중복도 없을 것. + +### 5. 나머지 재분류 + +지면 제거 후 남은 메시에서 건물/수목/차량 분리. +지면이 없으면 연결 성분(connected component)으로 물체가 자연히 나뉜다. +이 단계는 4번이 끝나야 의미가 있다. + +## 주의사항 + +- WSL2에서 VRAM 초과는 OOM을 안 낸다. 호스트 RAM으로 흘려서 + 25~100배 느리게 조용히 완주한다. peak VRAM과 전력(W)으로 판정해라. +- 파일을 만들었으면 scripts/verify_outputs.sh 로 읽히는지 확인해라. + "만들었다"고 보고하기 전에 파서로 검증한다. +- 포인트 클라우드는 PLY로 내라. 면 없는 OBJ는 뷰어가 아무것도 안 보여준다. +- 각 단계 게이트를 통과 못 하면 다음으로 넘어가지 마라. + +## 판정 기준 + +bare earth의 성패는 precision이다. 지면이라 부른 것에 구조물이 섞이면 +지형이 왜곡된다. 반대로 지면을 놓쳐 생긴 구멍은 보간으로 메운다. + +현재 기준선: precision 95.24% / recall 66.10% +```` + +--- + +## 참고 — 이 세션에서 확인한 것 + +### PointVector 핵심 + +특징값(스칼라)을 3D 벡터로 확장해서 **방향**을 얻는다. +회전각 α, β 두 개만 MLP로 예측한다 — 회전행렬을 직접 예측하면 +원소끼리 종속이라 최적화가 어렵기 때문이다. + +attention이나 dynamic conv 없이 이방성(anisotropy)을 얻으므로 +**PointNeXt 파라미터의 58%로 더 높은 정확도**가 나온다. + +### 왜 우리 과제에 맞나 + +지면과 벽의 구분은 본질적으로 방향 문제다. +지면은 이웃이 수평으로, 벽은 수직으로 퍼져 있다. +등방적 집계는 이 차이를 뭉갠다. + +측정치가 이를 뒷받침한다 — 비지면이 지면으로 샌 26,217 포인트 중 +**49.90%가 facade_surface**였다. 벽 하단, 지면과 만나는 경계다. + +### 경계 5종의 성질이 다르다 + +| 경계 | 성질 | 난이도 | +|---|---|---| +| 지면–벽 | 법선 급변 | 쉬움 | +| 지면–수목 | 수직 이격 | 중간 | +| 지면–자동차 | 얹힌 물체 | 중간 | +| 지면–지장물 | 얹힌 물체 | 중간 | +| **지면–비탈면** | **둘 다 지면** | **가장 어려움** | + +비탈면만 성질이 반대다 — 나머지는 잘라내야 하고 비탈면은 남겨야 한다. +"수평이면 지면"으로 가면 도로 절·성토를 통째로 날린다. + +### SUM Parts의 한계 + +헬싱키 평지 도시라 **비탈면을 배울 수 없다.** +재학습으로도 안 고쳐진다. 데이터에 없는 개념이다. +→ 비탈면은 CSF 같은 기하 기법으로 보완한다. + +--- + +## 문서 + +| 문서 | 내용 | +|---|---| +| [docs/sum-parts-explained.html](docs/sum-parts-explained.html) | 학습 정리 — 용어, 모델 계보, PointVector 원리, bare earth 전략 | +| [STATUS.md](STATUS.md) | 작업 현황, 실측 수치, 미해결 문제 | +| [docs/pipeline.html](docs/pipeline.html) | 6단계 공정 정의 | +| [docs/papers/md/](docs/papers/md/) | 논문 6편 마크다운 + 그림 86개 (Vision 주석 완료) | +| [SETUP.md](SETUP.md) · [TRAIN.md](TRAIN.md) | 새 머신 구축 | diff --git a/README.md b/README.md index a04de05..3663695 100644 --- a/README.md +++ b/README.md @@ -9,9 +9,11 @@ | 경로 | 내용 | |---|---| +| [NEXT.md](NEXT.md) | **다음 세션 시작점 — 목표와 프롬프트** | | [STATUS.md](STATUS.md) | **현재 상태·결과·미해결 문제 — 이어받을 때 여기부터** | | [SETUP.md](SETUP.md) | 1단계 — 환경 구축 (GPU 불필요) | | [TRAIN.md](TRAIN.md) | 2단계 — 학습 (GPU 필요) | +| [docs/sum-parts-explained.html](docs/sum-parts-explained.html) | 학습 정리 — 용어·모델 계보·PointVector 원리·bare earth 전략 | | [docs/pipeline.html](docs/pipeline.html) | 전체 6단계 공정 정의 (브라우저로 열 것) | | [docs/SUM-Parts-검토노트.md](docs/SUM-Parts-검토노트.md) | 트러블슈팅 16건, 데이터 스키마 실측, 라이선스 | | [scripts/](scripts/) | 환경 구축 · 학습 · 평가 · 변환 스크립트 | diff --git a/docs/papers/ANNOTATE_PROMPT.md b/docs/papers/ANNOTATE_PROMPT.md new file mode 100644 index 0000000..0954dc8 --- /dev/null +++ b/docs/papers/ANNOTATE_PROMPT.md @@ -0,0 +1,177 @@ +# Gemini 이미지 주석 작업 프롬프트 + +`docs/papers/md/`의 논문 마크다운에 들어 있는 그림 86개를 +Vision으로 분석해 각 그림 아래에 한국어 설명을 붙이는 작업. + +아래 블록을 Antigravity에 그대로 넣으면 된다. + +--- + +## 프롬프트 + +```` +# 작업: 논문 마크다운의 그림에 Vision 분석 주석 달기 + +## 대상 + +D:\MYCLAUDE_PROJECT\sum-parts-test\docs\papers\md\ 아래 .md 파일 6개. +각 파일 옆에 같은 이름의 _images\ 폴더가 있고, 거기에 그림 파일이 들어 있다. + +| 파일 | 그림 수 | +|---|---| +| PointNet_1612.00593.md | 24 | +| PointNet++_1706.02413.md | 10 | +| PointNeXt_2206.04670.md | 5 | +| PointVector_2205.10528.md | 8 | +| SUM_2021_dataset.md | 14 | +| SUM-Parts_2503.15300.md | 25 | +| **합계** | **86** | + +## 마크다운의 현재 구조 + +그림은 항상 이 형태로 들어 있다. 앞에 `` 앵커가 붙기도 한다. + +``` +![](PointVector_2205.10528_images/_page_0_Figure_11.jpeg) + +Figure 1. Illustrations of the core operations of the different methods. (a) The +features of each point are calculated separately... (논문 원본 캡션) +``` + +- 이미지 경로는 **.md 파일 기준 상대경로**다 +- 이미지 바로 아래 빈 줄, 그 다음에 **논문 원본 캡션**이 이미 있다 +- 캡션이 없는 그림도 있다 (표 이미지, 장식용 등) + +## 해야 할 일 + +각 이미지를 Vision으로 열어보고, **원본 캡션 아래에** 한국어 설명을 삽입한다. + +### 삽입 형식 + +원본 캡션 다음 빈 줄 뒤에 인용 블록으로 넣는다. + +``` +![](PointVector_2205.10528_images/_page_0_Figure_11.jpeg) + +Figure 1. Illustrations of the core operations of the different methods. (a) The +features of each point are calculated separately... + +> **[그림 해설]** 4개 패널을 가로로 배열한 비교 다이어그램. (a) attention 방식은 +> 고정 커널로 각 점을 따로 계산한 뒤 입력에서 만든 가중치로 이방성을 부여한다. +> (b) 변위 벡터로 커널 패턴에 가까운 점을 골라 집계한다. (c) 점마다 다른 동적 +> 커널을 적용한다. (d) 제안 방식은 특징에서 벡터 표현을 만들고, 벡터의 방향 +> 자체가 이방성을 만든다. 화살표 색으로 등방성(검정)과 이방성(색)을 구분한다. +``` + +**원본 캡션이 없는 그림**은 이미지 바로 아래에 넣는다. + +### 규칙 + +1. **`> **[그림 해설]**` 마커로 시작한다.** 이 마커로 나중에 검색·제거가 가능해야 한다. +2. **원본 텍스트는 절대 수정하지 않는다.** 캡션도, 본문도, 앵커 ``도 그대로 둔다. + 오직 삽입만 한다. +3. **이미 `[그림 해설]`이 붙은 그림은 건너뛴다.** 재실행해도 중복되지 않아야 한다. +4. 한국어로 쓴다. 기술 용어(attention, MLP, IoU, voxel 등)는 원어 그대로 둔다. +5. 인용 블록 전체를 `> `로 시작하는 여러 줄로 쓴다. + +## 무엇을 쓸 것인가 + +**"그림이 무엇을 보여주는지"를 쓴다. 캡션을 번역하지 않는다.** +원본 캡션은 바로 위에 이미 있으므로, 번역만 하면 아무 가치가 없다. + +그림 유형별로 다르게 접근한다. + +### 아키텍처 다이어그램 (가장 중요) + +- 블록이 몇 개이고 무엇이 무엇으로 흘러가는가 +- 화살표 방향, 분기와 합류 지점 +- 텐서 shape 표기가 있으면 그대로 옮긴다 (예: `N×3 → N×64 → 1024`) +- 색이나 선 스타일이 구분하는 것 +- **논문의 주장과 직결되는 지점을 짚는다** (예: "여기가 skip connection이고, 이게 + PointNet++ 대비 추가된 부분이다") + +### 성능 차트 (막대·꺾은선·산점도) + +- 축이 무엇인가 (단위 포함) +- 비교 대상이 몇 개이고 각각 무엇인가 +- **읽을 수 있는 수치는 옮긴다** — 나중에 검색된다 +- 추세와 교차점 + +### 정성 결과 (세그멘테이션 시각화 등) + +- 몇 개 열/행이고 각각 무엇인가 (입력 / GT / 각 방법의 결과) +- 색상 범례가 있으면 클래스-색 대응을 옮긴다 +- 방법 간 눈에 띄는 차이가 어디서 나타나는가 + +### 표를 이미지로 캡처한 것 + +- **표 내용을 마크다운 표로 복원한다.** 이게 가장 가치 있다. +- 행·열 헤더와 수치를 정확히 옮긴다 + +### 수식 이미지 + +- 수식을 LaTeX로 옮긴다 +- 각 기호가 무엇을 뜻하는지 본문에서 찾아 붙인다 + +## 하지 말 것 + +- **읽을 수 없는 것을 지어내지 않는다.** 흐리거나 잘려서 안 보이면 + "해상도가 낮아 세부 수치는 판독 불가" 라고 명시한다. +- 원본 캡션을 그대로 번역하지 않는다. +- 논문에 없는 해석이나 평가를 덧붙이지 않는다. + ("이 방법이 우수하다" 같은 것 — 그림에서 읽히는 사실만 쓴다) +- 그림과 무관한 배경 설명을 늘어놓지 않는다. + +## 분량 + +그림당 **3~8줄**. 아키텍처 다이어그램과 표 이미지는 더 길어져도 된다 +(표는 완전히 복원할 것). 장식용 그림은 1~2줄로 짧게. + +## 진행 방식 + +1. 파일 하나씩 처리한다. 한 파일을 끝내고 다음으로 간다. +2. 파일 안에서는 위에서 아래 순서로 그림을 처리한다. +3. 이미지를 열 때는 **.md 파일이 있는 디렉토리 기준**으로 상대경로를 해석한다. +4. 파일 하나가 끝나면 몇 개 그림에 주석을 달았는지 보고한다. + +## 검증 + +작업 후 이것들이 성립해야 한다. + +- `[그림 해설]` 개수 == 그 파일의 이미지 개수 +- 원본 텍스트 줄 수가 줄지 않았다 (삽입만 했으므로 늘어나야 정상) +- 마크다운이 깨지지 않았다 (표, 코드 블록, 수식이 그대로) +```` + +--- + +## 작업 후 확인 명령 + +Git Bash 또는 WSL에서: + +```bash +cd /d/MYCLAUDE_PROJECT/sum-parts-test/docs/papers/md + +# 그림 수 대 주석 수 대조 +for md in *.md; do + img=$(grep -c '!\[' "$md") + ann=$(grep -c '\[그림 해설\]' "$md") + printf '%-34s 그림 %2s 주석 %2s %s\n' "$md" "$img" "$ann" \ + "$([ "$img" -eq "$ann" ] && echo OK || echo MISMATCH)" +done +``` + +## 되돌리기 + +주석만 지우려면: + +```bash +# 백업 먼저 +cp -r md md.backup + +# [그림 해설] 인용 블록 제거 +sed -i '/^> \*\*\[그림 해설\]\*\*/,/^$/d' md/*.md +``` + +원본 PDF가 `docs/papers/`에 있으므로 최악의 경우 +`scripts/convert_papers.sh`로 재생성하면 된다. diff --git a/docs/papers/md/PointNeXt_2206.04670.md b/docs/papers/md/PointNeXt_2206.04670.md new file mode 100644 index 0000000..b432149 --- /dev/null +++ b/docs/papers/md/PointNeXt_2206.04670.md @@ -0,0 +1,515 @@ +## PointNeXt: Revisiting PointNet++ with Improved Training and Scaling Strategies + +Guocheng Qian1∗ , Yuchen Li1∗ , Houwen Peng2† , Jinjie Mai1 , Hasan Abed Al Kader Hammoud1 , Mohamed Elhoseiny1 , Bernard Ghanem1† 1King Abdullah University of Science and Technology (KAUST), 2Microsoft Research + +## Abstract + +PointNet++ is one of the most influential neural architectures for point cloud understanding. Although the accuracy of PointNet++ has been largely surpassed by recent networks such as PointMLP and Point Transformer, we find that a large portion of the performance gain is due to improved training strategies, *i.e.* data augmentation and optimization techniques, and increased model sizes rather than architectural innovations. Thus, the full potential of PointNet++ has yet to be explored. In this work, we revisit the classical PointNet++ through a systematic study of model training and scaling strategies, and offer two major contributions. First, we propose a set of improved training strategies that significantly improve PointNet++ performance. For example, we show that, without any change in architecture, the overall accuracy (OA) of PointNet++ on ScanObjectNN object classification can be raised from 77.9% to 86.1%, even outperforming state-of-theart PointMLP. Second, we introduce an inverted residual bottleneck design and separable MLPs into PointNet++ to enable efficient and effective model scaling and propose *PointNeXt*, the next version of PointNets. PointNeXt can be flexibly scaled up and outperforms state-of-the-art methods on both 3D classification and segmentation tasks. For classification, PointNeXt reaches an overall accuracy of 87.7% on ScanObjectNN, surpassing PointMLP by 2.3%, while being 10× faster in inference. For semantic segmentation, PointNeXt establishes a new state-of-theart performance with 74.9% mean IoU on S3DIS (6-fold cross-validation), being superior to the recent Point Transformer. The code and models are available at . + +## 1 Introduction + +Recent advances in 3D data acquisition have led to a surge in interest for point cloud understanding. With the rise of PointNet [\[29\]](#page-11-0) and PointNet++ [\[30\]](#page-11-1), processing point clouds in their unstructured format using deep CNNs become possible. Subsequent to "PointNets", many point-based networks are introduced with the majority focusing on developing new and sophisticated modules to extract local structures, *e.g*. the pseudo-grid convolution in KPConv [\[43\]](#page-12-0) and the self-attention layer in Point Transformer [\[56\]](#page-12-1). These newly proposed methods outperform PointNet++ by a large margin in a variety of tasks, leaving the impression that the PointNet++ architecture is too simple to learn complex point cloud representations. In this work, we revisit PointNet++, the classical and widely used network, and find that its full potential has yet to be explored, mainly due to two factors that were not present at the time of PointNet++: (1) superior training strategies and (2) effective model scaling strategies. + +Through a comprehensive empirical study on various benchmarks, *e.g*., ScanObjecNN [\[44\]](#page-12-2) for object classification and S3DIS [\[1\]](#page-10-0) for semantic segmentation, we discover that training strategies, *i.e*., data augmentation and optimization techniques, play an important role in the network's performance. In fact, a large part of the performance gain of state-of-the-art (SOTA) methods [\[46,](#page-12-3) [43,](#page-12-0) [56\]](#page-12-1) over Point-Net++ [\[30\]](#page-11-1) is due to improved training strategies that are, unfortunately, less publicized compared to + +∗Equal contribution. †Corresponding authors. + +![](PointNeXt_2206.04670_images/_page_1_Figure_0.jpeg) + +Figure 1: Effects of training strategies and model scaling on PointNet++ [\[30\]](#page-11-1). We show that improved training strategies (data augmentation and optimization techniques) and model scaling can significantly boost PointNet++ performance. The average overall accuracy and mIoU (6-fold cross-validation) are reported on ScanObjectNN [\[44\]](#page-12-2) and S3DIS [\[1\]](#page-10-0). + +> **[그림 해설]** 학습 전략 개선 및 모델 스케일링이 PointNet++의 성능 향상에 미치는 기여도 분석 차트. +> - **ScanObjectNN (상단 막대, Overall Accuracy %)**: +> - 원본 PointNet++(노란색): 77.9% +> - Data Augmentation(초록색): +5.8% $\to$ 83.7% +> - Optimization Techniques(보라색): +2.4% $\to$ 86.1% (기존 SOTA인 PointMLP 85.4% 추월) +> - Model Scaling (PointNeXt 구조, 분홍색): +1.6% $\to$ **87.7%** 달성. +> - **S3DIS (하단 막대, 6-fold mIoU %)**: +> - 원본 PointNet++(노란색): 54.5% +> - Data Augmentation(초록색): +12.3% $\to$ 66.8% +> - Optimization Techniques(보라색): +1.3% $\to$ 68.1% +> - Model Scaling(분홍색): +6.8% $\to$ **74.9%** 달성 (기존 SOTA인 Point Transformer 73.5% 추월). + +architectural changes. For example, randomly dropping colors during training can unexpectedly boost the testing performance of PointNet++ by 5.9% mean IoU (mIoU) on S3DIS [\[1\]](#page-10-0), as demonstrated in Tab. [5.](#page-7-0) In addition, adopting label smoothing [\[39\]](#page-11-2) can improve the overall accuracy (OA) on ScanObjectNN [\[44\]](#page-12-2) by 1.3%. These findings inspire us to revisit PointNet++ and equip it with new advanced training strategies that are widely used today. Surprisingly, as shown in Fig. [1,](#page-1-0) utilizing the improved training strategies alone improves the OA of PointNet++ by 8.2% on ScanObjectNN (from 77.9% to 86.1%), establishing a new SOTA without introducing any changes to the architecture (refer to Sec. [4.4.1](#page-7-1) for details). For the S3DIS segmentation benchmark, the mIoU evaluated in all areas by 6-fold cross-validation can increase by 13.6% (from 54.5% to 68.1%), outperforming many modern architectures that are subsequent to PointNet++, such as PointCNN [\[22\]](#page-11-3) and DeepGCN [\[21\]](#page-11-4). + +Moreover, we observe that the current prevailing models [\[20,](#page-11-5) [43,](#page-12-0) [56\]](#page-12-1) for point cloud analysis have employed many more parameters than the original PointNets [\[29,](#page-11-0) [30\]](#page-11-1). Effectively expanding PointNet++ from its original small scale to a larger scale is a topic worth studying because larger models are generally expected to enable richer representations and perform better [\[2,](#page-10-1) [19,](#page-10-2) [55\]](#page-12-4). However, we find that the naive way of using more building blocks or increasing the channel size in PointNet++ only leads to an overhead in latency and no significant improvement in accuracy (see Sec. [4.4.2\)](#page-8-0). For effective and efficient model scaling, we introduce residual connections [\[13\]](#page-10-3), an inverted bottleneck design [\[36\]](#page-11-6), and separable MLPs [\[32\]](#page-11-7) into PointNet++. The modernized architecture is named PointNeXt, the next version of PointNets. PointNeXt can be scaled up flexibly and outperforms SOTA on various benchmarks. As demonstrated in Fig. [1,](#page-1-0) PointNeXt improves the original PointNet++ by 20.4% mIoU (from 54.5% to 74.9%) on *S3DIS* [\[1\]](#page-10-0) 6-fold and achieves 9.8% OA gains on *ScanObjecNN* [\[44\]](#page-12-2), surpassing SOTA Point Transformer [\[56\]](#page-12-1) and PointMLP [\[28\]](#page-11-8). We summarize our contributions next: + +- We present the first systematic study of training strategies in the point cloud domain and show that *PointNet++ strikes back* (+8.2% OA on ScanObjectNN and +13.6% mIoU on S3DIS) by simply adopting *improved training strategies alone*. The improved training strategies are general and can be easily applied to improve other methods [\[29,](#page-11-0) [46,](#page-12-3) [28\]](#page-11-8). +- We propose PointNeXt, the next version of PointNets. PointNeXt is scalable and surpasses SOTA on all tasks studied, including object classification [\[44,](#page-12-2) [49\]](#page-12-5), semantic segmentation [\[1,](#page-10-0) [5\]](#page-10-4), and part segmentation [\[53\]](#page-12-6), while being faster than SOTA in inference. + +## 2 Preliminary: A Review of PointNet++ + +Our PointNeXt is built upon PointNet++ [\[30\]](#page-11-1), which uses a U-Net [\[35\]](#page-11-9) like architecture with an encoder and a decoder, as visualized in Figure [2.](#page-3-0) The encoder part hierarchically abstracts features of point clouds using a number of *set abstraction* (SA) blocks, while the decoder gradually interpolates the abstracted features by the same number of *feature propagation* blocks. The SA block consists of a *subsampling* layer to downsample the incoming points, a *grouping* layer to query neighbors for each point, a set of shared multilayer perceptrons (*MLPs*) to extract features, and a *reduction* layer to aggregate features within the neighbors. The combination of the grouping layer, MLPs, and the reduction layer is formulated as: + + +$$\mathbf{x}_{i}^{l+1} = \mathcal{R}_{j:(i,j)\in\mathcal{N}} \left\{ h_{\Theta} \left( \left[ \mathbf{x}_{j}^{l}; \mathbf{p}_{j}^{l} - \mathbf{p}_{i}^{l} \right] \right) \right\}, \tag{1}$$ + +where $\mathcal{R}$ is the reduction layer (e.g. max-pooling) that aggregates features for point i from its neighbors denoted as $\{j:(i,j)\in\mathcal{N}\}$ . $\mathbf{p}_i^l,\mathbf{x}_i^l,\mathbf{x}_j^l$ are the input coordinates, the input features, and the features of neighbor j in the $l^{th}$ layer of the network, respectively. $h_{\Theta}$ denotes the shared MLPs that take the concatenation of $\mathbf{x}_j^l$ and the relative coordinates $(\mathbf{p}_j^l - \mathbf{p}_i^l)$ as input. Note that, since PointNet++ with single-scale grouping that uses one SA block per stage is the default architecture used in the original paper [30], we refer to it as PointNet++ throughout and use it as our baseline. + +## 3 Methodology: From PointNet++ to PointNeXt + +In this section, we present how to modernize the classical architecture PointNet++ [30] into PointNeXt, the next version of PointNet++ with SOTA performance. Our exploration mainly focuses on two aspects: (1) training modernization to improve data augmentation and optimization techniques, and (2) architectural modernization to probe receptive field scaling and model scaling. Both aspects have important impact on the model's performance, but were under-explored by previous studies. + +#### 3.1 Training Modernization: PointNet++ Strikes Back + +We conduct a systematic study to quantify the effect of each data augmentation and optimization technique used by modern point cloud networks [46, 43, 56] and propose a set of improved training strategies. The potential of PointNet++ can be unveiled by adopting our proposed training strategies. + +#### 3.1.1 Data Augmentation + +Data augmentation is one of the most important strategies to boost the performance of a neural network; thus we start our modernization from there. The original PointNet++ used simple combinations of data augmentations from random rotation, scaling, translation, and jittering for various benchmarks [30]. Recent methods adopt stronger augmentations than those used in PointNet++. For example, KPConv [43] randomly drops colors during training, Point-BERT [54] uses a common point resampling strategy to randomly sample 1,024 points from the original point cloud for data scaling, while RandLA-Net [15] and Point Transformer [56] load the entire scene as input in segmentation tasks. In this paper, we quantify the effect of each data augmentation through an additive study. + +We start our study with PointNet++ [30] as the baseline, which is trained with the original data augmentations and optimization techniques. We remove each data augmentation to check whether it is necessary or not. We add back the useful augmentations but remove the unnecessary ones. We then systematically study all the data augmentations used in the representative works [46, 43, 32, 56, 28, 54], including data scaling such as point resampling [54] and loading the entire scene as input [15], random rotation, random scaling, translation to shift point clouds, jittering to add independent noise to each point, height appending [43] (*i.e.*, appending the measurement of each point along the gravity direction of objects as additional input features), color auto-contrast to automatically adjust color contrast [56], and color drop that randomly replaces colors with zero values. We verify the effectiveness of data augmentation incrementally and only keep the augmentations that give a better validation accuracy. At the end of this study, we provide a collection of data augmentations for each task that allow for the highest boost in the model's performance. Sec. 4.4.1 presents and analyzes in detail the uncovered findings. + +#### 3.1.2 Optimization Techniques + +Optimization techniques including loss functions, optimizers, learning rate schedulers, and hyperparameters are also vital to the performance of a neural network. PointNet++ uses the same optimization techniques throughout its experiments: CrossEntropy loss, Adam optimizer [16], exponential learning rate decay (Step Decay), and the same hyperparmeters. Owing to the development of machine learning theory, modern neural networks can be trained with theoretically better optimizers (e.g. AdamW [27] vs. Adam [16]) and more advanced loss functions (CrossEntropy with label smoothing [39]). Similarly to our study on data augmentations, we also quantify the effect of each modern optimization technique on PointNet++. We first perform a sequential hyperparameter search for the learning rate and weight decay. We then conduct an additive study on label smoothing, optimizer, and learning rate scheduler. We discover a set of improved optimization techniques that further + +![](PointNeXt_2206.04670_images/_page_3_Figure_0.jpeg) + +Figure 2: **PointNeXt architecture.** PointNeXt shares the same Set Abstraction and Feature Propagation blocks as PointNet++ [30], while adding an additional MLP layer at the beginning and scaling the architecture with the proposed Inverted Residual MLP (InvResMLP) blocks. + +> **[그림 해설]** PointNeXt의 전체 세그멘테이션 아키텍처 다이어그램 (빨간색 테두리가 현대화된 핵심 구성 요소). +> - **인코더 (U-Net 형태 백본)**: +> - 입력 포인트 $\to$ 초기 임베딩 **MLP [N, 32]** 추가. +> - 4단계 계층적 다운샘플링: Set Abstraction + **InvResMLP** 블록을 거치며 $[N/4, 64] \to [N/16, 128] \to [N/64, 256] \to [N/256, 512]$로 점진적 축소. +> - **핵심 블록 상세 (하단)**: +> - **Set Abstraction**: Subsample $\to$ Grouping $\to$ MLPs (64) $\to$ Reduction(Max-pool). +> - **InvResMLP (Inverted Residual MLP)**: Residual Connection(지름길 연결)을 포함하며, Grouping $\to$ MLP(256) $\to$ Reduction $\to$ MLP(1024, 4배 확장 역병목) $\to$ MLP(256) 구조로 파라미터 효율성과 특징 표현력을 극대화. +> - **Feature Propagation (디코더)**: +> - Interpolate $\to$ Skip Connection 결합 $\to$ MLPs(128)를 거쳐 $[N/64, 256] \to [N/16, 128] \to [N/4, 64] \to [N, 32]$ 순으로 원본 해상도를 복원한 후 최종 세그멘테이션 레이블을 출력. + +boost performance by a decent margin. In general, CrossEntropy with label smoothing, AdamW, and Cosine Decay can decently optimize models in various tasks. See Sec. 4.4.1 for detailed findings. + +#### **3.2** Architecture Modernization: Small Modifications → Big Improvements + +In this subsection, we modernize PointNet++ [30] into the proposed PointNeXt. The modernization consists of two aspects: (1) receptive field scaling and (2) model scaling. + +#### 3.2.1 Receptive Field Scaling + +The receptive field is a significant factor in the design space of a neural network [38, 7]. There are at least two ways to scale the receptive field in point cloud processing: (1) adopting a larger radius to query the neighborhood, and (2) adopting a hierarchical architecture. Since the hierarchical architecture has been adopted in the original PointNet++, we mainly study (1) in this subsection. Note that the radius of PointNet++ is set to an initial value r that doubles when the point cloud is downsampled. We study a different initial value in each benchmark and discover that the radius is dataset-specific and can have significant influence on performance. This is elaborated in Sec. 4.4.2. + +Furthermore, we find that the relative coordinates $\Delta_p = \mathbf{p}_j^l - \mathbf{p}_i^l$ in Eq. (1) make network optimization harder, leading to a decrease in performance. Thus, we propose relative position normalization ( $\Delta_p$ normalization) to divide relative position by the neighborhood query radius: + + +$$\mathbf{x}_{i}^{l+1} = \mathcal{R}_{j:(i,j)\in\mathcal{N}}\left\{h_{\Theta}\left(\left[\mathbf{x}_{j}^{l};(\mathbf{p}_{j}^{l} - \mathbf{p}_{i}^{l})/r^{l}\right]\right)\right\}. \tag{2}$$ + +Without normalization, values of relative positions $(\Delta_p = \mathbf{p}_j^l - \mathbf{p}_i^l)$ are considerably small (less than the radius), requiring the network to learn a larger weight to apply on $\Delta_p$ . This makes the optimization non-trivial, especially since weight decay is used to reduce the weights of the network and thus tends to ignore the effects of relative position. The proposed normalization alleviates this issue by rescaling and in the meantime reduces the variance of $\Delta_p$ among different stages. + +#### 3.2.2 Model Scaling + +PointNet++ is a relatively small network, where the encoder consists of only 2 stages in the classification architecture and 4 stages for segmentation. Each stage consists of only 1 SA block, and each block contains 3 layers of MLP. The model sizes of PointNet++ for both classification and segmentation are less than 2M, which is much smaller compared to modern networks that typically use more than 10M parameters [43, 28, 32]. Interestingly, we find that neither appending more SA blocks nor using more channels leads to a noticeable improvement in accuracy, while causing a significant drop in throughput (refer to Sec. 4.4.2), mainly due to vanishing gradient and overfitting. Therefore, in this subsection, we study how to scale up PointNet++ in an effective and efficient way. + +We propose an Inverted Residual MLP (InvResMLP) block to be appended after the first SA block, per stage, for effective and efficient model scaling. InvResMLP is built on the SA block and is + +illustrated at the bottom middle of Fig. [2.](#page-3-0) There are three differences between InvResMLP and SA. (1) A residual connection between the input and the output is added to alleviate the vanishing gradient problem [\[13\]](#page-10-3), especially when the network goes deeper. (2) Separable MLPs are introduced to reduce computation and reinforce pointwise feature extraction. While all 3 layers of MLPs in the original SA block are computed on the neighborhood features, InvResMLP separates the MLPs into a single layer computed on the neighborhood features (between the grouping and reduction layers) and two layers for point features (after reduction), as inspired by MobileNet [\[14\]](#page-10-8) and ASSANet [\[32\]](#page-11-7). (3) The inverted bottleneck design [\[36\]](#page-11-6) is leveraged to expand the output channels of the second MLP by 4 times to enrich feature extraction. Appending InvResMLP blocks is proven to significantly improve performance compared to the appending of the original SA blocks (see Sec. [4.4.2\)](#page-8-0). + +In addition to InvResMLP, we present three changes in the macro architecture. (1) We unify the design of PointNet++ encoder for classification and segmentation, *i.e*., scaling the number of SA blocks for classification from 2 to 4 while keeping the original number (4 blocks) for segmentation at each stage. (2) We utilize a symmetric decoder in which its channel size is changed to match the encoder. (3) We add a stem MLP, an additional MLP layer inserted at the beginning of the architecture, to map the input point cloud to a higher dimension. + +In summary, we present PointNeXt, the next version of PointNets [\[29,](#page-11-0) [52\]](#page-12-8), modified from PointNet++ by incorporating the proposed InvResMLP and the aforementioned macro-architectural changes. The architecture of PointNeXt is illustrated in Fig. [2.](#page-3-0) We denote the channel size of the stem MLP as C and the number of InvResMLP blocks as B. A larger C leads to an increase in the width of the network (*i.e*., width scaling), while a larger B leads to an increase in the depth of the network (*i.e*., depth scaling). Note that when B = 0, only one SA block and no InvResMLP blocks are used at each stage. The number of MLP layers in the SA block is set to 2, and a residual connection is added inside each SA block. When B 6= 0, InvResMLP blocks are appended after the original SA block. The number of MLP layers in the SA block in this case is set to 1 to save computation cost. The configuration of our PointNeXt family is summarized as follows: + +``` +• PointNeXt-S: C = 32, B = 0 +• PointNeXt-B: C = 32, B = (1, 2, 1, 1) + • PointNeXt-L: C = 32, B = (2, 4, 2, 2) + • PointNeXt-XL: C = 64, B = (3, 6, 3, 3) +``` + +## 4 Experiments + +We evaluate PointNeXt on five standard benchmarks: *S3DIS* [\[1\]](#page-10-0) and *ScanNet* [\[5\]](#page-10-4) for semantic segmentation, *ScanObjectNN* [\[44\]](#page-12-2) and *ModelNet40* [\[49\]](#page-12-5) for object classification, and *ShapeNetPart* [\[3\]](#page-10-9) for object part segmentation. + +Experimental Setups. We train PointNeXt using CrossEntropy loss with label smoothing [\[39\]](#page-11-2), AdamW optimizer [\[27\]](#page-11-10), an initial learning rate lr = 0.001, weight decay 10−4 , with Cosine Decay, and a batch size of 32, with a 32G V100 GPU, for all tasks, unless otherwise specified. The best model on the validation set is selected for testing. For S3DIS segmentation, point clouds are voxel downsampled with a voxel size of 0.04m following common practice [\[43,](#page-12-0) [32,](#page-11-7) [56\]](#page-12-1). PointNeXt is trained with an initial lr = 0.01, for 100 epochs (training set is repeated by 30 times), using a fixed number of points (24, 000) per batch with a batch size of 8 as input. During training, the input points are obtained by querying the nearest neighbors of a random point in each iteration. Following Point Transformer [\[56\]](#page-12-1), we evaluate PointNeXt using the entire voxel-downsampled scene as input. For ScanNet scene segmentation, we follow the Stratified Transformer [\[17\]](#page-10-10) and train PointNeXt with multi-step learning rate decay and decay at [70,90] epochs with a decay rate of 0.1 without label smoothing. The voxel size is set to 0.02m and input number of points in training is set to 64, 000. We train the model for 100 epochs (training set is repeated for 6 times) with a batch size of 2 per GPU with 8 GPUs. For ScanObjectNN classification, PointNeXt is trained with a weight decay of 0.05 for 250 epochs. Following Point-BERT [\[54\]](#page-12-7), the number of input points is set to 1, 024, where the points are randomly sampled during training and uniformly sampled during testing (denoted as point resampled augmentation). For ModelNet40 classification, PointNeXt is trained similarly as ScanObjectNN but for 600 epochs. For ShapeNetPart part segmentation, we train PointNeXt using a batch size of 8 per GPU with 4 GPUs, and Poly FocalLoss [\[18\]](#page-10-11) as criterion, for 400 epochs. Following PointNet++, 2,048 randomly sampled points with normals are used as input for training and testing. The details of data augmentations used in S3DIS, ScanNet, ScanObjectNN, ModelNet40 and ShapeNetPart are detailed in Sec. [4.4.1.](#page-7-1) + +Table 1: **3D semantic segmentation in S3DIS** (**evaluation by 6-Fold or in Area 5**) and **ScanNet V2.** For PointNeXt in S3DIS Area 5, the average results without voting in three random runs are reported. The improvements of PointNeXt over the original performance reported by PointNet++ [30] are highlighted in green color. PointNet++ (ours) denotes PointNet++ trained using our improved data augmentation and optmization techniques. Methods are in chronological order. + +| | S3DIS | 6-Fold | S3DIS | Area-5 | Scan | Net V2 | Doros | me FLO | Ps Throughput | +|------------------------|--------------|-------------|-------------------------|-----------------------|-------------|-------------|---------|-----------|------------------| +| Method | mIoU | OA | mIoU | OA | Val mIoU | Test mIoU | 1 ai ai | iiis. FLO | 1 8 Till oughput | +| | (%) | (%) | (%) | (%) | (%) | (%) | M | G | (ins./sec.) | +| PointNet [29] | 47.6 | 78.5 | 41.1 | - | - | - | 3.6 | 35.5 | 162 | +| PointCNN [22] | 65.4 | 88.1 | 57.3 | 85.9 | - | 45.8 | 0.6 | - | - | +| DGCNN [46] | 56.1 | 84.1 | 47.9 | 83.6 | - | - | 1.3 | - | 8 | +| DeepGCN [21] | 60.0 | 85.9 | 52.5 | - | - | - | 3.6 | - | 3 | +| KPConv [43] | 70.6 | - | 67.1 | - | 69.2 | 68.6 | 15.0 | - | 30 | +| RandLA-Net [15] | 70.0 | 88.0 | - | - | - | 64.5 | 1.3 | 5.8 | 159 | +| BAAF-Net [33] | 72.2 | 88.9 | 65.4 | 88.9 | - | - | 5.0 | - | 10 | +| Point Transformer [56] | 73.5 | 90.2 | 70.4 | 90.8 | 70.6 | - | 7.8 | 5.6 | 34 | +| CBL [41] | 73.1 | 89.6 | 69.4 | 90.6 | - | 70.5 | 18.6 | - | - | +| PointNet++ [30] | 54.5 | 81.0 | 53.5 | 83.0 | 53.5 | 55.7 | 1.0 | 7.2 | 186 | +| PointNet++ (ours) | 68.1(+13.6) | 87.6(+6.2) | $63.2\pm0.4(+9.7)$ | $87.5\pm0.2(+4.5)$ | 57.2(+3.7) | - | 1.0 | 7.2 | 186 | +| PointNeXt-S (ours) | 68.0(+13.5) | 87.4(+6.4) | 63.4±0.8(+9.9) | $87.9\pm0.3(+4.9)$ | 64.5(+11.0) | - | 0.8 | 3.6 | 227 | +| PointNeXt-B (ours) | 71.5(+17.0) | 88.8(+7.8) | $67.3\pm0.2(+13.8)$ | $89.4\pm0.1(+6.4)$ | 68.4(+14.9) | - | 3.8 | 8.9 | 158 | +| PointNeXt-L (ours) | 73.9(+19.4) | 89.8(+8.8) | 69.0±0.5(+15.5) | $90.0\pm0.1(+7.0)$ | 69.4(+15.9) | - | 7.1 | 15.2 | 115 | +| PointNeXt-XL (ours) | 74.9 (+20.4) | 90.3 (+9.3) | 70.5 ±0.3(+17.0) | $90.6 \pm 0.1 (+7.6)$ | 71.5(+18.0) | 71.2(+15.5) | 41.6 | 84.8 | 46 | + +For all experiments except ShapeNetPart segmentation, we do not conduct any voting $[23]^2$ , since it is more standard to compare the performance without using any ensemble methods as suggested by SimpleView [9]. However, we found that the performance in ShapeNetPart of nearly all models is quite close to each other, where it is hard to achieve state-of-the-art IoUs without voting. We also provide model parameters (Params.) and inference throughput (instances per second) for comparison. The throughput of all methods is measured using $128 \times 1024$ (batch size 128, number of points 1024) as input in ScanObjectNN and ModelNet40 and $64 \times 2048$ in ShapeNetPart. In S3DIS, $16 \times 15,000$ points are used to measure throughput following [32], since some methods [46, 20] could not process the whole scene due to memory constraints. The throughput of all methods is measured using an NVIDIA Tesla V100 32GB GPU and a 32 core Intel Xeon @ 2.80GHz CPU. + +#### 4.1 3D Semantic Segmentation in S3DIS and ScanNet + +S3DIS [1] (Stanford Large-Scale 3D Indoor Spaces) is a challenging benchmark composed of 6 large-scale indoor areas, 271 rooms, and 13 semantic categories in total. The standard 6-fold cross-validation results in S3DIS are reported in Tab. 1. Note that the official PointNet++ [30] did not conduct experiments in S3DIS. Here, we use the results reported by PointCNN [22] for comparison. Our PointNeXt-S, the smallest variant, outperforms PointNet++ by 13.5%, 6.4%, and 10.2% in terms of mean IoU (mIoU), overall accuracy (OA), and mean accuracy (mAcc), respectively, while being faster in terms of throughput. The increased speed is due to the reduced number of layers in the SA block for PointNeXt-S (see Sec. 3.2.2). With the proposed model scaling, the performance of PointNeXt can be gradually boosted. For example, PointNeXt-L outperforms SOTA Point Transformer [56] by 0.4% in mIoU while being 3× faster. Note that Point Transformer utilizes most of the improved training strategies of ours. PointNeXt-XL, the extra large variant, achieves mIoU/OA/mAcc of 74.9%/90.3%/83.0%, while running faster than Point Transformer. As a limitation, our PointNeXt-XL consists of more parameters and is more computationally expensive in terms of FLOPs, mainly due to channel expansion $(\times 4)$ in the inverted bottleneck and doubled initial channel size (C=64). We also provide the results of PointNeXt in S3DIS area 5 in the Tab. 1 with mean $\pm$ std in three random runs, where PointNeXt achieves similar improvements as the 6-fold experiments. + +ScanNet [5], another well-known large-scale segmentation dataset, contains 3D indoor scenes of various rooms with 20 semantic categories. We follow the public training, validation, and test splits, with 1201, 312 and 100 scans, respectively. For PointNet++, we use the results reported from the Stratified Transformer [17] for comparison. As shown in Tab. 1, we improve PointNet++ from 53.5% mIou to 57.2% mIoU in the validation set by adopting the improved training strategies (detailed in supplementary material). PointNeXt-S further gains +11.0 in val mIoU over the original PointNet++ mostly due to the use of a smaller radius $(0.1\text{m} \rightarrow 0.05\text{m})$ and relative position normalization. The performance in ScanNet improves steadily with the increase in model sizes. Our largest variant, + +<sup>2The voting strategy combines results by using randomly augmented points as input to enhance performance. + +Table 2: **3D object classification in ScanObjectNN and ModelNet40.** Averaged results in three random runs using 1024 points as input without normals and without voting are reported. + +| Method | ScanObjectNN
OA (%) | M (PB_T50_RS)
mAcc (%) | OA (%) | et40
mAcc (%) | Params.
M | FLOPs
G | Throughput (ins./sec.) | +|------------------------------|------------------------|---------------------------|----------------|------------------|--------------|------------|------------------------| +| PointNet [29] | 68.2 | 63.4 | 89.2 | 86.2 | 3.5 | 0.9 | 4212 | +| PointCNN [22] | 78.5 | 75.1 | 92.2 | 88.1 | 0.6 | - | 44 | +| DGCNN [46] | 78.1 | 73.6 | 92.9 | 90.2 | 1.8 | 4.8 | 402 | +| DeepGCN [20] | - | _ | 93.6 | 90.9 | 2.2 | 3.9 | 263 | +| KPConv [43] | - | _ | 92.9 | _ | 14.3 | - | - | +| ASSANet-L [32] | - | _ | 92.9 | _ | 118.4 | - | 153 | +| SimpleView [9] | 80.5±0.3 | _ | 93.0±0.4 | $90.5 \pm 0.8$ | 0.8 | - | - | +| MVTN [12] | 82.8 | _ | 93.5 | 92.2 | 3.5 | 1.8 | 236 | +| Point Cloud Transformer [11] | - | - | 93.2 | _ | 2.9 | 2.3 | - | +| CurveNet [50] | - | _ | 93.8 | _ | 2.0 | - | 22 | +| PointMLP [28] | 85.4±1.3 | $83.9 \pm 1.5$ | 94.1 | 91.3 | 13.2 | 31.3 | 191 | +| PointNet++ [30] | 77.9 | 75.4 | 91.9 | - | 1.5 | 1.7 | 1872 | +| PointNet++ (ours) | 86.1±0.7(+8.2) | $84.2 \pm 0.9 (+8.8)$ | 92.8±0.1(+0.9) | $89.9 \pm 0.8$ | 1.5 | 1.7 | 1872 | +| PointNeXt-S (ours) | 87.7±0.4(+9.8) | $85.8 \pm 0.6 (+10.4)$ | 93.2±0.1(+1.3) | $90.8 {\pm} 0.2$ | 1.4 | 1.6 | 2040 | + +PointNeXt-XL outperforms PointNet++ by 18.0% mIoU in validation and achieves 71.2% mIoU in testing, beating the recent methods Point Transformer [56] and CBL [41]. + +#### 4.2 3D Object Classification in ScanObjectNN and ModelNet40 + +ScanObjectNN [44] contains about 15,000 real scanned objects that are categorized into 15 classes with 2,902 unique object instances. Due to occlusions and noise, ScanObjectNN poses significant challenges to existing point cloud analysis methods. Following PointMLP [28], we experiment on PB\_T50\_RS, the hardest and most commonly used variant of ScanObjectNN. As reported in Tab. 2, the proposed PointNeXt-S surpasses existing methods by non-trivial margins in terms of both OA and mAcc, while using much fewer model parameters and running much faster. Built upon PointNet++ [30], PointNeXt achieves significant improvements over the originally reported performance of PointNet++, *i.e.* +9.8% OA and +10.4% mACC. This demonstrates the efficacy of the proposed training and model scaling strategies. PointNeXt also outperforms SOTA PointMLP [28] (*i.e.* +2.3% OA, +1.9% mACC), while running $10\times$ faster. This shows that PointNeXt is a simple, yet effective, and efficient baseline. Note that we did not experiment with upscaled variants of PointNeXt on this benchmark, since we found that the performance had saturated using PointNeXt-S mostly due to the limited scale of the dataset. + +ModelNet40 [49] was a commonly used 3D object classification dataset, which has 40 object categories, each of which contains 100 unique CAD models. However, recent works [12, 28, 34] show an increasing interest in the real-world scanned dataset ScanObejectNN compared to this synthesized dataset. Following this trend, we mainly benchmarked PointNeXt in ScanObjectNN. Here, we also provide our results in ModelNet40. Tab. 2 shows that advanced training strategies improve PointNet++ from 91.9% OA to 92.8% OA without any architecture change. PointNeXt-S (C=32) outperforms the original reported PointNet++ by 1.3% OA, while being faster. Note that PointNeXt-S with a larger width C=64 can achieve a higher overall accuracy (94.0%). + +## 4.3 3D Object Part Segmentation in ShapeNetPart + +ShapeNetPart [53] is a widely-used dataset for object-level part segmentation. It consists of 16,880 models from 16 different shape categories, 2-6 parts for each category, and 50 part labels in total. As shown in Tab. 3, our PointNeXt-S with default width (C=32) obtains a performance comparable + +Table 3: Part segmentation in ShapeNetPart. + + + +| Method | ins. mIoU | cls. mIoU | Params. | FLOPs | Throughput | +|------------------------|-----------------------|-----------------------|---------|-------|------------| +| PointNet [29] | 83.7 | 80.4 | 3.6 | 4.9 | 1184 | +| DGCNN [46] | 85.2 | 82.3 | 1.3 | 12.4 | 147 | +| KPConv [43] | 86.4 | 85.1 | - | - | 44 | +| CurveNet [50] | 86.8 | - | - | - | 97 | +| ASSANet-L [32] | 86.1 | - | - | - | 640 | +| Point Transformer [56] | 86.6 | 83.7 | 7.8 | - | 297 | +| PointMLP [28] | 86.1 | 84.6 | - | - | 270 | +| Stratifiedformer [17] | 86.6 | 85.1 | - | - | 398 | +| PointNet++ [30] | 85.1 | 81.9 | 1.0 | 4.9 | 708 | +| PointNeXt-S | $86.7 \pm 0.0 (+1.6)$ | $84.4 \pm 0.2 (+2.5)$ | 1.0 | 4.5 | 782 | +| PointNeXt-S (C=64) | $86.9\pm0.1(+1.8)$ | $84.8 \pm 0.5 (+2.9)$ | 3.7 | 17.8 | 331 | +| PointNeXt-S (C=160) | $87.0\pm0.1(+1.9)$ | $85.2 \pm 0.1 (+3.3)$ | 22.5 | 110.2 | 76 | + +to that of the SOTA CurveNet [50] and outperforms a large number of representative networks, such as KPConv [43] and ASSANet [32] in terms of both instance mean IoU (ins. mIoU) and throughput. Due to the small scale of ShapeNetPart, the model would overfit after being depth scaled. However, we find by increasing the width from 32 to 64 instead, PointNeXt can outperform CurveNet, while being over 4× faster. It is also worth highlighting that PointNeXt with an even larger width (C = 160) reaches 87.0% Ins. mIoU, whereas the performance of point-based methods has saturated below this value for years. We highlight that we used voting only in ShapeNetPart by averaging the results of 10 randomly scaled input point clouds, with scaling factors equal to [0.8,1.2]. Without voting, we notice a performance drop around 0.5 instance mIoU. + +#### 4.4 Ablation and Analysis + +Tab. 4 and Tab. 5 present additive studies for the proposed training and scaling strategies in ScanObjectNN [44] and S3DIS [1], respectively. We adopt the original PointNet++ as the baseline. In ScanObjectNN, PointNet++ was trained by [44] with CrossEntropy loss, Adam optimizer, a learning rate 1e-3, a weight decay of 1e-4, a step decay of 0.7 for every 20 epochs, and a batch size of 16, for 250 epochs, while using random rotation and jittering as data augmentations. The official PointNet++ did not conduct experiments in S3DIS dataset. We refer to the widely used reimplementation [52], where PointNet++ was trained with the same settings as ScanObjectNN except that only random rotation was used as augmentation. Note that for all experiments, we train all our models for 250 epochs in ScanObjectNN and for 100 epochs in S3DIS. + +#### 4.4.1 Training Strategies + +Data augmentation is the first aspect that we study to modernize PointNet++. We draw four conclusions based on observations in Tab. 4 and 5. (1) Data scaling improves performance for both classification and segmentation tasks. For example, point resampling is shown to boost the performance by 2.5% OA in ScanObjectNN. Taking the entire scene as input instead of using the block or sphere subsampled input as done in PointNet++ [30] and other previous works [43, 21, 32] improves the segmentation result by 1.1% mIoU. (2) Height appending improves performance, especially for object classification. Height appending makes the network aware of the actual size of the objects, thus leading to an increase in accuracy (+1.1% OA). (3) Color drop is a strong augmentation that significantly improves the performance of tasks where colors are available. Adopting color drop alone adds 5.9% mIoU in S3DIS area 5. We hypothesize that color drop forces the network to focus more on the geometric relationships between points, which in turn improves performance. (4) Larger models favor stronger data augmentation. Whereas random rotation drops the performance of PointNet++ by 0.3% mIoU in S3DIS ( $2^{nd}$ row in Tab. 5 data augmentation part), it is shown to be beneficial for larger-scale models (e.g. raises 1.5% mIoU on PointNeXt-B). Another example in ScanObjectNN shows that the removal of random jittering also adds 1.1% OA. In general, with the improved data augmentations, the OA of PointNet++ in ScanObjectNN and the mIoU in S3DIS area 5 are increased by 5.8% and 9.5%, respectively. + +Table 4: Additive study of sequentially applying train- Table 5: Additive study of sequentially applying and scaling strategies for classification on ScanOb- ing training and scaling strategies for segmenjectNN. We use light green, purple, yellow, and pink background colors to denote data augmentation, \top- ing/removing the strategy. timization techniques, receptive field scaling, and model scaling, respectively. + +| Improvements | OA (%) | Δ | +|-------------------------------------------|----------------|------| +| PointNet++ | 77.9 | - | +| + Point resampling | $81.4 \pm 0.6$ | +2.5 | +|
  • Jittering
| $82.5 \pm 0.4$ | +1.1 | +| + Height appending | $83.6 \pm 0.4$ | +1.1 | +| + Random scaling | $83.7 \pm 0.2$ | +0.1 | +| + Label Smoothing | $85.0 \pm 0.5$ | +1.3 | +| $+$ Adam $\rightarrow$ Adam $W$ | $85.6 \pm 0.1$ | +0.6 | +| $+$ Step Decay $\rightarrow$ Cosine Decay | $86.1 \pm 0.7$ | +0.5 | +| $+$ Radius $0.2 \rightarrow 0.15$ | $86.4 \pm 0.3$ | +0.3 | +| + Normalizing $\Delta_p$ (Eqn. (2)) | $86.7 \pm 0.3$ | +0.3 | +| + Scale up (PointNeXt-S) | $87.7 \pm 0.4$ | +1.0 | + +tation on S3DIS area 5. +/- denote adopt- + +| Improvements | mIoU (%) | Δ | +|-------------------------------------------|----------------|------| +| PointNet++ | 51.5 | - | +| + Entire scene as input | $52.6 \pm 0.5$ | +1.1 | +| - Rotation | $52.9 \pm 0.6$ | +0.3 | +| + Height appending | $53.4 \pm 0.4$ | +0.5 | +| + Color drop | $59.3 \pm 0.7$ | +5.9 | +| + Color auto-contrast | $61.0 \pm 0.4$ | +0.7 | +| $+ lr = 0.001 \rightarrow 0.01$ | $61.5 \pm 0.5$ | +0.5 | +| + Label Smoothing | $61.9 \pm 0.1$ | +0.4 | +| $+$ Adam $\rightarrow$ Adam $W$ | $62.5 \pm 0.6$ | +0.6 | +| $+$ Step Decay $\rightarrow$ Cosine Decay | $63.2 \pm 0.4$ | +0.7 | +| + Normalize $\Delta_p$ | $63.6 \pm 0.4$ | +0.4 | +| + Scale down (PointNeXt-S) | $63.4 \pm 0.8$ | -0.2 | +| + Scale up (PointNeXt-B) | $65.8 \pm 0.5$ | +2.4 | +| + Rotation | $67.3 \pm 0.2$ | +1.5 | +| + Scale up (PointNeXt-L) | $69.0 \pm 0.5$ | +1.7 | +| + Scale up (PointNeXt-XL) | $70.5 \pm 0.3$ | +1.5 | + +Optimization techniques involve loss functions, optimizers, learning rate schedulers, and hyperparameters. As shown in Tab. [4](#page-7-0) and [5,](#page-7-0) Label Smoothing, AdamW [\[27\]](#page-11-10) optimizer, and Cosine Decay consistently boost performance in both classification and segmentation tasks. This reveals that the more developed optimization methods such as label smoothing and AdamW are generally good for optimizing a neural network. Compared to Step Decay, Cosine Decay is also easier to tune (usually only the initial and minimum learning rates are required) and can achieve a performance similar to Step Decay. Regarding hyperparameters, using a learning rate greater than that used in PointNet++ improves the segmentation performance in S3DIS. + +In general, our training strategies consisted of stronger data augmentation and modern optimization techniques can increase the performance of PointNet++ from 77.9% to 86.1% OA in ScanObjectNN dataset, impressively surpassing SOTA PointMLP by 0.7%. The mIoUs in S3DIS area 5 and S3DIS 6-fold (illustrated in Fig. [1\)](#page-1-0) are boosted by 11.7 and 13.6 absolute percentage points, respectively. Our observations imply that *a significant portion of the performance gap between classical PointNet++ and SOTA is due to the training strategies.* + +Generalize to other networks. Although the training strategies are proposed for PointNet++ [\[30\]](#page-11-1), we find that they can be applied to other methods such as PointNet [\[29\]](#page-11-0), DGCNN [\[46\]](#page-12-3), and PointMLP [\[28\]](#page-11-8), and also improve their performance. Such generalizability is validated in ScanObjectNN [\[44\]](#page-12-2). As shown in Tab. [6,](#page-8-1) the OA of the representative methods can all be improved when equipped with our training strategies. + +Table 6: The generalizability of improved training strategies. OA on ScanObjectNN of networks trained with improved training strategies is reported. + +| Method | ours | ∆ | +|---------------|------------|------| +| PointNet [29] | 74.4 ± 0.9 | +6.2 | +| DGCNN [46] | 86.0 ± 0.5 | +7.9 | +| PointMLP [28] | 87.1 ± 0.7 | +1.7 | + +#### 4.4.2 Model Scaling + +Receptive field scaling includes both radius scaling and normalizing ∆p defined in Eqn. [\(2\)](#page-3-2), which are also validated in Tab. [4](#page-7-0) and [5.](#page-7-0) The radius is dataset specific, while down-scaling the radius from 0.2 to 0.15 improves 0.3% OA in ScanObjectNN, keeping the radius the same as 0.1 achieves the best performance in S3DIS. Regarding normalizing ∆p, it improves the performance in ScanObjectNN and S3DIS by 0.3 OA and 0.4 mIoU, respectively. Furthermore, in Tab. [7,](#page-8-2) we show that normalizing ∆p has a larger impact (2.3 mIoU in S3DIS dataset) on the bigger model PointNext-XL. + +Model scaling scales PointNet++ by the proposed InvResMLP and some macro-architectural changes (see Sec. [3.2.2\)](#page-3-1). In Tab. [4,](#page-7-0) we show that PointNeXt-S using the stem MLP, the symmetric decoder, and the residual connection in the SA block improves 1.0% OA in ScanObjectNN. Performance in the large-scale S3DIS dataset can be further unveiled (from 63.8% to 70.5% mIoU) by up-scaling PointNeXt-S using more blocks of the proposed InvResMLP, as demonstrated in Tab. [5.](#page-7-0) Furthermore, in Tab. [7,](#page-8-2) we ablate each component of the proposed InvResMLP + +block and different stage ratios in S3DIS area 5 using the best-performed model PointNeXt-XL as the baseline. As observed, each architectural change indeed contributes to increased performance. Among all changes, the residual connection is the most essential, without which the mIoU will drop from 70.5% to only 64.0%. The separable MLPs increase 3.9% mIoU while speeding up the network 3 times. Removing the inverted bottleneck from the baseline leads to a drop of 1.5% mIoU with less than a 1% gain in speed. Adding more blocks inside each stage after removing inverted bottleneck can improve its performance to 69.7 ± 0.3 but is still lower than the baseline. Tab. [7](#page-8-2) also shows the performance of naive width scaling that increases the width of PointNet++ from 32 to 256 to match the throughput of PointNeXt-XL, naive depth + +Table 7: Ablate architectural changes on S3DIS area 5. − denotes removing from baseline. TP denotes throughput. + +| Ablate | mIoU | ∆ | TP | +|-------------------------|------------|-------|----| +| baseline (PointNeXt-XL) | 70.5 ± 0.3 | - | 45 | +| − normalizing ∆p | 68.2 ± 0.7 | -2.3 | 45 | +| − residual connection | 64.0 ± 1.0 | -6.5 | 45 | +| − stem MLP | 70.1 ± 0.4 | -0.4 | 46 | +| − Separable MLPs | 66.6 ± 0.8 | -3.9 | 15 | +| − Inverted bottleneck | 69.0 ± 0.4 | -1.5 | 48 | +| − Inverted bottleneck | 69.7 ± 0.3 | -0.8 | 43 | +| stage ratio → (1:1:1:1) | 69.8 ± 0.6 | -0.7 | 52 | +| stage ratio → (2:1:1:1) | 69.4 ± 0.4 | -1.1 | 41 | +| stage ratio → (1:1:2:1) | 69.9 ± 0.6 | -0.6 | 47 | +| stage ratio → (1:1:1:2) | 69.5 ± 0.4 | -1.0 | 48 | +| stage ratio → (1:3:1:1) | 70.1 ± 0.4 | -0.4 | 39 | +| naive width scaling | 59.4 ± 0.1 | -11.1 | 43 | +| naive depth scaling | 63.4 ± 0.5 | -7.1 | 53 | +| naive compound scaling | 62.3 ± 1.2 | -8.2 | 24 | + +scaling to append more SA blocks in PointNet++ to obtain the same number of blocks of PointNext-XL whose B = (3, 6, 3, 3), and naive compound scaling to double the width of the naive depth scaled model to the same width as PointNeXt-XL (C = 64). Our proposed model scaling strategy achieves much higher performance than these naive scaling strategies, while being much faster. + +## 5 Related Work + +*Point-based methods* process point clouds directly using their unstructured format compared to voxel-based methods [\[10,](#page-10-15) [4\]](#page-10-16) and multi view-based methods [\[37,](#page-11-15) [12,](#page-10-13) [9\]](#page-10-12). PointNet [\[29\]](#page-11-0), the pioneering work of point-based methods, proposes to model the permutation invariance of points with shared MLPs by restricting feature extraction to be pointwise. PointNet++ [\[30\]](#page-11-1) is presented to improve PointNet by capturing local geometric structures. Currently, most point-based methods focus on the design of local modules. [\[46,](#page-12-3) [45,](#page-12-11) [31\]](#page-11-16) rely on graph neural networks. [\[51,](#page-12-12) [22,](#page-11-3) [43,](#page-12-0) [42\]](#page-12-13) project point clouds onto pseudo grids to allow for regular convolutions. [\[48,](#page-12-14) [23,](#page-11-13) [24\]](#page-11-17) adaptively aggregate neighborhood features through weights determined by the local structure. In addition, very recent methods leverage Transformer-like networks [\[56,](#page-12-1) [17\]](#page-10-10) to extract local information through self-attention. Our work does not follow this trend in local module design. In contrast, we shift our attention to another important but largely under-explored aspect, *i.e.*, the training and scaling strategies. + +*Training strategies* are studied recently in [\[2,](#page-10-1) [47,](#page-12-15) [26\]](#page-11-18) on image classification. In the point cloud domain, SimpleView [\[9\]](#page-10-12) is the first work to show that training strategies have a large impact on the performance of a neural network. However, SimpleView simply adopts the same training strategies as DGCNN [\[46\]](#page-12-3). On the contrary, we conducted a systematic study to quantify the effect of *each* data augmentation and optimization technique, and propose a set of improved training strategies that boost the performance of PointNet++ [\[30\]](#page-11-1) and other representative works [\[29,](#page-11-0) [46,](#page-12-3) [28\]](#page-11-8). + +*Model scaling* can significantly improve the performance of a network, as shown in pioneering works in various domains [\[40,](#page-12-16) [55,](#page-12-4) [21\]](#page-11-4). Compared to PointNet++ [\[30\]](#page-11-1) that uses parameters less than 2M, most current prevailing networks consist of parameters greater than 10 M, such as KPConv [\[43\]](#page-12-0) (15M) and PointMLP [\[28\]](#page-11-8) (13M). In our work, we explore model scaling strategies that can scale up PointNet++ in an effective and efficient manner. We offer practical suggestions on scaling technologies that improve performance, namely using residual connections and an inverted bottleneck design, while maintaining throughput by using separable MLPs. + +## 6 Conclusion and Discussion + +In this paper, we demonstrate that with improved training and scaling strategies, the performance of PointNet++ can be increased to exceed the current state of the art. More specifically, we quantify the effect of each data augmentation and optimization technique that are widely used today, and propose a set of improved training strategies. These strategies can be easily applied to boost the performance of PointNet++ and other representative works. We also introduce the Inverted Residual MLP block into PointNet++ to develop PointNeXt. We demonstrate that PointNeXt has superior performance and scalability over PointNet++ on various benchmarks while maintaining high throughput. This work aims to guide researchers toward paying more attention to the effects of training and scaling strategies and motivate future work in this direction. + +Limitation. Even though PointNeXt-XL is one of the largest models among all representative pointbased networks [\[30,](#page-11-1) [43,](#page-12-0) [15,](#page-10-5) [56\]](#page-12-1), its number of parameters (44M) is still below that of small networks in image classification such as Swin-S [\[25\]](#page-11-19) (50M), ConNeXt-S [\[26\]](#page-11-18) (50M), and ViT-B [\[8\]](#page-10-17) (87M), and is far from their large variants, including Swin-L (197M), ConvNeXt-XL (350M), and ViT-L (305M). In this work, we do not push the model size further, mainly due to the smaller-scale nature of point cloud datasets compared to their larger image counterparts, such as ImageNet [\[6\]](#page-10-18). Moreover, our work is limited to existing modules since the focus is not on introducing new architectural changes. + +Acknowledgement. The authors would like to thank the reviewers of NeurIPS'22 for their constructive suggestions. This work was supported by the King Abdullah University of Science and Technology (KAUST) Office of Sponsored Research through the Visual Computing Center (VCC) funding, as well as, the SDAIA-KAUST Center of Excellence in Data Science and Artificial Intelligence (SDAIA-KAUST AI). + +## References + +- [1] Iro Armeni, Ozan Sener, Amir R Zamir, Helen Jiang, Ioannis Brilakis, Martin Fischer, and Silvio Savarese. 3d semantic parsing of large-scale indoor spaces. In *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)*, pages 1534–1543, 2016. +- [2] Irwan Bello, William Fedus, Xianzhi Du, Ekin Dogus Cubuk, Aravind Srinivas, Tsung-Yi Lin, Jonathon Shlens, and Barret Zoph. Revisiting resnets: Improved training and scaling strategies. *Advances in Neural Information Processing Systems (NeurIPS)*, 34, 2021. +- [3] Angel X Chang, Thomas Funkhouser, Leonidas Guibas, Pat Hanrahan, Qixing Huang, Zimo Li, Silvio Savarese, Manolis Savva, Shuran Song, Hao Su, et al. Shapenet: An information-rich 3d model repository. *arXiv preprint arXiv:1512.03012*, 2015. +- [4] Christopher Choy, JunYoung Gwak, and Silvio Savarese. 4d spatio-temporal convnets: Minkowski convolutional neural networks. In *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)*, pages 3075–3084, 2019. +- [5] Angela Dai, Angel X. Chang, Manolis Savva, Maciej Halber, Thomas Funkhouser, and Matthias Nießner. ScanNet: Richly-annotated 3D reconstructions of indoor scenes. In *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)*, 2017. +- [6] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)*, pages 248–255. Ieee, 2009. +- [7] Xiaohan Ding, Xiangyu Zhang, Yizhuang Zhou, Jungong Han, Guiguang Ding, and Jian Sun. Scaling up your kernels to 31x31: Revisiting large kernel design in cnns. In *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)*, 2022. +- [8] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In *International Conference on Learning Representations (ICLR)*, 2021. +- [9] Ankit Goyal, Hei Law, Bowei Liu, Alejandro Newell, and Jia Deng. Revisiting point cloud shape classification with a simple and effective baseline. In *Proceedings of the International Conference on Machine Learning (ICML)*, pages 3809–3820. PMLR, 2021. +- [10] Benjamin Graham, Martin Engelcke, and Laurens Van Der Maaten. 3d semantic segmentation with submanifold sparse convolutional networks. In *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)*, pages 9224–9232, 2018. +- [11] Meng-Hao Guo, Jun-Xiong Cai, Zheng-Ning Liu, Tai-Jiang Mu, Ralph R Martin, and Shi-Min Hu. Pct: Point cloud transformer. *Computational Visual Media*, 7(2):187–199, 2021. +- [12] Abdullah Hamdi, Silvio Giancola, and Bernard Ghanem. Mvtn: Multi-view transformation network for 3d shape recognition. In *Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)*, pages 1–11, 2021. +- [13] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)*, pages 770–778, 2016. +- [14] Andrew G Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetto, and Hartwig Adam. Mobilenets: Efficient convolutional neural networks for mobile vision applications. *arXiv preprint arXiv:1704.04861*, 2017. +- [15] Qingyong Hu, Bo Yang, Linhai Xie, Stefano Rosa, Yulan Guo, Zhihua Wang, Niki Trigoni, and Andrew Markham. Randla-net: Efficient semantic segmentation of large-scale point clouds. In *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)*, pages 11108–11117, 2020. +- [16] Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In *International Conference on Learning Representations (ICLR)*, 2015. +- [17] Xin Lai, Jianhui Liu, Li Jiang, Liwei Wang, Hengshuang Zhao, Shu Liu, Xiaojuan Qi, and Jiaya Jia. Stratified transformer for 3d point cloud segmentation. In *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)*, 2022. +- [18] Zhaoqi Leng, Mingxing Tan, Chenxi Liu, Ekin Dogus Cubuk, Xiaojie Shi, Shuyang Cheng, and Drago Anguelov. Polyloss: A polynomial expansion perspective of classification loss functions. In *International Conference on Learning Representations (ICLR)*, 2022. +- [19] Guohao Li, Matthias Müller, Bernard Ghanem, and Vladlen Koltun. Training graph neural networks with 1000 layers. In *Proceedings of the International Conference on Machine Learning (ICML)*, volume 139, pages 6437–6449. PMLR, 2021. + +- [20] Guohao Li, Matthias Müller, Guocheng Qian, Itzel C. Delgadillo, Abdulellah Abualshour, Ali K. Thabet, and Bernard Ghanem. Deepgcns: Making gcns go as deep as cnns. *IEEE transactions on pattern analysis and machine intelligence (T-PAMI)*, PP, 2021. +- [21] Guohao Li, Matthias Muller, Ali Thabet, and Bernard Ghanem. Deepgcns: Can gcns go as deep as cnns? In *Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)*, pages 9267–9276, 2019. +- [22] Yangyan Li, Rui Bu, Mingchao Sun, Wei Wu, Xinhan Di, and Baoquan Chen. Pointcnn: Convolution on X -transformed points. *Advances in Neural Information Processing Systems (NeurIPS)*, 31, 2018. +- [23] Yongcheng Liu, Bin Fan, Shiming Xiang, and Chunhong Pan. Relation-shape convolutional neural network for point cloud analysis. In *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)*, pages 8887–8896, 2019. +- [24] Ze Liu, Han Hu, Yue Cao, Zheng Zhang, and Xin Tong. A closer look at local aggregation operators in point cloud analysis. In *Proceedings of the European Conference on Computer Vision (ECCV)*, pages 326–342. Springer, 2020. +- [25] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In *Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)*, pages 10012–10022, 2021. +- [26] Zhuang Liu, Hanzi Mao, Chao-Yuan Wu, Christoph Feichtenhofer, Trevor Darrell, and Saining Xie. A convnet for the 2020s. In *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)*, 2022. +- [27] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In *International Conference on Learning Representations (ICLR)*, 2019. +- [28] Xu Ma, Can Qin, Haoxuan You, Haoxi Ran, and Yun Fu. Rethinking network design and local geometry in point cloud: A simple residual MLP framework. In *International Conference on Learning Representations (ICLR)*, 2022. +- [29] Charles Ruizhongtai Qi, Hao Su, Kaichun Mo, and Leonidas J. Guibas. Pointnet: Deep learning on point sets for 3d classification and segmentation. In *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)*, 2017. +- [30] Charles Ruizhongtai Qi, Li Yi, Hao Su, and Leonidas J. Guibas. Pointnet++: Deep hierarchical feature learning on point sets in a metric space. In *Advances in Neural Information Processing Systems (NeurIPS)*, 2017. +- [31] Guocheng Qian, Abdulellah Abualshour, Guohao Li, Ali Thabet, and Bernard Ghanem. Pu-gcn: Point cloud upsampling using graph convolutional networks. In *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)*, pages 11683–11692, June 2021. +- [32] Guocheng Qian, Hasan Hammoud, Guohao Li, Ali Thabet, and Bernard Ghanem. Assanet: An anisotropic separable set abstraction for efficient point cloud representation learning. *Advances in Neural Information Processing Systems (NeurIPS)*, 34, 2021. +- [33] Shi Qiu, Saeed Anwar, and Nick Barnes. Semantic segmentation for real point cloud scenes via bilateral augmentation and adaptive fusion. In *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)*, pages 1757–1767, 2021. +- [34] Haoxi Ran, Jun Liu, and Chengjie Wang. Surface representation for point clouds. In *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)*, 2022. +- [35] Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In *International Conference on Medical image computing and computer-assisted intervention (MICCAI)*, 2015. +- [36] Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chieh Chen. Mobilenetv2: Inverted residuals and linear bottlenecks. In *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)*, pages 4510–4520, 2018. +- [37] Hang Su, Subhransu Maji, Evangelos Kalogerakis, and Erik G. Learned-Miller. Multi-view convolutional neural networks for 3d shape recognition. In *Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)*, 2015. +- [38] Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott E. Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. Going deeper with convolutions. In *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)*, 2015. +- [39] Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jon Shlens, and Zbigniew Wojna. Rethinking the inception architecture for computer vision. In *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)*, 2016. + +- [40] Mingxing Tan and Quoc V. Le. Efficientnet: Rethinking model scaling for convolutional neural networks. In *Proceedings of the International Conference on Machine Learning (ICML)*, volume 97, pages 6105–6114. PMLR, 2019. +- [41] Liyao Tang, Yibing Zhan, Zhe Chen, Baosheng Yu, and Dacheng Tao. Contrastive boundary learning for point cloud segmentation. In *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)*, 2022. +- [42] Maxim Tatarchenko, Jaesik Park, V. Koltun, and Qian-Yi Zhou. Tangent convolutions for dense prediction in 3d. In *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)*, pages 3887–3896, 2018. +- [43] Hugues Thomas, Charles R Qi, Jean-Emmanuel Deschaud, Beatriz Marcotegui, François Goulette, and Leonidas J Guibas. Kpconv: Flexible and deformable convolution for point clouds. In *Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)*, 2019. +- [44] Mikaela Angelina Uy, Quang-Hieu Pham, Binh-Son Hua, Duc Thanh Nguyen, and Sai-Kit Yeung. Revisiting point cloud classification: A new benchmark dataset and classification model on real-world data. In *Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)*, 2019. +- [45] Lei Wang, Yuchun Huang, Yaolin Hou, Shenman Zhang, and Jie Shan. Graph attention convolution for point cloud semantic segmentation. In *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)*, 2019. +- [46] Yue Wang, Yongbin Sun, Ziwei Liu, Sanjay E. Sarma, Michael M. Bronstein, and Justin M. Solomon. Dynamic graph cnn for learning on point clouds. *ACM Transactions on Graphics (TOG)*, 2019. +- [47] Ross Wightman, Hugo Touvron, and Hervé Jégou. Resnet strikes back: An improved training procedure in timm. *arXiv preprint arXiv:2110.00476*, 2021. +- [48] Wenxuan Wu, Zhongang Qi, and Li Fuxin. Pointconv: Deep convolutional networks on 3d point clouds. In *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)*, 2019. +- [49] Zhirong Wu, Shuran Song, Aditya Khosla, Fisher Yu, Linguang Zhang, Xiaoou Tang, and Jianxiong Xiao. 3d shapenets: A deep representation for volumetric shapes. In *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)*, 2015. +- [50] Tiange Xiang, Chaoyi Zhang, Yang Song, Jianhui Yu, and Weidong Cai. Walk in the cloud: Learning curves for point clouds shape analysis. In *Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)*, pages 915–924, 2021. +- [51] Yifan Xu, Tianqi Fan, Mingye Xu, L. Zeng, and Yu Qiao. Spidercnn: Deep learning on point sets with parameterized convolutional filters. In *Proceedings of the European Conference on Computer Vision (ECCV)*, 2018. +- [52] Xu Yan. Pointnet/pointnet++ pytorch. [https://github.com/yanx27/Pointnet\\_Pointnet2\\_](https://github.com/yanx27/Pointnet_Pointnet2_pytorch) [pytorch](https://github.com/yanx27/Pointnet_Pointnet2_pytorch), 2019. +- [53] Li Yi, Vladimir G Kim, Duygu Ceylan, I Shen, Mengyan Yan, Hao Su, ARCewu Lu, Qixing Huang, Alla Sheffer, Leonidas Guibas, et al. A scalable active framework for region annotation in 3d shape collections. *ACM Transactions on Graphics (TOG)*, 35(6):210, 2016. +- [54] Xumin Yu, Lulu Tang, Yongming Rao, Tiejun Huang, Jie Zhou, and Jiwen Lu. Point-bert: Pre-training 3d point cloud transformers with masked point modeling. In *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)*, 2022. +- [55] Xiaohua Zhai, Alexander Kolesnikov, Neil Houlsby, and Lucas Beyer. Scaling vision transformers. In *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)*, pages 12104–12113, 2022. +- [56] Hengshuang Zhao, Li Jiang, Jiaya Jia, Philip HS Torr, and Vladlen Koltun. Point transformer. In *Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)*, pages 16259–16268, 2021. + +# **PointNeXt: Revisiting PointNet++ with Improved Training and Scaling Strategies** + +## - Supplementary Material - + +In this appendix, we provide additional content to complement the main manuscript: + +- Appendix A: A detailed description of Tab. 7. +- Appendix B: Comparisons of training strategies for prior representative works and PointNeXt. +- Appendix C: Qualitative comparisons on S3DIS and ShapeNetPart. +- Appendix D: The architecture of PointNeXt for classification. +- Appendix E: Societal impact. + +## A Detailed Description for Manuscript Tab. 7 + +Naive width scaling increases the channel size of PointNet++ from 32 to 256 to match the throughput of the baseline model, PointNeXt-XL. Naive depth scaling refers to appending more SA blocks (B=(3,6,3,3)), the same as PointNext-XL) in PointNet++. Furthermore, naive compound scaling doubles the width of naive depth scaled model to the same as PointNeXt-XL (C=64). Compared to the PointNet++ trained with improved training strategies (63.2% mIoU, 186 ins./sec.), naive depth scaling (63.4% mIoU, 53 ins. / sec.) and naive width scaling (59.4% mIoU, 43 ins./sec.) only lead to a large overhead in throughput with insignificant improvement in accuracy. In contrast, our proposed model scaling strategy achieves much higher performance than the naive scaling strategies while being much faster. This can be observed by comparing PointNeXt-XL (70.5% mIoU, 45 ins./sec.) to the naive compound scaled PointNet++ (62.3% mIoU, 24 ins./sec.). + +#### **B** Training Strategies Comparison + +In this section, we summarize the training strategies used in representative point-based methods such as DGCNN [46], KPConv [43], PointMLP [28], Point Transformer [56], Stratified Transformer [17], PointNet++ [30], and our PointNeXt on S3DIS [1] in Tab. I, on ScanObjectNN [44] in Tab. II, on ScanNet [5] in Tab. III, and on ShapeNetPart [53] in Tab. IV, respectively. + +## C Qualitative Results + +We provide qualitative results of PointNeXt-XL for S3DIS (Fig. II) and PointNeXt-S (C=160) for ShapeNetPart (Fig. III). The qualitative results of PointNet++ trained with the original training strategies are also included in the figures for comparison. On both datasets, PointNeXt produces predictions closer to the ground truth compared to PointNet++. More specifically, on S3DIS shown in (Fig. II), PointNeXt is able to segment hard classes, including doors ( $1^{st}$ , $3^{rd}$ , and $4^{th}$ rows), clutter ( $1^{st}$ and $3^{rd}$ rows), chairs ( $2^{nd}$ row), and the board ( $4^{th}$ row), while PointNet++ fails to segment properly to some extent. On ShapeNetPart (Fig. III), PointNeXt precisely segments wings of an airplane ( $1^{st}$ row), microphone of an earphone( $2^{nd}$ row), body of a motorbike( $3^{rd}$ row), fin of a rocket( $4^{th}$ row), and bearing of a skateboard ( $5^{th}$ row). + +#### D Classification Architecture + +As illustrated in Fig. I, the classification architecture shares the same encoder as the segmentation one. The output features of the encoder are passed to a global pooling layer (*i.e.* global max-pooling) to acquire a global shape representation for classification. Note that the points are only downsampled by a factor of 2 in each stage, since the number of input points in classification tasks is usually small, *e.g.* 1024 or 2048 points. + +Table I: Training strategies used in different methods for S3DIS segmentation. + + + +| Method | DGCNN | KPConv | PointTransformer | PointNet++ | PointNeXt (Ours) | +|-------------------------------|--------------------|--------------------|------------------|--------------------|------------------| +| Epochs | 101 | 500 | 100 | 32 | 100 | +| Batch size | 12 | 10 | 16 | 16 | 8 | +| Optimizer | Adam | SGD | SGD | Adam | AdamW | +| LR | $1 \times 10^{-3}$ | $1 \times 10^{-2}$ | 0.5 | $1 \times 10^{-3}$ | 0.01 | +| LR decay | step | step | multi step | step | cosine | +| Weight decay | 0 | $10^{-3}$ | $10^{-4}$ | $10^{-4}$ | $10^{-4}$ | +| Label smoothing $\varepsilon$ | X | Х | × | × | 0.2 | +| Entire scene as input | X | Х | ✓ | × | ✓ | +| Random rotation | X | ✓ | X | ✓ | ✓ | +| Random scaling | X | [0.8, 1.2] | [0.9, 1.1] | X | [0.9, 1.1] | +| Random translation | X | X | X | X | X | +| Random jittering | X | 0.001 | X | X | ✓ | +| Height appending | X | ✓ | X | X | ✓ | +| Color drop | × | 0.2 | X | X | 0.2 | +| Color auto-contrast | X | × | ✓ | X | ✓ | +| Color jittering | X | X | ✓ | × | X | +| mIoU (%) | 56.1 | 70.6 | 73.5 | 54.5 | 74.9 | + +Table II: Training strategies used in different methods for ScanObecjectNN classification. + +| Method | DGCNN | PointMLP | PointNet++ | PointNeXt (Ours) | +|-------------------------------|--------------------|-----------|------------|--------------------| +| Epochs | 250 | 200 | 250 | 250 | +| Batch size | 32 | 32 | 16 | 32 | +| Optimizer | Adam | SGD | Adam | AdamW | +| LR | $1 \times 10^{-3}$ | 0.01 | $10^{-3}$ | $2 \times 10^{-3}$ | +| LR decay | step | cosine | step | cosine | +| Weight decay | $10^{-4}$ | $10^{-4}$ | $10^{-4}$ | 0.05 | +| Label smoothing $\varepsilon$ | 0.2 | 0.2 | × | 0.3 | +| Point resampling | Х | Х | × | ✓ | +| Random rotation | 1 | × | ✓ | ✓ | +| Random scaling | X | ✓ | X | ✓ | +| Random translation | X | ✓ | X | X | +| Random jittering | / | × | ✓ | X | +| Height appending | X | × | × | ✓ | +| OA (%) | 78.1 | 85.7 | 77.9 | 87.7 | + +## **E** Societal Impact + +We do not see an immediate negative societal impact from our work. We notice that the way we discover the improved training and scaling strategies may consume a little more computing resources and affect the environment. Nevertheless, the improved training and scaling strategies will make researchers pay more attention to aspects other than architectural changes, which in the long term makes research in computer vision more diverse and generally better. + +Table III: Training strategies used in different methods for ScanNet segmentation. + + + +| Method | KPConv | PointTransformer | Stratified Transformer | PointNet++ | PointNeXt (Ours) | +|-----------------------|--------------------|--------------------|-------------------------|--------------------|--------------------| +| Epochs | 500 | 100 | 100 | 200 | 100 | +| Batch size | 10 | 16 | 8 | 32 | 2 | +| Optimizer | SGD | SGD | AdamW | Adam | AdamW | +| LR | $1 \times 10^{-2}$ | $5 \times 10^{-1}$ | $6 \times 10^{-3}$ | $1 \times 10^{-3}$ | $1 \times 10^{-3}$ | +| LR decay | step | multi step | multi step with warm up | step | multi step | +| Weight decay | $10^{-3}$ | $10^{-4}$ | $5 \times 10^{-2}$ | $10^{-4}$ | $10^{-4}$ | +| Entire scene as input | X | 1 | / | × | / | +| Random rotation | 1 | X | ✓ | ✓ | ✓ | +| Random scaling | [0.9,1.1] | [0.9,1.1] | [0.8,1.2] | × | [0.8,1.2] | +| Random translation | X | X | × | × | Х | +| Random jittering | 0.001 | X | × | × | X | +| Height appending | / | X | × | × | ✓ | +| Color drop | X | X | 0.2 | × | 0.2 | +| Color auto-contrast | × | ✓ | × | × | ✓ | +| Color jittering | × | ✓ | × | / × | × | +| Test mIoU (%) | 68.6 | - | 73.7 | 55.7 | 71.2 | + +Table IV: Training strategies used in different methods for ShapeNetPart segmentation. + +| Method | DGCNN | KPConv | PointNet++ | PointNeXt (Ours) | +|-------------------------------|--------------------|--------------------|--------------------|------------------| +| Epochs | 201 | 500 | 201 | 300 | +| Batch size | 16 | 16 | 32 | 8 | +| Optimizer | Adam | SGD | Adam | AdamW | +| LR | $3 \times 10^{-3}$ | $1 \times 10^{-2}$ | $1 \times 10^{-3}$ | 0.001 | +| LR decay | step | step | step | multi step | +| Weight decay | 0.0 | $10^{-3}$ | 0.0 | $10^{-4}$ | +| Label smoothing $\varepsilon$ | X | X | × | × | +| Random rotation | Х | Х | × | ✓ | +| Random scaling | X | [0.9, 1.1] | X | [0.8, 1.2] | +| Random translation | X | × | X | X | +| Random jittering | X | 0.001 | ✓ | 0.001 | +| Normal Drop | X | X | X | ✓ | +| Height appending | X | ✓ | × | ✓ | +| mIoU (%) | 85.2 | 86.4 | 85.1 | 87.0 | + +![](PointNeXt_2206.04670_images/_page_15_Figure_4.jpeg) + +Figure I: **PointNeXt architecture for classification.** The classification architecture shares the same encoder as the segmentation architecture. + +> **[그림 해설]** PointNeXt의 3D 객체 분류(Classification) 아키텍처 다이어그램. +> - 입력 포인트 클라우드 $\to$ 초기 MLP $[N, 32]$. +> - 4단계 Set Abstraction + InvResMLP 인코더: $[N/2, 64] \to [N/4, 128] \to [N/8, 256] \to [N/16, 512]$로 계층적 특징 추출. +> - 최종단에 **Global Pooling**을 적용하여 전역 특징 벡터를 집계한 뒤 분류 점수를 도출 (세그멘테이션과 동일한 백본 인코더 설계 공유). + +![](PointNeXt_2206.04670_images/_page_16_Figure_0.jpeg) + +Figure II: Qualitative comparisons of PointNet++ ( $2^{nd}$ column), PointNeXt ( $3^{rd}$ column), and Ground Truth ( $4^{th}$ column) on S3DIS semantic segmentation. The input point cloud is visualized with original colors in the $1^{st}$ column. Differences between PointNet++ and PointNeXt are highlighted with red dash circles. Zoom-in for details. + +> **[그림 해설]** S3DIS 실내 씬 5개에 대한 시맨틱 세그멘테이션 정성적 결과 비교 (Input vs PointNet++ vs PointNeXt vs Ground Truth). +> - **색상 범례**: ceiling(초록), floor(파랑), wall(하늘), beam(노랑), column(자주), window(남색), door(황갈), table(보라), chair(빨강), sofa(연분홍), bookcase(청록), board(회색), clutter(검정). +> - 빨간 점선 원으로 표시된 확대 영역에서 PointNet++는 문(door), 의자(chair), 화이트보드(board) 경계를 벽면으로 오분류하거나 누락하는 반면, **PointNeXt**는 복잡한 가구의 세부 윤곽과 벽 부착물 경계를 Ground Truth와 거의 완벽히 일치하게 분할함. + +![](PointNeXt_2206.04670_images/_page_17_Figure_0.jpeg) + +Figure III: Qualitative comparisons of PointNet++ (left), PointNeXt (middle), and Ground Truth (right) on ShapeNetPart part segmentation. + +> **[그림 해설]** ShapeNetPart 3D 파트 분할 정성적 비교 (비행기, 헤드폰, 오토바이, 로켓, 스케이트보드 5개 객체). +> - **빨간 점선 원 표시 영역**: +> - 비행기 날개 밑 제트 엔진(노란색): PointNet++는 누락/과소예측하나 PointNeXt는 선명히 감지. +> - 헤드폰 이어패드와 헤드밴드 결합부: PointNeXt가 정밀한 경계 구분 성공. +> - 오토바이 핸들/미러 및 로켓 날개 핀, 스케이트보드 트럭(바퀴 축): PointNeXt가 미세한 구조적 파트를 GT와 일치하게 분할. \ No newline at end of file diff --git a/docs/papers/md/PointNeXt_2206.04670_images/_page_15_Figure_4.jpeg b/docs/papers/md/PointNeXt_2206.04670_images/_page_15_Figure_4.jpeg new file mode 100644 index 0000000..7a6fd3d Binary files /dev/null and b/docs/papers/md/PointNeXt_2206.04670_images/_page_15_Figure_4.jpeg differ diff --git a/docs/papers/md/PointNeXt_2206.04670_images/_page_16_Figure_0.jpeg b/docs/papers/md/PointNeXt_2206.04670_images/_page_16_Figure_0.jpeg new file mode 100644 index 0000000..a06af17 Binary files /dev/null and b/docs/papers/md/PointNeXt_2206.04670_images/_page_16_Figure_0.jpeg differ diff --git a/docs/papers/md/PointNeXt_2206.04670_images/_page_17_Figure_0.jpeg b/docs/papers/md/PointNeXt_2206.04670_images/_page_17_Figure_0.jpeg new file mode 100644 index 0000000..56d796a Binary files /dev/null and b/docs/papers/md/PointNeXt_2206.04670_images/_page_17_Figure_0.jpeg differ diff --git a/docs/papers/md/PointNeXt_2206.04670_images/_page_1_Figure_0.jpeg b/docs/papers/md/PointNeXt_2206.04670_images/_page_1_Figure_0.jpeg new file mode 100644 index 0000000..c4a1262 Binary files /dev/null and b/docs/papers/md/PointNeXt_2206.04670_images/_page_1_Figure_0.jpeg differ diff --git a/docs/papers/md/PointNeXt_2206.04670_images/_page_3_Figure_0.jpeg b/docs/papers/md/PointNeXt_2206.04670_images/_page_3_Figure_0.jpeg new file mode 100644 index 0000000..4f40367 Binary files /dev/null and b/docs/papers/md/PointNeXt_2206.04670_images/_page_3_Figure_0.jpeg differ diff --git a/docs/papers/md/PointNet++_1706.02413.md b/docs/papers/md/PointNet++_1706.02413.md new file mode 100644 index 0000000..9eb513d --- /dev/null +++ b/docs/papers/md/PointNet++_1706.02413.md @@ -0,0 +1,464 @@ +# PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Space + +Charles R. Qi Li Yi Hao Su Leonidas J. Guibas Stanford University + +# Abstract + +Few prior works study deep learning on point sets. PointNet [\[20\]](#page-8-0) is a pioneer in this direction. However, by design PointNet does not capture local structures induced by the metric space points live in, limiting its ability to recognize fine-grained patterns and generalizability to complex scenes. In this work, we introduce a hierarchical neural network that applies PointNet recursively on a nested partitioning of the input point set. By exploiting metric space distances, our network is able to learn local features with increasing contextual scales. With further observation that point sets are usually sampled with varying densities, which results in greatly decreased performance for networks trained on uniform densities, we propose novel set learning layers to adaptively combine features from multiple scales. Experiments show that our network called PointNet++ is able to learn deep point set features efficiently and robustly. In particular, results significantly better than state-of-the-art have been obtained on challenging benchmarks of 3D point clouds. + +# 1 Introduction + +We are interested in analyzing geometric point sets which are collections of points in a Euclidean space. A particularly important type of geometric point set is point cloud captured by 3D scanners, e.g., from appropriately equipped autonomous vehicles. As a set, such data has to be invariant to permutations of its members. In addition, the distance metric defines local neighborhoods that may exhibit different properties. For example, the density and other attributes of points may not be uniform across different locations — in 3D scanning the density variability can come from perspective effects, radial density variations, motion, etc. + +Few prior works study deep learning on point sets. PointNet [\[20\]](#page-8-0) is a pioneering effort that directly processes point sets. The basic idea of PointNet is to learn a spatial encoding of each point and then aggregate all individual point features to a global point cloud signature. By its design, PointNet does not capture local structure induced by the metric. However, exploiting local structure has proven to be important for the success of convolutional architectures. A CNN takes data defined on regular grids as the input and is able to progressively capture features at increasingly larger scales along a multi-resolution hierarchy. At lower levels neurons have smaller receptive fields whereas at higher levels they have larger receptive fields. The ability to abstract local patterns along the hierarchy allows better generalizability to unseen cases. + +We introduce a hierarchical neural network, named as PointNet++, to process a set of points sampled in a metric space in a hierarchical fashion. The general idea of PointNet++ is simple. We first partition the set of points into overlapping local regions by the distance metric of the underlying space. Similar to CNNs, we extract local features capturing fine geometric structures from small neighborhoods; such local features are further grouped into larger units and processed to produce higher level features. This process is repeated until we obtain the features of the whole point set. + +The design of PointNet++ has to address two issues: how to generate the partitioning of the point set, and how to abstract sets of points or local features through a local feature learner. The two issues + +are correlated because the partitioning of the point set has to produce common structures across partitions, so that weights of local feature learners can be shared, as in the convolutional setting. We choose our local feature learner to be PointNet. As demonstrated in that work, PointNet is an effective architecture to process an unordered set of points for semantic feature extraction. In addition, this architecture is robust to input data corruption. As a basic building block, PointNet abstracts sets of local points or features into higher level representations. In this view, PointNet++ applies PointNet recursively on a nested partitioning of the input set. + +One issue that still remains is how to generate overlapping partitioning of a point set. Each partition is defined as a neighborhood ball in the underlying Euclidean space, whose parameters include centroid location and scale. To evenly cover the whole set, the centroids are selected among input point set by a farthest point sampling (FPS) algorithm. Compared with volumetric CNNs that scan the space with fixed strides, our local receptive fields are dependent on both the input data and the metric, and thus more efficient and effective. + +![](PointNet++_1706.02413_images/_page_1_Picture_2.jpeg) + +Figure 1: Visualization of a scan captured from a Structure Sensor (left: RGB; right: point cloud). + +> **[그림 해설]** 실제 센서 측정 시 발생하는 포인트 클라우드의 불균일 밀도(Non-uniform density) 문제 시각화. +> - **좌측 (RGB 이미지)**: 실제 실내 환경 사진 (소파, 테이블, 의자). +> - **우측 (포인트 클라우드)**: 시점 및 거리에 따라 색상 코딩된 포인트 집합. 센서에 가까운 전경(파란색 영역)은 점 밀도가 매우 조밀한 반면, 거리가 먼 후경(연두색/흰색 영역)은 점이 희소하고 차폐(Occlusion)로 인한 결손이 크게 발생함을 보여줌. + +Deciding the appropriate scale of local neighborhood balls, however, is a more challenging yet intriguing problem, due to the entanglement of feature scale and non-uniformity of input point set. We assume that the input point set may have variable density at different areas, which is quite common in real data such as Structure Sensor scanning [18] (see Fig. 1). Our input point set is thus very different from CNN inputs which can be viewed as data defined on regular grids with uniform constant density. In CNNs, the counterpart to local partition scale is the size of kernels. [25] shows that using smaller kernels helps to improve the ability of CNNs. Our experiments on point set data, however, give counter evidence to this rule. Small neighborhood may consist of too few points due to sampling deficiency, which might be insufficient to allow PointNets to capture patterns robustly. + +A significant contribution of our paper is that PointNet++ leverages neighborhoods at multiple scales to achieve both robustness and detail capture. Assisted with random input dropout during training, the network learns to adaptively weight patterns detected at different scales and combine multi-scale features according to the input data. Experiments show that our PointNet++ is able to process point sets efficiently and robustly. In particular, results that are significantly better than state-of-the-art have been obtained on challenging benchmarks of 3D point clouds. + +#### 2 Problem Statement + +Suppose that $\mathcal{X}=(M,d)$ is a discrete metric space whose metric is inherited from a Euclidean space $\mathbb{R}^n$ , where $M\subseteq\mathbb{R}^n$ is the set of points and d is the distance metric. In addition, the density of M in the ambient Euclidean space may not be uniform everywhere. We are interested in learning set functions f that take such $\mathcal{X}$ as the input (along with additional features for each point) and produce information of semantic interest regrading $\mathcal{X}$ . In practice, such f can be classification function that assigns a label to $\mathcal{X}$ or a segmentation function that assigns a per point label to each member of M. + +#### 3 Method + +Our work can be viewed as an extension of PointNet [20] with added hierarchical structure. We first review PointNet (Sec. 3.1) and then introduce a basic extension of PointNet with hierarchical structure (Sec. 3.2). Finally, we propose our PointNet++ that is able to robustly learn features even in non-uniformly sampled point sets (Sec. 3.3). + +### 3.1 Review of PointNet [20]: A Universal Continuous Set Function Approximator + +Given an unordered point set $\{x_1, x_2, ..., x_n\}$ with $x_i \in \mathbb{R}^d$ , one can define a set function $f : \mathcal{X} \to \mathbb{R}$ that maps a set of points to a vector: + + +$$f(x_1, x_2, ..., x_n) = \gamma \left( \max_{i=1,...,n} \{ h(x_i) \} \right)$$ + (1) + +![](PointNet++_1706.02413_images/_page_2_Figure_0.jpeg) + +Figure 2: Illustration of our hierarchical feature learning architecture and its application for set segmentation and classification using points in 2D Euclidean space as an example. Single scale point grouping is visualized here. For details on density adaptive grouping, see Fig. 3 + +> **[그림 해설]** PointNet++의 계층적 신경망 구조도 (Set Abstraction 인코더와 태스크별 헤드). +> - **Hierarchical feature learning (좌측 회색 인코더 영역)**: +> - 입력: $(N, d+C)$ 포인트 ($d$차원 좌표 + $C$차원 특징). +> - **Set Abstraction (SA) 모듈**: Farthest Point Sampling(FPS)으로 중심점 선정 $ o$ 반경 기반 Grouping으로 국소 이웃 군집화 $ o$ 국소 PointNet을 적용해 특징 요약. +> - 점 수를 점진적으로 축소하며 다중 스케일 국소 기하 구조를 계층적으로 학습: $(N, d+C) \to (N_1, d+C_1) \to (N_2, d+C_2)$. +> - **Segmentation (우측 상단 디코더 영역)**: +> - Feature Propagation(FP) 계층: 역거리 가중치 기반 $k$-NN 보간(Interpolate)과 스킵 연결(Skip link concatenation)을 통해 축소된 해상도를 원래 포인트 개수 $N$으로 점진적 복원. +> - Unit PointNet(1x1 Conv)을 거쳐 최종 $(N, k)$ 점별 분류 점수(per-point scores) 도출. +> - **Classification (우측 하단 분류 헤드)**: +> - 마지막 SA 계층의 포인트들을 전체 집계하여 $(1, C_4)$ 글로벌 특징 벡터 생성 $ o$ Fully Connected 레이어 $ o (k)$개 클래스 점수 출력. + +where $\gamma$ and h are usually multi-layer perceptron (MLP) networks. + +The set function f in Eq. 1 is invariant to input point permutations and can arbitrarily approximate any continuous set function [20]. Note that the response of h can be interpreted as the spatial encoding of a point (see [20] for details). + +PointNet achieved impressive performance on a few benchmarks. However, it lacks the ability to capture local context at different scales. We will introduce a hierarchical feature learning framework in the next section to resolve the limitation. + +### 3.2 Hierarchical Point Set Feature Learning + +While PointNet uses a single max pooling operation to aggregate the whole point set, our new architecture builds a hierarchical grouping of points and progressively abstract larger and larger local regions along the hierarchy. + +Our hierarchical structure is composed by a number of *set abstraction* levels (Fig. 2). At each level, a set of points is processed and abstracted to produce a new set with fewer elements. The set abstraction level is made of three key layers: *Sampling layer*, *Grouping layer* and *PointNet layer*. The *Sampling layer* selects a set of points from input points, which defines the centroids of local regions. *Grouping layer* then constructs local region sets by finding "neighboring" points around the centroids. *PointNet layer* uses a mini-PointNet to encode local region patterns into feature vectors. + +A set abstraction level takes an $N \times (d+C)$ matrix as input that is from N points with d-dim coordinates and C-dim point feature. It outputs an $N' \times (d+C')$ matrix of N' subsampled points with d-dim coordinates and new C'-dim feature vectors summarizing local context. We introduce the layers of a set abstraction level in the following paragraphs. + +**Sampling layer.** Given input points $\{x_1, x_2, ..., x_n\}$ , we use iterative farthest point sampling (FPS) to choose a subset of points $\{x_{i_1}, x_{i_2}, ..., x_{i_m}\}$ , such that $x_{i_j}$ is the most distant point (in metric distance) from the set $\{x_{i_1}, x_{i_2}, ..., x_{i_{j-1}}\}$ with regard to the rest points. Compared with random sampling, it has better coverage of the entire point set given the same number of centroids. In contrast to CNNs that scan the vector space agnostic of data distribution, our sampling strategy generates receptive fields in a data dependent manner. + +**Grouping layer.** The input to this layer is a point set of size $N \times (d+C)$ and the coordinates of a set of centroids of size $N' \times d$ . The output are groups of point sets of size $N' \times K \times (d+C)$ , where each group corresponds to a local region and K is the number of points in the neighborhood of centroid points. Note that K varies across groups but the succeeding *PointNet layer* is able to convert flexible number of points into a fixed length local region feature vector. + +In convolutional neural networks, a local region of a pixel consists of pixels with array indices within certain Manhattan distance (kernel size) of the pixel. In a point set sampled from a metric space, the neighborhood of a point is defined by metric distance. + +Ball query finds all points that are within a radius to the query point (an upper limit of K is set in implementation). An alternative range query is K nearest neighbor (kNN) search which finds a fixed + +number of neighboring points. Compared with kNN, ball query's local neighborhood guarantees a fixed region scale thus making local region feature more generalizable across space, which is preferred for tasks requiring local pattern recognition (e.g. semantic point labeling). + +**PointNet layer.** In this layer, the input are N' local regions of points with data size $N' \times K \times (d+C)$ . Each local region in the output is abstracted by its centroid and local feature that encodes the centroid's neighborhood. Output data size is $N' \times (d+C')$ . + +The coordinates of points in a local region are firstly translated into a local frame relative to the centroid point: $x_i^{(j)} = x_i^{(j)} - \hat{x}^{(j)}$ for i=1,2,...,K and j=1,2,...,d where $\hat{x}$ is the coordinate of the centroid. We use PointNet [20] as described in Sec. 3.1 as the basic building block for local pattern learning. By using relative coordinates together with point features we can capture point-to-point relations in the local region. + +# 3.3 Robust Feature Learning under Non-Uniform Sampling Density + +As discussed earlier, it is common that a point set comes with non-uniform density in different areas. Such non-uniformity introduces a significant challenge for point set feature learning. Features learned in dense data may not generalize to sparsely sampled regions. Consequently, models trained for sparse point cloud may not recognize fine-grained local structures. + +Ideally, we want to inspect as closely as possible into a point set to capture finest details in densely sampled regions. However, such close inspect is prohibited at low density areas because local patterns may be corrupted by the sampling deficiency. In this case, we should look for larger scale patterns in greater vicinity. To achieve this goal we propose density adaptive PointNet layers (Fig. 3) that learn to + +![](PointNet++_1706.02413_images/_page_3_Figure_6.jpeg) + +Figure 3: (a) Multi-scale grouping (MSG); (b) Multi-resolution grouping (MRG). + +> **[그림 해설]** 밀도 불균일성에 대응하는 밀도 적응형 계층(Density Adaptive Layers) 2종 다이어그램. +> - **(a) Multi-scale grouping (MSG)**: 단일 중심점에 대해 크기가 다른 여러 동심 반경 구(sphere)를 정의하고, 각 반경별 국소 영역에서 PointNet 특징을 따로 추출한 뒤 하나로 결합(concat). 다양한 스케일의 기하 정보를 동시에 포착. +> - **(b) Multi-resolution grouping (MRG)**: 연산량을 줄이기 위해 이전 계층(하위 해상도)에서 이미 요약된 특징 벡터 집합과, 원본 국소 영역 포인트들로부터 직접 추출한 특징을 결합(concat). 저밀도 영역에서는 큰 영역 특징을, 고밀도 영역에서는 세밀한 국소 특징을 적응적으로 활용. + +combine features from regions of different scales when the input sampling density changes. We call our hierarchical network with density adaptive PointNet layers as *PointNet++*. + +Previously in Sec. 3.2, each abstraction level contains grouping and feature extraction of a single scale. In PointNet++, each abstraction level extracts multiple scales of local patterns and combine them intelligently according to local point densities. In terms of grouping local regions and combining features from different scales, we propose two types of density adaptive layers as listed below. + +**Multi-scale grouping (MSG).** As shown in Fig. 3 (a), a simple but effective way to capture multi-scale patterns is to apply grouping layers with different scales followed by according PointNets to extract features of each scale. Features at different scales are concatenated to form a multi-scale feature. + +We train the network to learn an optimized strategy to combine the multi-scale features. This is done by randomly dropping out input points with a randomized probability for each instance, which we call random input dropout. Specifically, for each training point set, we choose a dropout ratio $\theta$ uniformly sampled from [0,p] where $p \leq 1$ . For each point, we randomly drop a point with probability $\theta$ . In practice we set p=0.95 to avoid generating empty point sets. In doing so we present the network with training sets of various sparsity (induced by $\theta$ ) and varying uniformity (induced by randomness in dropout). During test, we keep all available points. + +**Multi-resolution grouping (MRG).** The MSG approach above is computationally expensive since it runs local PointNet at large scale neighborhoods for every centroid point. In particular, since the number of centroid points is usually quite large at the lowest level, the time cost is significant. + +Here we propose an alternative approach that avoids such expensive computation but still preserves the ability to adaptively aggregate information according to the distributional properties of points. In Fig. 3 (b), features of a region at some level $L_i$ is a concatenation of two vectors. One vector (left in figure) is obtained by summarizing the features at each subregion from the lower level $L_{i-1}$ using the set abstraction level. The other vector (right) is the feature that is obtained by directly processing all raw points in the local region using a single PointNet. + +When the density of a local region is low, the first vector may be less reliable than the second vector, since the subregion in computing the first vector contains even sparser points and suffers more from sampling deficiency. In such a case, the second vector should be weighted higher. On the other hand, + +when the density of a local region is high, the first vector provides information of finer details since it possesses the ability to inspect at higher resolutions recursively in lower levels. + +Compared with MSG, this method is computationally more efficient since we avoids the feature extraction in large scale neighborhoods at lowest levels. + +### 3.4 Point Feature Propagation for Set Segmentation + +In set abstraction layer, the original point set is subsampled. However in set segmentation task such as semantic point labeling, we want to obtain point features for *all* the original points. One solution is to always sample all points as centroids in all set abstraction levels, which however results in high computation cost. Another way is to propagate features from subsampled points to the original points. + +We adopt a hierarchical propagation strategy with distance based interpolation and across level skip links (as shown in Fig. [2\)](#page-2-1). In a *feature propagation* level, we propagate point features from Nl × (d + C) points to Nl−1 points where Nl−1 and Nl (with Nl ≤ Nl−1) are point set size of input and output of set abstraction level l. We achieve feature propagation by interpolating feature values f of Nl points at coordinates of the Nl−1 points. Among the many choices for interpolation, we use inverse distance weighted average based on k nearest neighbors (as in Eq. [2,](#page-4-0) in default we use p = 2, k = 3). The interpolated features on Nl−1 points are then concatenated with skip linked point features from the set abstraction level. Then the concatenated features are passed through a "unit pointnet", which is similar to one-by-one convolution in CNNs. A few shared fully connected and ReLU layers are applied to update each point's feature vector. The process is repeated until we have propagated features to the original set of points. + + +$$f^{(j)}(x) = \frac{\sum_{i=1}^{k} w_i(x) f_i^{(j)}}{\sum_{i=1}^{k} w_i(x)} \quad \text{where} \quad w_i(x) = \frac{1}{d(x, x_i)^p}, \ j = 1, ..., C$$ + (2) + +# 4 Experiments + +Datasets We evaluate on four datasets ranging from 2D objects (MNIST [\[11\]](#page-8-3)), 3D objects (Model-Net40 [\[31\]](#page-9-0) rigid object, SHREC15 [\[12\]](#page-8-4) non-rigid object) to real 3D scenes (ScanNet [\[5\]](#page-8-5)). Object classification is evaluated by accuracy. Semantic scene labeling is evaluated by average voxel classification accuracy following [\[5\]](#page-8-5). We list below the experiment setting for each dataset: + +- MNIST: Images of handwritten digits with 60k training and 10k testing samples. +- ModelNet40: CAD models of 40 categories (mostly man-made). We use the official split with 9,843 shapes for training and 2,468 for testing. +- SHREC15: 1200 shapes from 50 categories. Each category contains 24 shapes which are mostly organic ones with various poses such as horses, cats, etc. We use five fold cross validation to acquire classification accuracy on this dataset. +- ScanNet: 1513 scanned and reconstructed indoor scenes. We follow the experiment setting in [\[5\]](#page-8-5) and use 1201 scenes for training, 312 scenes for test. + +### 4.1 Point Set Classification in Euclidean Metric Space + +We evaluate our network on classifying point clouds sampled from both 2D (MNIST) and 3D (ModleNet40) Euclidean spaces. MNIST images are converted to 2D point clouds of digit pixel locations. 3D point clouds are sampled from mesh surfaces from ModelNet40 shapes. In default we use 512 points for MNIST and 1024 points for ModelNet40. In last row (ours normal) in Table [2,](#page-5-0) we use face normals as additional point features, where we also use more points (N = 5000) to further boost performance. All point sets are normalized to be zero mean and within a unit ball. We use a three-level hierarchical network with three fully connected layers [1](#page-4-1) + +Results. In Table [1](#page-5-1) and Table [2,](#page-5-0) we compare our method with a representative set of previous state of the arts. Note that PointNet (vanilla) in Table [2](#page-5-0) is the the version in [\[20\]](#page-8-0) that does not use transformation networks, which is equivalent to our hierarchical net with only one level. + +Firstly, our hierarchical learning architecture achieves significantly better performance than the non-hierarchical PointNet [\[20\]](#page-8-0). In MNIST, we see a relative 60.8% and 34.6% error rate reduction + +1 See supplementary for more details on network architecture and experiment preparation. + +| Method | Error rate (%) | +|-----------------------------|----------------| +| Multi-layer perceptron [24] | 1.60 | +| LeNet5 [11] | 0.80 | +| Network in Network [13] | 0.47 | +| PointNet (vanilla) [20] | 1.30 | +| PointNet [20] | 0.78 | +| Ours | 0.51 | + + + +| Table 1: | MNIST digi | t classification. | +|----------|------------|-------------------| + +| Input | Accuracy (%) | +|-------|------------------------| +| vox | 89.2 | +| img | 90.1 | +| pc | 87.2 | +| pc | 89.2 | +| рс | 90.7 | +| pc | 91.9 | +| | vox
img
pc
pc | + +Table 2: ModelNet40 shape classification. + +![](PointNet++_1706.02413_images/_page_5_Figure_4.jpeg) + +> **[그림 해설]** 무작위 점 탈락(Random Point Dropout)을 적용한 의자 포인트 클라우드 시각화. 1024개에서 512, 256, 128개로 점 수가 급감함에 따라 형상의 디테일이 점차 희소해지는 과정을 보여준다. + +![](PointNet++_1706.02413_images/_page_5_Figure_5.jpeg) + +Figure 4: Left: Point cloud with random point dropout. Right: Curve showing advantage of our density adaptive strategy in dealing with non-uniform density. DP means random input dropout during training; otherwise training is on uniformly dense points. See Sec.3.3 for details. + +> **[그림 해설]** 테스트 시 점 개수 감소(1000 $\to$ 128개)에 따른 ModelNet40 분류 정확도(Accuracy %) 평가 곡선. +> - DP(Random Point Dropout) 없이 학습한 모델: PointNet vanilla(파란색)와 Ours SSG(주황색)는 점 수가 줄어들면 500개 미만에서 성능이 75% 이하로 급격히 추락함. +> - DP 적용 모델: PointNet vanilla(DP, 초록) 및 Ours SSG+DP(노랑)는 점 감소에 대해 상대적으로 안정적임. +> - 밀도 적응형 모델: **Ours MSG+DP(빨간색)** 및 **Ours MRG+DP(청록색)**는 1000개 포인트에서 90% 이상의 최고 성능을 기록하며, 점 개수가 256개 이하로 떨어져도 88~89% 이상의 높은 정확도를 견고하게 유지하여 밀도 적응 전략의 우수성을 입증. + +from PointNet (vanilla) and PointNet to our method. In ModelNet40 classification, we also see that using same input data size (1024 points) and features (coordinates only), ours is remarkably stronger than PointNet. Secondly, we observe that point set based method can even achieve better or similar performance as mature image CNNs. In MNIST, our method (based on 2D point set) is achieving an accuracy close to the Network in Network CNN. In ModelNet40, ours with normal information significantly outperforms previous state-of-the-art method MVCNN [26]. + +**Robustness to Sampling Density Variation.** Sensor data directly captured from real world usually suffers from severe irregular sampling issues (Fig. 1). Our approach selects point neighborhood of multiple scales and learns to balance the descriptiveness and robustness by properly weighting them. + +We randomly drop points (see Fig. 4 left) during test time to validate our network's robustness to non-uniform and sparse data. In Fig. 4 right, we see MSG+DP (multi-scale grouping with random input dropout during training) and MRG+DP (multi-resolution grouping with random input dropout during training) are very robust to sampling density variation. MSG+DP performance drops by less than 1% from 1024 to 256 test points. Moreover, it achieves the best performance on almost all sampling densities compared with alternatives. PointNet vanilla [20] is fairly robust under density variation due to its focus on global abstraction rather than fine details. However loss of details also makes it less powerful compared to our approach. SSG (ablated PointNet++ with single scale grouping in each level) fails to generalize to sparse sampling density while SSG+DP amends the problem by randomly dropping out points in training time. + +## 4.2 Point Set Segmentation for Semantic Scene Labeling + +To validate that our approach is suitable for large scale point cloud analysis, we also evaluate on semantic scene labeling task. The goal is to predict semantic object label for points in indoor scans. [5] provides a baseline using fully convolutional neural network on voxelized scans. They purely rely on scanning geometry instead of RGB information and report the accuracy on a per-voxel basis. To make a fair comparison, + +![](PointNet++_1706.02413_images/_page_5_Figure_12.jpeg) + +Figure 5: Scannet labeling accuracy. + +> **[그림 해설]** ScanNet 3D 시맨틱 세그멘테이션 복셀 레이블링 정확도(Accuracy) 비교 막대 그래프. +> - **ScanNet (균일 샘플링, 파란색)**: 3DCNN(0.730), PointNet(0.739), Ours SSG(0.833), Ours MSG+DP(0.845), Ours MRG+DP(0.834). +> - **ScanNet non-uniform (불균일 밀도, 노란색)**: PointNet(0.680, 0.059 하락), Ours SSG(0.727, 0.106 하락), **Ours MSG+DP(0.804, 0.041 하락으로 불균일 환경에서 최고 성능 및 최고 강건성 달성)**, Ours MRG+DP(0.762). + +we remove RGB information in all our experiments and convert point cloud label prediction into voxel labeling following [5]. We also compare with [20]. The accuracy is reported on a per-voxel basis in Fig. 5 (blue bar). + +Our approach outperforms all the baseline methods by a large margin. In comparison with [5], which learns on voxelized scans, we directly learn on point clouds to avoid additional quantization error, + +and conduct data dependent sampling to allow more effective learning. Compared with [20], our approach introduces hierarchical feature learning and captures geometry features at different scales. This is very important for understanding scenes at multiple levels and labeling objects with various sizes. We visualize example scene labeling results in Fig. 6. + +Robustness to Sampling Density Variation To test how our trained model performs on scans with non-uniform sampling density, we synthesize virtual scans of Scannet scenes similar to that in Fig. 1 and evaluate our network on this data. We refer readers to supplementary material for how we generate the virtual scans. We evaluate our framework in three settings (SSG, MSG+DP, MRG+DP) and compare with a baseline approach [20]. + +Performance comparison is shown in Fig. 5 (yellow bar). We see that SSG performance greatly falls due to the sampling density shift from uniform point cloud to virtually scanned scenes. MRG network, on the other hand, is more robust to the sampling density shift since it is able to automatically switch to features depicting coarser granularity when the sampling is sparse. Even though there is a domain + +![](PointNet++_1706.02413_images/_page_6_Figure_3.jpeg) + +• Wall • Floor • Chair • Desk • Bed • Door • Table Figure 6: Scannet labeling results. [20] captures the overall layout of the room correctly but fails to discover the furniture. Our approach, in contrast, is much better at segmenting objects besides the room layout. + +> **[그림 해설]** ScanNet 실내 씬 2개에 대한 시맨틱 분할 정성적 결과 비교 (PointNet vs Ours vs Ground Truth). +> - **색상 범례**: Wall(빨강), Floor(노랑), Chair(초록), Desk(파랑), Bed(보라), Door(하늘), Table(분홍). +> - PointNet(좌측)은 방의 기본 외곽(벽, 바닥)만 대략적으로 구분하고 실내 가구 객체들을 놓치거나 뭉개는 반면, **Ours(PointNet++, 중앙)**는 책상(파랑), 침대(보라), 의자(초록) 등 세부 가구들의 경계를 Ground Truth(우측)와 거의 일치하게 정확히 분할해냄. + +gap between training data (uniform points with random dropout) and scanned data with non-uniform density, our MSG network is only slightly affected and achieves the best accuracy among methods in comparison. These prove the effectiveness of our density adaptive layer design. + +#### 4.3 Point Set Classification in Non-Euclidean Metric Space + +In this section, we show generalizability of our approach to non-Euclidean space. In non-rigid shape classification (Fig. 7), a good classifier should be able to classify (a) and (c) in Fig. 7 correctly as the same category even given their difference in pose, which requires knowledge of intrinsic structure. Shapes in SHREC15 are 2D surfaces embedded in 3D space. Geodesic distances along the surfaces naturally induce a metric space. We show through experiments that adopting PointNet++ in this metric space is an effective way to capture intrinsic structure of the underlying point set. + +For each shape in [12], we firstly construct the metric space induced by pairwise geodesic distances. We follow [23] to obtain an embedding metric that mimics geodesic distance. Next we extract intrinsic point features in this metric space including WKS [1], HKS [27] and multi-scale Gaussian curvature [16]. We use these features as input and then sample and group points according to the underlying metric space. In this way, our network learns to capture multi-scale intrinsic structure that is not influenced by the specific pose of a shape. Alternative design choices include using XYZ coordinates as points feature or use Euclidean space $\mathbb{R}^3$ as the underlying metric space. We show below these are not optimal choices. + +![](PointNet++_1706.02413_images/_page_6_Figure_9.jpeg) + +(a) Horse (b) Cat (c) Horse Figure 7: An example of non-rigid shape classification. + +> **[그림 해설]** 비강체(Non-rigid) 3D 형상 분류 예시 (SHREC15 데이터셋). +> - (a) 뒷다리로 일어선 말(Horse), (b) 뒷다리로 일어선 고양이(Cat), (c) 네 발로 선 말(Horse). +> - 유클리드 좌표($XYZ$)만 사용하면 포즈가 유사한 (a)와 (b)를 같은 클래스로 오인하기 쉬우나, 표면 측지선 거리(Geodesic metric) 기반 내재적 특징(Intrinsic features)을 활용함으로써 포즈 변화가 극심한 (a)와 (c)를 동일한 말(Horse) 카테고리로 올바르게 분류 가능함을 설명. + +**Results.** We compare our methods with previous state-of-the-art method [14] in Table 3. [14] extracts geodesic moments as shape features and use a stacked sparse autoencoder to digest these features to predict shape category. Our approach using non-Euclidean metric space and intrinsic features achieves the best performance in all settings and outperforms [14] by a large margin. + +Comparing the first and second setting of our approach, we see intrinsic features are very important for non-rigid shape classification. XYZ feature fails to reveal intrinsic structures and is greatly influenced by pose variation. Comparing the second and third setting of our approach, we see using geodesic neighborhood is beneficial compared with Euclidean neighborhood. Euclidean neighborhood might include points far away on surfaces and this neighborhood could change dramatically when shape affords non-rigid deformation. This introduces difficulty for effective weight sharing since the local structure could become combinatorially complicated. Geodesic neighborhood on surfaces, on the other hand, gets rid of this issue and improves the learning effectiveness. + +| | Metric space | Input feature | Accuracy (%) | +|-------------|-----------------------------------------|-------------------------------------------------|--------------------------------| +| DeepGM [14] | - | Intrinsic features | 93.03 | +| Ours | Euclidean
Euclidean
Non-Euclidean | XYZ
Intrinsic features
Intrinsic features | 60.18
94.49
96.09 | + +Table 3: SHREC15 Non-rigid shape classification. + +#### 4.4 Feature Visualization. + +In Fig. 8 we visualize what has been learned by the first level kernels of our hierarchical network. We created a voxel grid in space and aggregate local point sets that activate certain neurons the most in grid cells (highest 100 examples are used). Grid cells with high votes are kept and converted back to 3D point clouds, which represents the pattern that neuron recognizes. Since the model is trained on ModelNet40 which is mostly consisted of furniture, we see structures of planes, double planes, lines, corners etc. in the visualization. + +![](PointNet++_1706.02413_images/_page_7_Figure_4.jpeg) + +### 5 Related Work + +The idea of hierarchical feature learning has been very successful. Among all the learning models, convolutional neural network [10, 25, 8] is one of the most prominent ones. However, convolution does not apply to unordered point sets with distance metrics, which is the focus of our work. + +Figure 8: 3D point cloud patterns learned from the first layer kernels. The model is trained for ModelNet40 shape classification (20 out of the 128 kernels are randomly selected). Color indicates point depth (red is near, blue is far). + +> **[그림 해설]** PointNet++ 1단계 계층의 커널들이 학습한 국소 3D 포인트 클라우드 기하학적 패턴 시각화 (무작위 20개 커널, 깊이에 따라 빨강(근거리)~파랑(원거리) 코딩). +> - 평면 조각, 원통/기둥, 모서리(Edge), 구면 곡면, 이중 원반(Disk), 쐐기형 코너 등 3D 공간을 구성하는 다양한 기본 기하 요소(Geometric primitives)를 1차 레이어에서 감지하도록 성공적으로 학습됨을 확인. + +A few very recent works [20, 28] have studied how to apply deep learning to unordered sets. They ignore the underlying distance metric even if the point set does possess one. As a result, they are unable to capture local context of points and are sensitive to global set translation and normalization. In this work, we target at points sampled from a metric space and tackle these issues by explicitly considering the underlying distance metric in our design. + +Point sampled from a metric space are usually noisy and with non-uniform sampling density. This affects effective point feature extraction and causes difficulty for learning. One of the key issue is to select proper scale for point feature design. Previously several approaches have been developed regarding this [19, 17, 2, 6, 7, 30] either in geometry processing community or photogrammetry and remote sensing community. In contrast to all these works, our approach learns to extract point features and balance multiple feature scales in an end-to-end fashion. + +In 3D metric space, other than point set, there are several popular representations for deep learning, including volumetric grids [21, 22, 29], and geometric graphs [3, 15, 33]. However, in none of these works, the problem of non-uniform sampling density has been explicitly considered. + +#### 6 Conclusion + +In this work, we propose PointNet++, a powerful neural network architecture for processing point sets sampled in a metric space. PointNet++ recursively functions on a nested partitioning of the input point set, and is effective in learning hierarchical features with respect to the distance metric. To handle the non uniform point sampling issue, we propose two novel set abstraction layers that intelligently aggregate multi-scale information according to local point densities. These contributions enable us to achieve state-of-the-art performance on challenging benchmarks of 3D point clouds. + +In the future, it's worthwhile thinking how to accelerate inference speed of our proposed network especially for MSG and MRG layers by sharing more computation in each local regions. It's also interesting to find applications in higher dimensional metric spaces where CNN based method would be computationally unfeasible while our method can scale well. + +# References + +- [1] M. Aubry, U. Schlickewei, and D. Cremers. The wave kernel signature: A quantum mechanical approach to shape analysis. In *Computer Vision Workshops (ICCV Workshops), 2011 IEEE International Conference on*, pages 1626–1633. IEEE, 2011. +- [2] D. Belton and D. D. Lichti. Classification and segmentation of terrestrial laser scanner point clouds using local variance information. *Iaprs, Xxxvi*, 5:44–49, 2006. +- [3] J. Bruna, W. Zaremba, A. Szlam, and Y. LeCun. Spectral networks and locally connected networks on graphs. *arXiv preprint arXiv:1312.6203*, 2013. +- [4] A. X. Chang, T. Funkhouser, L. Guibas, P. Hanrahan, Q. Huang, Z. Li, S. Savarese, M. Savva, S. Song, H. Su, J. Xiao, L. Yi, and F. Yu. ShapeNet: An Information-Rich 3D Model Repository. Technical Report arXiv:1512.03012 [cs.GR], 2015. +- [5] A. Dai, A. X. Chang, M. Savva, M. Halber, T. Funkhouser, and M. Nießner. Scannet: Richly-annotated 3d reconstructions of indoor scenes. *arXiv preprint arXiv:1702.04405*, 2017. +- [6] J. Demantké, C. Mallet, N. David, and B. Vallet. Dimensionality based scale selection in 3d lidar point clouds. *The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences*, 38(Part 5):W12, 2011. +- [7] A. Gressin, C. Mallet, J. Demantké, and N. David. Towards 3d lidar point cloud registration improvement using optimal neighborhood knowledge. *ISPRS journal of photogrammetry and remote sensing*, 79:240– 251, 2013. +- [8] K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In *Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition*, pages 770–778, 2016. +- [9] D. Kingma and J. Ba. Adam: A method for stochastic optimization. *arXiv preprint arXiv:1412.6980*. +- [10] A. Krizhevsky, I. Sutskever, and G. E. Hinton. Imagenet classification with deep convolutional neural networks. In *Advances in neural information processing systems*, pages 1097–1105, 2012. +- [11] Y. LeCun, L. Bottou, Y. Bengio, and P. Haffner. Gradient-based learning applied to document recognition. *Proceedings of the IEEE*, 86(11):2278–2324, 1998. +- [12] Z. Lian, J. Zhang, S. Choi, H. ElNaghy, J. El-Sana, T. Furuya, A. Giachetti, R. A. Guler, L. Lai, C. Li, H. Li, F. A. Limberger, R. Martin, R. U. Nakanishi, A. P. Neto, L. G. Nonato, R. Ohbuchi, K. Pevzner, D. Pickup, P. Rosin, A. Sharf, L. Sun, X. Sun, S. Tari, G. Unal, and R. C. Wilson. Non-rigid 3D Shape Retrieval. In I. Pratikakis, M. Spagnuolo, T. Theoharis, L. V. Gool, and R. Veltkamp, editors, *Eurographics Workshop on 3D Object Retrieval*. The Eurographics Association, 2015. +- [13] M. Lin, Q. Chen, and S. Yan. Network in network. *arXiv preprint arXiv:1312.4400*, 2013. +- [14] L. Luciano and A. B. Hamza. Deep learning with geodesic moments for 3d shape classification. *Pattern Recognition Letters*, 2017. +- [15] J. Masci, D. Boscaini, M. Bronstein, and P. Vandergheynst. Geodesic convolutional neural networks on riemannian manifolds. In *Proceedings of the IEEE International Conference on Computer Vision Workshops*, pages 37–45, 2015. +- [16] M. Meyer, M. Desbrun, P. Schröder, A. H. Barr, et al. Discrete differential-geometry operators for triangulated 2-manifolds. *Visualization and mathematics*, 3(2):52–58, 2002. +- [17] N. J. MITRA, A. NGUYEN, and L. GUIBAS. Estimating surface normals in noisy point cloud data. *International Journal of Computational Geometry & Applications*, 14(04n05):261–276, 2004. +- [18] I. Occipital. Structure sensor-3d scanning, augmented reality, and more for mobile devices, 2016. +- [19] M. Pauly, L. P. Kobbelt, and M. Gross. Point-based multiscale surface representation. *ACM Transactions on Graphics (TOG)*, 25(2):177–193, 2006. +- [20] C. R. Qi, H. Su, K. Mo, and L. J. Guibas. Pointnet: Deep learning on point sets for 3d classification and segmentation. *arXiv preprint arXiv:1612.00593*, 2016. +- [21] C. R. Qi, H. Su, M. Nießner, A. Dai, M. Yan, and L. Guibas. Volumetric and multi-view cnns for object classification on 3d data. In *Proc. Computer Vision and Pattern Recognition (CVPR), IEEE*, 2016. +- [22] G. Riegler, A. O. Ulusoys, and A. Geiger. Octnet: Learning deep 3d representations at high resolutions. *arXiv preprint arXiv:1611.05009*, 2016. +- [23] R. M. Rustamov, Y. Lipman, and T. Funkhouser. Interior distance using barycentric coordinates. In *Computer Graphics Forum*, volume 28, pages 1279–1288. Wiley Online Library, 2009. +- [24] P. Y. Simard, D. Steinkraus, and J. C. Platt. Best practices for convolutional neural networks applied to visual document analysis. In *ICDAR*, volume 3, pages 958–962, 2003. +- [25] K. Simonyan and A. Zisserman. Very deep convolutional networks for large-scale image recognition. *arXiv preprint arXiv:1409.1556*, 2014. +- [26] H. Su, S. Maji, E. Kalogerakis, and E. G. Learned-Miller. Multi-view convolutional neural networks for 3d shape recognition. In *Proc. ICCV, to appear*, 2015. +- [27] J. Sun, M. Ovsjanikov, and L. Guibas. A concise and provably informative multi-scale signature based on heat diffusion. In *Computer graphics forum*, volume 28, pages 1383–1392. Wiley Online Library, 2009. +- [28] O. Vinyals, S. Bengio, and M. Kudlur. Order matters: Sequence to sequence for sets. *arXiv preprint arXiv:1511.06391*, 2015. + +- [29] P.-S. WANG, Y. LIU, Y.-X. GUO, C.-Y. SUN, and X. TONG. O-cnn: Octree-based convolutional neural networks for 3d shape analysis. 2017. +- [30] M. Weinmann, B. Jutzi, S. Hinz, and C. Mallet. Semantic point cloud interpretation based on optimal neighborhoods, relevant features and efficient classifiers. *ISPRS Journal of Photogrammetry and Remote Sensing*, 105:286–304, 2015. +- [31] Z. Wu, S. Song, A. Khosla, F. Yu, L. Zhang, X. Tang, and J. Xiao. 3d shapenets: A deep representation for volumetric shapes. In *Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition*, pages 1912–1920, 2015. +- [32] L. Yi, V. G. Kim, D. Ceylan, I.-C. Shen, M. Yan, H. Su, C. Lu, Q. Huang, A. Sheffer, and L. Guibas. A scalable active framework for region annotation in 3d shape collections. *SIGGRAPH Asia*, 2016. +- [33] L. Yi, H. Su, X. Guo, and L. Guibas. Syncspeccnn: Synchronized spectral cnn for 3d shape segmentation. *arXiv preprint arXiv:1612.00606*, 2016. + +# Supplementary + +# A Overview + +This supplementary material provides more details on experiments in the main paper and includes more experiments to validate and analyze our proposed method. + +In Sec [B](#page-10-0) we provide specific network architectures used for experiments in the main paper and also describe details in data preparation and training. In Sec [C](#page-12-0) we show more experimental results including benchmark performance on part segmentation and analysis on neighborhood query, sensitivity to sampling randomness and time space complexity. + +# B Details in Experiments + +Architecture protocol. We use following notations to describe our network architecture. + +SA(K,r,[l1, ..., ld]) is a set abstraction (SA) level with K local regions of ball radius r using PointNet of d fully connected layers with width li (i = 1, ..., d). SA([l1, ...ld]) is a global set abstraction level that converts set to a single vector. In multi-scale setting (as in MSG), we use SA(K, [r (1), ..., r(m) ], [[l (1) 1 , ..., l(1) d ],...,[l (m) 1 , ..., l(m) d ]]) to represent MSG with m scales. + +FC(l,dp) represents a fully connected layer with width l and dropout ratio dp. FP(l1, ..., ld) is a feature propagation (FP) level with d fully connected layers. It is used for updating features concatenated from interpolation and skip link. All fully connected layers are followed by batch normalization and ReLU except for the last score prediction layer. + +# B.1 Network Architectures + +For all classification experiments we use the following architecture (Ours SSG) with different K (number of categories): + +``` +SA(512, 0.2, [64, 64, 128]) → SA(128, 0.4, [128, 128, 256]) → SA([256, 512, 1024]) → +F C(512, 0.5) → F C(256, 0.5) → F C(K) +``` + +The multi-scale grouping (MSG) network (PointNet++) architecture is as follows: + +``` +SA(512, [0.1, 0.2, 0.4], [[32, 32, 64], [64, 64, 128], [64, 96, 128]]) → +SA(128, [0.2, 0.4, 0.8], [[64, 64, 128], [128, 128, 256], [128, 128, 256]]) → +SA([256, 512, 1024]) → F C(512, 0.5) → F C(256, 0.5) → F C(K) +``` + +The cross level multi-resolution grouping (MRG) network's architecture uses three branches: + +``` +Branch 1: SA(512, 0.2, [64, 64, 128]) → SA(64, 0.4, [128, 128, 256]) +``` + +Branch 2: SA(512, 0.4, [64, 128, 256]) using r = 0.4 regions of original points + +Branch 3: SA(64, 128, 256, 512) using all original points. + +Branch 4: SA(256, 512, 1024). + +Branch 1 and branch 2 are concatenated and fed to branch 4. Output of branch 3 and branch4 are then concatenated and fed to F C(512, 0.5) → F C(256, 0.5) → F C(K) for classification. + +Network for semantic scene labeling (last two fully connected layers in FP are followed by dropout layers with drop ratio 0.5): + +``` +SA(1024, 0.1, [32, 32, 64]) → SA(256, 0.2, [64, 64, 128]) → +SA(64, 0.4, [128, 128, 256]) → SA(16, 0.8, [256, 256, 512]) → +F P(256, 256) → F P(256, 256) → F P(256, 128) → F P(128, 128, 128, 128, K) +``` + +Network for semantic and part segmentation (last two fully connected layers in FP are followed by dropout layers with drop ratio 0.5): + +``` +SA(512, 0.2, [64, 64, 128]) → SA(128, 0.4, [128, 128, 256]) → SA([256, 512, 1024]) → +F P(256, 256) → F P(256, 128) → F P(128, 128, 128, 128, K) +``` + +# B.2 Virtual Scan Generation + +In this section, we describe how we generate labeled virtual scan with non-uniform sampling density from ScanNet scenes. For each scene in ScanNet, we set camera location 1.5m above the centroid of the floor plane and rotate the camera orientation in the horizontal plane evenly in 8 directions. In each direction, we use a image plane with size 100px by 75px and cast rays from camera through each pixel to the scene. This gives a way to select visible points in the scene. We could then generate 8 virtual scans for each test scene similar and an example is shown in Fig. [9.](#page-11-0) Notice point samples are denser in regions closer to the camera. + +![](PointNet++_1706.02413_images/_page_11_Figure_2.jpeg) + +Figure 9: Virtual scan generated from ScanNet + +> **[그림 해설]** ScanNet 데이터로부터 합성한 가상 스캔(Virtual scan) 포인트 클라우드 비교. +> - **(a) ScanNet labeled scene (좌측)**: 완전한 3D 메시에서 균일하게 샘플링된 고밀도 실내 씬 포인트 클라우드. +> - **(b) ScanNet non-uniform (우측)**: 단일 가상 센서 시점에서 레이 캐스팅(Ray casting)을 시뮬레이션하여 거리에 따른 점 밀도 감소와 시점 가림(Occlusion)에 의한 포인트 결손을 사실적으로 반영한 불균일 스캔 데이터. + +## B.3 MNIST and ModelNet40 Experiment Details + +For MNIST images, we firstly normalize all pixel intensities to range [0, 1] and then select all pixels with intensities larger than 0.5 as valid digit pixels. Then we convert digit pixels in an image into a 2D point cloud with coordinates within [−1, 1], where the image center is the origin point. Augmented points are created to add the point set up to a fixed cardinality (512 in our case). We jitter the initial point cloud (with random translation of Gaussian distribution N (0, 0.01) and clipped to 0.03) to generate the augmented points. For ModelNet40, we uniformly sample N points from CAD models surfaces based on face area. + +For all experiments, we use Adam [\[9\]](#page-8-26) optimizer with learning rate 0.001 for training. For data augmentation, we randomly scale object, perturb the object location as well as point sample locations. We also follow [\[21\]](#page-8-8) to randomly rotate objects for ModelNet40 data augmentation. We use Tensor-Flow and GTX 1080, Titan X for training. All layers are implemented in CUDA to run GPU. It takes around 20 hours to train our model to convergence. + +### B.4 ScanNet Experiment Details + +To generate training data from ScanNet scenes, we sample 1.5m by 1.5m by 3m cubes from the initial scene and then keep the cubes where ≥ 2% of the voxels are occupied and ≥ 70% of the surface voxels have valid annotations (this is the same set up in [\[5\]](#page-8-5)). We sample such training cubes on the fly and random rotate it along the up-right axis. Augmented points are added to the point set to make a fixed cardinality (8192 in our case). During test time, we similarly split the test scene into smaller cubes and get label prediction for every point in the cubes first, then merge label prediction in all the cubes from a same scene. If a point get different labels from different cubes, we will just conduct a majority voting to get the final point label prediction. + +### B.5 SHREC15 Experiment Details + +We randomly sample 1024 points on each shape both for training and testing. To generate the input intrinsic features, we to extract 100 dimensional WKS, HKS and multiscale Gaussian curvature respectively, leading to a 300 dimensional feature vector for each point. Then we conduct PCA to reduce the feature dimension to 64. We use a 8 dimensional embedding following [\[23\]](#page-8-10) to mimic the geodesic distance, which is used to describe our non-Euclidean metric space while choosing the point neighborhood. + +# C More Experiments + +In this section we provide more experiment results to validate and analyze our proposed network architecture. + +## C.1 Semantic Part Segmentation + +Following the setting in [\[32\]](#page-9-4), we evaluate our approach on part segmentation task assuming category label for each shape is already known. Taken shapes represented by point clouds as input, the task is to predict a part label for each point. The dataset contains 16,881 shapes from 16 classes, annotated with 50 parts in total. We use the official train test split following [\[4\]](#page-8-27). + +We equip each point with its normal direction to better depict the underlying shape. This way we could get rid of hand-crafted geometric features as is used in [\[32,](#page-9-4) [33\]](#page-9-3). We compare our framework with traditional learning based techniques [\[32\]](#page-9-4), as well as state-of-the-art deep learning approaches [\[20,](#page-8-0) [33\]](#page-9-3) in Table [4.](#page-12-1) Point intersection over union (IoU) is used as the evaluation metric, averaged across all part classes. Cross-entropy loss is minimized during training. On average, our approach achieves the best performance. In comparison with [\[20\]](#page-8-0), our approach performs better on most of the categories, which proves the importance of hierarchical feature learning for detailed semantic understanding. Notice our approach could be viewed as implicitly building proximity graphs at different scales and operating on these graphs, thus is related to graph CNN approaches such as [\[33\]](#page-9-3). Thanks to the flexibility of our multi-scale neighborhood selection as well as the power of set operation units, we could achieve better performance compared with [\[33\]](#page-9-3). Notice our set operation unit is much simpler compared with graph convolution kernels, and we do not need to conduct expensive eigen decomposition as opposed to [\[33\]](#page-9-3). These make our approach more suitable for large scale point cloud analysis. + +| | mean | aero | bag | cap | car | chair ear | phone | | guitar knife | lamp | laptop motor | | mug | | pistol rocket skate | board | table | +|--------------------|--------------|--------------|--------------|--------------|--------------|--------------|--------------|--------------|--------------|--------------|--------------|--------------|--------------|--------------|---------------------|--------------|--------------| +| Yi [32]
PN [20] | 81.4
83.7 | 81.0
83.4 | 78.4
78.7 | 77.7
82.5 | 75.7
74.9 | 87.6
89.6 | 61.9
73.0 | 92.0
91.5 | 85.4
85.9 | 82.5
80.8 | 95.7
95.3 | 70.6
65.2 | 91.9
93.0 | 85.9
81.2 | 53.1
57.9 | 69.8
72.8 | 75.3
80.6 | +| SSCNN [33] | 84.7 | 81.6 | 81.7 | 81.9 | 75.2 | 90.2 | 74.9 | 93.0 | 86.1 | 84.7 | 95.6 | 66.7 | 92.7 | 81.6 | 60.6 | 82.9 | 82.1 | +| Ours | 85.1 | 82.4 | 79.0 | 87.7 | 77.3 | 90.8 | 71.8 | 91.0 | 85.9 | 83.7 | 95.3 | 71.6 | 94.1 | 81.3 | 58.7 | 76.4 | 82.6 | + +Table 4: Segmentation results on ShapeNet part dataset. + +# C.2 Neighborhood Query: kNN v.s. Ball Query. + +Here we compare two options to select a local neighborhood. We used radius based ball query in our main paper. Here we also experiment with kNN based neighborhood search and also play with different search radius and k. In this experiment all training and testing are on ModelNet40 shapes with uniform sampling density. 1024 points are used. As seen in Table [5,](#page-12-2) radius based ball query is slightly better than kNN based method. However, we speculate in very non-uniform point set, kNN based query will results in worse generalization ability. Also we observe that a slightly large radius is helpful for performance probably because it captures richer local patterns. + +| kNN (k=16) | kNN (k=64) | radius (r=0.1) | radius (r=0.2) | +|------------|------------|----------------|----------------| +| 89.3 | 90.3 | 89.1 | 90.7 | + +Table 5: Effects of neighborhood choices. Evaluation metric is classification accuracy (%) on ModelNet 40 test set. + +### C.3 Effect of Randomness in Farthest Point Sampling. + +For the *Sampling layer* in our set abstraction level, we use farthest point sampling (FPS) for point set sub sampling. However FPS algorithm is random and the subsampling depends on which point is selected first. Here we evaluate the sensitivity of our model to this randomness. In Table [6,](#page-13-0) we test our model trained on ModelNet40 for feature stability and classification stability. + +To evaluate feature stability we extract global features of all test samples for 10 times with different random seed. Then we compute mean features for each shape across the 10 sampling. Then we compute standard deviation of the norms of feature's difference from the mean feature. At last we average all std. in all feature dimensions as reported in the table. Since features are normalized into 0 to 1 before processing, the 0.021 difference means a 2.1% deviation of feature norm. + +For classification, we observe only a 0.17% standard deviation in test accuracy on all ModelNet40 test shapes, which is robust to sampling randomness. + + + +| Feature difference std. | Accuracy std. | +|-------------------------|---------------| +| 0.021 | 0.0017 | + +Table 6: Effects of randomness in FPS (using ModelNet40). + +# C.4 Time and Space Complexity. + +Table [7](#page-13-1) summarizes comparisons of time and space cost between a few point set based deep learning method. We record forward time with a batch size 8 using TensorFlow 1.1 with a single GTX 1080. The first batch is neglected since there is some preparation for GPU. While PointNet (vanilla) [\[20\]](#page-8-0) has the best time efficiency, our model without density adaptive layers achieved smallest model size with fair speed. + +It's worth noting that ours MSG, while it has good performance in non-uniformly sampled data, it's 2x expensive than SSG version due the multi-scale region feature extraction. Compared with MSG, MRG is more efficient since it uses regions across layers. + +| | PointNet (vanilla) 1 | PointNet 1 | Ours (SSG) | Ours (MSG) | Ours (MRG) | +|-------------------|----------------------|------------|------------|------------|------------| +| Model size (MB) | 9.4 | 40 | 8.7 | 12 | 24 | +| Forward time (ms) | 11.6 | 25.3 | 82.4 | 163.2 | 87.0 | + +Table 7: Model size and inference time (forward pass) of several networks. \ No newline at end of file diff --git a/docs/papers/md/PointNet++_1706.02413_images/_page_11_Figure_2.jpeg b/docs/papers/md/PointNet++_1706.02413_images/_page_11_Figure_2.jpeg new file mode 100644 index 0000000..a711614 Binary files /dev/null and b/docs/papers/md/PointNet++_1706.02413_images/_page_11_Figure_2.jpeg differ diff --git a/docs/papers/md/PointNet++_1706.02413_images/_page_1_Picture_2.jpeg b/docs/papers/md/PointNet++_1706.02413_images/_page_1_Picture_2.jpeg new file mode 100644 index 0000000..43cf999 Binary files /dev/null and b/docs/papers/md/PointNet++_1706.02413_images/_page_1_Picture_2.jpeg differ diff --git a/docs/papers/md/PointNet++_1706.02413_images/_page_2_Figure_0.jpeg b/docs/papers/md/PointNet++_1706.02413_images/_page_2_Figure_0.jpeg new file mode 100644 index 0000000..d62a0af Binary files /dev/null and b/docs/papers/md/PointNet++_1706.02413_images/_page_2_Figure_0.jpeg differ diff --git a/docs/papers/md/PointNet++_1706.02413_images/_page_3_Figure_6.jpeg b/docs/papers/md/PointNet++_1706.02413_images/_page_3_Figure_6.jpeg new file mode 100644 index 0000000..92d59c5 Binary files /dev/null and b/docs/papers/md/PointNet++_1706.02413_images/_page_3_Figure_6.jpeg differ diff --git a/docs/papers/md/PointNet++_1706.02413_images/_page_5_Figure_12.jpeg b/docs/papers/md/PointNet++_1706.02413_images/_page_5_Figure_12.jpeg new file mode 100644 index 0000000..5107446 Binary files /dev/null and b/docs/papers/md/PointNet++_1706.02413_images/_page_5_Figure_12.jpeg differ diff --git a/docs/papers/md/PointNet++_1706.02413_images/_page_5_Figure_4.jpeg b/docs/papers/md/PointNet++_1706.02413_images/_page_5_Figure_4.jpeg new file mode 100644 index 0000000..0d1c114 Binary files /dev/null and b/docs/papers/md/PointNet++_1706.02413_images/_page_5_Figure_4.jpeg differ diff --git a/docs/papers/md/PointNet++_1706.02413_images/_page_5_Figure_5.jpeg b/docs/papers/md/PointNet++_1706.02413_images/_page_5_Figure_5.jpeg new file mode 100644 index 0000000..5f50360 Binary files /dev/null and b/docs/papers/md/PointNet++_1706.02413_images/_page_5_Figure_5.jpeg differ diff --git a/docs/papers/md/PointNet++_1706.02413_images/_page_6_Figure_3.jpeg b/docs/papers/md/PointNet++_1706.02413_images/_page_6_Figure_3.jpeg new file mode 100644 index 0000000..3c5a4a0 Binary files /dev/null and b/docs/papers/md/PointNet++_1706.02413_images/_page_6_Figure_3.jpeg differ diff --git a/docs/papers/md/PointNet++_1706.02413_images/_page_6_Figure_9.jpeg b/docs/papers/md/PointNet++_1706.02413_images/_page_6_Figure_9.jpeg new file mode 100644 index 0000000..2fb710f Binary files /dev/null and b/docs/papers/md/PointNet++_1706.02413_images/_page_6_Figure_9.jpeg differ diff --git a/docs/papers/md/PointNet++_1706.02413_images/_page_7_Figure_4.jpeg b/docs/papers/md/PointNet++_1706.02413_images/_page_7_Figure_4.jpeg new file mode 100644 index 0000000..20e2633 Binary files /dev/null and b/docs/papers/md/PointNet++_1706.02413_images/_page_7_Figure_4.jpeg differ diff --git a/docs/papers/md/PointNet_1612.00593.md b/docs/papers/md/PointNet_1612.00593.md new file mode 100644 index 0000000..347747f --- /dev/null +++ b/docs/papers/md/PointNet_1612.00593.md @@ -0,0 +1,727 @@ +# PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation + +Charles R. Qi\* Hao Su\* Kaichun Mo Leonidas J. Guibas Stanford University + +### Abstract + +*Point cloud is an important type of geometric data structure. Due to its irregular format, most researchers transform such data to regular 3D voxel grids or collections of images. This, however, renders data unnecessarily voluminous and causes issues. In this paper, we design a novel type of neural network that directly consumes point clouds, which well respects the permutation invariance of points in the input. Our network, named PointNet, provides a unified architecture for applications ranging from object classification, part segmentation, to scene semantic parsing. Though simple, PointNet is highly efficient and effective. Empirically, it shows strong performance on par or even better than state of the art. Theoretically, we provide analysis towards understanding of what the network has learnt and why the network is robust with respect to input perturbation and corruption.* + +## 1. Introduction + +In this paper we explore deep learning architectures capable of reasoning about 3D geometric data such as point clouds or meshes. Typical convolutional architectures require highly regular input data formats, like those of image grids or 3D voxels, in order to perform weight sharing and other kernel optimizations. Since point clouds or meshes are not in a regular format, most researchers typically transform such data to regular 3D voxel grids or collections of images (e.g, views) before feeding them to a deep net architecture. This data representation transformation, however, renders the resulting data unnecessarily voluminous — while also introducing quantization artifacts that can obscure natural invariances of the data. + +For this reason we focus on a different input representation for 3D geometry using simply point clouds – and name our resulting deep nets *PointNets*. Point clouds are simple and unified structures that avoid the combinatorial irregularities and complexities of meshes, and thus are easier to learn from. The PointNet, however, + +![](PointNet_1612.00593_images/_page_0_Figure_9.jpeg) + +Figure 1. Applications of PointNet. We propose a novel deep net architecture that consumes raw point cloud (set of points) without voxelization or rendering. It is a unified architecture that learns both global and local point features, providing a simple, efficient and effective approach for a number of 3D recognition tasks. + +> **[그림 해설]** PointNet이 다루는 세 가지 핵심 3D 태스크를 요약한 다이어그램. +> - **Classification(분류)**: 컵(mug), 테이블(table), 자동차(car) 등 단일 3D 포인트 클라우드 입력을 받아 객체의 전체 클래스 레이블을 판별. +> - **Part Segmentation(부품 분할)**: 램프, 비행기(동체·날개·엔진), 탁자(상판·다리) 등 단일 객체 내 각 부품 영역을 포인트 단위로 분할하여 색상별로 구분. +> - **Semantic Segmentation(시맨틱 분할)**: 실내 공간 전체 포인트 클라우드에서 바닥, 벽, 의자, 테이블 등 복합 환경의 각 구성 요소를 포인트별 시맨틱 클래스로 분류. + +still has to respect the fact that a point cloud is just a set of points and therefore invariant to permutations of its members, necessitating certain symmetrizations in the net computation. Further invariances to rigid motions also need to be considered. + +Our PointNet is a unified architecture that directly takes point clouds as input and outputs either class labels for the entire input or per point segment/part labels for each point of the input. The basic architecture of our network is surprisingly simple as in the initial stages each point is processed identically and independently. In the basic setting each point is represented by just its three coordinates (x, y, z). Additional dimensions may be added by computing normals and other local or global features. + +Key to our approach is the use of a single symmetric function, max pooling. Effectively the network learns a set of optimization functions/criteria that select interesting or informative points of the point cloud and encode the reason for their selection. The final fully connected layers of the network aggregate these learnt optimal values into the global descriptor for the entire shape as mentioned above (shape classification) or are used to predict per point labels (shape segmentation). + +Our input format is easy to apply rigid or affine transformations to, as each point transforms independently. Thus we can add a data-dependent spatial transformer network that attempts to canonicalize the data before the PointNet processes them, so as to further improve the results. + +\* indicates equal contributions. + +We provide both a theoretical analysis and an experimental evaluation of our approach. We show that our network can approximate any set function that is continuous. More interestingly, it turns out that our network learns to summarize an input point cloud by a sparse set of key points, which roughly corresponds to the skeleton of objects according to visualization. The theoretical analysis provides an understanding why our PointNet is highly robust to small perturbation of input points as well as to corruption through point insertion (outliers) or deletion (missing data). + +On a number of benchmark datasets ranging from shape classification, part segmentation to scene segmentation, we experimentally compare our PointNet with state-ofthe-art approaches based upon multi-view and volumetric representations. Under a unified architecture, not only is our PointNet much faster in speed, but it also exhibits strong performance on par or even better than state of the art. + +The key contributions of our work are as follows: + +- We design a novel deep net architecture suitable for consuming unordered point sets in 3D; +- We show how such a net can be trained to perform 3D shape classification, shape part segmentation and scene semantic parsing tasks; +- We provide thorough empirical and theoretical analysis on the stability and efficiency of our method; +- We illustrate the 3D features computed by the selected neurons in the net and develop intuitive explanations for its performance. + +The problem of processing unordered sets by neural nets is a very general and fundamental problem – we expect that our ideas can be transferred to other domains as well. + +### 2. Related Work + +Point Cloud Features Most existing features for point cloud are handcrafted towards specific tasks. Point features often encode certain statistical properties of points and are designed to be invariant to certain transformations, which are typically classified as intrinsic [\[2,](#page-8-0) [24,](#page-8-1) [3\]](#page-8-2) or extrinsic [\[20,](#page-8-3) [19,](#page-8-4) [14,](#page-8-5) [10,](#page-8-6) [5\]](#page-8-7). They can also be categorized as local features and global features. For a specific task, it is not trivial to find the optimal feature combination. + +Deep Learning on 3D Data 3D data has multiple popular representations, leading to various approaches for learning. *Volumetric CNNs:* [\[28,](#page-8-8) [17,](#page-8-9) [18\]](#page-8-10) are the pioneers applying 3D convolutional neural networks on voxelized shapes. However, volumetric representation is constrained by its resolution due to data sparsity and computation cost of 3D convolution. FPNN [\[13\]](#page-8-11) and Vote3D [\[26\]](#page-8-12) proposed special methods to deal with the sparsity problem; however, their operations are still on sparse volumes, it's challenging for them to process very large point clouds. *Multiview CNNs:* [\[23,](#page-8-13) [18\]](#page-8-10) have tried to render 3D point cloud or shapes into 2D images and then apply 2D conv nets to classify them. With well engineered image CNNs, this line of methods have achieved dominating performance on shape classification and retrieval tasks [\[21\]](#page-8-14). However, it's nontrivial to extend them to scene understanding or other 3D tasks such as point classification and shape completion. *Spectral CNNs:* Some latest works [\[4,](#page-8-15) [16\]](#page-8-16) use spectral CNNs on meshes. However, these methods are currently constrained on manifold meshes such as organic objects and it's not obvious how to extend them to non-isometric shapes such as furniture. *Feature-based DNNs:* [\[6,](#page-8-17) [8\]](#page-8-18) firstly convert the 3D data into a vector, by extracting traditional shape features and then use a fully connected net to classify the shape. We think they are constrained by the representation power of the features extracted. + +Deep Learning on Unordered Sets From a data structure point of view, a point cloud is an unordered set of vectors. While most works in deep learning focus on regular input representations like sequences (in speech and language processing), images and volumes (video or 3D data), not much work has been done in deep learning on point sets. + +One recent work from Oriol Vinyals et al [\[25\]](#page-8-19) looks into this problem. They use a read-process-write network with attention mechanism to consume unordered input sets and show that their network has the ability to sort numbers. However, since their work focuses on generic sets and NLP applications, there lacks the role of geometry in the sets. + +### 3. Problem Statement + +We design a deep learning framework that directly consumes unordered point sets as inputs. A point cloud is represented as a set of 3D points {Pi | i = 1, ..., n}, where each point Pi is a vector of its (x, y, z) coordinate plus extra feature channels such as color, normal etc. For simplicity and clarity, unless otherwise noted, we only use the (x, y, z) coordinate as our point's channels. + +For the object classification task, the input point cloud is either directly sampled from a shape or pre-segmented from a scene point cloud. Our proposed deep network outputs k scores for all the k candidate classes. For semantic segmentation, the input can be a single object for part region segmentation, or a sub-volume from a 3D scene for object region segmentation. Our model will output n × m scores for each of the n points and each of the m semantic subcategories. + +![](PointNet_1612.00593_images/_page_2_Figure_0.jpeg) + +> **[그림 해설]** 'Classification Network' 텍스트 배너. 아래 Figure 2 다이어그램의 상단 파란색 블록에 해당하는 객체 분류 네트워크 영역을 지칭한다. + +![](PointNet_1612.00593_images/_page_2_Figure_1.jpeg) + +Figure 2. **PointNet Architecture.** The classification network takes n points as input, applies input and feature transformations, and then aggregates point features by max pooling. The output is classification scores for k classes. The segmentation network is an extension to the classification net. It concatenates global and local features and outputs per point scores. "mlp" stands for multi-layer perceptron, numbers in bracket are layer sizes. Batchnorm is used for all layers with ReLU. Dropout layers are used for the last mlp in classification net. + +> **[그림 해설]** PointNet의 전체 신경망 구조도. +> - **Classification Network (상단 파란색 영역)**: +> - 입력: $n \times 3$ 점 좌표. +> - **Input Transform**: T-Net을 통해 $3 \times 3$ 변환 행렬을 예측하여 입력 좌표 공간을 정렬 $\to n \times 3$. +> - **Shared MLP (64, 64)**: 각 점을 독립적으로 $64$차원 공간으로 매핑 $\to n \times 64$. +> - **Feature Transform**: T-Net을 통해 $64 \times 64$ 특징 변환 행렬을 예측하고 직교 정규화($L_{reg}$)를 적용하여 특징 공간 정렬 $\to n \times 64$. +> - **Shared MLP (64, 128, 1024)**: 점별 특징을 1024차원으로 확장 $\to n \times 1024$. +> - **Max Pool**: 대칭 함수(Symmetric Function)로 점 순서 불변성을 확보하며 전체 $n$개 점의 최댓값을 집계 $\to 1024$차원 Global Feature 생성. +> - **MLP (512, 256, k)**: 완전연결 레이어와 Dropout을 거쳐 $k$개 클래스 분류 점수(output scores) 출력. +> - **Segmentation Network (하단 노란색 영역)**: +> - $n \times 64$의 로컬 점 특징과 1024차원의 글로벌 특징 벡터를 결합(concatenation)하여 $n \times 1088$ 크기의 통합 특징 구성. +> - **Shared MLP (512, 256, 128)** $\to n \times 128$. +> - **Shared MLP (128, m)** $\to n \times m$ 출력 점수를 계산하여 $n$개의 각 점마다 $m$개 시맨틱/부품 범주 점수를 도출. + +### 4. Deep Learning on Point Sets + +The architecture of our network (Sec 4.2) is inspired by the properties of point sets in $\mathbb{R}^n$ (Sec 4.1). + +#### **4.1. Properties of Point Sets in** $\mathbb{R}^n$ + +Our input is a subset of points from an Euclidean space. It has three main properties: + +- Unordered. Unlike pixel arrays in images or voxel arrays in volumetric grids, point cloud is a set of points without specific order. In other words, a network that consumes N 3D point sets needs to be invariant to N! permutations of the input set in data feeding order. +- Interaction among points. The points are from a space with a distance metric. It means that points are not isolated, and neighboring points form a meaningful subset. Therefore, the model needs to be able to capture local structures from nearby points, and the combinatorial interactions among local structures. +- Invariance under transformations. As a geometric object, the learned representation of the point set should be invariant to certain transformations. For example, rotating and translating points all together should not modify the global point cloud category nor the segmentation of the points. + +#### 4.2. PointNet Architecture + +Our full network architecture is visualized in Fig 2, where the classification network and the segmentation network share a great portion of structures. Please read the caption of Fig 2 for the pipeline. + +Our network has three key modules: the max pooling layer as a symmetric function to aggregate information from + +all the points, a local and global information combination structure, and two joint alignment networks that align both input points and point features. + +We will discuss our reason behind these design choices in separate paragraphs below. + +Symmetry Function for Unordered Input In order to make a model invariant to input permutation, three strategies exist: 1) sort input into a canonical order; 2) treat the input as a sequence to train an RNN, but augment the training data by all kinds of permutations; 3) use a simple symmetric function to aggregate the information from each point. Here, a symmetric function takes n vectors as input and outputs a new vector that is invariant to the input order. For example, + and \* operators are symmetric binary functions. + +While sorting sounds like a simple solution, in high dimensional space there in fact does not exist an ordering that is stable w.r.t. point perturbations in the general sense. This can be easily shown by contradiction. If such an ordering strategy exists, it defines a bijection map between a high-dimensional space and a 1d real line. It is not hard to see, to require an ordering to be stable w.r.t point perturbations is equivalent to requiring that this map preserves spatial proximity as the dimension reduces, a task that cannot be achieved in the general case. Therefore, sorting does not fully resolve the ordering issue, and it's hard for a network to learn a consistent mapping from input to output as the ordering issue persists. As shown in experiments (Fig 5), we find that applying a MLP directly on the sorted point set performs poorly, though slightly better than directly processing an unsorted input. + +The idea to use RNN considers the point set as a sequential signal and hopes that by training the RNN + +with randomly permuted sequences, the RNN will become invariant to input order. However in "OrderMatters" [25] the authors have shown that order does matter and cannot be totally omitted. While RNN has relatively good robustness to input ordering for sequences with small length (dozens), it's hard to scale to thousands of input elements, which is the common size for point sets. Empirically, we have also shown that model based on RNN does not perform as well as our proposed method (Fig 5). + +Our idea is to approximate a general function defined on a point set by applying a symmetric function on transformed elements in the set: + +$$f(\lbrace x_1, \dots, x_n \rbrace) \approx g(h(x_1), \dots, h(x_n)), \tag{1}$$ + +where +$$f: 2^{\mathbb{R}^N} \to \mathbb{R}$$ +, $h: \mathbb{R}^N \to \mathbb{R}^K$ and $g: \mathbb{R}^K \times \cdots \times \mathbb{R}^K \to \mathbb{R}$ is a symmetric function. + +Empirically, our basic module is very simple: we approximate h by a multi-layer perceptron network and g by a composition of a single variable function and a max pooling function. This is found to work well by experiments. Through a collection of h, we can learn a number of f's to capture different properties of the set. + +While our key module seems simple, it has interesting properties (see Sec 5.3) and can achieve strong performace (see Sec 5.1) in a few different applications. Due to the simplicity of our module, we are also able to provide theoretical analysis as in Sec 4.3. + +**Local and Global Information Aggregation** The output from the above section forms a vector $[f_1,\ldots,f_K]$ , which is a global signature of the input set. We can easily train a SVM or multi-layer perceptron classifier on the shape global features for classification. However, point segmentation requires a combination of local and global knowledge. We can achieve this by a simple yet highly effective manner. + +Our solution can be seen in Fig 2 (Segmentation Network). After computing the global point cloud feature vector, we feed it back to per point features by concatenating the global feature with each of the point features. Then we extract new per point features based on the combined point features - this time the per point feature is aware of both the local and global information. + +With this modification our network is able to predict per point quantities that rely on both local geometry and global semantics. For example we can accurately predict per-point normals (fig in supplementary), validating that the network is able to summarize information from the point's local neighborhood. In experiment session, we also show that our model can achieve state-of-the-art performance on shape part segmentation and scene segmentation. **Joint Alignment Network** The semantic labeling of a point cloud has to be invariant if the point cloud undergoes certain geometric transformations, such as rigid transformation. We therefore expect that the learnt representation by our point set is invariant to these transformations. + +A natural solution is to align all input set to a canonical space before feature extraction. Jaderberg et al. [9] introduces the idea of spatial transformer to align 2D images through sampling and interpolation, achieved by a specifically tailored layer implemented on GPU. + +Our input form of point clouds allows us to achieve this goal in a much simpler way compared with [9]. We do not need to invent any new layers and no alias is introduced as in the image case. We predict an affine transformation matrix by a mini-network (T-net in Fig 2) and directly apply this transformation to the coordinates of input points. The mininetwork itself resembles the big network and is composed by basic modules of point independent feature extraction, max pooling and fully connected layers. More details about the T-net are in the supplementary. + +This idea can be further extended to the alignment of feature space, as well. We can insert another alignment network on point features and predict a feature transformation matrix to align features from different input point clouds. However, transformation matrix in the feature space has much higher dimension than the spatial transform matrix, which greatly increases the difficulty of optimization. We therefore add a regularization term to our softmax training loss. We constrain the feature transformation matrix to be close to orthogonal matrix: + +$$L_{reg} = ||I - AA^T||_F^2, (2)$$ + +where A is the feature alignment matrix predicted by a mini-network. An orthogonal transformation will not lose information in the input, thus is desired. We find that by adding the regularization term, the optimization becomes more stable and our model achieves better performance. + +#### 4.3. Theoretical Analysis + +**Universal approximation** We first show the universal approximation ability of our neural network to continuous set functions. By the continuity of set functions, intuitively, a small perturbation to the input point set should not greatly change the function values, such as classification or segmentation scores. + +Formally, let $\mathcal{X} = \{S : S \subseteq [0,1]^m \text{ and } |S| = n\}, f : \mathcal{X} \to \mathbb{R}$ is a continuous set function on $\mathcal{X}$ w.r.t to Hausdorff distance $d_H(\cdot,\cdot)$ , i.e., $\forall \epsilon > 0, \exists \delta > 0$ , for any $S, S' \in \mathcal{X}$ , if $d_H(S,S') < \delta$ , then $|f(S) - f(S')| < \epsilon$ . Our theorem says that f can be arbitrarily approximated by our network given enough neurons at the max pooling layer, i.e., K in (1) is sufficiently large. + +![](PointNet_1612.00593_images/_page_4_Figure_0.jpeg) + +Figure 3. Qualitative results for part segmentation. We visualize the CAD part segmentation results across all 16 object categories. We show both results for partial simulated Kinect scans (left block) and complete ShapeNet CAD models (right block). + +> **[그림 해설]** 16개 카테고리에 대한 3D 파트 분할(Part Segmentation) 정성적 결과 시각화. +> - **좌측 (Partial Inputs, 불완전 입력)**: 가상 Kinect 스캔으로 생성된 한쪽 면만 스캔되고 결손이 있는 포인트 클라우드에 대한 결과(table, motorbike, car, airplane, mug, lamp, guitar, chair 8종). 결손 및 가림이 있는 상태에서도 바퀴, 손잡이, 날개 등의 부품이 정확한 색상으로 분할됨. +> - **우측 (Complete Inputs, 완전 입력)**: ShapeNet 3D CAD 모델의 완전한 포인트 클라우드에 대한 결과(bag, knife, cap, skateboard, pistol, rocket, earphone, laptop 8종). 칼날과 손잡이, 모자 챙과 본체, 노트북 모니터와 본체 등 복잡한 기하학적 세부 부품이 정밀하게 구분됨. + +**Theorem 1.** Suppose $f: \mathcal{X} \to \mathbb{R}$ is a continuous set function w.r.t Hausdorff distance $d_H(\cdot, \cdot)$ . $\forall \epsilon > 0$ , $\exists$ a continuous function h and a symmetric function $g(x_1, \ldots, x_n) = \gamma \circ MAX$ , such that for any $S \in \mathcal{X}$ , + +$$\left| f(S) - \gamma \left( \max_{x_i \in S} \{h(x_i)\} \right) \right| < \epsilon$$ + +where $x_1, \ldots, x_n$ is the full list of elements in S ordered arbitrarily, $\gamma$ is a continuous function, and MAX is a vector max operator that takes n vectors as input and returns a new vector of the element-wise maximum. + +The proof to this theorem can be found in our supplementary material. The key idea is that in the worst case the network can learn to convert a point cloud into a volumetric representation, by partitioning the space into equal-sized voxels. In practice, however, the network learns a much smarter strategy to probe the space, as we shall see in point function visualizations. + +**Bottleneck dimension and stability** Theoretically and experimentally we find that the expressiveness of our network is strongly affected by the dimension of the max pooling layer, i.e., K in (1). Here we provide an analysis, which also reveals properties related to the stability of our model. + +We define $\mathbf{u} = \max_{x_i \in S} \{h(x_i)\}$ to be the sub-network of f which maps a point set in $[0,1]^m$ to a K-dimensional vector. The following theorem tells us that small corruptions or extra noise points in the input set are not likely to change the output of our network: + +**Theorem 2.** Suppose $\mathbf{u}: \mathcal{X} \to \mathbb{R}^K$ such that $\mathbf{u} = \max_{x \in S} \{h(x_i)\}$ and $f = \gamma \circ \mathbf{u}$ . Then, + +(a) +$$\forall S, \exists C_S, \mathcal{N}_S \subseteq \mathcal{X}, f(T) = f(S) \text{ if } C_S \subseteq T \subseteq \mathcal{N}_S;$$ + +(b) +$$|\mathcal{C}_S| \leq K$$ + +| | input | #views | accuracy | accuracy | +|------------------|--------|--------|------------|----------| +| | | | avg. class | overall | +| SPH [11] | mesh | - | 68.2 | - | +| 3DShapeNets [28] | volume | 1 | 77.3 | 84.7 | +| VoxNet [17] | volume | 12 | 83.0 | 85.9 | +| Subvolume [18] | volume | 20 | 86.0 | 89.2 | +| LFD [28] | image | 10 | 75.5 | - | +| MVCNN [23] | image | 80 | 90.1 | - | +| Ours baseline | point | - | 72.6 | 77.4 | +| Ours PointNet | point | 1 | 86.2 | 89.2 | + +Table 1. Classification results on ModelNet40. Our net achieves state-of-the-art among deep nets on 3D input. + +We explain the implications of the theorem. (a) says that f(S) is unchanged up to the input corruption if all points in $\mathcal{C}_S$ are preserved; it is also unchanged with extra noise points up to $\mathcal{N}_S$ . (b) says that $\mathcal{C}_S$ only contains a bounded number of points, determined by K in (1). In other words, f(S) is in fact totally determined by a finite subset $\mathcal{C}_S \subseteq S$ of less or equal to K elements. We therefore call $\mathcal{C}_S$ the critical point set of S and K the bottleneck dimension of f. + +Combined with the continuity of h, this explains the robustness of our model w.r.t point perturbation, corruption and extra noise points. The robustness is gained in analogy to the sparsity principle in machine learning models. Intuitively, our network learns to summarize a shape by a sparse set of key points. In experiment section we see that the key points form the skeleton of an object. + +#### 5. Experiment + +Experiments are divided into four parts. First, we show PointNets can be applied to multiple 3D recognition tasks (Sec 5.1). Second, we provide detailed experiments to validate our network design (Sec 5.2). At last we visualize what the network learns (Sec 5.3) and analyze time and space complexity (Sec 5.4). + +#### 5.1. Applications + +In this section we show how our network can be trained to perform 3D object classification, object part segmentation and semantic scene segmentation 1. Even though we are working on a brand new data representation (point sets), we are able to achieve comparable or even better performance on benchmarks for several tasks. + +**3D Object Classification** Our network learns global point cloud feature that can be used for object classification. We evaluate our model on the ModelNet40 [28] shape classification benchmark. There are 12,311 CAD models from 40 man-made object categories, split into 9,843 for + +&lt;sup>1More application examples such as correspondence and point cloud based CAD model retrieval are included in supplementary material. + + + +| | mean | aero | bag | cap | car | chair | ear | guitar knife | | lamp | laptop | motor | | mug pistol | rocket | skate | table | +|----------|------|------|------|------|------|-------|-------|--------------|------|------|--------|-------|-----|------------|--------|-------|-------| +| | | | | | | | phone | | | | | | | | | board | | +| # shapes | | 2690 | 76 | 55 | 898 | 3758 | 69 | 787 | 392 | 1547 | 451 | 202 | 184 | 283 | 66 | 152 | 5271 | +| Wu [27] | - | 63.2 | - | - | - | 73.5 | - | - | - | 74.4 | - | - | - | - | - | - | 74.8 | +| Yi [29] | 81.4 | 81.0 | 78.4 | 77.7 | 75.7 | 87.6 | 61.9 | 92.0 | 85.4 | 82.5 | 95.7 | 70.6 | | 91.9 85.9 | 53.1 | 69.8 | 75.3 | +| 3DCNN | 79.4 | 75.1 | 72.8 | 73.3 | 70.0 | 87.2 | 63.5 | 88.4 | 79.6 | 74.4 | 93.9 | 58.7 | | 91.8 76.4 | 51.2 | 65.3 | 77.1 | +| Ours | 83.7 | 83.4 | 78.7 | 82.5 | 74.9 | 89.6 | 73.0 | 91.5 | 85.9 | 80.8 | 95.3 | 65.2 | | 93.0 81.2 | 57.9 | 72.8 | 80.6 | + +Table 2. Segmentation results on ShapeNet part dataset. Metric is mIoU(%) on points. We compare with two traditional methods [\[27\]](#page-8-22) and [\[29\]](#page-8-23) and a 3D fully convolutional network baseline proposed by us. Our PointNet method achieved the state-of-the-art in mIoU. + +training and 2,468 for testing. While previous methods focus on volumetric and mult-view image representations, we are the first to directly work on raw point cloud. + +We uniformly sample 1024 points on mesh faces according to face area and normalize them into a unit sphere. During training we augment the point cloud on-the-fly by randomly rotating the object along the up-axis and jitter the position of each points by a Gaussian noise with zero mean and 0.02 standard deviation. + +In Table [1,](#page-4-2) we compare our model with previous works as well as our baseline using MLP on traditional features extracted from point cloud (point density, D2, shape contour etc.). Our model achieved state-of-the-art performance among methods based on 3D input (volumetric and point cloud). With only fully connected layers and max pooling, our net gains a strong lead in inference speed and can be easily parallelized in CPU as well. There is still a small gap between our method and multi-view based method (MVCNN [\[23\]](#page-8-13)), which we think is due to the loss of fine geometry details that can be captured by rendered images. + +3D Object Part Segmentation Part segmentation is a challenging fine-grained 3D recognition task. Given a 3D scan or a mesh model, the task is to assign part category label (e.g. chair leg, cup handle) to each point or face. + +We evaluate on ShapeNet part data set from [\[29\]](#page-8-23), which contains 16,881 shapes from 16 categories, annotated with 50 parts in total. Most object categories are labeled with two to five parts. Ground truth annotations are labeled on sampled points on the shapes. + +We formulate part segmentation as a per-point classification problem. Evaluation metric is mIoU on points. For each shape S of category C, to calculate the shape's mIoU: For each part type in category C, compute IoU between groundtruth and prediction. If the union of groundtruth and prediction points is empty, then count part IoU as 1. Then we average IoUs for all part types in category C to get mIoU for that shape. To calculate mIoU for the category, we take average of mIoUs for all shapes in that category. + +In this section, we compare our segmentation version PointNet (a modified version of Fig [2,](#page-2-2) *Segmentation Network*) with two traditional methods [\[27\]](#page-8-22) and [\[29\]](#page-8-23) that both take advantage of point-wise geometry features and correspondences between shapes, as well as our own 3D CNN baseline. See supplementary for the detailed modifications and network architecture for the 3D CNN. + +In Table [2,](#page-5-0) we report per-category and mean IoU(%) scores. We observe a 2.3% mean IoU improvement and our net beats the baseline methods in most categories. + +We also perform experiments on simulated Kinect scans to test the robustness of these methods. For every CAD model in the ShapeNet part data set, we use Blensor Kinect Simulator [\[7\]](#page-8-24) to generate incomplete point clouds from six random viewpoints. We train our PointNet on the complete shapes and partial scans with the same network architecture and training setting. Results show that we lose only 5.3% mean IoU. In Fig [3,](#page-4-3) we present qualitative results on both complete and partial data. One can see that though partial data is fairly challenging, our predictions are reasonable. + +Semantic Segmentation in Scenes Our network on part segmentation can be easily extended to semantic scene segmentation, where point labels become semantic object classes instead of object part labels. + +We experiment on the Stanford 3D semantic parsing data set [\[1\]](#page-8-25). The dataset contains 3D scans from Matterport scanners in 6 areas including 271 rooms. Each point in the scan is annotated with one of the semantic labels from 13 categories (chair, table, floor, wall etc. plus clutter). + +To prepare training data, we firstly split points by room, and then sample rooms into blocks with area 1m by 1m. We train our segmentation version of PointNet to predict + + + +| | mean IoU | overall accuracy | +|---------------|----------|------------------| +| Ours baseline | 20.12 | 53.19 | +| Ours PointNet | 47.71 | 78.62 | + +Table 3. Results on semantic segmentation in scenes. Metric is average IoU over 13 classes (structural and furniture elements plus clutter) and classification accuracy calculated on points. + +| | table | chair | sofa | board | mean | +|-------------------|-------|-------|------|-------|-------| +| # instance | 455 | 1363 | 55 | 137 | | +| Armeni et al. [1] | 46.02 | 16.15 | 6.78 | 3.91 | 18.22 | +| Ours | 46.67 | 33.80 | 4.76 | 11.72 | 24.24 | + +Table 4. Results on 3D object detection in scenes. Metric is average precision with threshold IoU 0.5 computed in 3D volumes. + +![](PointNet_1612.00593_images/_page_6_Figure_0.jpeg) + +Figure 4. Qualitative results for semantic segmentation. Top row is input point cloud with color. Bottom row is output semantic segmentation result (on points) displayed in the same camera viewpoint as input. + +> **[그림 해설]** Stanford 3D Semantic Parsing 데이터셋 실내 환경 3개 씬(오피스 2곳, 회의실 1곳)에 대한 Semantic Segmentation 결과. +> - **상단 (Input)**: RGB 컬러 정보가 포함된 원본 실내 공간 3D 포인트 클라우드 입력. +> - **하단 (Output)**: PointNet이 예측한 포인트별 시맨틱 레이블. +> - 색상 대응: 바닥(파란색), 벽(하늘색/청록색), 천장(초록색), 테이블(보라색), 의자(빨간색), 보드(회색), 책장/도어(노란색/연두색) 등으로 객체 및 실내 구조물이 명확히 분리됨. + +per point class in each block. Each point is represented by a 9-dim vector of XYZ, RGB and normalized location as to the room (from 0 to 1). At training time, we randomly sample 4096 points in each block on-the-fly. At test time, we test on all the points. We follow the same protocol as [\[1\]](#page-8-25) to use k-fold strategy for train and test. + +We compare our method with a baseline using handcrafted point features. The baseline extracts the same 9 dim local features and three additional ones: local point density, local curvature and normal. We use standard MLP as the classifier. Results are shown in Table [3,](#page-5-1) where our PointNet method significantly outperforms the baseline method. In Fig [4,](#page-6-2) we show qualitative segmentation results. Our network is able to output smooth predictions and is robust to missing points and occlusions. + +Based on the semantic segmentation output from our network, we further build a 3D object detection system using connected component for object proposal (see supplementary for details). We compare with previous stateof-the-art method in Table [4.](#page-5-2) The previous method is based on a sliding shape method (with CRF post processing) with SVMs trained on local geometric features and global room context feature in voxel grids. Our method outperforms it by a large margin on the furniture categories reported. + +### 5.2. Architecture Design Analysis + +In this section we validate our design choices by control experiments. We also show the effects of our network's hyperparameters. + +### Comparison with Alternative Order-invariant Methods + +As mentioned in Sec [4.2,](#page-2-0) there are at least three options for consuming unordered set inputs. We use the ModelNet40 shape classification problem as a test bed for comparisons of those options, the following two control experiment will also use this task. + +The baselines (illustrated in Fig [5\)](#page-6-0) we compared with include multi-layer perceptron on unsorted and sorted + +| rnn
rnn
rnn
MLP | | +|--------------------------------------------------------|--| +| cell
cell

cell | | +| MLP
MLP
MLP | | +|
(1,2,3)
(2,3,4)
(1,3,1)
sequential model | | +| sorted | | +| (1,2,3)
(1,2,3)
MLP
(1,3,1)
(2,3,4)
MLP | | +| MLP
MLP
(1,3,1)
(2,3,4) | | +| MLP
sorting
symmetry function | | + +Figure 5. Three approaches to achieve order invariance. Multilayer perceptron (MLP) applied on points consists of 5 hidden layers with neuron sizes 64,64,64,128,1024, all points share a single copy of MLP. The MLP close to the output consists of two layers with sizes 512,256. + +points as n×3 arrays, RNN model that considers input point as a sequence, and a model based on symmetry functions. The symmetry operation we experimented include max pooling, average pooling and an attention based weighted sum. The attention method is similar to that in [\[25\]](#page-8-19), where a scalar score is predicted from each point feature, then the score is normalized across points by computing a softmax. The weighted sum is then computed on the normalized scores and the point features. As shown in Fig [5,](#page-6-0) maxpooling operation achieves the best performance by a large winning margin, which validates our choice. + +### Effectiveness of Input and Feature Transformations In + +Table [5](#page-6-3) we demonstrate the positive effects of our input and feature transformations (for alignment). It's interesting to see that the most basic architecture already achieves quite reasonable results. Using input transformation gives a 0.8% performance boost. The regularization loss is necessary for the higher dimension transform to work. By combining both transformations and the regularization term, we achieve the best performance. + +Robustness Test We show our PointNet, while simple and effective, is robust to various kinds of input corruptions. We use the same architecture as in Fig [5'](#page-6-0)s max pooling network. Input points are normalized into a unit sphere. Results are in Fig [6.](#page-7-2) + +As to missing points, when there are 50% points missing, the accuracy only drops by 2.4% and 3.8% w.r.t. furthest and random input sampling. Our net is also robust to outlier + +| Transform | accuracy | +|------------------------|----------| +| none | 87.1 | +| input (3x3) | 87.9 | +| feature (64x64) | 86.9 | +| feature (64x64) + reg. | 87.4 | +| both | 89.2 | + +Table 5. Effects of input feature transforms. Metric is overall classification accuracy on ModelNet40 test set. + +![](PointNet_1612.00593_images/_page_7_Figure_0.jpeg) + +Figure 6. PointNet robustness test. The metric is overall classification accuracy on ModelNet40 test set. Left: Delete points. Furthest means the original 1024 points are sampled with furthest sampling. Middle: Insertion. Outliers uniformly scattered in the unit sphere. Right: Perturbation. Add Gaussian noise to each point independently. + +> **[그림 해설]** 포인트 결손, 이상치, 노이즈에 대한 PointNet의 강건성(Robustness) 평가 그래프 (ModelNet40 테스트셋 기준). +> - **좌측 (Missing data ratio, 데이터 결손)**: 점을 무작위(Random, 빨간 사각) 또는 최원점 샘플링(Furthest, 파란 원)으로 삭제했을 때의 정확도 변화. 50%의 점이 누락되어도 정확도는 약 86% 수준을 유지하며, 75% 누락 시에도 74~81%로 유지되다가 90% 이상 누락 시 급격히 하락. +> - **중앙 (Outlier ratio, 이상치 비율)**: 공간 내 임의의 노이즈 점 추가 시 정확도 변화. $XYZ$ 좌표만 사용한 경우(파란 사각)와 밀도 정보($XYZ+\text{density}$, 갈색 다이아몬드)를 함께 사용한 경우 모두 이상치 비율 30%까지 70% 이상의 정확도를 견고하게 유지. +> - **우측 (Perturbation noise std, 점 섭동 노이즈)**: 점 좌표에 가우시안 노이즈(표준편차 $0 \sim 0.1$)를 가했을 때의 정확도 변화. 표준편차 0.05까지 약 80% 이상을 유지하다가 0.1에 도달하면 약 30%로 감소. + +points, if it has seen those during training. We evaluate two models: one trained on points with (x, y, z) coordinates; the other on (x, y, z) plus point density. The net has more than 80% accuracy even when 20% of the points are outliers. Fig [6](#page-7-2) right shows the net is robust to point perturbations. + +### 5.3. Visualizing PointNet + +In Fig [7,](#page-7-3) we visualize *critical point sets* CS and *upperbound shapes* NS (as discussed in Thm [2\)](#page-4-4) for some sample shapes S. The point sets between the two shapes will give exactly the same global shape feature f(S). + +We can see clearly from Fig [7](#page-7-3) that the *critical point sets* CS, those contributed to the max pooled feature, summarizes the skeleton of the shape. The *upper-bound shapes* NS illustrates the largest possible point cloud that give the same global shape feature f(S) as the input point cloud S. CS and NS reflect the robustness of PointNet, meaning that losing some non-critical points does not change the global shape signature f(S) at all. + +The NS is constructed by forwarding all the points in a edge-length-2 cube through the network and select points p whose point function values (h1(p), h2(p), · · · , hK(p)) are no larger than the global shape descriptor. + +![](PointNet_1612.00593_images/_page_7_Figure_7.jpeg) + +Figure 7. Critical points and upper bound shape. While critical points jointly determine the global shape feature for a given shape, any point cloud that falls between the critical points set and the upper bound shape gives exactly the same feature. We color-code all figures to show the depth information. + +> **[그림 해설]** PointNet이 학습한 임계 포인트 세트($\mathcal{C}_S$)와 상한 형상($\mathcal{N}_S$)의 시각화 (테이블, 권총, 램프, 스탠드 4종, 깊이에 따라 무지개색 코딩). +> - **1행 (Original Shape, $S$)**: 원본 입력 포인트 클라우드. +> - **2행 (Critical Point Sets, $\mathcal{C}_S$)**: Max Pooling 레이어의 1024개 글로벌 특징을 결정짓는 핵심 포인트들만 추출한 서브셋. 물체의 외곽 윤곽(스켈레톤) 형태를 띄며, 입력 포인트의 극히 일부만으로 구성됨. +> - **3행 (Upper-bound Shapes, $\mathcal{N}_S$)**: 글로벌 특징 벡터 $\mathbf{u}$의 출력을 변화시키지 않으면서 최대로 추가할 수 있는 포인트들의 점유 영역. 원본 형태를 둘러싼 두꺼운 볼륨 형태를 형성하여, 노이즈나 추가 점이 이 영역 내에 존재해도 네트워크 출력이 불변함을 입증. + +### 5.4. Time and Space Complexity Analysis + +Table [6](#page-7-4) summarizes space (number of parameters in the network) and time (floating-point operations/sample) complexity of our classification PointNet. We also compare PointNet to a representative set of volumetric and multiview based architectures in previous works. + +While MVCNN [\[23\]](#page-8-13) and Subvolume (3D CNN) [\[18\]](#page-8-10) achieve high performance, PointNet is orders more efficient in computational cost (measured in FLOPs/sample: *141x* and *8x* more efficient, respectively). Besides, PointNet is much more space efficient than MVCNN in terms of #param in the network (*17x* less parameters). Moreover, PointNet is much more scalable – it's space and time complexity is O(N) – *linear* in the number of input points. However, since convolution dominates computing time, multi-view method's time complexity grows *squarely* on image resolution and volumetric convolution based method grows *cubically* with the volume size. + +Empirically, PointNet is able to process more than one million points per second for point cloud classification (around 1K objects/second) or semantic segmentation (around 2 rooms/second) with a 1080X GPU on Tensor-Flow, showing great potential for real-time applications. + +| | #params | FLOPs/sample | +|--------------------|---------|--------------| +| PointNet (vanilla) | 0.8M | 148M | +| PointNet | 3.5M | 440M | +| Subvolume [18] | 16.6M | 3633M | +| MVCNN [23] | 60.0M | 62057M | + +Table 6. Time and space complexity of deep architectures for 3D data classification. PointNet (vanilla) is the classification PointNet without input and feature transformations. FLOP stands for floating-point operation. The "M" stands for million. Subvolume and MVCNN used pooling on input data from multiple rotations or views, without which they have much inferior performance. + +### 6. Conclusion + +In this work, we propose a novel deep neural network *PointNet* that directly consumes point cloud. Our network provides a unified approach to a number of 3D recognition tasks including object classification, part segmentation and semantic segmentation, while obtaining on par or better results than state of the arts on standard benchmarks. We also provide theoretical analysis and visualizations towards understanding of our network. + +Acknowledgement. The authors gratefully acknowledge the support of a Samsung GRO grant, ONR MURI N00014- 13-1-0341 grant, NSF grant IIS-1528025, a Google Focused Research Award, a gift from the Adobe corporation and hardware donations by NVIDIA. + +### References + +- [1] I. Armeni, O. Sener, A. R. Zamir, H. Jiang, I. Brilakis, M. Fischer, and S. Savarese. 3d semantic parsing of large-scale indoor spaces. In *Proceedings of the IEEE International Conference on Computer Vision and Pattern Recognition*, 2016. [6,](#page-5-3) [7](#page-6-4) +- [2] M. Aubry, U. Schlickewei, and D. Cremers. The wave kernel signature: A quantum mechanical approach to shape analysis. In *Computer Vision Workshops (ICCV Workshops), 2011 IEEE International Conference on*, pages 1626–1633. IEEE, 2011. [2](#page-1-0) +- [3] M. M. Bronstein and I. Kokkinos. Scale-invariant heat kernel signatures for non-rigid shape recognition. In *Computer Vision and Pattern Recognition (CVPR), 2010 IEEE Conference on*, pages 1704–1711. IEEE, 2010. [2](#page-1-0) +- [4] J. Bruna, W. Zaremba, A. Szlam, and Y. LeCun. Spectral networks and locally connected networks on graphs. *arXiv preprint arXiv:1312.6203*, 2013. [2](#page-1-0) +- [5] D.-Y. Chen, X.-P. Tian, Y.-T. Shen, and M. Ouhyoung. On visual similarity based 3d model retrieval. In *Computer graphics forum*, volume 22, pages 223–232. Wiley Online Library, 2003. [2](#page-1-0) +- [6] Y. Fang, J. Xie, G. Dai, M. Wang, F. Zhu, T. Xu, and E. Wong. 3d deep shape descriptor. In *Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition*, pages 2319–2328, 2015. [2](#page-1-0) +- [7] M. Gschwandtner, R. Kwitt, A. Uhl, and W. Pree. BlenSor: Blender Sensor Simulation Toolbox Advances in Visual Computing. volume 6939 of *Lecture Notes in Computer Science*, chapter 20, pages 199–208. Springer Berlin / Heidelberg, Berlin, Heidelberg, 2011. [6](#page-5-3) +- [8] K. Guo, D. Zou, and X. Chen. 3d mesh labeling via deep convolutional neural networks. *ACM Transactions on Graphics (TOG)*, 35(1):3, 2015. [2](#page-1-0) +- [9] M. Jaderberg, K. Simonyan, A. Zisserman, et al. Spatial transformer networks. In *NIPS 2015*. [4](#page-3-2) +- [10] A. E. Johnson and M. Hebert. Using spin images for efficient object recognition in cluttered 3d scenes. *IEEE Transactions on pattern analysis and machine intelligence*, 21(5):433– 449, 1999. [2](#page-1-0) +- [11] M. Kazhdan, T. Funkhouser, and S. Rusinkiewicz. Rotation invariant spherical harmonic representation of 3 d shape descriptors. In *Symposium on geometry processing*, volume 6, pages 156–164, 2003. [5](#page-4-5) +- [12] Y. LeCun, L. Bottou, Y. Bengio, and P. Haffner. Gradientbased learning applied to document recognition. *Proceedings of the IEEE*, 86(11):2278–2324, 1998. [13](#page-12-0) +- [13] Y. Li, S. Pirk, H. Su, C. R. Qi, and L. J. Guibas. Fpnn: Field probing neural networks for 3d data. *arXiv preprint arXiv:1605.06240*, 2016. [2](#page-1-0) +- [14] H. Ling and D. W. Jacobs. Shape classification using the inner-distance. *IEEE transactions on pattern analysis and machine intelligence*, 29(2):286–299, 2007. [2](#page-1-0) +- [15] L. v. d. Maaten and G. Hinton. Visualizing data using t-sne. *Journal of Machine Learning Research*, 9(Nov):2579–2605, 2008. [15](#page-14-0) + +- [16] J. Masci, D. Boscaini, M. Bronstein, and P. Vandergheynst. Geodesic convolutional neural networks on riemannian manifolds. In *Proceedings of the IEEE International Conference on Computer Vision Workshops*, pages 37–45, 2015. [2](#page-1-0) +- [17] D. Maturana and S. Scherer. Voxnet: A 3d convolutional neural network for real-time object recognition. In *IEEE/RSJ International Conference on Intelligent Robots and Systems*, September 2015. [2,](#page-1-0) [5,](#page-4-5) [10,](#page-9-0) [11](#page-10-0) +- [18] C. R. Qi, H. Su, M. Nießner, A. Dai, M. Yan, and L. Guibas. Volumetric and multi-view cnns for object classification on 3d data. In *Proc. Computer Vision and Pattern Recognition (CVPR), IEEE*, 2016. [2,](#page-1-0) [5,](#page-4-5) [8](#page-7-5) +- [19] R. B. Rusu, N. Blodow, and M. Beetz. Fast point feature histograms (fpfh) for 3d registration. In *Robotics and Automation, 2009. ICRA'09. IEEE International Conference on*, pages 3212–3217. IEEE, 2009. [2](#page-1-0) +- [20] R. B. Rusu, N. Blodow, Z. C. Marton, and M. Beetz. Aligning point cloud views using persistent feature histograms. In *2008 IEEE/RSJ International Conference on Intelligent Robots and Systems*, pages 3384–3391. IEEE, 2008. [2](#page-1-0) +- [21] M. Savva, F. Yu, H. Su, M. Aono, B. Chen, D. Cohen-Or, W. Deng, H. Su, S. Bai, X. Bai, et al. Shrec16 track largescale 3d shape retrieval from shapenet core55. [2](#page-1-0) +- [22] P. Y. Simard, D. Steinkraus, and J. C. Platt. Best practices for convolutional neural networks applied to visual document analysis. In *ICDAR*, volume 3, pages 958–962, 2003. [13](#page-12-0) +- [23] H. Su, S. Maji, E. Kalogerakis, and E. G. Learned-Miller. Multi-view convolutional neural networks for 3d shape recognition. In *Proc. ICCV, to appear*, 2015. [2,](#page-1-0) [5,](#page-4-5) [6,](#page-5-3) [8](#page-7-5) +- [24] J. Sun, M. Ovsjanikov, and L. Guibas. A concise and provably informative multi-scale signature based on heat diffusion. In *Computer graphics forum*, volume 28, pages 1383–1392. Wiley Online Library, 2009. [2](#page-1-0) +- [25] O. Vinyals, S. Bengio, and M. Kudlur. Order matters: Sequence to sequence for sets. *arXiv preprint arXiv:1511.06391*, 2015. [2,](#page-1-0) [4,](#page-3-2) [7](#page-6-4) +- [26] D. Z. Wang and I. Posner. Voting for voting in online point cloud object detection. *Proceedings of the Robotics: Science and Systems, Rome, Italy*, 1317, 2015. [2](#page-1-0) +- [27] Z. Wu, R. Shou, Y. Wang, and X. Liu. Interactive shape cosegmentation via label propagation. *Computers & Graphics*, 38:248–254, 2014. [6,](#page-5-3) [10](#page-9-0) +- [28] Z. Wu, S. Song, A. Khosla, F. Yu, L. Zhang, X. Tang, and J. Xiao. 3d shapenets: A deep representation for volumetric shapes. In *Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition*, pages 1912–1920, 2015. [2,](#page-1-0) [5,](#page-4-5) [11](#page-10-0) +- [29] L. Yi, V. G. Kim, D. Ceylan, I.-C. Shen, M. Yan, H. Su, C. Lu, Q. Huang, A. Sheffer, and L. Guibas. A scalable active framework for region annotation in 3d shape collections. *SIGGRAPH Asia*, 2016. [6,](#page-5-3) [10,](#page-9-0) [18](#page-17-0) + +### Supplementary + +### A. Overview + +This document provides additional quantitative results, technical details and more qualitative test examples to the main paper. + +In Sec [B](#page-9-1) we extend the robustness test to compare PointNet with VoxNet on incomplete input. In Sec [C](#page-9-2) we provide more details on neural network architectures, training parameters and in Sec [D](#page-10-1) we describe our detection pipeline in scenes. Then Sec [E](#page-10-2) illustrates more applications of PointNet, while Sec [F](#page-11-0) shows more analysis experiments. Sec [G](#page-12-1) provides a proof for our theory on PointNet. At last, we show more visualization results in Sec [H.](#page-14-1) + +## B. Comparison between PointNet and VoxNet (Sec 5.2) + +We extend the experiments in Sec 5.2 Robustness Test to compare PointNet and VoxNet [\[17\]](#page-8-9) (a representative architecture for volumetric representation) on robustness to missing data in the input point cloud. Both networks are trained on the same train test split with 1024 number of points as input. For VoxNet we voxelize the point cloud to 32 × 32 × 32 occupancy grids and augment the training data by random rotation around up-axis and jittering. + +At test time, input points are randomly dropped out by a certain ratio. As VoxNet is sensitive to rotations, its prediction uses average scores from 12 viewpoints of a point cloud. As shown in Fig [8,](#page-9-3) we see that our PointNet is much more robust to missing points. VoxNet's accuracy dramatically drops when half of the input points are missing, from 86.3% to 46.0% with a 40.3% difference, while our PointNet only has a 3.7% performance drop. This can be explained by the theoretical analysis and explanation of our PointNet – it is learning to use a collection of *critical points* to summarize the shape, thus it is very robust to missing data. + +## C. Network Architecture and Training Details (Sec 5.1) + +PointNet Classification Network As the basic architecture is already illustrated in the main paper, here we provides more details on the joint alignment/transformation network and training parameters. + +The first transformation network is a mini-PointNet that takes raw point cloud as input and regresses to a 3 × 3 matrix. It's composed of a shared MLP(64, 128, 1024) network (with layer output sizes 64, 128, 1024) on each point, a max pooling across points and two fully connected layers with output sizes 512, 256. The output matrix is initialized as an identity matrix. All layers, except the last one, include ReLU and batch normalization. The second + +![](PointNet_1612.00593_images/_page_9_Figure_10.jpeg) + +Figure 8. PointNet v.s. VoxNet [\[17\]](#page-8-9) on incomplete input data. Metric is overall classification accurcacy on ModelNet40 test set. Note that VoxNet is using 12 viewpoints averaging while PointNet is using only one view of the point cloud. Evidently PointNet presents much stronger robustness to missing points. + +> **[그림 해설]** 데이터 결손율(Missing Data Ratio, 0~1.0)에 따른 PointNet(파란 원)과 3D 복셀 기반 VoxNet(빨간 사각)의 분류 정확도(Accuracy %) 비교 그래프. +> - 데이터 결손이 0일 때 두 모델 모두 약 87%의 정확도로 시작함. +> - 데이터 결손율이 50%(0.5)일 때 PointNet은 약 84%의 높은 정확도를 유지하는 반면, VoxNet은 46% 수준으로 급락함. +> - 결손율 75%(0.75)에서도 PointNet은 약 74%를 기록하나 VoxNet은 18%로 추락하여, 불완전/부분 스캔 데이터에 대한 PointNet의 압도적인 강건성을 실증함. + +transformation network has the same architecture as the first one except that the output is a 64 × 64 matrix. The matrix is also initialized as an identity. A regularization loss (with weight 0.001) is added to the softmax classification loss to make the matrix close to orthogonal. + +We use dropout with keep ratio 0.7 on the last fully connected layer, whose output dimension 256, before class score prediction. The decay rate for batch normalization starts with 0.5 and is gradually increased to 0.99. We use adam optimizer with initial learning rate 0.001, momentum 0.9 and batch size 32. The learning rate is divided by 2 every 20 epochs. Training on ModelNet takes 3-6 hours to converge with TensorFlow and a GTX1080 GPU. + +PointNet Segmentation Network The segmentation network is an extension to the classification PointNet. Local point features (the output after the second transformation network) and global feature (output of the max pooling) are concatenated for each point. No dropout is used for segmentation network. Training parameters are the same as the classification network. + +As to the task of shape part segmentation, we made a few modifications to the basic segmentation network architecture (Fig 2 in main paper) in order to achieve best performance, as illustrated in Fig [9.](#page-10-3) We add a one-hot vector indicating the class of the input and concatenate it with the max pooling layer's output. We also increase neurons in some layers and add skip links to collect local point features in different layers and concatenate them to form point feature input to the segmentation network. + +While [\[27\]](#page-8-22) and [\[29\]](#page-8-23) deal with each object category independently, due to the lack of training data for some categories (the total number of shapes for all the categories in the data set are shown in the first line), we train our PointNet across categories (but with one-hot vector input to indicate category). To allow fair comparison, when testing + +![](PointNet_1612.00593_images/_page_10_Figure_0.jpeg) + +Figure 9. **Network architecture for part segmentation.** T1 and T2 are alignment/transformation networks for input points and features. FC is fully connected layer operating on each point. MLP is multi-layer perceptron on each point. One-hot is a vector of size 16 indicating category of the input shape. + +> **[그림 해설]** ShapeNet Part Segmentation을 위한 심층 PointNet 세그멘테이션 네트워크 구조 다이어그램. +> - 입력 $n \times 3$ $\to$ T1(Spatial Transform) $\to n \times 3 \to$ FC(64) $\to n \times 64 \to$ FC(128) $\to n \times 128 \to$ FC(128) $\to n \times 128 \to$ T2(Feature Transform) $\to n \times 128 \to$ FC(512) $\to n \times 512 \to$ FC(2048) $\to n \times 2048 \to$ Max Pooling $\to 2048$차원 Global Feature. +> - 다중 계층 특징 결합: 각 단계의 점별 특징($64 + 128 + 128 + 128 + 512 = 960$), 2048차원 글로벌 특징, 그리고 객체 카테고리를 나타내는 one-hot 벡터를 모두 결합하여 $n \times 3024$ 통합 특징 벡터 구성. +> - 최종 MLP(256, 256, 128)을 거쳐 $n \times 50$ 파트 예측 점수(part scores) 출력. + +![](PointNet_1612.00593_images/_page_10_Figure_2.jpeg) + +Figure 10. **Baseline 3D CNN segmentation network.** The network is fully convolutional and predicts part scores for each voxel. + +> **[그림 해설]** 파트 분할 성능 비교를 위한 3D 복셀 기반 CNN(Voxel-CNN) 베이스라인 아키텍처 다이어그램. +> - $32 \times 32 \times 32$ 크기의 3D 복셀 그리드 입력을 받음. +> - 인코더: 32개 필터(커널 5, stride 1) 컨볼루션 4회 $\to$ 32개 필터(커널 3, stride 1) 컨볼루션 1회를 거쳐 형상 특징 추출. +> - 디코더: 1개 크기의 특징을 스킵 연결(Skip Connection) 및 64 필터 $\to$ 64 필터 $\to$ 50 필터(커널 1, stride 1) 컨볼루션을 거쳐 $32 \times 32 \times 32$ 복셀 공간에 대해 각 복셀의 파트 카테고리를 예측(in-category prediction). + +these two models, we only predict part labels for the given specific object category. + +As to semantic segmentation task, we used the architecture as in Fig 2 in the main paper. + +It takes around six to twelve hours to train the model on ShapeNet part dataset and around half a day to train on the Stanford semantic parsing dataset. + +Baseline 3D CNN Segmentation Network In ShapeNet part segmentation experiment, we compare our proposed segmentation version PointNet to two traditional methods as well as a 3D volumetric CNN network baseline. In Fig 10, we show the baseline 3D volumetric CNN network we use. We generalize the well-known 3D CNN architectures, such as VoxNet [17] and 3DShapeNets [28] to a fully convolutional 3D CNN segmentation network. + +For a given point cloud, we first convert it to the volumetric representation as a occupancy grid with resolution $32 \times 32 \times 32$ . Then, five 3D convolution operations each with 32 output channels and stride of 1 are sequentially applied to extract features. The receptive field is 19 for each voxel. Finally, a sequence of 3D convolutional layers with kernel size $1 \times 1 \times 1$ is appended to the computed feature map to predict segmentation label for each voxel. ReLU and + +batch normalization are used for all layers except the last one. The network is trained across categories, however, in order to compare with other baseline methods where object category is given, we only consider output scores in the given object category. + +### D. Details on Detection Pipeline (Sec 5.1) + +We build a simple 3D object detection system based on the semantic segmentation results and our object classification PointNet. + +We use connected component with segmentation scores to get object proposals in scenes. Starting from a random point in the scene, we find its predicted label and use BFS to search nearby points with the same label, with a search radius of 0.2 meter. If the resulted cluster has more than 200 points (assuming a 4096 point sample in a 1m by 1m area), the cluster's bounding box is marked as one object proposal. For each proposed object, it's detection score is computed as the average point score for that category. Before evaluation, proposals with extremely small areas/volumes are pruned. For tables, chairs and sofas, the bounding boxes are extended to the floor in case the legs are separated with the seat/surface. + +We observe that in some rooms such as auditoriums lots of objects (e.g. chairs) are close to each other, where connected component would fail to correctly segment out individual ones. Therefore we leverage our classification network and uses sliding shape method to alleviate the problem for the chair class. We train a binary classification network for each category and use the classifier for sliding window detection. The resulted boxes are pruned by non-maximum suppression. The proposed boxes from connected component and sliding shapes are combined for final evaluation. + +In Fig 11, we show the precision-recall curves for object detection. We trained six models, where each one of them is trained on five areas and tested on the left area. At test phase, each model is tested on the area it has never seen. The test results for all six areas are aggregated for the PR curve generation. + +#### E. More Applications (Sec 5.1) + +Model Retrieval from Point Cloud Our PointNet learns a global shape signature for every given input point cloud. We expect geometrically similar shapes have similar global signature. In this section, we test our conjecture on the shape retrieval application. To be more specific, for every given query shape from ModelNet test split, we compute its global signature (output of the layer before the score prediction layer) given by our classification PointNet and retrieve similar shapes in the train split by nearest neighbor search. Results are shown in Fig 12. + +![](PointNet_1612.00593_images/_page_11_Figure_0.jpeg) + +Figure 11. Precision-recall curves for object detection in 3D point cloud. We evaluated on all six areas for four categories: table, chair, sofa and board. IoU threshold is 0.5 in volume. + +> **[그림 해설]** 3D 씬 객체 검출(Object Detection)에서 주요 4개 카테고리에 대한 Precision-Recall(정밀도-재현율) PR 곡선. +> - **table (상단 좌측)**: Recall 0.5 부근까지 Precision 0.7~0.8 이상을 유지하다가 점진적으로 하강 (최종 Recall 약 0.68). +> - **chair (상단 우측)**: Precision이 0.8에서 출발하여 Recall 0.65 부근(Precision 약 0.2)까지 완만하게 선형 하강. +> - **sofa (하단 좌측)**: 초기 급격한 하강 후 낮은 Precision 영역에서 Recall 0.24 부근까지 형성. +> - **board (하단 우측)**: 초기 Precision 약 0.8까지 상승 후 Recall 0.2 부근에서 급격히 감소. + +![](PointNet_1612.00593_images/_page_11_Picture_2.jpeg) + +Figure 12. Model retrieval from point cloud. For every given point cloud, we retrieve the top-5 similar shapes from the ModelNet test split. From top to bottom rows, we show examples of chair, plant, nightstand and bathtub queries. Retrieved results that are in wrong category are marked by red boxes. + +> **[그림 해설]** 불완전 쿼리 포인트 클라우드에 대한 PointNet 글로벌 특징 기반 Top-5 CAD 모델 검색(3D Shape Retrieval) 결과. +> - 4개 쿼리(좌측: 불완전 스캔 의자, 식물, 캐비닛/수납장, 세면대)에 대해 가장 유사한 CAD 모델 5개를 우측에 순서대로 배열. +> - 1~3행의 의자, 식물, 가구는 모두 올바른 카테고리의 유사 형상이 완벽히 검색됨. +> - 4행(세면대 쿼리)의 경우 1번째, 3번째, 5번째는 세면대(sink, 빨간 사각 박스)로 올바르게 검색되었으나, 2번째와 4번째는 외형이 유사한 욕조(bathtub)가 검색된 검색 실패/혼동 사례를 표시. + +Shape Correspondence In this section, we show that point features learnt by PointNet can be potentially used to compute shape correspondences. Given two shapes, we compute the correspondence between their *critical point sets* CS's by matching the pairs of points that activate the same dimensions in the global features. Fig [13](#page-11-3) and Fig [14](#page-11-4) show the detected shape correspondence between two similar chairs and tables. + +### F. More Architecture Analysis (Sec 5.2) + +Effects of Bottleneck Dimension and Number of Input Points Here we show our model's performance change with regard to the size of the first max layer output as well as the number of input points. In Fig [15](#page-11-5) we see that performance grows as we increase the number of points however it saturates at around 1K points. The max layer size plays an important role, increasing the layer size from + +![](PointNet_1612.00593_images/_page_11_Figure_7.jpeg) + +Figure 13. Shape correspondence between two chairs. For the clarity of the visualization, we only show 20 randomly picked correspondence pairs. + +> **[그림 해설]** 서로 다른 두 개의 의자 포인트 클라우드(빨간색, 파란색) 간의 기하학적 부품 대응점(Shape Correspondence) 시각화. +> - 무작위로 선택된 20개 대응 쌍을 색상 직선으로 연결. +> - 등받이 상단, 좌판 모서리, 의자 다리 끝단 등 구조적으로 일치하는 부품 위치끼리 평행하게 연결되어 PointNet 특징이 의미론적 형상 대응을 정확히 학습했음을 입증. + +![](PointNet_1612.00593_images/_page_11_Figure_9.jpeg) + +Figure 14. Shape correspondence between two tables. For the clarity of the visualization, we only show 20 randomly picked correspondence pairs. + +> **[그림 해설]** 형태가 서로 다른 두 개의 테이블 포인트 클라우드(빨간색 직사각형 테이블, 파란색 원형/타원형 테이블) 간 형상 대응 시각화. +> - 무작위 20개 포인트 쌍이 상판의 둘레, 모서리, 테이블 다리 하단부 등 대응하는 구조적 위치로 정확히 매핑되어 연결선을 형성. + +64 to 1024 results in a 2−4% performance gain. It indicates that we need enough point feature functions to cover the 3D space in order to discriminate different shapes. + +It's worth notice that even with 64 points as input (obtained from furthest point sampling on meshes), our network can achieve decent performance. + +![](PointNet_1612.00593_images/_page_11_Figure_13.jpeg) + +Figure 15. Effects of bottleneck size and number of input points. The metric is overall classification accuracy on Model-Net40 test set. + +> **[그림 해설]** 병목 차원(Bottleneck size, 64~1024) 및 입력 포인트 수(#points: 64, 128, 512, 1024, 2048)에 따른 ModelNet40 테스트 분류 정확도(Accuracy %) 변화 그래프. +> - X축: Bottleneck size (0, 200, 400, 600, 800, 1000). Y축: Accuracy % (81% ~ 88%). +> - 입력 포인트 수가 64개(하늘색 사각)일 때는 정확도 82~84.5% 수준이나, 1024개(초록 삼각) 및 2048개(주황 원)로 증가하면 87% 이상으로 향상됨. +> - 병목 크기 256~512 이상에서 성능이 포화(87.3% 수준)에 도달하며, 1024 차원에서 최대 성능을 기록함. + +MNIST Digit Classification While we focus on 3D point cloud learning, a sanity check experiment is to apply our network on a 2D point clouds - pixel sets. + +To convert an MNIST image into a 2D point set we threshold pixel values and add the pixel (represented as a point with (x, y) coordinate in the image) with values larger than 128 to the set. We use a set size of 256. If there are more than 256 pixels int he set, we randomly sub-sample it; if there are less, we pad the set with the one of the pixels in the set (due to our max operation, which point to use for the padding will not affect outcome). + +As seen in Table [7,](#page-12-2) we compare with a few baselines including multi-layer perceptron that considers input image as an ordered vector, a RNN that consider input as sequence from pixel (0,0) to pixel (27,27), and a vanilla version CNN. While the best performing model on MNIST is still well engineered CNNs (achieving less than 0.3% error rate), it's interesting to see that our PointNet model can achieve reasonable performance by considering image as a 2D point set. + +| | input | error (%) | +|-----------------------------|-----------|-----------| +| Multi-layer perceptron [22] | vector | 1.60 | +| LeNet5 [12] | image | 0.80 | +| Ours PointNet | point set | 0.78 | + +Table 7. MNIST classification results. We compare with vanilla versions of other deep architectures to show that our network based on point sets input is achieving reasonable performance on this traditional task. + +Normal Estimation In segmentation version of PointNet, local point features and global feature are concatenated in order to provide context to local points. However, it's unclear whether the context is learnt through this concatenation. In this experiment, we validate our design by showing that our segmentation network can be trained to predict point normals, a local geometric property that is determined by a point's neighborhood. + +We train a modified version of our segmentation Point-Net in a supervised manner to regress to the groundtruth point normals. We just change the last layer of our segmentation PointNet to predict normal vector for each point. We use absolute value of cosine distance as loss. + +Fig. [16](#page-12-3) compares our PointNet normal prediction results (the left columns) to the ground-truth normals computed from the mesh (the right columns). We observe a reasonable normal reconstruction. Our predictions are more smooth and continuous than the ground-truth which includes flipped normal directions in some region. + +Segmentation Robustness As discussed in Sec 5.2 and Sec [B,](#page-9-1) our PointNet is less sensitive to data corruption and missing points for classification tasks since the global shape feature is extracted from a collection of *critical points* from the given input point cloud. In this section, we show that the robustness holds for segmentation tasks too. The per-point part labels are predicted based on the combination of perpoint features and the learnt global shape feature. In Fig [17,](#page-13-0) + +![](PointNet_1612.00593_images/_page_12_Figure_8.jpeg) + +Figure 16. PointNet normal reconstrution results. In this figure, we show the reconstructed normals for all the points in some sample point clouds and the ground-truth normals computed on the mesh. + +> **[그림 해설]** PointNet을 이용한 포인트별 표면 법선 벡터(Normal Vector) 재구성 결과 비교. +> - 3개 객체 샘플(의자 등받이/좌판, 비행기, 변기)에 대해 PointNet이 예측한 법선 벡터(Prediction, 좌측)와 3D 메시에서 계산된 실제 정답 법선 벡터(Ground-truth, 우측)를 파란색 선분으로 시각화. +> - 각 점의 국소 기하 구조(평면, 곡면, 경계선)에 수직인 법선 방향이 GT와 거의 완벽하게 일치하게 재구성됨. + +we illustrate the segmentation results for the given input point clouds S (the left-most column), the *critical point sets* CS (the middle column) and the *upper-bound shapes* NS. + +Network Generalizability to Unseen Shape Categories In Fig [18,](#page-13-1) we visualize the *critical point sets* and the *upperbound shapes* for new shapes from unseen categories (face, house, rabbit, teapot) that are not present in ModelNet or ShapeNet. It shows that the learnt per-point functions are generalizable. However, since we train mostly on manmade objects with lots of planar structures, the reconstructed upper-bound shape in novel categories also contain more planar surfaces. + +### G. Proof of Theorem (Sec 4.3) + +Let +$$\mathcal{X} = \{S : S \subseteq [0, 1] \text{ and } |S| = n\}.$$ + +f : X → R is a continuous function on X w.r.t to Hausdorff distance dH(·, ·) if the following condition is satisfied: + +$$\forall \epsilon > 0, \exists \delta > 0$$ +, for any $S, S' \in \mathcal{X}$ , if $d_H(S, S') < \delta$ , then $|f(S) - f(S')| < \epsilon$ . + +We show that f can be approximated arbitrarily by composing a symmetric function and a continuous function. + +![](PointNet_1612.00593_images/_page_13_Figure_0.jpeg) + +Input Point Cloud Critical Point Sets Upper-bound Shapes + +Figure 17. The consistency of segmentation results. We illustrate the segmentation results for some sample given point clouds S, their *critical point sets* $\mathcal{C}_S$ and *upper-bound shapes* $\mathcal{N}_S$ . We observe that the shape family between the $\mathcal{C}_S$ and $\mathcal{N}_S$ share a consistent segmentation results. + +> **[그림 해설]** 형상 집합군에서의 파트 세그멘테이션 일관성 시각화 (탁자, 머그컵, 자동차 3개 예시). +> - **1열 (Input Point Cloud, $S$)**: 원본 포인트 클라우드와 예측된 파트 분할 결과 (색상별 부품 분할). +> - **2열 (Critical Point Sets, $\mathcal{C}_S$)**: 해당 형상의 임계 포인트들만 남긴 서브셋에서도 원본과 동일한 부품 경계와 분할 결과가 유지됨. +> - **3열 (Upper-bound Shapes, $\mathcal{N}_S$)**: 포인트가 두껍게 확장된 상한 형상에서도 동일한 파트 분할 레이블이 완벽하게 일관성을 유지하며 보존됨을 확인. + +![](PointNet_1612.00593_images/_page_13_Figure_3.jpeg) + +Figure 18. The critical point sets and the upper-bound shapes for unseen objects. We visualize the *critical point sets* and the *upper-bound shapes* for teapot, bunny, hand and human body, which are not in the ModelNet or ShapeNet shape repository to test the generalizability of the learnt per-point functions of our PointNet on other unseen objects. The images are color-coded to reflect the depth information. + +> **[그림 해설]** 훈련 세트(ModelNet/ShapeNet)에 포함되지 않은 새로운 미학습 객체 4종(주전자 Teapot, 토끼 Stanford Bunny, 손 Hand, 인체 Human body)에 대한 일반화 성능 평가. +> - **1행 (Original Shape)**: 원본 객체 포인트 클라우드 (깊이 정보에 따른 무지개색 코딩). +> - **2행 (Critical Point Sets)**: 임계 포인트 세트가 주전자 주구/손잡이, 토끼 귀/발, 손가락 끝/관절, 인체 사지 등 핵심 외곽 스켈레톤 구조를 정확히 포착. +> - **3행 (Upper-bound Shapes)**: 점유 상한 형상 또한 각 객체의 전체 볼륨 윤곽을 충실하게 감싸며 모델의 뛰어난 일반화 능력을 입증. + +**Theorem 1.** Suppose $f: \mathcal{X} \to \mathbb{R}$ is a continuous set function w.r.t Hausdorff distance $d_H(\cdot,\cdot)$ . $\forall \epsilon > 0$ , $\exists$ a continuous function h and a symmetric function $g(x_1, \dots, x_n) = \gamma \circ MAX$ , where $\gamma$ is a continuous function, MAX is a vector max operator that takes n vectors as input and returns a new vector of the element-wise maximum, such that for any $S \in \mathcal{X}$ , + +$$|f(S) - \gamma(MAX(h(x_1), \dots, h(x_n)))| < \epsilon$$ + +where $x_1, \ldots, x_n$ are the elements of S extracted in certain + +order, + +*Proof.* By the continuity of f, we take $\delta_{\epsilon}$ so that $|f(S) - f(S')| < \epsilon$ for any $S, S' \in \mathcal{X}$ if $d_H(S, S') < \delta_{\epsilon}$ . + +Define $K = \lceil 1/\delta_{\epsilon} \rceil$ , which split [0,1] into K intervals evenly and define an auxiliary function that maps a point to the left end of the interval it lies in: + +$$\sigma(x) = \frac{\lfloor Kx \rfloor}{K}$$ + +Let $\tilde{S} = {\sigma(x) : x \in S}$ , then + +$$|f(S) - f(\tilde{S})| < \epsilon$$ + +because $d_H(S, \tilde{S}) < 1/K \le \delta_{\epsilon}$ . + +Let $h_k(x) = e^{-d(x, \lfloor \frac{k-1}{K}, \frac{k}{K} \rfloor)}$ be a soft indicator function where d(x, I) is the point to set (interval) distance. Let $\mathbf{h}(x) = [h_1(x); \dots; h_K(x)]$ , then $\mathbf{h} : \mathbb{R} \to \mathbb{R}^K$ . + +Let $v_j(x_1, \ldots, x_n) = \max\{\hat{h}_j(x_1), \ldots, \hat{h}_j(x_n)\}$ , indicating the occupancy of the j-th interval by points in S. Let $\mathbf{v} = [v_1; \ldots; v_K]$ , then $\mathbf{v} : \underbrace{\mathbb{R} \times \ldots \times \mathbb{R}}_n \to \{0, 1\}^K$ + +is a symmetric function, indicating the occupancy of each interval by points in S. + +Define $\tau:\{0,1\}^K\to\mathcal{X}$ as $\tau(v)=\{\frac{k-1}{K}:v_k\geq 1\}$ , which maps the occupancy vector to a set which contains the left end of each occupied interval. It is easy to show: + +$$\tau(\mathbf{v}(x_1,\ldots,x_n)) \equiv \tilde{S}$$ + +where $x_1, \ldots, x_n$ are the elements of S extracted in certain order + +Let $\gamma:\mathbb{R}^K\to\mathbb{R}$ be a continuous function such that $\gamma(\mathbf{v})=f(\tau(\mathbf{v}))$ for $v\in\{0,1\}^K$ . Then, + +$$|\gamma(\mathbf{v}(x_1,\ldots,x_n)) - f(S)|$$ + +=|f(\tau(\mathbf{v}(x\_1,\ldots,x\_n))) - f(S)| < \epsilon + +Note that $\gamma(\mathbf{v}(x_1,\ldots,x_n))$ can be rewritten as follows: + +$$\gamma(\mathbf{v}(x_1,\ldots,x_n)) = \gamma(\mathbf{MAX}(\mathbf{h}(x_1),\ldots,\mathbf{h}(x_n)))$$ +$$= (\gamma \circ \mathbf{MAX})(\mathbf{h}(x_1),\ldots,\mathbf{h}(x_n))$$ + +Obviously $\gamma \circ MAX$ is a symmetric function. $\square$ + +Next we give the proof of Theorem 2. We define $\mathbf{u} = \underset{x_i \in S}{\operatorname{MAX}} \{h(x_i)\}$ to be the sub-network of f which maps a point set in $[0,1]^m$ to a K-dimensional vector. The following theorem tells us that small corruptions or extra noise points in the input set is not likely to change the output of our network: + +**Theorem 2.** Suppose $\mathbf{u}: \mathcal{X} \to \mathbb{R}^K$ such that $\mathbf{u} = \max_{x \in S} \{h(x_i)\}$ and $f = \gamma \circ \mathbf{u}$ . Then, + +*(a)* ∀S, ∃ CS, NS ⊆ X *,* f(T) = f(S) *if* CS ⊆ T ⊆ NS*;* + +*(b)* |CS| ≤ K + +*Proof.* Obviously, ∀S ∈ X , f(S) is determined by u(S). So we only need to prove that ∀S, ∃ CS, NS ⊆ X , f(T) = f(S)if CS ⊆ T ⊆ NS. + +For the jth dimension as the output of u, there exists at least one xj ∈ X such that hj (xj ) = uj , where hj is the jth dimension of the output vector from h. Take CS as the union of all xj for j = 1, . . . , K. Then, CS satisfies the above condition. + +Adding any additional points x such that h(x) ≤ u(S) at all dimensions to CS does not change u, hence f. Therefore, TS can be obtained adding the union of all such points to NS. + +![](PointNet_1612.00593_images/_page_14_Picture_5.jpeg) + +Figure 19. Point function visualization. For each per-point function h, we calculate the values h(p) for all the points p in a cube of diameter two located at the origin, which spatially covers the unit sphere to which our input shapes are normalized when training our PointNet. In this figure, we visualize all the points p that give h(p) > 0.5 with function values color-coded by the brightness of the voxel. We randomly pick 15 point functions and visualize the activation regions for them. + +> **[그림 해설]** PointNet 내부의 per-point 함수 $h$ 중 무작위로 선택된 15개 뉴런의 3D 공간 활성화 영역($h(p) > 0.5$) 시각화. +> - 원점을 중심으로 한 3D 큐브 공간 내에서 각 뉴런이 활성화되는 3D 볼륨(회색 음영 영역)을 3행 5열 격자로 배열. +> - 특정 뉴런은 구의 상단 반구, 모서리 단면, 쐐기형 슬라이스, 중심부 국소 구체 등 다양한 형태의 기하학적 3D 공간 필터(공간 분할 함수) 역할을 수행함을 확인. + +### H. More Visualizations + +Classification Visualization We use t-SNE[\[15\]](#page-8-28) to embed point cloud global signature (1024-dim) from our classification PointNet into a 2D space. Fig [20](#page-15-0) shows the embedding space of ModelNet 40 test split shapes. Similar shapes are clustered together according to their semantic categories. + +Segmentation Visualization We present more segmentation results on both complete CAD models and simulated Kinect partial scans. We also visualize failure cases with error analysis. Fig [21](#page-16-0) and Fig [22](#page-16-1) show more segmentation results generated on complete CAD models and their simulated Kinect scans. Fig [23](#page-17-1) illustrates some failure cases. Please read the caption for the error analysis. + +Scene Semantic Parsing Visualization We give a visualization of semantic parsing in Fig [24](#page-18-0) where we show input point cloud, prediction and ground truth for both semantic segmentation and object detection for two office rooms and one conference room. The area and the rooms are unseen in the training set. + +Point Function Visualization Our classification Point-Net computes K (we take K = 1024 in this visualization) dimension point features for each point and aggregates all the per-point local features via a max pooling layer into a single K-dim vector, which forms the global shape descriptor. + +To gain more insights on what the learnt per-point functions h's detect, we visualize the points pi's that give high per-point function value f(pi) in Fig [19.](#page-14-2) This visualization clearly shows that different point functions learn to detect for points in different regions with various shapes scattered in the whole space. + +![](PointNet_1612.00593_images/_page_15_Figure_0.jpeg) + +Figure 20. 2D embedding of learnt shape global features. We use t-SNE technique to visualize the learnt global shape features for the shapes in ModelNet40 test split. + +> **[그림 해설]** ModelNet40 테스트셋 전체 형상들에 대해 PointNet이 학습한 1024차원 글로벌 형상 특징 벡터의 2D t-SNE 임베딩 시각화. +> - 비행기, 의자, 테이블, 자동차, 병, 화분 등 동일 범주 및 유사 기하 구조를 가진 3D 객체들이 2차원 공간 상에서 밀집된 클러스터를 형성하고 상호 분리됨을 보여줌. + +![](PointNet_1612.00593_images/_page_16_Figure_0.jpeg) + +Figure 21. PointNet segmentation results on complete CAD models. + +> **[그림 해설]** 완전한 ShapeNet 3D CAD 모델에 대한 파트 분할 결과 (총 16개 카테고리, 카테고리당 3개씩 총 48개 객체 시각화). +> - 좌측: airplane, bag, cap, car, chair, earphone, guitar, knife. +> - 우측: rocket, pistol, table, skateboard, motorbike, mug, laptop, lamp. +> - 세부 부품(비행기 엔진, 가방 손잡이, 자동차 바퀴/유리, 의자 등받이/다리, 헤드폰 밴드/패드, 기타 넥/바디 등)이 선명한 고유 색상으로 정확하게 분할됨. + +![](PointNet_1612.00593_images/_page_16_Figure_2.jpeg) + +Figure 22. PointNet segmentation results on simulated Kinect scans. + +> **[그림 해설]** 시뮬레이션된 Kinect 스캔(단면 결손 및 가림이 존재하는 부분 스캔 데이터)에 대한 파트 분할 결과 (16개 카테고리별 각 3개씩 시각화). +> - 심한 점 결손과 가림 현상 속에서도 비행기 날개, 자동차 프레임, 의자 좌판, 모자 챙 등 부품 영역을 견고하게 인식하여 분할함. + +![](PointNet_1612.00593_images/_page_17_Figure_0.jpeg) + +Figure 23. PointNet segmentation failure cases. In this figure, we summarize six types of common errors in our segmentation application. The prediction and the ground-truth segmentations are given in the first and second columns, while the difference maps are computed and shown in the third columns. The red dots correspond to the wrongly labeled points in the given point clouds. (a) illustrates the most common failure cases: the points on the boundary are wrongly labeled. In the examples, the label predictions for the points near the intersections between the table/chair legs and the tops are not accurate. However, most segmentation algorithms suffer from this error. (b) shows the errors on exotic shapes. For examples, the chandelier and the airplane shown in the figure are very rare in the data set. (c) shows that small parts can be overwritten by nearby large parts. For example, the jet engines for airplanes (yellow in the figure) are mistakenly classified as body (green) or the plane wing (purple). (d) shows the error caused by the inherent ambiguity of shape parts. For example, the two bottoms of the two tables in the figure are classified as table legs and table bases (category *other* in [\[29\]](#page-8-23)), while ground-truth segmentation is the opposite. (e) illustrates the error introduced by the incompleteness of the partial scans. For the two caps in the figure, almost half of the point clouds are missing. (f) shows the failure cases when some object categories have too less training data to cover enough variety. There are only 54 bags and 39 caps in the whole dataset for the two categories shown here. + +> **[그림 해설]** PointNet 세그멘테이션의 6가지 주요 실패/오류 유형 분석 ((a)~(f) 각 유형별로 1열: Prediction, 2열: Ground-truth, 3열: 빨간 점으로 표시된 Difference map). +> - **(a) Boundary Error (경계 오류)**: 탁자/의자 상판과 다리 결합 부위 등 경계면 포인트의 오분류 (가장 흔한 오류). +> - **(b) Exotic Shapes (특이 형상)**: 샹들리에나 특이한 형태의 비행기 등 훈련 세트에 드문 희귀 형상에서의 오류. +> - **(c) Overwriting Small Parts (소형 부품 덮어쓰기)**: 제트 엔진(노란색) 같은 작은 부품이 인접한 큰 부품(동체/날개) 레이블로 덮어쓰여지는 현상. +> - **(d) Inherent Part Ambiguity (부품 정의의 모호성)**: 테이블 하단 지지 구조를 다리(leg)로 보는지 받침대(base)로 보는지에 대한 GT와의 정의 불일치. +> - **(e) Incomplete Scans (불완전 스캔)**: 모자 포인트 클라우드의 절반 가까이가 누락되어 형상 전체 맥락 파악 실패. +> - **(f) Lack of Training Data (데이터 부족)**: 가방(54개), 모자(39개) 등 훈련 데이터 수가 극히 적은 클래스에서 다양성 부족으로 인한 실패. + +![](PointNet_1612.00593_images/_page_18_Figure_0.jpeg) + +Figure 24. Examples of semantic segmentation and object detection. First row is input point cloud, where walls and ceiling are hided for clarity. Second and third rows are prediction and ground-truth of semantic segmentation on points, where points belonging to different semantic regions are colored differently (chairs in red, tables in purple, sofa in orange, board in gray, bookcase in green, floors in blue, windows in violet, beam in yellow, column in magenta, doors in khaki and clutters in black). The last two rows are object detection with bounding boxes, where predicted boxes are from connected components based on semantic segmentation prediction. + +> **[그림 해설]** 실내 대규모 3D 씬(3개 방/오피스)에 대한 시맨틱 분할 및 3D 바운딩 박스 객체 검출(Object Detection) 결과 (5개 행으로 구성). +> - **1행 (Input Point Cloud)**: 천장과 벽을 일부 제거하여 내부를 가시화한 실내 포인트 클라우드 입력. +> - **2행 (pred - Semantic Segmentation)**: PointNet의 포인트별 시맨틱 클래스 예측 결과 (의자: 빨강, 테이블: 보라, 책장: 초록, 바닥: 파랑, 빔: 노랑, 소파: 주황 등). +> - **3행 (GT - Semantic Segmentation)**: 실제 정답 시맨틱 분할. +> - **4행 (pred - Object Detection)**: 시맨틱 분할의 연결 요소(Connected Components)를 기반으로 추출된 3D 바운딩 박스 객체 검출 결과. +> - **5행 (GT - Object Detection)**: 실제 정답 3D 바운딩 박스. 예측 박스가 정답 박스의 위치와 크기를 거의 정확히 추정함. \ No newline at end of file diff --git a/docs/papers/md/PointNet_1612.00593_images/_page_0_Figure_9.jpeg b/docs/papers/md/PointNet_1612.00593_images/_page_0_Figure_9.jpeg new file mode 100644 index 0000000..f0bb14d Binary files /dev/null and b/docs/papers/md/PointNet_1612.00593_images/_page_0_Figure_9.jpeg differ diff --git a/docs/papers/md/PointNet_1612.00593_images/_page_10_Figure_0.jpeg b/docs/papers/md/PointNet_1612.00593_images/_page_10_Figure_0.jpeg new file mode 100644 index 0000000..8db41be Binary files /dev/null and b/docs/papers/md/PointNet_1612.00593_images/_page_10_Figure_0.jpeg differ diff --git a/docs/papers/md/PointNet_1612.00593_images/_page_10_Figure_2.jpeg b/docs/papers/md/PointNet_1612.00593_images/_page_10_Figure_2.jpeg new file mode 100644 index 0000000..0251fe0 Binary files /dev/null and b/docs/papers/md/PointNet_1612.00593_images/_page_10_Figure_2.jpeg differ diff --git a/docs/papers/md/PointNet_1612.00593_images/_page_11_Figure_0.jpeg b/docs/papers/md/PointNet_1612.00593_images/_page_11_Figure_0.jpeg new file mode 100644 index 0000000..3810157 Binary files /dev/null and b/docs/papers/md/PointNet_1612.00593_images/_page_11_Figure_0.jpeg differ diff --git a/docs/papers/md/PointNet_1612.00593_images/_page_11_Figure_13.jpeg b/docs/papers/md/PointNet_1612.00593_images/_page_11_Figure_13.jpeg new file mode 100644 index 0000000..905bde2 Binary files /dev/null and b/docs/papers/md/PointNet_1612.00593_images/_page_11_Figure_13.jpeg differ diff --git a/docs/papers/md/PointNet_1612.00593_images/_page_11_Figure_7.jpeg b/docs/papers/md/PointNet_1612.00593_images/_page_11_Figure_7.jpeg new file mode 100644 index 0000000..c35966a Binary files /dev/null and b/docs/papers/md/PointNet_1612.00593_images/_page_11_Figure_7.jpeg differ diff --git a/docs/papers/md/PointNet_1612.00593_images/_page_11_Figure_9.jpeg b/docs/papers/md/PointNet_1612.00593_images/_page_11_Figure_9.jpeg new file mode 100644 index 0000000..a557edb Binary files /dev/null and b/docs/papers/md/PointNet_1612.00593_images/_page_11_Figure_9.jpeg differ diff --git a/docs/papers/md/PointNet_1612.00593_images/_page_11_Picture_2.jpeg b/docs/papers/md/PointNet_1612.00593_images/_page_11_Picture_2.jpeg new file mode 100644 index 0000000..3147f68 Binary files /dev/null and b/docs/papers/md/PointNet_1612.00593_images/_page_11_Picture_2.jpeg differ diff --git a/docs/papers/md/PointNet_1612.00593_images/_page_12_Figure_8.jpeg b/docs/papers/md/PointNet_1612.00593_images/_page_12_Figure_8.jpeg new file mode 100644 index 0000000..1ef6ccf Binary files /dev/null and b/docs/papers/md/PointNet_1612.00593_images/_page_12_Figure_8.jpeg differ diff --git a/docs/papers/md/PointNet_1612.00593_images/_page_13_Figure_0.jpeg b/docs/papers/md/PointNet_1612.00593_images/_page_13_Figure_0.jpeg new file mode 100644 index 0000000..5720959 Binary files /dev/null and b/docs/papers/md/PointNet_1612.00593_images/_page_13_Figure_0.jpeg differ diff --git a/docs/papers/md/PointNet_1612.00593_images/_page_13_Figure_3.jpeg b/docs/papers/md/PointNet_1612.00593_images/_page_13_Figure_3.jpeg new file mode 100644 index 0000000..bd1b9e0 Binary files /dev/null and b/docs/papers/md/PointNet_1612.00593_images/_page_13_Figure_3.jpeg differ diff --git a/docs/papers/md/PointNet_1612.00593_images/_page_14_Picture_5.jpeg b/docs/papers/md/PointNet_1612.00593_images/_page_14_Picture_5.jpeg new file mode 100644 index 0000000..89a65b3 Binary files /dev/null and b/docs/papers/md/PointNet_1612.00593_images/_page_14_Picture_5.jpeg differ diff --git a/docs/papers/md/PointNet_1612.00593_images/_page_15_Figure_0.jpeg b/docs/papers/md/PointNet_1612.00593_images/_page_15_Figure_0.jpeg new file mode 100644 index 0000000..c41a460 Binary files /dev/null and b/docs/papers/md/PointNet_1612.00593_images/_page_15_Figure_0.jpeg differ diff --git a/docs/papers/md/PointNet_1612.00593_images/_page_16_Figure_0.jpeg b/docs/papers/md/PointNet_1612.00593_images/_page_16_Figure_0.jpeg new file mode 100644 index 0000000..575a1a5 Binary files /dev/null and b/docs/papers/md/PointNet_1612.00593_images/_page_16_Figure_0.jpeg differ diff --git a/docs/papers/md/PointNet_1612.00593_images/_page_16_Figure_2.jpeg b/docs/papers/md/PointNet_1612.00593_images/_page_16_Figure_2.jpeg new file mode 100644 index 0000000..52ae92e Binary files /dev/null and b/docs/papers/md/PointNet_1612.00593_images/_page_16_Figure_2.jpeg differ diff --git a/docs/papers/md/PointNet_1612.00593_images/_page_17_Figure_0.jpeg b/docs/papers/md/PointNet_1612.00593_images/_page_17_Figure_0.jpeg new file mode 100644 index 0000000..4facdac Binary files /dev/null and b/docs/papers/md/PointNet_1612.00593_images/_page_17_Figure_0.jpeg differ diff --git a/docs/papers/md/PointNet_1612.00593_images/_page_18_Figure_0.jpeg b/docs/papers/md/PointNet_1612.00593_images/_page_18_Figure_0.jpeg new file mode 100644 index 0000000..f2f6f00 Binary files /dev/null and b/docs/papers/md/PointNet_1612.00593_images/_page_18_Figure_0.jpeg differ diff --git a/docs/papers/md/PointNet_1612.00593_images/_page_2_Figure_0.jpeg b/docs/papers/md/PointNet_1612.00593_images/_page_2_Figure_0.jpeg new file mode 100644 index 0000000..33581c6 Binary files /dev/null and b/docs/papers/md/PointNet_1612.00593_images/_page_2_Figure_0.jpeg differ diff --git a/docs/papers/md/PointNet_1612.00593_images/_page_2_Figure_1.jpeg b/docs/papers/md/PointNet_1612.00593_images/_page_2_Figure_1.jpeg new file mode 100644 index 0000000..07ea2f5 Binary files /dev/null and b/docs/papers/md/PointNet_1612.00593_images/_page_2_Figure_1.jpeg differ diff --git a/docs/papers/md/PointNet_1612.00593_images/_page_4_Figure_0.jpeg b/docs/papers/md/PointNet_1612.00593_images/_page_4_Figure_0.jpeg new file mode 100644 index 0000000..ec6cd6f Binary files /dev/null and b/docs/papers/md/PointNet_1612.00593_images/_page_4_Figure_0.jpeg differ diff --git a/docs/papers/md/PointNet_1612.00593_images/_page_6_Figure_0.jpeg b/docs/papers/md/PointNet_1612.00593_images/_page_6_Figure_0.jpeg new file mode 100644 index 0000000..e4d37a1 Binary files /dev/null and b/docs/papers/md/PointNet_1612.00593_images/_page_6_Figure_0.jpeg differ diff --git a/docs/papers/md/PointNet_1612.00593_images/_page_7_Figure_0.jpeg b/docs/papers/md/PointNet_1612.00593_images/_page_7_Figure_0.jpeg new file mode 100644 index 0000000..209e249 Binary files /dev/null and b/docs/papers/md/PointNet_1612.00593_images/_page_7_Figure_0.jpeg differ diff --git a/docs/papers/md/PointNet_1612.00593_images/_page_7_Figure_7.jpeg b/docs/papers/md/PointNet_1612.00593_images/_page_7_Figure_7.jpeg new file mode 100644 index 0000000..32b0df1 Binary files /dev/null and b/docs/papers/md/PointNet_1612.00593_images/_page_7_Figure_7.jpeg differ diff --git a/docs/papers/md/PointNet_1612.00593_images/_page_9_Figure_10.jpeg b/docs/papers/md/PointNet_1612.00593_images/_page_9_Figure_10.jpeg new file mode 100644 index 0000000..2bb6b63 Binary files /dev/null and b/docs/papers/md/PointNet_1612.00593_images/_page_9_Figure_10.jpeg differ diff --git a/docs/papers/md/PointVector_2205.10528.md b/docs/papers/md/PointVector_2205.10528.md new file mode 100644 index 0000000..e46eeb6 --- /dev/null +++ b/docs/papers/md/PointVector_2205.10528.md @@ -0,0 +1,586 @@ +# PointVector: A Vector Representation In Point Cloud Analysis + +Xin Deng\* WenYu Zhang\* Qing Ding† XinMing Zhang† University of Science and Technology of China + +{xin deng, wenyuz}@mail.ustc.edu.cn, {dingqing, xinming}@ustc.edu.cn + +## Abstract + +*In point cloud analysis, point-based methods have rapidly developed in recent years. These methods have recently focused on concise MLP structures, such as Point-NeXt, which have demonstrated competitiveness with Convolutional and Transformer structures. However, standard MLPs are limited in their ability to extract local features effectively. To address this limitation, we propose a Vectororiented Point Set Abstraction that can aggregate neighboring features through higher-dimensional vectors. To facilitate network optimization, we construct a transformation from scalar to vector using independent angles based on 3D vector rotations. Finally, we develop a PointVector model that follows the structure of PointNeXt. Our experimental results demonstrate that PointVector achieves state-of-theart performance 72.3% mIOU on the S3DIS Area 5 and 78.4% mIOU on the S3DIS (6-fold cross-validation) with only 58% model parameters of PointNeXt. We hope our work will help the exploration of concise and effective feature representations. The code will be released soon.* + +## 1. Introduction + +Point cloud analysis is a cornerstone of various downstream tasks. With the introduction of PointNet [\[25\]](#page-12-0) and PointNet++ [\[26\]](#page-12-1), the direct processing of unstructured point clouds has become a hot topic. Many point-based networks introduced novel and sophisticated modules to extract local features, e.g., attention-based methods [\[53\]](#page-13-0) explore attention mechanisms as Fig[.1a](#page-0-0) with lower consumption, convolution-based methods [\[36\]](#page-12-2) explore the dynamic convolution kernel as Fig[.1c,](#page-0-0) and graph-based methods [\[39\]](#page-12-3) [\[54\]](#page-13-1) use graph to model relationships of points. The application of these methods to the feature extraction module of PointNet++ brings an improvement in feature quality. However, they are somewhat complicated to design in terms of network structure. PointNeXt [\[28\]](#page-12-4) adapts the SetAbstraction (SA) module of PointNet++ [\[26\]](#page-12-1) and proposes the Inverted Residual MLP (InvResMLP) module. The simple design of MLP network achieves good results. Motivated by this work, we try to further explore the potential of the MLP structure. + +![](PointVector_2205.10528_images/_page_0_Figure_11.jpeg) + +Figure 1. Illustrations of the core operations of the different methods. (a) The features of each point are calculated separately by applying a fixed/isotropic kernel (black arrow) like Linear layer. Then, it imparts anisotropy by weights generated from inputs. (b) The displacement vector is used to filter points that approximate the kernel pattern for features aggregation. (c) It applies unique dynamic kernels with anisotropy for each point feature. (d) Differently, we generate vector representations based on features, and the aggregation methods for vectors are anisotropic due to the direction of the vectors. + +> **[그림 해설]** 포인트 클라우드 특징 집계 방식 4종 비교 다이어그램 (이방성/등방성 표현 방식의 차이). +> - **(a) Attention**: 고정 커널로 각 점 특징을 독립 계산한 뒤, 입력에서 생성된 스칼라 가중치($w_1, w_2, w_3$)를 곱해 이방성을 부여. +> - **(b) Templated-based method**: 고정 템플릿 커널과 입력 점들 사이의 변위 벡터를 기반으로 유사 패턴 점을 매칭하여 집계. +> - **(c) Dynamic Conv**: 점마다 서로 다른 동적 커널 가중치($\hat{e}_1, \hat{e}_2, \hat{e}_3$)를 동적으로 생성하여 적용. +> - **(d) Vector (제안 방식)**: 특징으로부터 크기와 방향을 갖는 3D 벡터 표현을 생성하고, 국소 좌표계 상에서 벡터의 고유 방향성 자체를 활용해 점들 간의 기하학적 관계 및 이방성을 자연스럽게 포착 (검은색 화살표: 등방성, 유색 화살표: 이방성). + +PointNeXt uses all standard MLPs, which has insufficient feature extraction capability. In addition to attention and dynamic convolution mechanisms, template-based methods as Fig[.1b](#page-0-0) such as 3D-GCN [\[19\]](#page-11-0) employ relative displacement vectors to modulate the association between input points and the convolutional kernel. We introduce a vector representation of features to extend the range of feature variation with the intention of more effectively regulating the connections between local features. Our approach as Fig[.1d](#page-0-0) differs from template-based methods. Instead of using displacement vectors as a property of the kernel, we + +\*Co-first authors with equal contribution to refining the theory and experimental design + +†Corresponding authors + +generate a vector representation for each neighboring point and aggregate them. Our method introduces less inductive bias, resulting in improved generalization capabilities. Furthermore, we enhance the generation of 3D vector representations by utilizing a vector rotation matrix with two independent angles in 3D space. This method facilitates the network to find the better solution. + +Influenced by PointNeXt [\[28\]](#page-12-4) and PointNet++ [\[26\]](#page-12-1), we present the VPSA module. This module adheres to the structure of Point Set Abstraction (SA) module of the Point-Net series. Vector representations are obtained from input features and aggregated using a reduction function. The vector of each channel is then projected into a scalar to derive local features. By combining VPSA and SA modules, we construct a PointVector model with an architecture akin to that of PointNeXt. + +Our model undergoes comprehensive validation on public benchmark datasets. It achieves state-of-the-art performance on the S3DIS [\[1\]](#page-11-1) semantic segmentation benchmark and competitive results on the ScanObjectNN [\[48\]](#page-13-2) and ShapeNetPart [\[49\]](#page-13-3) datasets. By incorporating a priori knowledge of vectors, our model attains superior results with fewer parameters on S3DIS. Detailed ablation experiments further demonstrate the efficacy of our methodology. The contributions are summarized below: + +- We propose a novel immediate vector representation with relative features and positions to better guide local feature aggregation. +- We explore the method of obtaining vector representation and propose the generation method of 3D vector by utilizing the vector rotation matrix in 3D space. +- Our proposed PointVector model achieves 72.3% mean Intersection over Union (mIOU) on S3DIS area5 and 78.4% mIOU on S3DIS (6-fold cross-validation) with only 58% model parameters of PointNeXt. + +## 2. Related work + +Point-based network. In contrast to the voxelization [\[55\]](#page-13-4) [\[15\]](#page-11-2) [\[31\]](#page-12-5) and multiview [\[32\]](#page-12-6) [\[10\]](#page-11-3) [\[41\]](#page-12-7) methods, pointbased methods deal directly with point clouds. PointNet first proposes using MLP to process point clouds directly. PointNet++ subsequently introduces a hierarchical structure to improve the feature extraction. Subsequent works focused on the design of fine-grained local feature extractors. Graph-based methods [\[39\]](#page-12-3) [\[38\]](#page-12-8) rely on a graph neural network and introduce point features and edge features to model local relationships. Conv-based methods [\[36\]](#page-12-2) [\[46\]](#page-13-5) [\[42\]](#page-12-9) [\[2\]](#page-11-4) [\[17\]](#page-11-5) propose several dynamic convolution kernels to adaptively aggregate neighborhood features. Many transformer-like networks [\[11\]](#page-11-6) [\[51\]](#page-13-6) [\[50\]](#page-13-7) [\[9\]](#page-11-7) [\[14\]](#page-11-8) extract local features with self-attention. Recently, MLP-like networks are able to obtain good results with simple networks by enhancing the features. PointMLP [\[24\]](#page-12-10) proposes a geometric affine module to normalize the feature. Rep-Surf [\[30\]](#page-12-11) fits the surface information through the triangular plane, models umbrella surfaces to provide geometric information. PointNeXt [\[28\]](#page-12-4) integrates training strategies and model scaling. + +MLP-like Architecture. The MLP-like structure has recently shown the ability to rival the Transformer with simple architecture. In the image field, MLP-Mixer [\[37\]](#page-12-12) first use the combination of Spatial MLP and Channel MLP. The subsequent works [\[3\]](#page-11-9) [\[18\]](#page-11-10) reduce computational complexity by selecting objects for the spatial MLP while maintaining a large perceptual field to preserve accuracy. Since the point cloud is too large, the MLP-like network determines the perceptual field generally using K-Nearest neighbor sampling or ball sampling methods. The MLP structure in point cloud analysis starts with PointNet [\[25\]](#page-12-0) and PointNet++ [\[26\]](#page-12-1), using MLPs to extract features and aggregating them by symmetric functions. Point-Mixer [\[6\]](#page-11-11) proposes three point-set operators, PointMLP [\[24\]](#page-12-10) to modify the distribution of features by geometric affine module, and PointNeXt [\[28\]](#page-12-4) to scale up the PointNet++ model and improve the performance using by training strategies and model scaling. + +Feature Aggregation. PosPool [\[21\]](#page-11-12) improves the reduction function defined in PointNet++ by providing a parameter-free position-adaptive pooling operation. AS-SANet [\[27\]](#page-12-13) introduces a new anisotropic reduction function. Also, the introduction of the attention mechanism [\[47\]](#page-13-8) provides new dynamic weights for the reduction function. Vectors have direction, and this property is naturally satisfied for anisotropic aggregation functions. GeoCNN [\[4\]](#page-11-13) projects features based on vectors and angles of neighbor points and centroids in six directions and sums them. WaveMLP [\[35\]](#page-12-14) represents image patches as waves and describes feature aggregation using wave phase and amplitude. The Vector Neuron [\[7\]](#page-11-14) constructs a triad of neurons to reconstruct standard neural networks and represent features through vector transformations. The template-based methods represented by 3DGCN [\[19\]](#page-11-0) uses the cosine value of the relative displacement vectors to filter for aggregation features from neighbors that more conform to the pattern of the kernel. Local displacements [\[40\]](#page-12-15) use local displacement vectors to update features by combining the weights of fixed kernels. In our method, an intermediate vector representation is generated by modifying the point feature extraction function. The vector direction is determined based on both features and position to fulfill the anisotropic aggregation function. + +#### 3. Method + +We propose an intermediate vector representation to enhance local feature aggregation in point cloud analysis. This section includes a review of the Point Set Abstraction(SA) operator of the PointNet family in Section 3.1, the presentation of our Vector-oriented Point Set Abstraction module in Section 3.2, a description of our method of extending vectors from scalars in Section 3.3, and the network structure of PointVector in Section 3.4. + +#### 3.1. Preliminary + +The SA module include a grouping layer (K-NN or Ball-Query) to query each point's neighbors, shared MLPs, and a reduction layer to aggregate neighbor features. The SA module has an subsample layer to downsample the point cloud in the first layer. We denote $f_i^{l+1}$ as the extracted feature of point i after stage l+1, $N_i$ as the neighbors of point i and n is the number of incoming points. The content of the SA module can be formulated as follows: + +$$f_i^{l+1} = R\{H\{[f_j^l, p_j - p_i]\}|j \in N_i\},\tag{1}$$ + +where R is the reduction function that aggregates features for point i from its neighbors $N_i$ and H means the shared MLPs. $f_j^l, p_j, p_i$ denote the input features of point j, the position of point j and the position of point i, respectively. + +In the local aggregation operation, the classical method assigns weights to components of c-dimensional features as shown in Eq.2 and sums the neighboring features in spatial dimensions. We consider the component $f_i$ of the c-dimensional feature f as a base vector with only one non-zero value, and define the vector transformation as follows: + +$$f_i * w = w f_i, i = 0 \cdots c, \qquad (2)$$ + +$$\begin{bmatrix} f_i \ 0 \cdots \ 0 \end{bmatrix} \begin{bmatrix} w \ 0 \cdots \ 0 \\ \vdots \ \vdots \ \ddots \ \vdots \\ 0 \ 0 \cdots \ 0 \end{bmatrix} = \begin{bmatrix} w f_i \ 0 \cdots \ 0 \end{bmatrix}, \quad (3)$$ + +where w is the scalar weight. In Eq.3, the transformation changes one value of the vector. The two equations above are equivalent. Unchanged zeros in the equation do not contribute to subsequent operations and can be disregarded. In Physics, the degree of freedom of a motion is equal to the number of state quantities that the motion causes the system to change. A greater number of degrees of freedom in a physical system indicates a larger range of independent variation in the parameters that define its state. Similarly, the degrees of freedom of a vector transformation refer to the number of values in the vector that can change independently. So, the 3D vector we mentioned means the degrees of freedom of the vector transformation is 3. + +![](PointVector_2205.10528_images/_page_2_Picture_10.jpeg) + +Figure 2. The vector-oriented point set abstraction (VPSA) module of PointVector. It illustrates that VPSA module obtains vector representations from input features, aggregates them, and projects them back to the original feature style. As shown in the figure, each channel of the feature can be considered a 3D vector, with channels being independent of one another. + +> **[그림 해설]** PointVector의 핵심 단위인 VPSA(Vector-oriented Point Set Abstraction) 모듈 다이어그램. +> - 입력: $c$개 채널의 위치(position) 및 특징(feature). +> - **Vector encoder**: 스칼라 특징에 예측된 회전각($\alpha, \beta$)을 적용하여 $c \times 3$ 채널의 3D 벡터 $\vec{r}_1, \vec{r}_2$ 표현으로 확장. +> - **Aggregation ($\oplus$)**: 채널별 독립 3D 벡터들을 집계/합산하여 $c \times 3$ 집계 벡터 구성. +> - **Project**: 집계된 벡터 텐서를 그룹 컨볼루션을 통해 원래 형태의 $c$개 채널 스칼라 특징으로 다시 사영(Project). + +#### 3.2. Vector-oriented Point Set Abstraction + +As discussed in Section 3.1, feature components can be represented as vectors. A higher degree of freedom in vector transformations allows for increased variation and improved representation of connections between neighboring elements. Vectors, with their size and direction properties, are more expressive than scalars for representing features. When aggregated, they exhibit anisotropy due to their directional nature. So, we introduce an intermediate vector representation as Fig.2. + +It should be noted that in our assumptions, the component of a c-dimensional feature represents the projection of the feature vector along the c coordinate axes. After aggregating the vectors to obtain the $c \times 3$ centroid feature, where the number of changing values in the component vectors is 3. To merge them into a c-dimensional feature vector requires aligning the c components and then summing them. Due to the difficulty in implementing component alignment with this method, we directly project the c components into scalars and combine them into centroid features. Similar to the intermediate features in a convolutional network, the values on each channel's feature map represent the response strength to a specific feature at that location. + +The input features in our method are transformed into a series of vectors and then aggregated by the reduction function. Note that the element in each channel of the vector representation is vector. We obtain a vector representation that is channel independent. We denote $fp_j$ as a mixed feature of relative features $f_j - f_i$ and relative positions $p_j - p_i$ . The content of the vector-guided aggregation module can be formulated as: + + $f_i^{l+1} = \eta(f_i^l) + H_c\{H_p\{R\{H_v(fp_j)|j\in N_i\}\}\}, \quad \text{(4)}$ where $H_v$ is the function that generates the vector representation, $H_p$ denotes the projection Linear transform vector to a scalar, and $H_c$ is the channel mixing Linear that + +![](PointVector_2205.10528_images/_page_3_Figure_0.jpeg) + +Figure 3. Extension from general feature to vector representation. For simplicity, we tentatively set c=4 and m=3. The left side represents the process of generating features by standard MLP, and the right side adds 2 components to each scalar of the features to form a vector and then rotates it. + +> **[그림 해설]** 일반 스칼라 특징에서 3D 벡터 표현으로 확장하는 원리 비교 ($c=4$ 채널, $m=3$ 차원). +> - **좌측 (표준 MLP)**: 4개 입력 채널이 완전 연결 가중치에 의해 4개의 새로운 스칼라 출력 채널로 선형 변환. +> - **우측 (Vector 확장)**: 각 스칼라 특징에 0 성분 2개를 덧붙여 3차원 축 벡터 $(0, \text{feat}, 0)$를 구성한 뒤, 예측된 회전 행렬을 적용하여 3차원 방향 벡터 성분(분홍, 회색, 노랑, 하늘색)으로 변환. + +interacts with the information of each channel while transforming dimensions to fit the network. However, the feature representation we introduce is actually represented using a triplet form. We denote m as the dimension of the vector, and c is the channel of the feature. In fact, the set of c m-dimensional vectors is represented in the same form as the $(m \times c)$ -dimensional feature vectors. The reduction function is followed by a grouped convolution [13] that transforms the vectors to scalars for each channel, which distinguishes the intermediate vector representation from the general feature vector. + +When the reduction function R selects sum, the R and $H_p$ functions together constitute a special case of Group-Conv [13]. Let k denote the number of neighbor features. For one group, the convolution kernel of Group-Conv is a $k \times m \times 1$ parameter matrix, while our method can be viewed as k identical $m \times 1$ parameter matrices. This is because we treat vectors as wholes and assign equal weight to each element. We will explain in the supplementary material why the original group-conv operation is not suitable for our vector-guided feature aggregation. + +#### 3.3. Extended Vector From Scalar + +The simplest idea for the $H_v$ function defined in Eq.4 is to obtain c m-dimensional vectors of point j directly with MLPs. However, while single-layer MLPs may have limited expressive capability, multi-layer MLPs can be resource-intensive. As discussed in Section 3.1, input features are considered as vectors and we aim to design a transformation with high degrees of freedom. This transformation combines rotation and scaling, represented by a rotation matrix and a learnable parameter respectively. This method achieves better results with lower resource consumption. + +As shown in Fig.3, a scalar can be directly converted into an m-dimensional vector by adding m-1 zero-value components. Each channel of the extended vector representation can then be considered as an m-dimensional vector along a specific coordinate axis direction. Therefore, we can obtain the proper vector direction by additionally training a rotation matrix. Directly predicting the rotation matrix can cause difficulties for nonlinear optimization because the + +![](PointVector_2205.10528_images/_page_3_Picture_7.jpeg) + +Figure 4. The rotation of a 3D vector. The vector $\vec{r}$ can be obtained by two rotations to obtain another vector $\vec{r''}$ + +> **[그림 해설]** 3D 벡터의 2단계 공간 회전 기하 다이어그램. +> - $y$축 상의 기본 벡터 $\vec{r} = (0, zx, 0)$에서 시작. +> - **1단계 ($Rot_x$)**: $x$축을 중심으로 각도 $\frac{\pi}{2} - \beta$만큼 회전하여 중간 벡터 $\vec{r}'$ 형성. +> - **2단계 ($Rot_z$)**: $z$축을 중심으로 방위각 $\alpha$만큼 회전하여 최종 벡터 $\vec{r}'' = Rot_z Rot_x \vec{r}$ 완성. + +matrix elements are interdependent. Instead, we first predict the rotation angle and then derive the rotation matrix based on this angle. The rotation of a 3D vector can be decomposed into rotations around three axes. However, we have not yet determined how to represent the rotation of a 4D vector around a plane. As shown in Fig.4, since the extended 3D vector is on the coordinate axis, one rotation around that axis can be omitted. We keep the default rotation direction as counterclockwise. The vector $\vec{r}$ is first rotated around the *x*-axis by an angle $\pi/2 - \beta$ and then rotated around the *z*-axis by an angle $\pi/2 - \beta$ to finally obtain the vector $\vec{r}$ . The rotation can be formulated as follows: + + +$$\overrightarrow{r}'' = Rot_z Rot_x \overrightarrow{r}$$ + +$$= \begin{bmatrix} \cos(\alpha) - \sin(\alpha) & 0 \\ \sin(\alpha) & \cos(\alpha) & 0 \\ 0 & 0 & 1 \end{bmatrix} \begin{bmatrix} 1 & 0 & 0 \\ 0 & \sin(\beta) - \cos(\beta) \\ 0 & \cos(\beta) & \sin(\beta) \end{bmatrix} \begin{bmatrix} 0 \\ zx \\ 0 \end{bmatrix}$$ + +$$= \begin{bmatrix} \cos(\alpha) - \sin(\alpha)\sin(\beta) & \sin(\alpha)\cos(\beta) \\ \sin(\alpha) & \cos(\alpha)\sin(\beta) & -\cos(\alpha)\cos(\beta) \\ 0 & \cos(\beta) & \sin(\beta) \end{bmatrix} \begin{bmatrix} 0 \\ zx \\ 0 \end{bmatrix}$$ + +$$= \begin{bmatrix} -zx \cdot \sin(\alpha)\sin(\beta) \\ zx \cdot \cos(\alpha)\sin(\beta) \\ zx \cdot \cos(\alpha)\sin(\beta) \\ zx \cdot \cos(\beta) \end{bmatrix},$$ +(5) + +where $Rot_x, Rot_z$ denote the rotation matrix rotated around the x-axis and the rotation matrix rotated around the z-axis, respectively, and zx is generated by Linear. The independence of $\alpha$ and $\beta$ facilitates network optimization. + +Therefore, we can expand each scalar value of the features into a 3D vector according to Fig.4 and Eq.5. The feature aggregation in a local area is influenced by the relationship between neighboring points and centroids. Methods such as PointTransformer [53], PAConv [45], and Adaptconv [54] model this relationship using relative position and features. Our approach also extracts rotation angles using MLP on relative positions and features. The acquisition of the vector can be formulated as follows: + + +$$zx_j = Linear(fp_j)$$ + +$$[\alpha_j, \beta_j] = Relu(BN(Linear([fp_j]))),$$ +(6) + +where $fp_j$ denotes a mixed feature of relative features $f_j - f_i$ and relative positions $p_j - p_i$ , and $f_j$ means the fea- + +![](PointVector_2205.10528_images/_page_4_Figure_0.jpeg) + +Figure 5. Overall Architecture. We reuse the SA module and Feature Propagation module of PointNet++ and propose the VPSA module to improve the feature extraction of sampled point clouds. + +> **[그림 해설]** PointVector의 전체 세그멘테이션 U-Net 신경망 구조도. +> - **인코더 (좌측)**: +> - 입력 포인트 $\to$ 초기 MLP $[N, 32]$. +> - 4단계 계층적 인코더: SetAbstraction + **LocalVector** 모듈을 거쳐 $[N/4, 64] \to [N/16, 128] \to [N/64, 256] \to [N/256, 512]$로 점진적 축소. +> - **LocalVector 모듈 상세 (하단 중앙)**: Grouping $\to$ Vector Encoder $\to$ Reduction $\to$ Group Conv $\to$ Linear 및 Residual 지름길 연결로 구성. +> - **디코더 (우측)**: 4단계 Feature Propagation(FP) 계층(Interpolate + Skip Connection Concat + MLPs)을 거쳐 원본 해상도 $[N, 32]$로 복원 후 Segment Head에서 $[N, \text{Categories}]$ 예측. + +ture of point *j*. Therefore, we can obtain the intermediate vector representation from the input features and positions by using Eq[.5](#page-3-3) and Eq[.6.](#page-3-4) + +### 3.4. Architecture + +In summary, we propose PointVector, modified from PointNeXt [\[28\]](#page-12-4) by replacing its InvResMLP module with our proposed VPSA module, we defining its vector dimension m = 3. The architecture is illustrated in Fig[.5.](#page-4-1) Referring to the classical PointNet++, we use a hierarchical structure containing an encoder and a decoder. For the segmentation task, we use an encoder and a decoder. For the classification task, we only use an encoder. For a fair comparison with PointNeXt, we set up three sizes of models with reference to the parameter settings of PointNeXt. We denote C as the channel of embedding MLP in the beginning, S as the numbers of the SA module, V as the numbers of the VPSA module. The three sizes of models are shown as follows: + +• PointVector-S: C=32, S=0, V=[1,1,1,1] + +• PointVector-L: C=32, S=[1,1,1,1], V=[2,4,2,2] + +• PointVector-XL: C=64, S=[1,1,1,1], V=[3,6,3,3] + +Since PointNeXt uses only the PointNeXt-S model for classification, we use our VPSA module instead of the SA module in PointVector-S for a fair comparison. The detailed structure of the classification tasks will appear in the supplementary material. There is a skip connection path in the VPSA module in Fig[.5,](#page-4-1) which is added to the main path and then through a ReLU layer. The reason for using this summation method is that RepSurf [\[30\]](#page-12-11) indicates how two features with different distributions should be combined. For the segmentation task, finer local information is needed, and we set reduction function as sum. For the classification task, which favors aggregating global information, we choose the original reduction function such as max. + +## 4. Experiments + +We evaluate our model on three standard benchmarks: S3DIS [\[1\]](#page-11-1) for semantic segmentation and ScanObjectNN [\[48\]](#page-13-2) for real-world object classification and ShapeNetPart [\[49\]](#page-13-3) for part segmentation. Note that our model is implemented on the basis of PointNeXt. Since we use the training strategy provided by PointNeXt, we refer to the metrics reported by PointNeXt for a fair comparison. + +Experimental setups. We train PointVector using CrossEntropy loss with label smoothing [\[33\]](#page-12-16), AdamW optimizer [\[23\]](#page-11-16), and initial learning rate lr=0.002, weight decay 10−4 , with Cosine Decay, and a batch size of 32. The above are the base settings for all tasks, and specific parameters will be changed for specific tasks. We follow the train, valid, and test divisions for the dataset. The best model on the validation set will be evaluated on the test set. For S3DIS segmentation task, point clouds are downsampled with a voxel size of 0.4 m following previous methods [\[36\]](#page-12-2) [\[27\]](#page-12-13) [\[53\]](#page-13-0). The initial learning rate on this task is set to 0.01. For 100 epochs, we use a fixed 24000 points as a batch and set batch size to 8. During training, the input points are selected from the nearest neighbors of the random points. Similar to Point Transformer [\[53\]](#page-13-0), we evaluate our model using the entire scene as input. For ScanObjectNN [\[48\]](#page-13-2) classification task, we set the weight decay to 0.05 for 250 epochs. Following Point-BERT [\[51\]](#page-13-6), the number of input points is 1024. The training points are randomly sampled from the point cloud, and the testing points are uniformly sampled during evaluation. The details of data augmentation are the same as those in PointNeXt. For ShapeNetPart part segmentation, we train PointVector-S with a batch size of 32 for 300 epochs. Following PointNet++ [26], 2,048 randomly sampled points with normals are used as input for training and testing. + +For voting strategy [20], we keep it the same as Point-NeXt and use it only on part segmentation task. To ensure a fair comparison with standard methods, we do not use any ensemble methods, such as SimpleView [8]. We also provide the model parameters (Params) and GFLOPs. We additionally, similar to PointNeXt, provide throughput (instance per second) as an indicator of inference speed. The input data for the throughput calculation are kept consistent with PointNeXt for fair comparison. The throughput of all methods is measured using $128 \times 1024$ (batch size 128, number of points 1024) as input on ScanObjectNN and $64 \times 2048$ on ShapeNetPart. On S3DIS, $16 \times 15,000$ points are used to measure the throughput following [28] [27]. We evaluate our model using an NVIDIA Tesla V100 32 GB GPU and a 48 core Intel Xeon @ 2.10 Hz CPU. + +### 4.1. 3D Semantic segmentation on S3DIS + +S3DIS [1] (Stanford Large-Scale 3D Indoor Spaces) is a challenging benchmark composed of 6 large-scale indoor areas, 271 rooms, and 13 semantic categories in total. For our models in S3DIS, the number of neighbors in SetAbstraction is 32, and the number of neighbors in the Local Vector module is 8. PointTransformer [53] also employs most of the training strategies and data enhancements used by PointNeXt, so it is fair for us to compare with it. For a comprehensive comparison, we report the experimental results of PointVector-L and PointVector-XL on S3DIS with 6-fold cross-validation in Table 1 and S3DIS Area 5 in Table 2, respectively. As shown in table 1&2, we achieve stateof-the-art performance on both validation options. Table 1 shows that our largest mode PointVector-XL outperforms PointNeXt-XL by 1.6%, 3.1% and 3.5% in terms of overall accuracy (OA), mean accuracy(mAcc) and mIOU, respectively, while has only 58% Params. At the same time, the computational consumption of ours is only 69% of PointNeXt-XL in terms of GFLOPs. The reduction in computational consumption because the number of neighbors is reduced to 8. The limitation is that we make heavy use of GroupConv (groups=channel), which is not well optimized in PyTorch and is slower than standard convolution. Therefore, our inference speed is 6 instances/second lower than PointNeXt-XL. Our model shows better results at all sizes. + +On S3DIS Area 5, we selected the best results reported by PointNeXt for comparison and did not repeat the experiment. Our PointVector-XL model outperforms StratifiedFormer [14] and PointNeXt-XL by **0.3% and 1.8%** in mIOU, respectively. StratifiedFormer expands the scope of + + + +| Method | OA | mAcc | mIOU | Params | FLOPs | Throughput | +|------------------------|------|------|------|--------|-------|-------------| +| | % | % | % | M | G | (ins./sec.) | +| PointNet [25] | 78.5 | 66.2 | 47.6 | 3.6 | 35.5 | 162 | +| PointCNN [17] | 88.1 | 75.6 | 65.4 | 0.6 | - | - | +| DGCNN [39] | 84.1 | - | 56.1 | 1.3 | - | 8 | +| DeepGCN [16] | 85.9 | - | 60.0 | 3.6 | - | 3 | +| KPConv [36] | - | 79.1 | 70.6 | 15.0 | - | 30 | +| RandLA-Net [12] | 88.0 | 82.0 | 70.0 | 1.3 | 5.8 | 159 | +| Point Transformer [53] | 90.2 | 81.9 | 73.5 | 7.8 | 5.6 | 34 | +| CBL [34] | 89.6 | 79.4 | 73.1 | 18.6 | - | - | +| RepSurf [30] | 90.9 | 82.6 | 74.3 | 0.976 | - | - | +| PointNet++ [26] | 81.0 | 67.1 | 54.5 | 1.0 | 7.2 | 186 | +| PointNeXt-L [28] | 89.8 | 82.2 | 73.9 | 7.1 | 15.2 | 115 | +| PointNeXt-XL [28] | 90.3 | 83.0 | 74.9 | 41.6 | 84.8 | 46 | +| PointVector-L | 91.4 | 85.5 | 77.4 | 4.2 | 10.7 | 98 | +| PointVector-XL(Ours) | 91.9 | 86.1 | 78.4 | 24.1 | 58.5 | 40 | + +Table 1. Semantic segmentation on S3DIS with 6-fold cross-validation. Methods are in chronological order. The highest and second scores are marked in bold. + +the query by combining high-resolution and low-resolution keys while efficiently extracting contextual information. Even though its receptive field is much wilder than our model, we still show a competitive performance. Additionally, there are some differences in the experimental setup between our model and it, in which it has 80k points of input, much larger than our 24k points of input. In addition it uses KPConv [36] instead of Linear in the first layer. It seems that these measures have significant effects. However, the comparison is not fair enough for us due to the difference of the experimental configurations. We will synchronize its experimental configuration later. Additionally, our models of the same size on Area 5 show better results than PointNeXt. PointVector-L and PointVector-XL perform better than PointNeXt-L and PointNeXt-XL by 1.7% and 1.5% in mIOU, respectively, and we performs better on most of categories. + +#### 4.2. 3D Object Classification on ScanObjectNN + +ScanObjectNN [48] contains approximately 15,000 real scanned objects that are categorized into 15 classes with 2,902 unique object instances. The dataset has significant challenges due to occlusion and noise. As with PointNeXt, we chose the hardest variant PB\_T50\_RS of ScanObjectNN and report the mean±std Overall Accuracy and Mean Accuracy score. For our model in ScanObjectNN, the number of neighbors in SetAbstraction is 32. As shown in table.3, our PointVector-S model achieves a comparable performance on ScanObjectNN in OA, while outperforms PointNeXt-S by 0.4% in mAcc. This illustrates that our approach is not more biased toward certain categories and is relatively robust. Our approach is at a disadvantage in terms of speed and scale compared to the SA module. Since we introduce high-dimensional vectors, we generate more computations before the reduction compared to the standard SA module. Due to group convolution operations and trigonometric functions, there is a speed bottleneck. Although the inference speed is slower than PointNeXt, we are still faster + + + +| Method | N OA | mAcc | nolm 1 | ceiling | floor | wall | beam | column | window | door | table | chair | sofa | bookcase | board | clutter | +|------------------------|------|------|--------|---------|-------|------|------|--------|--------|------|-------------|-------|------|----------|-------|---------| +| D 1 07 (50.5) | % | % | % | 00.0 | 07.0 | | 0.4 | 2.0 | 16.0 | 10.0 | 50.0 | | | 10.2 | 26.4 | | +| PointNet [25] | - | 49.0 | 41.1 | 88.8 | 97.3 | 69.8 | 0.1 | 3.9 | 46.3 | 10.8 | 59.0 | 52.6 | 5.9 | 40.3 | 26.4 | 33.2 | +| PointCNN [17] | 85.9 | 63.9 | 57.3 | 92.3 | 98.2 | 79.4 | 0.0 | 17.6 | 22.8 | 62.1 | 74.4 | 80.6 | 31.7 | 66.7 | 62.1 | 56.7 | +| DGCNN [39] | 83.6 | - | 47.9 | - | - | - | - | - | - | - | - | - | - | - | - | - | +| DeepGCN [16] | - | - | 52.5 | - | - | - | - | - | - | - | - | - | - | - | - | - | +| KPConv [36] | - | 72.8 | 67.1 | 92.8 | 97.3 | 82.4 | 0.0 | 23.9 | 58.0 | 69.0 | 81.5 | 91.0 | 75.4 | 75.3 | 66.7 | 58.9 | +| PVCNN [22] | 87.1 | - | 59.0 | - | - | - | - | - | - | - | - | - | - | - | - | - | +| PAConv [45] | - | 73.0 | 66.6 | 94.6 | 98.6 | 82.4 | 0.0 | 26.4 | 58.0 | 60.0 | 89.7 | 80.4 | 74.3 | 69.8 | 73.5 | 57.7 | +| ASSANet-L [27] | - | - | 66.8 | - | - | - | - | - | - | - | - | - | - | - | - | - | +| Point Transformer [53] | 90.8 | 76.5 | 70.4 | 94.0 | 98.5 | 86.3 | 0.0 | 38.0 | 63.4 | 74.3 | 89.1 | 82.4 | 74.3 | 80.2 | 76.0 | 59.3 | +| PatchFormer [52] | - | - | 68.1 | - | - | - | - | - | - | - | - | - | - | - | - | - | +| CBL [34] | 90.6 | 75.2 | 69.4 | 93.9 | 98.4 | 84.2 | 0.0 | 37.0 | 57.7 | 71.9 | 91.7 | 81.8 | 77.8 | 75.6 | 69.1 | 62.9 | +| RepSurf-U [30] | 90.2 | 76.0 | 68.9 | - | - | - | - | - | - | - | - | - | - | - | - | - | +| StratifiedFormer* [14] | 91.5 | 78.1 | 72.0 | 96.2 | 98.7 | 85.6 | 0.0 | 46.1 | 60.0 | 76.8 | 92.6 | 84.5 | 77.8 | 75.2 | 78.1 | 64.0 | +| PointNet++ [26] | 83.0 | - | 53.5 | - | - | - | - | - | - | - | - | - | - | - | - | - | +| PointNeXt-L [28] | 90.1 | 76.1 | 69.5 | 94.0 | 98.5 | 83.5 | 0.0 | 30.3 | 57.3 | 74.2 | 82.1 | 91.2 | 74.5 | 75.5 | 76.7 | 58.9 | +| PointNeXt-XL [28] | 90.7 | 77.5 | 70.8 | 94.2 | 98.5 | 84.4 | 0.0 | 37.7 | 59.3 | 74.0 | 83.1 | 91.6 | 77.4 | 77.2 | 78.8 | 60.6 | +| PointVector-L(Ours) | 90.8 | 77.3 | 71.2 | 94.8 | 98.2 | 84.1 | 0.0 | 31.7 | 60.0 | 77.7 | 83.7 | 91.9 | 81.8 | 78.9 | 79.9 | 63.3 | +| PointVector-XL(Ours) | 91.0 | 78.1 | 72.3 | 95.1 | 98.6 | 85.1 | 0.0 | 41.4 | 60.8 | 76.7 | 84.4 | 92.1 | 82.0 | 77.2 | 85.1 | 61.4 | + +Table 2. Semantic segmentation on S3DIS Area5. \* denotes StratifiedFormer use 80k points as input points. The highest and second scores are marked in bold. + + + +| Method | OA | mAcc | Params. | Throughput | +|---------------------|----------------|----------------|---------|------------| +| | % | % | M | ins./sec. | +| PointNet [25] | 68.2 | 63.4 | 3.5 | 4212 | +| PointCNN [17] | 78.5 | 75.1 | 0.6 | 44 | +| DGCNN [39] | 78.1 | 73.6 | 1.8 | 402 | +| GBNet [29] | 80.5 | 77.8 | 8.8 | 194 | +| PRANet [5] | 82.1 | 79.1 | 2.3 | 493 | +| PointMLP [24] | $85.4 \pm 1.3$ | $83.9 \pm 1.5$ | 13.2 | 191 | +| RepSurf-U [30] | 86.0 | 83.1 | 6.8 | - | +| PointNet++ [26] | 77.9 | 75.4 | 1.5 | 1872 | +| PointNeXt-S [28] | $87.7 \pm 0.4$ | $85.8 \pm 0.6$ | 1.4 | 2040 | +| PointVector-S(Ours) | $87.8 \pm 0.4$ | $86.2 \pm 0.5$ | 1.55 | 901 | + +Table 3. **Object classification on ScanObjectNN.** The highest and second scores are marked in bold. + +than other methods [24] [39] [29]. Our method does not perform well on the classification task, where the downsampling phase of the classification task requires a max reduction function to retain salient contour information. + +## 4.3. 3D Object Part Segmentation on ShapeNetPart + +ShapeNetPart [49] is an object-level dataset for part segmentation. It consists of 16,880 models from 16 different shape categories, 2-6 parts for each category, and 50 part labels in total. As shown in Tab.4, our PointVector-S and PointVector-S\_C64 models both achieve results that are comparable to PointNeXt. For the PointNeXt-S model with C=160, the number of parameters is large, and we do not give a corresponding version of the model. + +#### 4.4. Ablation Study + +We perform ablation experiments at S3DIS to verify the effectiveness of the module, and because PointVector-XL is too large, we make changes to PointVector-L. To make the comparison fair, we did not change the training parameters. + + + +| Method | Ins.mIoU | Throughput | +|------------------------|----------------|------------| +| PointNet [25] | 83.7 | 1184 | +| DGCNN [39] | 85.2 | 147 | +| KPConv [36] | 86.2 | 44 | +| 3D-GCN [19] | 85.1 | - | +| CurveNet [44] | 86.8 | 97 | +| ASSANet-L [27] | 86.1 | 640 | +| Point Transformer [53] | 86.6 | 297 | +| PointMLP [24] | 86.1 | 270 | +| Stratifiedformer [14] | 86.6 | 398 | +| PointNet++ [26] | 85.1 | 560 | +| PointNeXt-S* [28] | 86.5 | 776 | +| PointNeXt-S* (C=64) | $86.9 \pm 0.1$ | 330 | +| PointNeXt-S* (C=160) | 87.2 | 75 | +| PointVector-S(Ours) | 86.5 | 446 | +| PointVector-S(C=64) | 86.9 | 211 | + +Table 4. **Object Part Segmentation on ShapeNetPart.** \*Our evaluation results on this task alone are not consistent with the throughput results derived from that paper. Other works we did not test one by one. + +**Vector-oriented Point Set Abstraction.** We abstract the module into two key operations: sum and Group-Conv(groups=Channel), which shows that this part of the module is channel independent, so we add a FC to mix the channel information. Considering that the channel information is already mixed using non-GroupConv operations, the channel mixing Linear will be removed. The convolution and grouped convolution parts have a convolution kernel size of $1 \times k$ and a stride size of 1. As shown in Tab.5, the direct use of fixed convolution brings a large number of parameters and fits very poorly with the irregular structure of the point cloud. max+FC shows better performance + + + +| Method | OA
% | mAcc
% | mIOU
M | Params | +|---------------|---------|-----------|-----------|--------| +| max+FC* | 90.6 | 76.4 | 70.6 | 6.35 | +| Conv | 90.4 | 75.7 | 69.4 | 24.56 | +| GroupConv | 90.6 | 76.5 | 70.8 | 4.76 | +| sum+FC | 90.7 | 76.6 | 71.0 | 6.35 | +| max+GroupConv | 90.6 | 76.2 | 70.6 | 4.71 | +| sum+GroupConv | 90.8 | 77.3 | 71.2 | 4.71 | + +Table 5. Core operation of VPSA. We abstract the module into sum and GroupConv operations, and replace this part. FC means Channel-FC as Linear. \* means it acts as a baseline. + +because intuitively aggregating features with higher dimensionality retains more information. GroupConv obtains a lower mIOU because it assigns independent weights to each element of the group; however, the three elements of a 3D vector of a channel should be given the same weight when summing. Furthermore, sum+FC is not very different from sum+GroupConv because GroupConv and channel mixing Linear can be combined into a specific layer of FC. In contrast, sum+GroupConv has the smallest number of parameters and best performance, so we chose it. + + + +| Method | OA
% | mAcc
% | mIOU
% | Params
M | +|------------------|---------|--------------|-----------|-------------| +| MLP | 91.0 | 76.5 | 70.8 | 5.55 | +| Linear+direction | 90.8 | 76.5
76.6 | 70.8 | 5.55 | +| | 90.8 | 77.3 | 71.2 | 4.71 | + +Table 6. Methods for obtaining vector representations. + +**Extended Vector From Scalar.** To verify the effectiveness of our vector rotation-based method, we compare it with two other methods. As shown in Tab.6, MLP is represented by two Linear layers and a ReLU activation and BatchNorm layers. Linear+direction means that Linear predicts the vector modulus length, then uses MLP to obtain the unit vector as direction, and the final modulus length is multiplied by the unit vector. The rotation-based vector expansion method proposed in Section 3.3 is ahead of other methods and has fewer parameters. This shows that the rotation-based approach can use fewer parameters to obtain a vector representation more suitable for neighbor features. + + + +| Method | OA
% | mAcc
% | mIOU
% | Params
M | +|-----------|---------|-----------|-----------|-------------| +| Scalar | 90.4 | 76.1 | 69.8 | 3.87 | +| 2D vector | 90.4 | 77.2 | 70.9 | 3.9 | +| 3D vector | 90.8 | 77.3 | 71.2 | 4.7 | + +Table 7. Different dimensional vector. + +**Vector dimension.** We need to explore the connection between the effect of vector representation and dimension. Intuitively, higher dimensional vectors will be more expressive of features than lower dimensional vectors. Tab.7 shows that the 3D vector has a better ability to express the features and that the increase in the number of parameters is not very large. The mIOU without our vector representation is still higher than the results of PointNeXt. We will discuss the validity of the other parts of our network in the supplementary material. + +**Robustness.** Table.8 shows that our method is extremely robust to various perturbations as Stratified Transformer. The ball query we use cannot get the same neighbors in the scaled point cloud. If the query radius is scaled together, then mIOU is invariant. It indicates that our method also has scale invariance. + + + +| | None | | | | | | | | | +|-----------------|-------|-------|-------|-------|-------|-------|-------|-------|-------| +| PointNet++ [25] | | | | | | | | | | +| PointTr [51] | 70.36 | 65.94 | 67.78 | 65.72 | 70.44 | 70.43 | 65.73 | 66.15 | 59.67 | +| Stratified | 71.96 | 72.59 | 72.37 | 71.86 | 71.99 | 71.93 | 70.42 | 71.21 | 72.02 | +| Ours | 72.29 | 72.27 | 72.30 | 72.32 | 72.29 | 72.29 | 69.34 | 69.26 | 72.16 | + +Table 8. Robustness study on S3DIS (mIOU %). We apply z-axis rotation ( $\pi/2$ , $\pi$ , $3\pi/2$ ), shifting ( $\pm 0.2$ ), scaling ( $\times 0.8$ , $\times 1.2$ ) and jitter in testing. PointTr: Point Transformer. Stratified: Stratified Transformer. + +#### 5. Conclusion and Limitation. + +We introduce PointVector, which achieves state-of-theart results on the S3DIS semantic segmentation task. Our vector-oriented point set abstraction improves local feature aggregation with fewer parameters. The rotation-based vector expansion method bridges the gap between vector representation and standard feature forms. By optimizing two independent perspectives, it achieves better results. Additionally, our method exhibits robustness to various perturbations. It is noteworthy that further exploration of vector representation's meaning may reveal additional applications, i.e. dominant neighbor selection. + +The speed of our approach is constrained by the grouped convolution implementation. An interesting avenue for future work includes exploring rotations above three dimensions and decomposing four-dimensional rotations into combinations of plane rotations. Additionally, summing after component alignment aligns with our assumptions better than scalar projection. + +### Acknowledgement + +This work was supported in part by the National Key Research and Development Program of China under Grant 2020YFB2103803. + +## A. Preliminary + +## A.1. Problem of WaveMLP. + +WaveMLP [\[35\]](#page-12-14) views the patch of each picture as a wave representation, and considers that the feature of that patch should have two attributes, phase and amplitude, with amplitude representing the actual property of the feature and phase modulating the amplitude that this wave exhibits at a moment. It thus considers that the feature extraction of the patches can be viewed as a superposition of waves. However, there is an important problem, WaveMLP gets an absolute representation of a patch, i.e. the patch is the same when participating in aggregation in any local region. The representation of a patch should be different in different local regions, so we focus on modulating the feature aggregation in local regions. That is, we use a vector representation to better express the relative relationship between neighbor points and centroids in the local region. + +In addition, WaveMLP use GroupConv [\[13\]](#page-11-15) to implement the aggregation and projection process with kernel sizes of 1 × 7 and 7 × 1. In this paper we take the form of a combination of the reduction function and GroupConv for aggregation. We give an example of why the original GroupConv is not suitable for this representation of vectors. We take two-dimensional vectors (x1, y1) and (x2, y2) as an example. The vectors are represented in coordinate form, and then the original vector aggregation method can be formulated as: + +$$f_{12} = (w_1(x_1, y_1) + w_2(x_2, y_2)) \cdot (w_3, w_4)^T$$ + +$$= w_3 w_1 x_1 + w_3 w_2 x_2 + w_4 w_1 y_1 + w_4 w_2 y_2$$ + +$$= a_1 x_1 + a_2 x_2 + a_3 y_1 + a_4 y_2,$$ +(7) + +where f12 denotes the result of aggregating two vectors, w1 and w2 are the weights of two vectors in summation, {w3, w4} is the projection matrix, and ai is the weight of each component. We can obtain the equation that should be satisfied between the coefficients of each component: a1 ∗ a4 = a2 ∗ a3. That is, the final trained weights need to satisfy this equation for the weighted summation formula of the vectors to hold. However, the network does not impose this restriction on these parameters. So the original groupconv does not preserve the totality of the vector. + +### A.2. Methodology Review. + +MLPs. + +The point-based approach was first introduced by Point-Net [\[25\]](#page-12-0). We denote f l+1 i as the extracted feature of point *i* after stage *l+1*, Ni as the neighbors of point *i* and n is the number of incoming points. The simplest point-set operator can be expressed as follows: + +$$f_i^{l+1} = R\{H\{[f_j^l, p_j - p_i]\}|j \in N_i\},$$ + (8) where $R$ is the reduction function that aggregates features for point $i$ from its neighbors $N_i$ and $H$ means the shared + +The subsequent dynamic convolution-based network [\[36\]](#page-12-2) [\[45\]](#page-13-9) can be similarly represented as PointNet-like point set operators: + +f l+1 i = Sum{φ{f l j , pj − pi} · f j |j ∈ Ni}, (9) where φ() means the dynamic weight generation function that generates dynamic weights for each point based on the input feature and location information. Eq[.9](#page-8-0) shows that the reduction function of dynamic convolution chooses sum and uses dynamic weights to generate a new fj . + +Similarly, the attention network [\[53\]](#page-13-0) can be expressed as a similar point set operator. The core operation can be formulated as follows: + +f l+1 i = Sum{att{f l j , fl i , pos}·σ{f l j , pos}|j ∈ Ni}, (10) where att() means the attention function that generates attention weights for each point, pos denotes the position information, and σ() means the linear transform function without anisotropy. Eq[.10](#page-8-1) shows that it uses the attention mechanism to update the features of each point *j* and then uses sum as the reduction function. + +Furthermore, template-based methods such as 3D-GCN make use of kernels with relative displacement vectors and weights. These weights are influenced by the cosine similarity between the relative displacement vector of the input features and the relative displacement vector of the kernel. The core operation can be formulated as follows: + +$$f_i^{l+1} = f_i \cdot kernel_c + \sum_{m=1}^k \max\{sim\{kernel_m, f_j\} | j \in N_i\},\$$ + +$$sim\{kernerl_m, f_j\} = cos\{dk_m, dp_j\} \cdot kernerl_m \cdot f_j,$$ +(11) + +where *k* means the kernel size, Ni means the neighbors of point *i*, kernelc means the center element of kernel, cos{dkm, dpj} means Cosine similarity of *m*-th kernerl element and *j*-th point feature, kernelm means *m*-th kernel element, dkm, dpj means displacement vector of *m*-th kernel element and *j*-th point feature respectively. + +We propose a unique method for generating new features fj by introducing a vector representation, where the direction of the vector guides the aggregation method. + +## B. Architecture + +### B.1. Vector encoder + +![](PointVector_2205.10528_images/_page_8_Figure_19.jpeg) + +Figure 6. The Vector encoder module. Two angles are predicted by MLP and zx is transformed by linear. + +> **[그림 해설]** Vector Encoder 모듈의 내부 연산 흐름도. +> - 중심점(1)과 이웃점(3) 간 상대 특징 $f_3 - f_1$ 및 상대 좌표 $p_{13}$을 결합($\oplus$, sum + ReLU). +> - 세 개 경로로 분기: +> - MLP 1 $\to$ 회전각 $\alpha$ 도출. +> - MLP 2 $\to$ 회전각 $\beta$ 도출. +> - Linear $\to$ 크기 계수 $zx$ 도출. +> - 최종적으로 $(zx \sin\beta \cos\alpha, zx \sin\beta \sin\alpha, zx \cos\beta)$의 3D 회전 벡터 표현을 생성. + +We provide detailed definitions in the manuscript, and we provide illustrations to illustrate the exact process. As shown in Fig[.6,](#page-8-2) the local information is obtained by a combination of relative features and relative positions. Note that the sum symbol in the figure means sum and ReLU operations. We use the simplest method to predict the angles using MLP, and by default the two angles are independent of each other. For zx, a simple transformation is performed with linear, and then a vector representation is obtained by rotation. The vector representation v ∈ RB,C×3,N , where B is the batch size, C is the channel of module and N is the spatial size of the input feature of the module. + +### B.2. Classification architecture. + +![](PointVector_2205.10528_images/_page_9_Figure_2.jpeg) + +Figure 7. The Classification architecture PointVector-S. For comparison with PointNext [\[28\]](#page-12-4), we replaced the SetAbstract module with the LocalVector module, keeping the other parameters the same. + +> **[그림 해설]** PointVector-S 객체 분류 네트워크 아키텍처 다이어그램. +> - 입력 $\to$ MLP $[N, 32]$. +> - 4단계 LocalVector 다운샘플링 블록: $[N/2, 64] \to [N/4, 128] \to [N/8, 256] \to [N/16, 512]$. +> - 마지막 전역 특징 집계용 SetAbstraction 블록 $[N/16, 512] \to$ Cls Head $\to [\text{Categories}]$ 분류 결과 출력. + +As shown in Fig[.7,](#page-9-0) we use the LocalVector module to replace the 4 SetAbstract modules and keep the downsampling parameters unchanged. The last SetAbstract was originally used to aggregate all the remaining points, so we leave it as it is. In the classification task, the max reduction fuction has a greater advantage by retaining the most intense part of the variation. + +## C. Experiments + +### C.1. Classification on ModelNet40 + +ModelNet40 [\[43\]](#page-12-20) is a commonly used dataset for object classification, which is generated by 3D graphic CAD models. It has 40 object categories, each of which contains 100 unique CAD models. Recent works [\[24\]](#page-12-10) [\[30\]](#page-12-11) [\[10\]](#page-11-3) show an increasing interest in the real-world scanned dataset ScanObejectNN [\[48\]](#page-13-2) than this synthesized 3D dataset ModelNet40. Therefore, we choose to report the results on ScanObjectNN in the manuscript. Furthermore, we report the results of our PointVector-S model on ModelNet40. We use the same parameters as PointNext: CrossEntropy loss with label smoothing, AdamW optimizer, a learning rate of 1e-3, a weight decay of 0.05, cosine learning rate decay, and a batch size of 32 for 600 epochs, while using random scaling and translation as data augmentations. As shown in table [9,](#page-9-1) the relatively poor performance of our model on + + + +| Method | mAcc | OA | +|-----------------------------|------------|------------| +| | % | % | +| PointNet [25] | 86.2 | 89.2 | +| PointCNN [17] | 88.1 | 92.2 | +| PointConv [42] | – | 92.5 | +| KPConv [36] | - | 92.9 | +| DGCNN [39] | 90.2 | 92.9 | +| DeepGCN [16] | 90.9 | 93.6 | +| ASSANet-L [27] | - | 92.9 | +| Point Cloud Transformer [9] | - | 93.2 | +| Point Transformer [53] | 90.6 | 93.7 | +| CurveNet [44] | - | 93.8 | +| PointMLP [24] | 90.9±0.4 | 93.7±0.2 | +| PointNet++ | - | 91.9 | +| PointNet++(PointNext) | 89.9 ± 0.8 | 92.8 ± 0.1 | +| PointNext(C=32) | 90.8 ± 0.2 | 93.2 ± 0.1 | +| PointNext(C=64) | 90.9 ± 0.5 | 93.7 ± 0.3 | +| PointVector-S(C=32) | 90.3 ± 0.2 | 93.2 ± 0.2 | +| PointVector-S(C=64) | 91.0 ± 0.5 | 93.5 ± 0.2 | + +Table 9. Object Classification on ModelNet40. + +the ModelNet40 dataset indicates the limitation of the proposed local vector representation in aggregating global information. We used hyperparameters consistent with Point-Next and a training strategy that may not be suitable for our model, which may also account for the relatively poor performance. Note that our network structure on the classification task directly takes vector feature aggregation for downsampling, but max-pooling is probably the simplest and most effective method for downsampling. + +### C.2. Ablation study + +There is a slight problem with the experimental setup in the manuscript, in the 6-fold cross-validation experiment we report the PointVector-L as the standard setup mentioned in the manuscript, but in the S3DIS Area5 and ablation experiments we report the setup of PointVector-L as V=[2, 2, 4, 2]. But, the max+groupconv in the manuscript is reported as V=[2,4,2,2]. + + + +| Method | size | OA | mAcc | mIOU | +|-----------------|-------------|------|-------|------| +| | | % | % | % | +| PointVector-L | V=[2,4,2,2] | 90.6 | 76.2 | 70.6 | +| (max+groupconv) | V=[2,2,4,2] | 90.6 | 77.1 | 71.1 | +| PointVector-L | V=[2,4,2,2] | 90.3 | 77.21 | 70.8 | +| (sum+groupconv) | V=[2,2,4,2] | 90.8 | 77.3 | 71.2 | +| | V=[3,5,3,3] | 90.8 | 78.3 | 72.3 | +| PointVector-XL | V=[3,3,5,3] | 91.0 | 76.7 | 71.1 | + +Table 10. Results for models with different number of stagess on S3DIS Area5. + +Number of stages. Since the PointVector-L with max+groupconv is reported by another configuration in the manuscript, we compare the two configurations here. As the tab[.10](#page-9-2) shows, the two reduction functions, max and sum, obtain very similar results, but sum has a higher mAcc and OA. This is consistent with our assumption that better results can be obtained by simply using groupconv to process vectors of each channel independently. Small and large models do not behave consistently in terms of the number of stages. This is an interesting phenomenon, but not the main point of our statement, so it will not be discussed for now. + +The following experiments are reported by default as PointVector-XL [3,5,3,3], PointVector-L [2,2,4,2] if no special instructions are given. + + + +| Method | OA | mAcc | mIOU | Params | +|------------------|------|------|------|--------| +| | % | % | % | M | +| PointNeXt-XL | 90.7 | 77.5 | 70.8 | 41.6 | +| PointVector-base | 90.9 | 77.0 | 71.4 | 37.2 | +| PointVector-XL | 90.8 | 78.3 | 72.3 | 24.1 | + +Table 11. Baseline. The same experimental configuration was used for all three models. + +Baseline. Our model has some gaps in channel variations and inputs with PointNeXt. To really evaluate whether our model has a greater advantage, we reset a baseline. We take our core operations i.e. Vector encoder and reduction+groupconv+channel mixing Linear was removed and replaced with PointNeXt's MLP+max+MLPs, where the channel of first MLP was transformed from c to 3c. The new model is named PointVector-base. The tab[.11](#page-10-0) shows that our model has a large improvement in each metric compared to baseline. Also this shows that the other parts of our model are superior compared to the original PointNeXt. + + + +| type | Method | OA | mAcc | mIOU | +|----------|------------------|------|------|------| +| | | % | % | % | +| | fj
− fi+pos | 90.8 | 77.3 | 71.2 | +| feature | [fj
− fi,pos] | 88.8 | 70.5 | 64.9 | +| | fj+pos | 90.9 | 76.6 | 70.5 | +| | linear | 90.8 | 77.3 | 71.2 | +| residual | identity | 90.3 | 75.8 | 69.3 | + +Table 12. Other Components. + means that the two are added together and then passed through the relu layer. [,] means to directly concatenate two elements. + +Other Components. The manuscript mentions that other operations of our model have a larger role, so we conducted ablation experiments on PointVector-L to explore the effect of both input features and residuals on the S3DIS segmentation task. Tab[.12](#page-10-1) shows that the two parts of the features are added together and then relu can better fuse their information. In addition relative features are more robust than absolute features. The key is that residual uses linear compared to identity, which is a huge improvement. + +## D. Visualization + +![](PointVector_2205.10528_images/_page_10_Figure_10.jpeg) + +Figure 8. Qualitative comparisons of PointNext (2 nd column), PointVector++ (3 rd column), and Ground Truth (4 th column) on S3DIS semantic segmentation. The input point cloud is visualized with original colors in the 1 st column. We have circled the different places with a paintbrush. + +> **[그림 해설]** S3DIS 시맨틱 세그멘테이션 정성적 비교 (PointNeXt vs PointVector++ vs Ground Truth, 4개 실내 씬). +> - **노란색 박스 영역 비교**: +> - 1행: 벽면에서 돌출된 기둥(자주색)을 PointNeXt는 평면 벽(하늘색)으로 오인했으나 PointVector++는 기둥으로 정확히 판별. +> - 2행: 복도 상단 빔/덕트 구조(검정색)를 PointNeXt는 오분류했으나 PointVector++는 GT와 동일하게 정확히 분할. +> - 3~4행: 복잡한 실내 출입구와 코너 구조에서 방향성 벡터 기반 특징 추출을 통해 PointVector++가 미세 기하 구조를 한층 더 우수하게 분할함을 입증. + +As shown in the Fig[.8,](#page-10-2) it can be found that our model performs a little better in complex areas. This shows that we are able to extract more detail in such intensely varied areas than the max-pooling operation of PointNeXt. But we are also prone to miscalculation in flat areas, which is our disadvantage. + +## E. Code release + +Since our model is based on PointNext, we used their code and added a PointVector model. Since our classification and part segmentation and semantic segmentation tasks use different model compositions, the model code is also different, and the corresponding PointVector.py needs to be replaced at runtime. We have not organized the code yet, where PATM represents the core part of our LocalVector module. In the classification and part segmentation tasks, it replaces the convs+max pooling operation in SetAbstraction. See the official instructions for PointNext for related running instructions. And on s3dis our gravity dim is set to 1. The code of each task is a little different, on ScanObjectNN classification task we insert leakyrelu in the two linear after the reduction function, and the relative features of the input after BN, encoder's activation function all use leakyrelu can reach 88.4% OA, but this is not the main point of our statement, so we do not discuss it for now. The code takes time to organize and we will make it public later. + +## References + +- [1] Iro Armeni, Ozan Sener, Amir R. Zamir, Helen Jiang, Ioannis Brilakis, Martin Fischer, and Silvio Savarese. 3d semantic parsing of large-scale indoor spaces. In *Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)*, June 2016. [2,](#page-1-0) [5,](#page-4-2) [6](#page-5-1) +- [2] Matan Atzmon, Haggai Maron, and Yaron Lipman. Point convolutional neural networks by extension operators. *ACM Transactions on Graphics*, 37(4), 2018. [2](#page-1-0) +- [3] Shoufa Chen, Enze Xie, Chongjian GE, Runjian Chen, Ding Liang, and Ping Luo. CycleMLP: A MLP-like architecture for dense prediction. In *International Conference on Learning Representations*, 2022. [2](#page-1-0) +- [4] Yuedong Chen, Guoxian Song, Zhiwen Shao, Jianfei Cai, Tat-Jen Cham, and Jianmin Zheng. Geoconv: Geodesic guided convolution for facial action unit recognition. *Pattern Recognition*, 122, 2022. [2](#page-1-0) +- [5] Silin Cheng, Xiwu Chen, Xinwei He, Zhe Liu, and Xiang Bai. Pra-net: Point relation-aware network for 3d point cloud analysis. *IEEE Transactions on Image Processing*, 30:4436 – 4448, 2021. [7](#page-6-3) +- [6] Jaesung Choe, Chunghyun Park, Francois Rameau, Jaesik Park, and In So Kweon. Pointmixer: Mlp-mixer for point cloud understanding. 2021. [2](#page-1-0) +- [7] Congyue Deng, Or Litany, Yueqi Duan, Adrien Poulenard, Andrea Tagliasacchi, and Leonidas J. Guibas. Vector neurons: A general framework for so(3)-equivariant networks. In *Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)*, pages 12200–12209, October 2021. [2](#page-1-0) +- [8] Ankit Goyal, Hei Law, Bowei Liu, Alejandro Newell, and Jia Deng. Revisiting point cloud shape classification with a simple and effective baseline. In Marina Meila and Tong Zhang, editors, *Proceedings of the 38th International Conference on Machine Learning*, volume 139, pages 3809–3820. PMLR, 2021. [6](#page-5-1) +- [9] Meng-Hao Guo, Jun-Xiong Cai, Zheng-Ning Liu, Tai-Jiang Mu, Ralph R. Martin, and Shi-Min Hu. Pct: Point cloud transformer. *Computational Visual Media*, 7(2):187 – 199, 2021. [2,](#page-1-0) [10](#page-9-3) +- [10] Abdullah Hamdi, Silvio Giancola, and Bernard Ghanem. Mvtn: Multi-view transformation network for 3d shape recognition. In *Proceedings of the IEEE International Conference on Computer Vision*, pages 1 – 11, Virtual, Online, Canada, 2021. [2,](#page-1-0) [10](#page-9-3) +- [11] Qingdong He, Zhengning Wang, Hao Zeng, Yi Zeng, and Yijun Liu. Svga-net: Sparse voxel-graph attention network for 3d object detection from point clouds. 2020. [2](#page-1-0) +- [12] Qingyong Hu, Bo Yang, Linhai Xie, Stefano Rosa, Yulan Guo, Zhihua Wang, Niki Trigoni, and Andrew Markham. + +- Randla-net: Efficient semantic segmentation of large-scale point clouds. In *Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition*, pages 11105 – 11114, Virtual, Online, United states, 2020. [6](#page-5-1) +- [13] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. In F. Pereira, C.J. Burges, L. Bottou, and K.Q. Weinberger, editors, *Advances in Neural Information Processing Systems*, volume 25. Curran Associates, Inc., 2012. [4,](#page-3-5) [9](#page-8-3) +- [14] Xin Lai, Jianhui Liu, Li Jiang, Liwei Wang, Hengshuang Zhao, Shu Liu, Xiaojuan Qi, and Jiaya Jia. Stratified transformer for 3d point cloud segmentation. In *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)*, pages 8500–8509, June 2022. [2,](#page-1-0) [6,](#page-5-1) [7](#page-6-3) +- [15] Alex H. Lang, Sourabh Vora, Holger Caesar, Lubing Zhou, Jiong Yang, and Oscar Beijbom. Pointpillars: Fast encoders for object detection from point clouds. In *Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition*, volume 2019-June, pages 12689 – 12697, Long Beach, CA, United states, 2019. [2](#page-1-0) +- [16] Guohao Li, Matthias Mueller, Guocheng Qian, Itzel Carolina Delgadillo Perez, Abdulellah Abualshour, Ali Kassem Thabet, and Bernard Ghanem. Deepgcns: Making gcns go as deep as cnns. *IEEE Transactions on Pattern Analysis and Machine Intelligence*, 2021. [6,](#page-5-1) [7,](#page-6-3) [10](#page-9-3) +- [17] Yangyan Li, Rui Bu, Mingchao Sun, Wei Wu, Xinhan Di, and Baoquan Chen. Pointcnn: Convolution on x-transformed points. In *Advances in Neural Information Processing Systems*, volume 2018-December, pages 820 – 830, Montreal, QC, Canada, 2018. [2,](#page-1-0) [6,](#page-5-1) [7,](#page-6-3) [10](#page-9-3) +- [18] Dongze Lian, Zehao Yu, Xing Sun, and Shenghua Gao. Asmlp: An axial shifted mlp architecture for vision. In *International Conference on Learning Representations (ICLR)*, 2022. [2](#page-1-0) +- [19] Zhi-Hao Lin, Sheng-Yu Huang, and Yu-Chiang Frank Wang. Convolution in the cloud: Learning deformable kernels in 3d graph convolution networks for point cloud analysis. In *2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)*, pages 1797–1806, 2020. [1,](#page-0-1) [2,](#page-1-0) [7](#page-6-3) +- [20] Yongcheng Liu, Bin Fan, Shiming Xiang, and Chunhong Pan. Relation-shape convolutional neural network for point cloud analysis. In *Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition*, volume 2019-June, pages 8887 – 8896, Long Beach, CA, United states, 2019. [6](#page-5-1) +- [21] Ze Liu, Han Hu, Yue Cao, Zheng Zhang, and Xin Tong. A closer look at local aggregation operators in point cloud analysis. In *Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XXIII 16*, pages 326–342. Springer, 2020. [2](#page-1-0) +- [22] Zhijian Liu, Haotian Tang, Yujun Lin, and Song Han. Pointvoxel cnn for efficient 3d deep learning. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alche-Buc, E. Fox, and R. ´ Garnett, editors, *Advances in Neural Information Processing Systems*, volume 32. Curran Associates, Inc., 2019. [7](#page-6-3) +- [23] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In *7th International Conference on Learn-* + +- *ing Representations, ICLR 2019*, New Orleans, LA, United states, 2019. [5](#page-4-2) +- [24] Xu Ma, Can Qin, Haoxuan You, Haoxi Ran, and Yun Fu. Rethinking network design and local geometry in point cloud: A simple residual mlp framework, 2022. [2,](#page-1-0) [7,](#page-6-3) [10](#page-9-3) +- [25] Charles R. Qi, Hao Su, Kaichun Mo, and Leonidas J. Guibas. Pointnet: Deep learning on point sets for 3d classification and segmentation. In *Proceedings - 30th IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017*, volume 2017-January, pages 77 – 85, Honolulu, HI, United states, 2017. [1,](#page-0-1) [2,](#page-1-0) [6,](#page-5-1) [7,](#page-6-3) [9,](#page-8-3) [10](#page-9-3) +- [26] Charles Ruizhongtai Qi, Li Yi, Hao Su, and Leonidas J Guibas. Pointnet++: Deep hierarchical feature learning on point sets in a metric space. In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, *Advances in Neural Information Processing Systems*, volume 30. Curran Associates, Inc., 2017. [1,](#page-0-1) [2,](#page-1-0) [6,](#page-5-1) [7](#page-6-3) +- [27] Guocheng Qian, Hasan Hammoud, Guohao Li, Ali Thabet, and Bernard Ghanem. Assanet: An anisotropic separable set abstraction for efficient point cloud representation learning. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan, editors, *Advances in Neural Information Processing Systems*, volume 34, pages 28119–28130. Curran Associates, Inc., 2021. [2,](#page-1-0) [5,](#page-4-2) [6,](#page-5-1) [7,](#page-6-3) [10](#page-9-3) +- [28] Guocheng Qian, Yuchen Li, Houwen Peng, Jinjie Mai, Hasan Hammoud, Mohamed Elhoseiny, and Bernard Ghanem. Pointnext: Revisiting pointnet++ with improved training and scaling strategies. In *Advances in Neural Information Processing Systems (NeurIPS)*, 2022. [1,](#page-0-1) [2,](#page-1-0) [5,](#page-4-2) [6,](#page-5-1) [7,](#page-6-3) [10](#page-9-3) +- [29] Shi Qiu, Saeed Anwar, and Nick Barnes. Geometric backprojection network for point cloud classification. *IEEE Transactions on Multimedia*, 24:1943 – 1955, 2022. [7](#page-6-3) +- [30] Haoxi Ran, Jun Liu, and Chengjie Wang. Surface representation for point clouds. In *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)*, pages 18942–18952, June 2022. [2,](#page-1-0) [5,](#page-4-2) [6,](#page-5-1) [7,](#page-6-3) [10](#page-9-3) +- [31] Shaoshuai Shi, Chaoxu Guo, Li Jiang, Zhe Wang, Jianping Shi, Xiaogang Wang, and Hongsheng Li. Pv-rcnn: Pointvoxel feature set abstraction for 3d object detection. In *Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition*, pages 10526 – 10535, Virtual, Online, United states, 2020. [2](#page-1-0) +- [32] Hang Su, Subhransu Maji, Evangelos Kalogerakis, and Erik Learned-Miller. Multi-view convolutional neural networks for 3d shape recognition. In *Proceedings of the IEEE International Conference on Computer Vision (ICCV)*, December 2015. [2](#page-1-0) +- [33] Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jon Shlens, and Zbigniew Wojna. Rethinking the inception architecture for computer vision. In *Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)*, June 2016. [5](#page-4-2) +- [34] Liyao Tang, Yibing Zhan, Zhe Chen, Baosheng Yu, and Dacheng Tao. Contrastive boundary learning for point cloud segmentation. In *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)*, pages 8489–8499, June 2022. [6,](#page-5-1) [7](#page-6-3) + +- [35] Yehui Tang, Kai Han, Jianyuan Guo, Chang Xu, Yanxi Li, Chao Xu, and Yunhe Wang. An image patch is a wave: Phase-aware vision mlp. In *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)*, pages 10935–10944, June 2022. [2,](#page-1-0) [9](#page-8-3) +- [36] Hugues Thomas, Charles R. Qi, Jean-Emmanuel Deschaud, Beatriz Marcotegui, Francois Goulette, and Leonidas Guibas. Kpconv: Flexible and deformable convolution for point clouds. In *Proceedings of the IEEE International Conference on Computer Vision*, volume 2019-October, pages 6410 – 6419, Seoul, Korea, Republic of, 2019. [1,](#page-0-1) [2,](#page-1-0) [5,](#page-4-2) [6,](#page-5-1) [7,](#page-6-3) [9,](#page-8-3) [10](#page-9-3) +- [37] Ilya O Tolstikhin, Neil Houlsby, Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai, Thomas Unterthiner, Jessica Yung, Andreas Steiner, Daniel Keysers, Jakob Uszkoreit, Mario Lucic, and Alexey Dosovitskiy. Mlp-mixer: An all-mlp architecture for vision. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan, editors, *Advances in Neural Information Processing Systems*, volume 34, pages 24261–24272. Curran Associates, Inc., 2021. [2](#page-1-0) +- [38] Lei Wang, Yuchun Huang, Yaolin Hou, Shenman Zhang, and Jie Shan. Graph attention convolution for point cloud semantic segmentation. In *Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition*, volume 2019-June, pages 10288 – 10297, Long Beach, CA, United states, 2019. [2](#page-1-0) +- [39] Yue Wang, Yongbin Sun, Ziwei Liu, Sanjay E. Sarma, Michael M. Bronstein, and Justin M. Solomon. Dynamic graph cnn for learning on point clouds. *ACM Transactions on Graphics*, 38(5), 2019. [1,](#page-0-1) [2,](#page-1-0) [6,](#page-5-1) [7,](#page-6-3) [10](#page-9-3) +- [40] Yida Wang, David Joseph Tan, Nassir Navab, and Federico Tombari. Learning local displacements for point cloud completion. In *Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition*, 2022. [2](#page-1-0) +- [41] Yan Wang, Wanxia Zhong, Hang Su, Fujiang Zheng, Yiran Pang, Hongchuan Wen, and Kun Cai. An improved mvcnn for 3d shape recognition. In *Proceedings of 2021 IEEE International Conference on Emergency Science and Information Technology, ICESIT 2021*, pages 469 – 472, Chongqing, China, 2021. [2](#page-1-0) +- [42] Wenxuan Wu, Zhongang Qi, and Li Fuxin. Pointconv: Deep convolutional networks on 3d point clouds. In *Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition*, volume 2019-June, pages 9613 – 9622, Long Beach, CA, United states, 2019. [2,](#page-1-0) [10](#page-9-3) +- [43] Zhirong Wu, Shuran Song, Aditya Khosla, Fisher Yu, Linguang Zhang, Xiaoou Tang, and Jianxiong Xiao. 3d shapenets: A deep representation for volumetric shapes. In *Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition*, volume 07-12- June-2015, pages 1912 – 1920, Boston, MA, United states, 2015. [10](#page-9-3) +- [44] Tiange Xiang, Chaoyi Zhang, Yang Song, Jianhui Yu, and Weidong Cai. Walk in the cloud: Learning curves for point clouds shape analysis. In *Proceedings of the IEEE International Conference on Computer Vision*, pages 895 – 904, Virtual, Online, Canada, 2021. [7,](#page-6-3) [10](#page-9-3) + +- [45] Mutian Xu, Runyu Ding, Hengshuang Zhao, and Xiaojuan Qi. Paconv: Position adaptive convolution with dynamic kernel assembling on point clouds. In *Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition*, pages 3172 – 3181, Virtual, Online, United states, 2021. [4,](#page-3-5) [7,](#page-6-3) [9](#page-8-3) +- [46] Yifan Xu, Tianqi Fan, Mingye Xu, Long Zeng, and Yu Qiao. Spidercnn: Deep learning on point sets with parameterized convolutional filters. In *Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)*, volume 11212 LNCS, pages 90 – 105, Munich, Germany, 2018. [2](#page-1-0) +- [47] Jiancheng Yang, Qiang Zhang, Bingbing Ni, Linguo Li, Jinxian Liu, Mengdie Zhou, and Qi Tian. Modeling point clouds with self-attention and gumbel subset sampling. In *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)*, June 2019. [2](#page-1-0) +- [48] Li Yi, Vladimir G. Kim, Duygu Ceylan, I-Chao Shen, Mengyan Yan, Hao Su, Cewu Lu, Qixing Huang, Alla Sheffer, and Leonidas Guibas. A scalable active framework for region annotation in 3d shape collections. *ACM Transactions on Graphics*, 35(6), 2016. [2,](#page-1-0) [5,](#page-4-2) [6,](#page-5-1) [10](#page-9-3) +- [49] Li Yi, Vladimir G. Kim, Duygu Ceylan, I-Chao Shen, Mengyan Yan, Hao Su, Cewu Lu, Qixing Huang, Alla Sheffer, and Leonidas Guibas. A scalable active framework for region annotation in 3d shape collections. *ACM Transactions on Graphics*, 35(6), 2016. [2,](#page-1-0) [5,](#page-4-2) [7](#page-6-3) +- [50] Xumin Yu, Yongming Rao, Ziyi Wang, Zuyan Liu, Jiwen Lu, and Jie Zhou. Pointr: Diverse point cloud completion with geometry-aware transformers. In *Proceedings of the IEEE International Conference on Computer Vision*, pages 12478 – 12487, Virtual, Online, Canada, 2021. [2](#page-1-0) +- [51] Xumin Yu, Lulu Tang, Yongming Rao, Tiejun Huang, Jie Zhou, and Jiwen Lu. Point-bert: Pre-training 3d point cloud transformers with masked point modeling. In *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)*, pages 19313–19322, June 2022. [2,](#page-1-0) [5](#page-4-2) +- [52] Cheng Zhang, Haocheng Wan, Xinyi Shen, and Zizhao Wu. Patchformer: An efficient point transformer with patch attention. In *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)*, pages 11799– 11808, June 2022. [7](#page-6-3) +- [53] Hengshuang Zhao, Li Jiang, Jiaya Jia, Philip H.S. Torr, and Vladlen Koltun. Point transformer. In *Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)*, pages 16259–16268, October 2021. [1,](#page-0-1) [4,](#page-3-5) [5,](#page-4-2) [6,](#page-5-1) [7,](#page-6-3) [9,](#page-8-3) [10](#page-9-3) +- [54] Haoran Zhou, Yidan Feng, Mingsheng Fang, Mingqiang Wei, Jing Qin, and Tong Lu. Adaptive graph convolution for point cloud analysis. In *Proceedings of the IEEE International Conference on Computer Vision*, pages 4945 – 4954, Virtual, Online, Canada, 2021. [1,](#page-0-1) [4](#page-3-5) +- [55] Yin Zhou and Oncel Tuzel. Voxelnet: End-to-end learning for point cloud based 3d object detection. In *Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition*, pages 4490 – 4499, Salt Lake City, UT, United states, 2018. [2](#page-1-0) \ No newline at end of file diff --git a/docs/papers/md/PointVector_2205.10528_images/_page_0_Figure_11.jpeg b/docs/papers/md/PointVector_2205.10528_images/_page_0_Figure_11.jpeg new file mode 100644 index 0000000..6a65073 Binary files /dev/null and b/docs/papers/md/PointVector_2205.10528_images/_page_0_Figure_11.jpeg differ diff --git a/docs/papers/md/PointVector_2205.10528_images/_page_10_Figure_10.jpeg b/docs/papers/md/PointVector_2205.10528_images/_page_10_Figure_10.jpeg new file mode 100644 index 0000000..78d1adf Binary files /dev/null and b/docs/papers/md/PointVector_2205.10528_images/_page_10_Figure_10.jpeg differ diff --git a/docs/papers/md/PointVector_2205.10528_images/_page_2_Picture_10.jpeg b/docs/papers/md/PointVector_2205.10528_images/_page_2_Picture_10.jpeg new file mode 100644 index 0000000..d2e539f Binary files /dev/null and b/docs/papers/md/PointVector_2205.10528_images/_page_2_Picture_10.jpeg differ diff --git a/docs/papers/md/PointVector_2205.10528_images/_page_3_Figure_0.jpeg b/docs/papers/md/PointVector_2205.10528_images/_page_3_Figure_0.jpeg new file mode 100644 index 0000000..03ba6bd Binary files /dev/null and b/docs/papers/md/PointVector_2205.10528_images/_page_3_Figure_0.jpeg differ diff --git a/docs/papers/md/PointVector_2205.10528_images/_page_3_Picture_7.jpeg b/docs/papers/md/PointVector_2205.10528_images/_page_3_Picture_7.jpeg new file mode 100644 index 0000000..e782806 Binary files /dev/null and b/docs/papers/md/PointVector_2205.10528_images/_page_3_Picture_7.jpeg differ diff --git a/docs/papers/md/PointVector_2205.10528_images/_page_4_Figure_0.jpeg b/docs/papers/md/PointVector_2205.10528_images/_page_4_Figure_0.jpeg new file mode 100644 index 0000000..7c8b363 Binary files /dev/null and b/docs/papers/md/PointVector_2205.10528_images/_page_4_Figure_0.jpeg differ diff --git a/docs/papers/md/PointVector_2205.10528_images/_page_8_Figure_19.jpeg b/docs/papers/md/PointVector_2205.10528_images/_page_8_Figure_19.jpeg new file mode 100644 index 0000000..6d14d81 Binary files /dev/null and b/docs/papers/md/PointVector_2205.10528_images/_page_8_Figure_19.jpeg differ diff --git a/docs/papers/md/PointVector_2205.10528_images/_page_9_Figure_2.jpeg b/docs/papers/md/PointVector_2205.10528_images/_page_9_Figure_2.jpeg new file mode 100644 index 0000000..213349e Binary files /dev/null and b/docs/papers/md/PointVector_2205.10528_images/_page_9_Figure_2.jpeg differ diff --git a/docs/papers/md/SUM-Parts_2503.15300.md b/docs/papers/md/SUM-Parts_2503.15300.md new file mode 100644 index 0000000..fa70663 --- /dev/null +++ b/docs/papers/md/SUM-Parts_2503.15300.md @@ -0,0 +1,761 @@ +# SUM Parts: Benchmarking Part-Level Semantic Segmentation of Urban Meshes + +# Weixiao Gao, Liangliang Nan, Hugo Ledoux Delft University of Technology + +{w.gao-1,liangliang.nan,h.ledoux}@tudelft.nl + +![](SUM-Parts_2503.15300_images/_page_0_Figure_4.jpeg) + +Figure 1. SUM Parts provides part-level semantic segmentation of urban textured meshes, covering 2.5 km2 with 21 classes. From left to right: textured mesh, face-based annotations, and texture-based annotations. See Tab. [1](#page-4-0) for class definitions. + +> **[그림 해설]** SUM-Parts 벤치마크 데이터셋의 3가지 뷰 비교 (항공 조감도 및 2개 상세 확대 영역). +> - **좌측 (Textured Mesh)**: 헬싱키 도심 $2.5\,\text{km}^2$ 영역의 원본 3D 텍스처 메시. +> - **중앙 (Face-based annotations)**: 13개 클래스(외벽, 지붕, 지형, 도로, 수목, 수면, 차량 등)의 폴리곤 페이스 단위 부품 분할. +> - **우측 (Texture-based annotations)**: 19/21개 클래스의 텍스처(픽셀) 기반 초정밀 파트 분할 (창문, 출입문, 굴뚝, 차선 마킹, 보도, 잔디밭 등 미세 구성 요소 완벽 분할). + +# Abstract + +*Semantic segmentation in urban scene analysis has mainly focused on images or point clouds, while textured meshes—offering richer spatial representation—remain underexplored. This paper introduces SUM Parts, the first large-scale dataset for urban textured meshes with partlevel semantic labels, covering about* 2.5 *km*2 *with 21 classes. The dataset was created using our own annotation tool, which supports both face- and texture-based annotations with efficient interactive selection. We also provide a comprehensive evaluation of 3D semantic segmentation and interactive annotation methods on this dataset. Our project page is available at [https://tudelft3d.github.io/SUMParts/.](https://tudelft3d.github.io/SUMParts/)* + +## 1. Introduction + +Semantic segmentation is crucial to understanding urban scenes by accurately classifying objects and improving data usability. Recent advances have led to the development of datasets and methods primarily for images and point clouds [\[15,](#page-8-0) [35,](#page-9-0) [40\]](#page-9-1). Research on textured meshes has focused mainly on small-scale indoor settings [\[5,](#page-8-1) [11,](#page-8-2) [16\]](#page-8-3), with limited work on large outdoor environments [\[19,](#page-8-4) [20,](#page-8-5) [42,](#page-9-2) [52\]](#page-10-0). Furthermore, a critical gap in urban scene understanding is part-level semantic segmentation, which decomposes urban objects into functional components (e.g., windows, chimneys, road markings) following the international CityGML standard [\[45\]](#page-9-3). To address this, we introduce the first large-scale benchmark dataset providing part-level semantic labels for urban textured meshes. + +Obtaining ground truth labels for urban scene understanding often relies on manual annotation, which is timeconsuming and expensive [\[19\]](#page-8-4). Labeling large-scale 3D scenes poses significant challenges compared to 2D image annotation, requiring flexible viewpoint management, specialized interaction methods, and advanced rendering techniques. Although considerable research has focused on interactive point cloud annotation [\[29,](#page-9-4) [36,](#page-9-5) [54\]](#page-10-1), efforts to annotate textured meshes [\[19,](#page-8-4) [49\]](#page-9-6) remain very sparse. Existing studies often label mesh vertices [\[16,](#page-8-3) [31\]](#page-9-7) or faces [\[19,](#page-8-4) [49\]](#page-9-6), neglecting the richer details provided by textures. + +Details such as building windows and road markings are better represented in texture images than in unconstrained mesh faces (see Fig. [2\)](#page-0-0). To address this, we introduce an efficient interactive tool + +![](SUM-Parts_2503.15300_images/_page_0_Picture_13.jpeg) + +Figure 2. Mesh textures and wireframes (black). for textured meshes, enabling both face- and texture-based annotations. + +> **[그림 해설]** 메시 기하 구조(와이어프레임)와 텍스처 해상도 간의 불일치(Resolution mismatch) 시각화. +> - **좌측**: 단일 창문 텍스처 위에 걸쳐진 듬성듬성한 삼각 폴리곤 와이어프레임. +> - **우측**: 횡단보도 줄무늬 텍스처 위에 교차하는 불규칙한 삼각 메시. +> - 페이스 단위 분할만으로는 텍스처 내 미세 부품 경계를 표현할 수 없어 텍스처(픽셀) 레벨 주석이 필수적임을 설명. + +The main contributions are (1) the first large-scale dataset with part-level semantic labels for urban textured meshes, (2) an efficient interactive annotation tool, and (3) a comprehensive analysis of state-of-the-art 3D semantic segmentation and interactive annotation methods. + +### 2. Related work + +Annotation for semantic segmentation. Annotation for semantic segmentation is essential in computer vision, leading to the development of various interactive methods for both 2D images and 3D data. + +Early interactive image segmentation methods include region-growing [\[2,](#page-8-6) [44\]](#page-9-8), contour-based [\[43\]](#page-9-9), graph-cut [\[6,](#page-8-7) [50\]](#page-10-2), and random walk approaches [\[21\]](#page-8-8). While effective, they often require significant user input or struggle with complex images. Deep learning models like DEXTR [\[39\]](#page-9-10) and SAM [\[27\]](#page-8-9) have advanced the field but depend on large datasets and may perform poorly on unseen categories. Our interactive annotation method for mesh texture images overcomes these limitations, offering greater generalizability without relying on extensive training data. + +Interactive 3D annotation uses data like multi-view images, point clouds, and meshes. Manual methods [\[26\]](#page-8-10) are labor-intensive; graph-cut approaches [\[36\]](#page-9-5) depend heavily on user input quality; region-based methods [\[19\]](#page-8-4) can lead to segmentation errors. Deep learning techniques [\[29,](#page-9-4) [61\]](#page-10-3) require extensive training data and struggle with new data. In contrast, our unsupervised approach requires no prior labeling and operates independently of the original imagery, using approximate user selections and template matching to enhance efficiency and accuracy in complex 3D scenes. + +Semantic 3D urban datasets. While several semantic 3D urban datasets exist for LiDAR and photogrammetric point clouds [\[3,](#page-8-11) [10,](#page-8-12) [24,](#page-8-13) [34,](#page-9-11) [35,](#page-9-0) [53\]](#page-10-4), they lack fine-grained partlevel semantics essential for comprehensive urban analysis. This is partly due to inherent point cloud limitations, such as low resolution and missing data from occlusions, which hinder capturing detailed structures and small object boundaries [\[18\]](#page-8-14). Additionally, point clouds have larger data volumes than textured meshes, leading to longer processing times and higher storage requirements without added informational benefit. In contrast, textured meshes offer better resolution and completeness, but existing mesh datasets [\[9,](#page-8-15) [19\]](#page-8-4) typically lack part-level semantic annotations and cover limited categories or scales, focusing on specific objects [\[28,](#page-9-12) [62\]](#page-10-5), which limits their applicability. Moreover, existing annotations frequently overlook the rich texture information in meshes. To address these gaps, we introduce the first part-level benchmark dataset of largescale urban meshes for comprehensive urban analysis. + +## 3. The SUM parts dataset + +We aim to use our developed interactive 3D annotation tool to create ground truth for urban textured meshes. Using Helsinki city's mesh [\[4\]](#page-8-16) as input, the output includes meshes with face labels and semantic texture masks. The textured meshes were generated using Bentley's ContextCapture [\[57\]](#page-10-6), reconstructed from oblique aerial imagery with a ground sampling distance of approximately 7.5 cm. The annotations were conducted in three representative areas of central Helsinki, comprising 40 tiles of 62, 500 m2 each, covering a total area of approximately 2.5 km2 . + +### 3.1. Annotation + +Our annotation aims to achieve precise semantic labeling with significantly improved efficiency for urban meshes. Our tool features two main modules for part-level semantic annotation: face-based annotation for triangle faces and texture-based annotation for texture pixels. We enhance the efficiency of both modules by incorporating interactive selection and template-matching strategies. We invited five individuals with experience in remote sensing to manually annotate the dataset using our tool: two focused on facebased annotation, two on texture pixel-based annotation, and one reviewed and corrected the annotations. The entire annotation process took approximately 640 hours in total. + +#### 3.1.1. Face-based annotation + +The face-based annotation aims to assign labels to each face through user interaction, using tools like brushes, strokes, and lassos. To minimize interactions, we developed interactive 3D selection and template-matching algorithms. We first over-segment the mesh into planar segments via region growing, enabling quick selection of large areas, while protrusions are selected semi-automatically based on geometric features. Leveraging the repetitive nature of urban structures, we use structural features for template matching to facilitate rapid annotation. + +1) Interactive 3D selection. We propose an interactive protrusion extraction method to address the challenges in over-segmented urban textured meshes, which often struggle with non-planar areas, sharp features, and small-scale structures due to under- or over-segmentation [\[20\]](#page-8-5). Our method aims to efficiently identify protrusions not part of the support plane, similar to foreground-background separation in image segmentation [\[50\]](#page-10-2). It involves two main steps that work seamlessly to enhance annotation efficiency. + +First, during interactive selection, users employ a lasso or stroke to generate candidate faces for labeling. The algorithm distinguishes these inputs based on the ratio of contour endpoints' distance to the bounding box diagonal and ensures consistent candidate face extraction. For lasso input, all planar segments within the lasso are selected. For stroke input, the selection expands to neighboring faces along the stroke's path, selecting their corresponding planar segments as candidate faces (see Fig. [3\)](#page-2-0). + +Second, we formulate protrusion selection as a binary labeling problem l f = {support plane, protrusion}. The can- + +![](SUM-Parts_2503.15300_images/_page_2_Figure_0.jpeg) + +(a) Input (b) Lasso/Stroke (c) Candidates (d) Protrusions + +Figure 3. Interactive 3D selection. The user performs a lasso (green) or stroke selection (yellow) (b), which generates candidate faces (red) (c). Binary labeling is then applied to these candidate faces to extract protrusions (red) (d). + +> **[그림 해설]** 3D 대화형 선택 및 돌출물(Protrusion) 자동 분할 파이프라인. +> - **1행 (지붕 위 굴뚝 돌출물)**: (a) 원본 $\to$ (b) 초록색 라쏘 선택 $\to$ (c) 후보 평면 페이스(빨간색) 생성 $\to$ (d) 이진 그래프 컷을 통한 돌출 굴뚝 부품(빨간색) 분리. +> - **2행 (공원 수목)**: (a) 원본 $\to$ (b) 노란색 스트로크 선 긋기 $\to$ (c) 후보 페이스 $\to$ (d) 바닥 지면과 얽힌 복잡한 나무 군집 정밀 추출. + +didate segments $\{f_i\}$ are ordered by area, with the largest serving as the support plane. We construct a dual graph $\mathcal{G}^f = \{\nu^f, \xi^f\}$ for all candidate faces $f = \{f_i\}$ each represented by a node in the graph and connected to adjacent faces by graph edges. + +The data term $D^f$ evaluates the likelihood that face $f_i$ belongs to a protrusion: + +$$D^f(l_i^f) = \eta \times \begin{cases} p_i & \text{if } l_i^f \text{ is support plane,} \\ 1 - p_i & \text{if } l_i^f \text{ is protrusion,} \end{cases}$$ + +where $\eta$ modulates sensitivity to various geometric characteristics, and $p_i=d_i+\omega_i\theta_i$ is the protrusion score. Here, $d_i$ is the maximum distance from face $f_i$ to the support plane $P_k^f$ ; $\theta_i$ is the minimum angle that quantifies the orientation deviation between the normals of face $f_i$ and the support plane $P_k^f$ , and $\omega_i$ is defined as: + +$$\omega_i = \begin{cases} 1 & \text{if } d_i > 1, \\ 1 - d_i & \text{otherwise.} \end{cases}$$ + +The smoothness term $V^f$ measures geometric similarity between adjacent faces: + +$$V^{f}(l_{i}^{f}, l_{j}^{f}) = R_{i,j} \cdot 1_{\{l_{i}^{f} \neq l_{j}^{f}\}},$$ + +where $z_{\rm max}$ and $z_{\rm min}$ correspond to the range of z-values of all vertices in $\{f_i\}$ , and $R_{i,j}=1-\min\left(1,\frac{2|r_i-r_j|}{z_{\rm max}-z_{\rm min}}\right)$ , and $r_i,r_j$ are shrinking ball radii computed via the 3D medial axis transform [56]. This accounts for local geometric consistency. By combining the above terms, we define the objective function as: + +$$E^{f}(l^{f}) = \sum_{i} D^{f}(l^{f}_{i}) + \lambda^{f} \sum_{\{i,j\}} V^{f}(l^{f}_{i}, l^{f}_{j}),$$ + +which we minimize using a graph-cut algorithm [7]. The parameter $\lambda^f$ adjusts the weight of the smoothness term, controlling the influence of geometric similarity. + +![](SUM-Parts_2503.15300_images/_page_2_Figure_13.jpeg) + +Figure 4. 3D template matching. When the user selects a planar segment by clicking on it (a), the matched segments are automatically identified (b). A similar matching process also applies to protrusions via a user-drawn stroke, as shown in (c) and (d). + +> **[그림 해설]** 3D 구조 인식 기반 템플릿 일괄 매칭(Template matching) 기능. +> - **(a) $\to$ (b) 평면 세그먼트 매칭**: 사용자가 건물 외벽 하나를 클릭하면(a), 동일 블록 내 유사한 면적·방위·형상을 가진 모든 외벽 평면(빨간색)이 일괄 매칭(b). +> - **(c) $\to$ (d) 돌출물 매칭**: 도로변에 나열된 차량/돌출물 중 하나를 스트로크로 지정하면(c), 같은 열의 모든 유사 차량 돌출물(빨간색)이 자동 일괄 선택(d). + +2) 3D template matching. To leverage repetitive structures in urban scenes and reduce annotation efforts, we employ 3D template matching using structural awareness features from user-selected faces, matching them with similar structures in the scene. This strategy unifies both planar segment and protrusion matching. + +For planar segment matching (see Fig. 4a to Fig. 4b), we treat the user-selected segment $P^{(t)}$ as a template and compare it with candidate segments $\{P_k^{(c)}\}$ based on feature similarity. We assess characteristics such as geometric homogeneity (comparing surface areas), spatial distribution (weighted average heights), orientation (vertical orientations), shape sphericity (based on eigenvalues), and optionally photometric coherence (similarity in color), which constitute a feature vector $\mathbf{F}^{(\text{seg})}$ . A match is determined when the Euclidean norm $\|\mathbf{F}^{(\text{seg})}\| < \epsilon^{(\text{seg})}$ where $\epsilon^{(\text{seg})}$ is user-defined depending on input quality. + +In protrusion matching (see Fig. 4c to Fig. 4d), we use the user-extracted protrusions as templates to find similar structures. We first decompose the template protrusion into planar segments and match them with segments in the scene. The matched segments serve as seeds, which are expanded to neighboring segments to generate candidate regions. We apply spatial and segment scale constraints to limit the expansion: + +$$\left\| O_k^{(e)} - O_j^{(a)} \right\| < \sqrt{s} \cdot \max_i \left\| O^{(t)} - O_i^{(t)} \right\|,$$ + +where $O_k^{(e)}$ is the center of the seed segment $P_k^{(e)}$ , $O_j^{(a)}$ is the center of face $f_j^{(a)}$ in the neighboring segment $P_k^{(a)}$ , $O^{(t)}$ is the center of the template faces $f^{(t)}$ , $O_i^{(t)}$ is the center of individual template face $f_i^{(t)}$ , and s is the structural + +scale parameter controlling the expansion based on the template size. The segment scale constraint ensures that neighboring segments are comparable in size to the templates: + +$$\frac{A_k^{(a)}}{A_k^{(e)}} < s \cdot \frac{\max_j A_j^{(t)}}{\min_j A_j^{(t)}},$$ + +where $A_k^{(a)}$ and $A_k^{(e)}$ are the areas of the neighboring segment $P_k^{(a)}$ and seed segment $P_k^{(e)}$ , respectively, and $A_j^{(t)}$ are the areas of the template's planar segments. + +We then extract candidate protrusions based on structural features such as spatial compactness (comparing the volume occupied by the protrusion relative to its bounding box), surface complexity (assessed by the number of planar segments composing the protrusion), and eigenvalue-based characteristics like linearity, planarity, and sphericity (derived from the covariance of vertex positions). These features form a vector $\mathbf{F}^{(\mathrm{str})}$ , and a match is accepted when the Euclidean norm $\|\mathbf{F}^{(\mathrm{str})}\| < \epsilon^{(str)}$ , where $\epsilon^{(str)}$ is determined by user interaction and data quality. + +#### 3.1.2. Texture-based annotation + +Mesh textures capture fine details more effectively and avoid the redundancies and discontinuities often found in image-based annotations. However, direct texture annotation is challenging due to discontinuities and computational demands. To address this, we propose a mesh texture annotation strategy based on planar segments, allowing flexible splitting and merging of segments. Our efficient interactive annotation leverages local region extraction and 2D template matching, operating under the assumption that semantic components consist of superpixels with similar geometric and color features. + +1) Interactive 2D selection. We aim to capture the region of interest through user clicks. Unlike traditional methods, our approach requires only positive samples. As shown in Fig. 5, our method consists of two main steps: local expansion and fine segmentation. + +In the first local expansion step, we expand the seed superpixels selected by user clicks to encompass the entire area of interest. We first apply Simple Linear Iterative Clustering (SLIC) [1] to generate homogeneous superpixels from the textured planar segment. We then construct a local adjacency graph $\mathcal{G}^s = \{\nu^s, \xi^s\}$ , where each superpixel is a node connected to its adjacent superpixels. We formulate the expansion as a binary labeling problem $l^s = \{\text{similar}, \text{non-similar}\}$ , aiming to label adjacent superpixels based on their similarity to the initial seed superpixel $S_0$ . The data term $D^s$ measures this similarity using the average Wasserstein distance of their Gaussian mixture models (GMMs) over the RGB channels: + +$$D^s(l^s_j) = \alpha \times \begin{cases} 1 - w_j & \text{if } l^s_j \text{ is non-similar,} \\ w_j & \text{if } l^s_j \text{ is similar,} \end{cases}$$ + +![](SUM-Parts_2503.15300_images/_page_3_Figure_9.jpeg) + +Figure 5. Interactive 2D selection. The user selects a texture segment (green) (a). Superpixels are generated (blue), and the user clicks on the region of interest (green star) (b). This triggers local expansion, yielding a coarse segmentation (red) (c), followed by fine segmentation for the final selection (red) (d). + +> **[그림 해설]** 대화형 2D 텍스처 주석 파이프라인 (공원 잔디밭 및 산책로 예시). +> - **(a)** 텍스처 평면 세그먼트(초록색) 선택. +> - **(b)** 슈퍼픽셀 격자(파란색) 위 관심 영역 클릭(초록색 별표). +> - **(c)** 색상 및 텍스처 유사도 기반 국소 영역 확장 (Coarse 세그멘테이션, 빨간색). +> - **(d)** 그래프 컷 기반 정밀 세그멘테이션으로 산책로 경계와 완벽히 분리된 잔디 영역(Fine 세그멘테이션, 빨간색) 완성. + +where $w_j = \frac{1}{K} \sum_{k=1}^K W(G_k(S_j), G_k(S_0))$ , K=3 is the number of channels, and W denotes the Wasserstein distance between the GMMs of the superpixels in each channel. The smoothness term $V^s$ captures color differences between neighboring superpixels: + +$$V^{s}(l_{j}^{s}, l_{j'}^{s}) = H_{j,j'} \cdot 1_{\{l_{j}^{s} \neq l_{j'}^{s}\}},$$ + +where $H_{j,j'}=|\rho_j-\rho_{j'}|$ , with $\rho_j$ being the color distance (CIEDE2000 [38]) between superpixel $S_j$ and the seed $S_0$ . The energy function combines these terms: + +$$E^{s}(l^{s}) = \sum_{j \in \nu^{s}} D^{s}(l^{s}_{j}) + \lambda^{s} \sum_{\{j,j'\} \in \xi^{s}} V^{s}(l^{s}_{j}, l^{s}_{j'}),$$ + +where $\lambda^s$ adjusts the weight of $V^s$ . We minimize this energy using graph cuts [7] to obtain the expanded region. + +Second, we perform fine segmentation to refine the coarse results from the local expansion, + +which may be imprecise at object boundaries due to superpixel resolution and user input. Building on GrabCut [50], we perform detailed pixel-level segmentation using foreground samples while automatically + +![](SUM-Parts_2503.15300_images/_page_3_Figure_18.jpeg) + +generating background samples from the area beyond the optimal bounding box of the coarse results. + +> **[그림 해설]** GrabCut 기반 전경/배경 초기화 및 바운딩 박스 모델 다이어그램. +> - Bounding box(녹색/빨간 테두리) 외부: 확정 Background (파란색). +> - Bounding box 내부: 확정 Foreground (연두색) 및 미결정 Unknown 영역(회색)을 구분하여 최적의 에너지 최소화 경계 검출 수행. + +2) 2D template matching. To improve annotation efficiency for texture images with repetitive structures like windows and road markings, we adopt fast-matching techniques based on 2D structural awareness. We use user- + +![](SUM-Parts_2503.15300_images/_page_4_Figure_0.jpeg) + +Figure 6. 2D template matching: Extracted regions with green bounding boxes: top shows optimal bounding boxes (rotational invariance), bottom shows vertically aligned bounding boxes (scale invariance) compared to NCC-based methods [\[8\]](#page-8-19). + +> **[그림 해설]** 2D 텍스처 템플릿 매칭 결과 비교 (상단: 도로 차선 마킹, 하단: 건물 파사드 창문). +> - **(a) Selected template**: 사용자가 지정한 단일 템플릿(차선 한 칸, 창문 하나). +> - **(b) Our matching**: 제안하는 구조적 템플릿 매칭 (회전 불변 최적 바운딩 박스로 곡선 도로 차선 마킹 완벽 추적, 크기 불변성으로 다층 창문 일괄 매칭). +> - **(c) NCC matching**: 기존 정규화 상호상관(NCC) 방식 (곡선 회전 시 차선 누락 및 크기 변경 창문 오탐 다수 발생). + +defined templates to find similar structures within the textured planar segment. The template can be a user-selected region or any arbitrary shape drawn by the user. Normalized Cross-Correlation (NCC) [\[8\]](#page-8-19) can be used to identify potential matches. However, NCC performs poorly when faced with rotation and scale changes, especially in 3D urban scenes with varying orientations (see Fig. [6\)](#page-4-1). To overcome this, we propose a region-based template matching approach by extracting structural features from the region R(t) created by local expansion and matching them to similar regions within the textured planar segments. We calculate the Gaussian Mixture Model (GMM) Gk(R(t) ) of the user-selected region. We then use the Wasserstein distance to filter candidate superpixels: + +$$W(G_k(R^{(t)}), G_k(S_i^{(c)})) < \epsilon^{\text{seed}},$$ + +where ϵ seed is a user-adjustable threshold, and Gk(S (c) i ) represents the GMM of candidate superpixel S (c) i . From the qualified superpixels, we extract candidate regions {R (c) i } and compute their similarity to the template region to form a feature vector F (reg) , which includes: shape index (measuring elongation or flatness), shape regularity (how well the region fills its bounding box), contextual features (similarity in internal and external color distributions). We accept matches where the Euclidean norm ∥F (reg)∥ < ϵreg , with ϵ reg depending on user input and image resolution. We also constrain the scale of matching regions using a scaling range based on the number of template pixels: + +$$s^{\mathrm{range}} \in \left\lceil \frac{N(R^{(t)})}{s^{\mathrm{reg}}}, s^{\mathrm{reg}} \cdot N(R^{(t)}) \right\rceil,$$ + +where N(R(t) ) is the number of pixels in the template region, and s reg is the scaling factor. Notably, unlike NCCbased template matching, our method offers rotational and scale invariance, allowing it to match targets of varying orientation and size within textured planar segments. + + + +| Color Name | Explanation | +|-------------------|--------------------------------------------| +| terrain | Ground surfaces. | +| high vegetation | Tall plants such as trees. | +| water | Bodies of water. | +| car | Road vehicles. | +| boat | Watercraft. | +| wall | Vertical barriers. | +| roof surface | Building roofs. | +| facade surface | Exterior building walls. | +| chimney | Roof vents for smoke. | +| dormer | Roof projections. | +| balcony | Outdoor platforms on buildings. | +| roof installation | Fixtures on roofs. | +| window | Glass openings in buildings. | +| door | Entrances to buildings. | +| low vegetation | Short plants like grass. | +| | impervious surface Non-permeable surfaces. | +| road | Vehicular paths. | +| road marking | Markings on roads. | +| cycle lane | Bicycle paths. | +| sidewalk | Pedestrian paths beside roads. | +| unclassified | Elements not classified elsewhere. | + +Table 1. The top 12 are face label definitions, the middle 8 are pixel label definitions, and the last 'unclassified' applies to both. + +### 3.2. Label definition + +In the semantic annotation process, we defined two label types: face and pixel labels. Each triangle mesh face is assigned one of 13 semantic face labels listed in the top part of Tab. [1.](#page-4-0) Based on these, we defined pixel labels for mesh textures to capture part-level details, introducing 8 new categories shown in the bottom part of Tab. [1.](#page-4-0) We differentiate between two types of labeled scenes: meshes with only face labels, featuring 12 semantic classes (excluding 'unclassified'), and meshes with both face and pixel labels, comprising 19 semantic classes (excluding 'unclassified', and 'terrain' as these are broken down into more specific pixel labels). Fig. [7](#page-5-0) show the details of class distribution. + +### 4. Benchmarks + +### 4.1. Evaluation of semantic segmentation + +Research shows that semantic segmentation can be performed on point clouds sampled from mesh surfaces [\[19,](#page-8-4) [55\]](#page-10-8), mapping results back to meshes via nearest neighbor or voting methods. Traditional sampling methods (e.g., facecentered, random, Poisson-disk [\[14\]](#page-8-20)) often miss fine details or exhibit density sensitivity issues in complex environments (see Fig. [8\)](#page-5-1). We apply SLIC [\[1\]](#page-8-18) over-segmentation on texture images to accurately capture boundaries, then use superpixel centers to generate a texture-based point cloud, ensuring precise semantics, fewer samples, reduced computational load, and efficient pixel-to-point label transfer. + +![](SUM-Parts_2503.15300_images/_page_5_Figure_0.jpeg) + +Figure 7. Statistical distribution of semantic classes across the entire dataset for face (top) and pixel-level labels (bottom). + +> **[그림 해설]** SUM-Parts 데이터셋의 클래스별 면적 및 픽셀 분포 로그 스케일 막대 그래프. +> - **상단 (Face-level labels, 12개 클래스 표면적 $\text{m}^2$)**: 3(Facade/Building, $10^6\,\text{m}^2$ 이상)이 가장 크고, 1(Ground), 7(Roof), 2(Vegetation), 4(Water) 순. +> - **하단 (Pixel-level labels, 19개 클래스 픽셀 수)**: 2(Facade), 6(Roof), 1(Road), 12(Grass), 15(Water) 등이 $10^7 \sim 10^8$ 픽셀 수준으로 풍부한 데이터량 보유. + +![](SUM-Parts_2503.15300_images/_page_5_Picture_2.jpeg) + +Figure 8. Comparison of mesh sampling methods. From left to right: input meshes, face-centered sampling, random sampling, Poisson-disk sampling [\[14\]](#page-8-20), and our superpixel texture sampling. + +> **[그림 해설]** 3D 메시 $\to$ 포인트 클라우드 샘플링 기법 5종 비교 (횡단보도 도로 및 건물 창문 파사드). +> - 1열: 원본 텍스처 메시 (Input meshes). +> - 2열: Face-centered sampling (페이스 중심점 샘플링 - 텍스처 파트 표현 불가). +> - 3열: Random sampling (무작위 샘플링). +> - 4열: Poisson-disk sampling (포아송 디스크 균일 샘플링). +> - 5열: **Our superpixel texture sampling** (제안하는 슈퍼픽셀 텍스처 샘플링 - 창문 프레임 및 차선 텍스처 경계를 가장 선명하게 보존). + +We evaluate state-of-the-art 3D semantic segmentation methods on our datasets, including mesh-based methods like RF-MRF [\[52\]](#page-10-0), SUM-RF [\[19\]](#page-8-4), and PSSNet [\[20\]](#page-8-5), and point cloud based approaches like PointNet [\[12\]](#page-8-21), PointNet++ [\[47\]](#page-9-14), superpoint graphs (SPG) [\[30\]](#page-9-15), SparseConvUnet [\[22\]](#page-8-22), RandLA-Net [\[25\]](#page-8-23), KPConv [\[58\]](#page-10-9), Point-Next [\[48\]](#page-9-16), Point Transformer V3 (PointTransV3) [\[60\]](#page-10-10), and PointVector [\[17\]](#page-8-24). It is worth noting that there are currently no semantic segmentation methods specifically designed for meshes with textured-based pixel labels. These point cloud segmentation methods were evaluated on both the face and pixel labeling tracks. We divide our data into three random splits: 24 tiles for training, 8 for validation, and 8 for testing. To address the class imbalance, we applied class weights to each method, and to reduce randomness in network predictions, results are averaged over five training runs. + +We used established semantic segmentation evaluation metrics, conducting a detailed analysis that included Intersection over Union (IoU) per class, overall accuracy (OA), mean accuracy (mAcc), and mean IoU (mIoU). For the face + + + +| | | | Face Labeling | Pixel Labeling | | | | | | +|---------------------------------------------------|-----|-----|---------------|----------------|-----|------------------------------------|-----|--|--| +| | Fc. | Rd. | Po. | Sp. | Rd. | Po. | Sp. | | | +| PointNet | 5.1 | 8.5 | 5.1 | 15.1 | 2.5 | 5.7 | 2.6 | | | +| PointNet++ | | | | | | 27.1 17.3 18.4 33.1 20.6 19.5 24.7 | | | | +| SPG | | | | | | 29.9 29.7 31.5 31.7 16.0 20.0 19.2 | | | | +| SparseUNet | | | | | | 60.5 47.6 38.6 49.9 34.5 13.3 23.3 | | | | +| Randla-net | | | | | | 57.4 49.8 49.1 54.4 36.9 39.9 42.1 | | | | +| KPConv | | | | | | 57.5 56.4 46.5 52.9 38.9 26.1 42.6 | | | | +| PointNext | | | | | | 65.3 51.3 50.4 47.7 42.9 44.7 43.0 | | | | +| PointTransV3 | | | | | | 59.1 49.5 51.7 54.0 38.0 36.1 37.8 | | | | +| PointVector | | | | | | 70.0 56.1 52.8 57.1 44.7 45.1 47.9 | | | | +| Average (mIoU) 48.0 40.7 38.2 44.0 30.6 27.8 31.5 | | | | | | | | | | + +Table 2. Evaluation against different sampling strategies on semantic segmentation using mIoU. 'Fc.' represents face-centered sampling, 'Rd.' random sampling, 'Po.' Poisson-disk sampling [\[14\]](#page-8-20), and 'Sp.' superpixel texture sampling. The highest and average mIoUs for each method are highlighted in bold. + +labeling track, we determine the final labels of triangles using a voting method based on point cloud prediction results, with the area of each triangle serving as a weight for semantic evaluation metrics. For the pixel labeling track, we assign final labels to each pixel in the texture image using the nearest neighbor method based on point cloud prediction results, evaluating pixel-level semantic metrics. + +*1) Face labeling track.* The face labeling track includes 12 labels, excluding 'unclassified'. We evaluated the impact of four mesh point cloud sampling strategies on semantic segmentation. To control point cloud density, the number of points for random and Poisson-disk samples matched the superpixel texture sample size, while face-centered samples always matched the number of mesh faces. + +Tab. [2](#page-5-2) shows that using face-centered point clouds leads to optimal performance for most methods. This is because they adapt well to the geometric characteristics of triangulated meshes, where triangle density is lower in flat areas and higher in non-flat areas. Such a distribution enables deep learning networks to learn rich geometric features. However, this does not apply to uniform triangular meshes, indicating that the impact of point cloud sampling density on semantic segmentation is far less significant than the impact of point cloud distribution. The average mIoU in Tab. [2](#page-5-2) also indicates that our proposed superpixel texture sampling method outperforms other mesh sampling methods (except for face-centered point clouds). Tab. [3](#page-6-0) shows that PointVector [\[17\]](#page-8-24) surpasses all competing methods with the highest mAcc of 80.7% and mIoU of 70.0%. Fig. [9](#page-6-1) presents a qualitative analysis of the top three methods. + +*2) Pixel labeling track.* The pixel labeling track consists of 19 semantic classes, as described in Sec. [3.2,](#page-4-2) including all semantic labels except for 'terrain' and 'unclassified'. We + + + +| | | | | Face Labeling Track Pixel Labeling Track | | | | | | | | +|------------------|-----|----------|------|------------------------------------------|----------|------|--|--|--|--|--| +| | | Sa. mAcc | mIoU | | Sa. mAcc | mIoU | | | | | | +| RF_MRF | - | 45.3 | 39.5 | - | - | - | | | | | | +| SUM_RF | - | 53.6 | 46.0 | - | - | - | | | | | | +| PSSNet | - | 56.4 | 47.0 | - | - | - | | | | | | +| PointNet | Sp. | 22.0 | 15.1 | Sp. | 9.8 | 2.6 | | | | | | +| PointNet++ | Sp. | 46.9 | 33.1 | Sp. | 35.2 | 24.7 | | | | | | +| SPG | Sp. | 55.0 | 31.7 | Sp. | 34.5 | 19.2 | | | | | | +| SparseUNet | Fc. | 71.7 | 60.5 | Rd. | 45.1 | 34.5 | | | | | | +| Randla-net | Fc. | 76.3 | 57.4 | Sp. | 57.7 | 42.1 | | | | | | +| KPConv | Fc. | 64.7 | 57.5 | Sp. | 58.3 | 42.6 | | | | | | +| PointNext | Fc. | 77.2 | 65.3 | Po. | 57.6 | 44.7 | | | | | | +| PointTransV3 Fc. | | 70.2 | 59.1 | Rd. | 54.1 | 38.0 | | | | | | +| PointVector | Fc. | 80.7 | 70.0 | Sp. | 63.8 | 47.9 | | | | | | + +Table 3. Evaluation of semantic segmentation performance for face labeling and pixel labeling tracks. 'Sa.' represents sampling methods, including 'Fc.' for face-centered sampling, 'Rd.' for random sampling, 'Po.' for Poisson-disk sampling [\[14\]](#page-8-20), and 'Sp.' for superpixel texture sampling. '-' indicates not applicable. + +![](SUM-Parts_2503.15300_images/_page_6_Figure_2.jpeg) + +Figure 9. Qualitative analysis of face and pixel labeling. The top two rows show the three best methods for face labeling: SparseUNetF c. [\[22\]](#page-8-22), PointNextF c. [\[48\]](#page-9-16), and PointVectorF c. [\[17\]](#page-8-24). The bottom two rows show the three best methods for pixel labeling: KPConvSp. [\[58\]](#page-10-9), PointNextP o. [\[48\]](#page-9-16), and PointVectorSp. [\[17\]](#page-8-24). 'Fc.' represents face-centered sampling, 'Po.' Poisson-disk sampling [\[14\]](#page-8-20), and 'Sp.' superpixel texture sampling. + +> **[그림 해설]** Face 및 Pixel 트랙 상위 3개 모델의 정성적 파트 분할 결과 비교. +> - **상단 (Face labeling 트랙)**: Input vs SparseUNet$^{Fc}$ vs PointNext$^{Fc}$ vs PointVector$^{Fc}$ vs Truth. +> - 지붕 위 미세 굴뚝/환기창(빨강/보라) 및 벽체(노랑) 분할에서 PointVector$^{Fc}$가 Ground Truth에 가장 근접한 복원력 시현. +> - **하단 (Pixel labeling 트랙)**: Input vs KPConv$^{Sp}$ vs PointNext$^{Po}$ vs PointVector$^{Sp}$ vs Truth. +> - 파사드 창문(파랑), 문(갈색), 횡단보도(보라), 차량(자주) 분할에서 PointVector$^{Sp}$가 창문 격자 배열과 차선 선형을 최고 정밀도로 복원. + +evaluated the impact of three sampling methods—random, Poisson-disk, and superpixel texture sampling—on semantic segmentation performance. Because face-centered point clouds cannot represent semantic components with pixel labels, they were excluded from testing. + +The average mIoU in Tab. [2](#page-5-2) shows most methods achieve optimal performance with our proposed sampling, which + + + +| | M(%) B(%) | | O | T(s) | S(%) | +|---------------|-----------|------|-----|-------------|------| +| Manual | - | - | | 6754 4183.2 | - | +| Segment-based | 91.6 | 74.7 | | 6119 3565.8 | - | +| Ours | 92.2 | 75.0 | | 2992 2992.1 | 83.0 | +| Manual | - | - | 653 | 777.1 | - | +| GrabCut | 88.1 | 47.1 | 711 | 780.9 | 30.2 | +| SAM | 84.4 | 29.8 | 716 | 640.6 | 71.7 | +| SimpleClick | 81.4 | 30.9 | 252 | 861.3 | - | +| Ours | 87.9 | 49.3 | 582 | 663.3 | 40.3 | + +Table 4. Comprehensive performance evaluation of interactive face (top) and texture image annotation (bottom) methods across different test scenarios. The highest values are given in bold. + +![](SUM-Parts_2503.15300_images/_page_6_Figure_8.jpeg) + +Figure 10. Errors (red) in interactive face annotation. + +> **[그림 해설]** 수목/식생 영역 대화형 페이스 주석 방식 비교 (Input vs Segment-based vs Ours vs Manual). +> - **Segment-based**: 나무 아래 지면까지 초록색으로 뭉개지는 과다 분할 오류 다수 발생. +> - **Ours (제안 기법)**: 수목 기하 경계를 정밀 분리하여 오류(빨간색 점)를 최소화하고 수동 주석(Manual) 수준의 정밀도 달성. + +precisely captures the boundaries of part-level objects. Tab. [3](#page-6-0) shows that PointVector [\[17\]](#page-8-24) continues to outperform all other methods, achieving a mIoU of 47.9%, significantly higher than the others. We also conducted a qualitative analysis of the top three methods, shown in Fig. [9.](#page-6-1) + +### 4.2. Evaluation of interactive annotation + +We evaluated existing interactive annotation methods for meshes and textured images in real-world scenarios and proposed new evaluation criteria based on user studies. Traditional metrics like the number of clicks to achieve a certain IoU or Average Precision (AP) do not fully capture annotation efficiency due to limitations in evaluation comprehensiveness, interaction complexity, and efficiency measurement. To address these issues, we developed new evaluation metrics including average mean IoU (M), average boundary mean IoU (B), average number of mean user interactions (O), average user annotation time (T(s)), and average percentage of using smart interaction tools (S(%)). + +In our experiments, five users were invited to annotate four representative scenes for face annotation and six for texture annotation using various methods. For interactive face annotation, we compared our method with manual [\[52\]](#page-10-0) and segment-based interactive annotation [\[19\]](#page-8-4). Our method outperformed both in all metrics, reducing the number of interactions and annotation time significantly—about 1.73 times faster than manual annotation and 1.32 times faster than segment-based methods (see Tab. [4](#page-6-2) top and Fig. [10\)](#page-6-3). The smart interaction ratio exceeded 80% in most scenarios, indicating minimal manual intervention was needed. + +![](SUM-Parts_2503.15300_images/_page_7_Figure_0.jpeg) + +Figure 11. Boundary errors (red) in interactive texture annotation. + +> **[그림 해설]** 대화형 텍스처 주석 기법 간 경계 오류(빨간색 표시) 정성 비교 (Input vs GrabCut vs Simclick vs SAM vs Ours vs Manual). +> - **상단 (건물 파사드 창문/외벽)**: GrabCut, Simclick, SAM 대비 제안 기법(Ours)이 창문 경계 오류(빨간색)를 거의 남기지 않고 깔끔한 격자형 창문 분할 완수. +> - **하단 (교차로 도로/횡단보도)**: 복잡한 차선 및 보도 경계에서 제안 기법(Ours)이 오차를 최소화하며 정밀 분할 달성. + +For interactive texture annotation, we compared with manual annotation, GrabCut [50], SAM [27], and SimpleClick [37]. Our method surpassed deep learning methods in annotation quality and was comparable to the fastest annotation times of SAM (see Tab. 4 bottom and Fig. 11). It excelled in boundary accuracy due to our fine segmentation and template matching. Notably, for objects with regular shapes or repetitive structures, interactive clicking can be inefficient, requiring users to manually trace shape boundaries for higher accuracy. Our method addresses this by enabling efficient annotation of similar structures using reusable templates, reducing repetitive interactions. + +#### 4.3. Sensitivity analysis + +Conducting ablation studies on energy function-based methods is challenging due to the high interdependence among terms and the complexity of the optimization, complicating the evaluation of individual parts. We conducted a qualitative analysis to assess the impact of parameter adjustments and data quality on our method's outcomes. + +In face-based analysis, increasing the balance parameter $\lambda^f$ improves object boundary clarity while adjusting segment matching thresholds ( $\epsilon^{(\text{seg})}$ and $\epsilon^{(\text{str})}$ ) enhances coverage of repetitive structures (see Fig. 12 top). Our method demonstrates significant noise tolerance, accurately extracting protrusions despite added Gaussian noise, which is crucial for effective template matching (see Fig. 13 top). + +In texture-based analysis, a larger balance parameter $\lambda^s$ smooths local regions, while higher thresholds ( $\epsilon^{\rm seed}$ and $\epsilon^{\rm reg}$ ) increase region matches but may reduce seed quality (see Fig. 12 bottom). Default parameters perform well with minimal adjustments. Our method, using few interpretable parameters, often outperforms deep learning approaches that rely on inconsistent user clicks, effectively identifying target regions even under high noise and varying user interactions (see Fig. 13 bottom). + +#### 5. Summary + +We introduced SUM Parts, a part-level semantic segmentation dataset for urban meshes covering 2.5 km2 with 21 classes. A novel annotation tool facilitates the semantic labeling of mesh faces and texture pixels with efficient 2D/3D selection strategies, streamlining the annotation of 3D ur- + +![](SUM-Parts_2503.15300_images/_page_7_Figure_9.jpeg) + +Figure 12. Sensitivity analysis of parameters in face (top) and texture annotation (bottom) with the same user interactions. + +> **[그림 해설]** 동일한 사용자 인터랙션 하에서 주석 파라미터 민감도 분석 시각화. +> - **상단 (Face 주석)**: 돌출물 임계값 $\lambda^f$ (0.3 vs 0.6), 세그먼트 매칭 오차 $\epsilon^{(seg)}$ (30 vs 80), 구조 유사도 오차 $\epsilon^{(str)}$ (20 vs 80)에 따른 선택 영역 변화. +> - **하단 (Texture 주석)**: 시드 확장 비율 $\lambda^s$ (0.2 vs 0.4), 시드 매칭 오차 $\epsilon^{(seed)}$ (15 vs 30), 정규성 오차 $\epsilon^{(reg)}$ (30 vs 60)에 따른 주석 범위 변화. + +![](SUM-Parts_2503.15300_images/_page_7_Figure_11.jpeg) + +Figure 13. Sensitivity analysis of data quality in protrusion extraction (top) and local region extraction (bottom). $\sigma^m$ and $\sigma^t$ represent the standard deviations of Gaussian noise of different inputs. + +> **[그림 해설]** 데이터 품질 노이즈에 대한 추출 알고리즘의 강건성(Robustness) 분석. +> - **상단 (메시 기하 노이즈 $\sigma^m = 0.0, 0.2, 0.4$)**: 지붕 돌출물 표면에 심한 기하 노이즈가 추가되어도 돌출물 일괄 매칭(빨간색)이 안정적으로 유지됨. +> - **하단 (텍스처 이미지 노이즈 $\sigma^t = 0, 1, 3$)**: 텍스처에 가우시안 블러/노이즈가 심화되어도 공원 산책로 영역(빨간색) 분할이 견고하게 추출됨. + +![](SUM-Parts_2503.15300_images/_page_7_Figure_13.jpeg) + +Figure 14. Applications of our annotation tool for indoor meshes (top left), building models (top right), and images (bottom). + +> **[그림 해설]** 제안 주석 도구의 다양한 도메인 확장 적용 사례. +> - **상단 좌측 (실내 3D 메시)**: 사무실 의자/책상 부품 분할. +> - **상단 우측 (콤팩트 3D 빌딩 모델)**: 지붕 위 굴뚝 구조물 일괄 분할 매칭. +> - **하단 좌측 (도심 고층 빌딩 2D 사진)**: 대규모 외벽 창문 격자 일괄 매칭 주석. +> - **하단 우측 (실내 방 2D 사진)**: 붉은색 커튼 영역의 세밀한 텍스처 세그멘테이션. + +ban scenes. Our evaluations show that our approach outperforms existing interactive annotation methods. + +**Applications.** Our interactive annotation method can also handle complex indoor scenes, compact building models, and images (see Fig. 14). The SUM Parts dataset advances lightweight semantic city modeling by providing part-level semantics, enabling automated reconstruction of CityGML LoD3 city models [23, 45]. + +Limitations. Our annotation method relies on geometric precision and structural clarity and may be less effective for triangle meshes with topological errors, low resolution, or poor planar over-segmentation. It is also not applicable to natural scenes (e.g., mountains) or complex structures (e.g., palaces) that do not conform to planar and protrusion-based assumptions. In texture annotation, performance can degrade with complex textures, cluttered backgrounds, smaller superpixels (increasing processing time) or larger ones (risking under-segmentation), shadows, or regions with minimal color differentiation. + +## References + +- [1] Radhakrishna Achanta, Appu Shaji, Kevin Smith, Aurelien Lucchi, Pascal Fua, and Sabine Süsstrunk. SLIC superpixels compared to state-of-the-art superpixel methods. *IEEE Transactions on Pattern Analysis and Machine Intelligence*, 34(11):2274–2282, 2012. [4,](#page-3-1) [5](#page-4-3) +- [2] R. Adams and L. Bischof. Seeded region growing. *IEEE Transactions on Pattern Analysis and Machine Intelligence*, 16(6):641–647, 1994. [2](#page-1-0) +- [3] AHN. Actueel Hoogtebestand Nederland (AHN). [https:](https://www.ahn.nl/) [//www.ahn.nl/](https://www.ahn.nl/), 2019. Accessed: 2021-04-16. [2](#page-1-0) +- [4] E. Airaksinen, M. Bergström, H. Heinonen, K. Kaisla, K. Lahti, and J. Suomisto. The Kalasatama digital twins project—The final report of the KIRA-digi pilot project. Technical report, City of Helsinki, 2019. [2](#page-1-0) +- [5] I. Armeni, A. Sax, A. R. Zamir, and S. Savarese. Joint 2D-3D-semantic data for indoor scene understanding. *ArXiv eprints*, 2017. [1](#page-0-1) +- [6] Y.Y. Boykov and M.-P. Jolly. Interactive graph cuts for optimal boundary & region segmentation of objects in N-D images. In *Proceedings Eighth IEEE International Conference on Computer Vision. ICCV 2001*, pages 105–112 vol.1, 2001. [2](#page-1-0) +- [7] Y. Boykov, O. Veksler, and R. Zabih. Fast approximate energy minimization via graph cuts. *IEEE Transactions on Pattern Analysis and Machine Intelligence*, 23(11):1222–1239, 2001. [3,](#page-2-2) [4](#page-3-1) +- [8] Kai Briechle and Uwe D Hanebeck. Template matching using fast normalized cross correlation. In *Optical pattern recognition XII*, pages 95–102. SPIE, 2001. [5](#page-4-3) +- [9] Gabriel J. Brostow, Julien Fauqueur, and Roberto Cipolla. Semantic object classes in video: A high-definition ground truth database. *Pattern Recognition Letters*, 30(2):88–97, 2009. Video-based Object and Event Analysis. [2](#page-1-0) +- [10] Gülcan Can, Dario Mantegazza, Gabriele Abbate, Sébastien Chappuis, and Alessandro Giusti. Semantic segmentation on swiss3Dcities: A benchmark study on aerial photogrammetric 3D pointcloud dataset. *Pattern Recognition Letters*, 150: 108–114, 2021. [2](#page-1-0) +- [11] Angel Chang, Angela Dai, Thomas Funkhouser, Maciej Halber, Matthias Niessner, Manolis Savva, Shuran Song, Andy Zeng, and Yinda Zhang. Matterport3D: Learning from RGB-D data in indoor environments. *International Conference on 3D Vision (3DV)*, 2017. [1](#page-0-1) +- [12] R. Qi Charles, Hao Su, Mo Kaichun, and Leonidas J. Guibas. PointNet: Deep learning on point sets for 3D classification and segmentation. In *2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)*, pages 77–85, 2017. [6,](#page-5-3) +- [13] Bowen Cheng, Ross Girshick, Piotr Dollár, Alexander C. Berg, and Alexander Kirillov. Boundary IoU: Improving object-centric image segmentation evaluation. In *2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)*, pages 15329–15337, 2021. +- [14] Robert L. Cook. Stochastic sampling in computer graphics. *ACM Trans. Graph.*, 5(1):51–72, 1986. [5,](#page-4-3) [6,](#page-5-3) [7,](#page-6-4) + +- [15] M. Cordts, M. Omran, S. Ramos, T. Rehfeld, M. Enzweiler, R. Benenson, U. Franke, S. Roth, and B. Schiele. The cityscapes dataset for semantic urban scene understanding. In *2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)*, pages 3213–3223, Los Alamitos, CA, USA, 2016. IEEE Computer Society. [1](#page-0-1) +- [16] Angela Dai, Angel X. Chang, Manolis Savva, Maciej Halber, Thomas Funkhouser, and Matthias Nießner. ScanNet: Richly-annotated 3D reconstructions of indoor scenes. In *2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)*, pages 2432–2443, 2017. [1](#page-0-1) +- [17] Xin Deng, WenYu Zhang, Qing Ding, and XinMing Zhang. Pointvector: A vector representation in point cloud analysis. In *2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)*, pages 9455–9465, 2023. [6,](#page-5-3) [7,](#page-6-4) +- [18] Ben Fei, Weidong Yang, Wen-Ming Chen, Zhijun Li, Yikang Li, Tao Ma, Xing Hu, and Lipeng Ma. Comprehensive review of deep learning-based 3D point cloud completion processing and analysis. *IEEE Transactions on Intelligent Transportation Systems*, 23(12):22862–22883, 2022. [2](#page-1-0) +- [19] Weixiao Gao, Liangliang Nan, Bas Boom, and Hugo Ledoux. SUM: A benchmark dataset of Semantic Urban Meshes. *ISPRS Journal of Photogrammetry and Remote Sensing*, 179:108–120, 2021. [1,](#page-0-1) [2,](#page-1-0) [5,](#page-4-3) [6,](#page-5-3) [7,](#page-6-4) +- [20] Weixiao Gao, Liangliang Nan, Bas Boom, and Hugo Ledoux. PSSNet: Planarity-sensible semantic segmentation of large-scale urban meshes. *ISPRS Journal of Photogrammetry and Remote Sensing*, 196:32–44, 2023. [1,](#page-0-1) [2,](#page-1-0) [6,](#page-5-3) +- [21] L. Grady. Random walks for image segmentation. *IEEE Transactions on Pattern Analysis and Machine Intelligence*, 28(11):1768–1783, 2006. [2](#page-1-0) +- [22] Benjamin Graham, Martin Engelcke, and Laurens van der Maaten. 3D semantic segmentation with submanifold sparse convolutional networks. In *2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition*, pages 9224– 9232, 2018. [6,](#page-5-3) [7](#page-6-4) +- [23] Gerhard Gröger and Lutz Plümer. CityGML – interoperable semantic 3D city models. *ISPRS Journal of Photogrammetry and Remote Sensing*, 71:12–33, 2012. [8](#page-7-4) +- [24] T. Hackel, N. Savinov, L. Ladicky, J. D. Wegner, K. Schindler, and M. Pollefeys. Semantic3d.net: A new largescale point cloud classification benchmark. *ISPRS Annals of the Photogrammetry, Remote Sensing and Spatial Information Sciences*, IV-1/W1:91–98, 2017. [2](#page-1-0) +- [25] Qingyong Hu, Bo Yang, Linhai Xie, Stefano Rosa, Yulan Guo, Zhihua Wang, Niki Trigoni, and Andrew Markham. RandLA-Net: Efficient semantic segmentation of large-scale point clouds. In *2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)*, pages 11105– 11114, 2020. [6](#page-5-3) +- [26] Muhammad Ibrahim, Naveed Akhtar, Michael Wise, and Ajmal Mian. Annotation tool and urban dataset for 3D point cloud semantic segmentation. *IEEE Access*, 9:35984–35996, 2021. [2](#page-1-0) +- [27] Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C. Berg, Wan-Yen Lo, Piotr Dollár, and + +- Ross Girshick. Segment anything. In *2023 IEEE/CVF International Conference on Computer Vision (ICCV)*, pages 3992–4003, 2023. [2,](#page-1-0) [8,](#page-7-4) +- [28] Michael Kölle, Dominik Laupheimer, Stefan Schmohl, Norbert Haala, Franz Rottensteiner, Jan Dirk Wegner, and Hugo Ledoux. The Hessigheim 3D (H3D) benchmark on semantic segmentation of high-resolution 3D point clouds and textured meshes from UAV LiDAR and Multi-View-Stereo. *IS-PRS Open Journal of Photogrammetry and Remote Sensing*, 1:100001, 2021. [2](#page-1-0) +- [29] Theodora Kontogianni, Ekin Celikkan, Siyu Tang, and Konrad Schindler. Interactive object segmentation in 3D point clouds. In *2023 IEEE International Conference on Robotics and Automation (ICRA)*, pages 2891–2897, 2023. [1,](#page-0-1) [2](#page-1-0) +- [30] Loic Landrieu and Martin Simonovsky. Large-scale point cloud semantic segmentation with superpoint graphs. In *2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition*, pages 4558–4567, 2018. [6](#page-5-3) +- [31] Itai Lang, Fei Xu, Dale Decatur, Sudarshan Babu, and Rana Hanocka. iSeg: Interactive 3D segmentation via interactive attention. In *SIGGRAPH Asia 2024 Conference Papers*, New York, NY, USA, 2024. Association for Computing Machinery. [1](#page-0-1) +- [32] Feng Li, Hao Zhang, Huaizhe Xu, Shilong Liu, Lei Zhang, Lionel M. Ni, and Heung-Yeung Shum. Mask DINO: Towards a unified transformer-based framework for object detection and segmentation. In *2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)*, pages 3041–3050, Los Alamitos, CA, USA, 2023. IEEE Computer Society. +- [33] Feng Li, Hao Zhang, Peize Sun, Xueyan Zou, Shilong Liu, Chunyuan Li, Jianwei Yang, Lei Zhang, and Jianfeng Gao. Segment and recognize anything at any granularity. In *Computer Vision – ECCV 2024*, pages 467–484, Cham, 2025. Springer Nature Switzerland. +- [34] Xinke Li, Chongshou Li, Zekun Tong, Andrew Lim, Junsong Yuan, Yuwei Wu, Jing Tang, and Raymond Huang. Campus3D: A photogrammetry point cloud benchmark for hierarchical understanding of outdoor scene. In *Proceedings of the 28th ACM International Conference on Multimedia*, page 238–246, New York, NY, USA, 2020. Association for Computing Machinery. [2](#page-1-0) +- [35] Yiyi Liao, Jun Xie, and Andreas Geiger. KITTI-360: A novel dataset and benchmarks for urban scene understanding in 2D and 3D. *IEEE Transactions on Pattern Analysis and Machine Intelligence*, 45(3):3292–3310, 2023. [1,](#page-0-1) [2,](#page-1-0) +- [36] K. Liu and J. Boehm. A new framework for interactive segmentation of point clouds. *The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences*, XL-5:357–362, 2014. [1,](#page-0-1) [2](#page-1-0) +- [37] Q. Liu, Z. Xu, G. Bertasius, and M. Niethammer. Simpleclick: Interactive image segmentation with simple vision transformers. In *2023 IEEE/CVF International Conference on Computer Vision (ICCV)*, pages 22233–22243, Los Alamitos, CA, USA, 2023. IEEE Computer Society. [8,](#page-7-4) +- [38] M Ronnier Luo, Guihua Cui, and Bryan Rigg. The development of the CIE 2000 colour-difference formula: + +- CIEDE2000. *Color Research & Application: Endorsed by Inter-Society Color Council, The Colour Group (Great Britain), Canadian Society for Color, Color Science Association of Japan, Dutch Society for the Study of Color, The Swedish Colour Centre Foundation, Colour Society of Australia, Centre Français de la Couleur*, 26(5):340–350, 2001. [4,](#page-3-1) +- [39] K. Maninis, S. Caelles, J. Pont-Tuset, and L. Van Gool. Deep extreme cut: From extreme points to object segmentation. In *2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)*, pages 616–625, Los Alamitos, CA, USA, 2018. IEEE Computer Society. [2](#page-1-0) +- [40] Andelo Martinovi ¯ c, Jan Knopp, Hayko Riemenschneider, ´ and Luc Van Gool. 3D all the way: Semantic segmentation of urban scenes from start to end in 3D. In *2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)*, pages 4456–4465, 2015. [1](#page-0-1) +- [41] Tom McKinnon and Paul Hoff. Comparing RGB-based vegetation indices with NDVI for drone based agricultural sensing. *Agribotix. Com*, 21(17):1–8, 2017. +- [42] Ondrej Miksik, Vibhav Vineet, Morten Lidegaard, Ram Prasaath, Matthias Nießner, Stuart Golodetz, Stephen L. Hicks, Patrick Pérez, Shahram Izadi, and Philip H.S. Torr. The semantic paintbrush: Interactive 3D mapping and recognition in large outdoor spaces. In *Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems*, page 3317–3326, New York, NY, USA, 2015. Association for Computing Machinery. [1](#page-0-1) +- [43] Eric N. Mortensen and William A. Barrett. Interactive segmentation with intelligent scissors. *Graphical Models and Image Processing*, 60(5):349–384, 1998. [2](#page-1-0) +- [44] Jifeng Ning, Lei Zhang, David Zhang, and Chengke Wu. Interactive image segmentation by maximal similarity based region merging. *Pattern Recognition*, 43(2):445–456, 2010. Interactive Imaging and Vision. [2](#page-1-0) +- [45] OGC. OGC City Geography Markup Language (CityGML) Part 1: Conceptual Model Standard. Open Geospatial Consortium inc., 2021. Document 20-010, version 3.0.0, available at [https://docs.ogc.org/is/20-010/20-](https://docs.ogc.org/is/20-010/20-010.html) [010.html](https://docs.ogc.org/is/20-010/20-010.html). [1,](#page-0-1) [8](#page-7-4) +- [46] Ravi Peters. *Geographical point cloud modelling with the 3D medial axis transform*. PhD thesis, Technische Universiteit Delft, 2018. +- [47] Charles R. Qi, Li Yi, Hao Su, and Leonidas J. Guibas. Point-Net++: Deep hierarchical feature learning on point sets in a metric space. In *Proceedings of the 31st International Conference on Neural Information Processing Systems*, page 5105–5114, Red Hook, NY, USA, 2017. Curran Associates Inc. [6](#page-5-3) +- [48] Guocheng Qian, Yuchen Li, Houwen Peng, Jinjie Mai, Hasan Hammoud, Mohamed Elhoseiny, and Bernard Ghanem. PointNext: Revisiting PointNet++ with improved training and scaling strategies. In *Advances in Neural Information Processing Systems*, pages 23192–23204. Curran Associates, Inc., 2022. [6,](#page-5-3) [7](#page-6-4) +- [49] A. Romanoni and M. Matteucci. A data-driven prior on facet orientation for semantic mesh labeling. In *2018 International* + +- *Conference on 3D Vision (3DV)*, pages 662–671, Los Alamitos, CA, USA, 2018. IEEE Computer Society. [1](#page-0-1) +- [50] Carsten Rother, Vladimir Kolmogorov, and Andrew Blake. GrabCut: interactive foreground extraction using iterated graph cuts. *ACM Trans. Graph.*, 23(3):309–314, 2004. [2,](#page-1-0) [4,](#page-3-1) [8,](#page-7-4) +- [51] Mohammad Rouhani, Florent Lafarge, and Pierre Alliez. Semantic segmentation of 3D textured meshes for urban scene analysis. *ISPRS Journal of Photogrammetry and Remote Sensing*, 123:124–139, 2017. +- [52] Mohammad Rouhani, Florent Lafarge, and Pierre Alliez. Semantic segmentation of 3D textured meshes for urban scene analysis. *ISPRS Journal of Photogrammetry and Remote Sensing*, 123:124–139, 2017. [1,](#page-0-1) [6,](#page-5-3) [7,](#page-6-4) +- [53] Xavier Roynard, Jean-Emmanuel Deschaud, and François Goulette. Paris-Lille-3D: A large and high-quality groundtruth urban point cloud dataset for automatic segmentation and classification. *The International Journal of Robotics Research*, 37(6):545–557, 2018. [2](#page-1-0) +- [54] Patric Schmitz, Sebastian Suder, Kersten Schuster, and Leif Kobbelt. Interactive segmentation of textured point clouds. In *Vision, Modeling, and Visualization*. The Eurographics Association, 2022. [1](#page-0-1) +- [55] Pratheba Selvaraju, Mohamed Nabail, Marios Loizou, Maria Maslioukova, Melinos Averkiou, Andreas Andreou, Siddhartha Chaudhuri, and Evangelos Kalogerakis. BuildingNet: Learning to label 3D buildings. In *2021 IEEE/CVF International Conference on Computer Vision (ICCV)*, pages 10377–10387, 2021. [5](#page-4-3) +- [56] E.C. Sherbrooke, N.M. Patrikalakis, and E. Brisson. An algorithm for the medial axis transform of 3d polyhedral solids. *IEEE Transactions on Visualization and Computer Graphics*, 2(1):44–61, 1996. [3,](#page-2-2) +- [57] BENTLEY SYSTEMS. Reality and spatial modeling software. [https://www.bentley.com/software/](https://www.bentley.com/software/reality-and-spatial-modeling/) [reality- and- spatial- modeling/](https://www.bentley.com/software/reality-and-spatial-modeling/), 2016. Accessed: 2025-03-14. [2](#page-1-0) +- [58] Hugues Thomas, Charles R. Qi, Jean-Emmanuel Deschaud, Beatriz Marcotegui, François Goulette, and Leonidas Guibas. KPConv: Flexible and deformable convolution for point clouds. In *2019 IEEE/CVF International Conference on Computer Vision (ICCV)*, pages 6410–6419, 2019. [6,](#page-5-3) [7](#page-6-4) +- [59] Martin Weinmann, Boris Jutzi, and Clément Mallet. Feature relevance assessment for the semantic interpretation of 3D point cloud data. *ISPRS Workshop Laser Scanning 2013. ISPRS Annals of the Photogrammetry, Remote Sensing and Spatial Information Sciences, Vol. II-5/W2*, pages 313–318, 2013. +- [60] Xiaoyang Wu, Li Jiang, Peng-Shuai Wang, Zhijian Liu, Xihui Liu, Yu Qiao, Wanli Ouyang, Tong He, and Hengshuang Zhao. Point Transformer V3: Simpler, Faster, Stronger . In *2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)*, pages 4840–4851, Los Alamitos, CA, USA, 2024. IEEE Computer Society. [6](#page-5-3) +- [61] Yuanwen Yue, Sabarinath Mahadevan, Jonas Schult, Francis Engelmann, Bastian Leibe, Konrad Schindler, and Theodora Kontogianni. AGILE3D: Attention Guided Interactive + +- Multi-object 3D Segmentation. In *International Conference on Learning Representations (ICLR)*, 2024. [2](#page-1-0) +- [62] SM Zolanvari, Susana Ruano, Aakanksha Rana, Alan Cummins, Rogerio Eduardo da Silva, Morteza Rahbar, and Aljosa Smolic. DublinCity: Annotated LiDAR point cloud and its applications. In *BMVC 30th British Machine Vision Conference*, 2019. [2](#page-1-0) + +# **SUM Parts: Benchmarking Part-Level Semantic Segmentation of Urban Meshes** + +# Supplementary Material + +#### 6. Details on annotation tool + +#### 6.1. Face-based annotation + +**Protrusion score.** By measuring the distance and angle from face $f_i$ to support plane $P_k^f$ , we define protrusion score $p_i = d_i + \omega_i \cdot \theta_i$ , where + +$$d_i = \max_{t \in \{0,1,2\}} \left( \operatorname{dist}(v_{t,i}, P_k^f) \right)$$ + +is the maximum Euclidean distance from the vertices $v_t$ of the face $f_i$ to the support plane $P_k^f$ . The angle weight $\theta_i$ is calculated by measuring the angle $\hat{\theta}_i = \cos^{-1}(\mathbf{n}_i \cdot \mathbf{n}_k)$ between the normal $\mathbf{n}_i$ of face $f_i$ and the normal $\mathbf{n}_k$ of support planar segment $P_k^f$ , defined as: + + +$$\theta_i = \frac{\min(\hat{\theta}_i, 180^\circ - \hat{\theta}_i)}{90^\circ}.$$ + +Geometric consistency. To measure the geometric consistency between adjacent faces, we utilize an interior shrinking ball algorithm derived from the 3D medial axis transform to compute the ball radii for each face [46, 56]. + +In urban mesh scenarios, larger shrinking balls typically correspond to major geometric structures such as the terrain or main surfaces of buildings, whereas smaller balls indicate sharp structures or protrusions (as shown in Fig. 15). Con- Figure 15. Cross-sectional view balls can indirectly reflect urban scenarios. the local structural scale, + +![](SUM-Parts_2503.15300_images/_page_11_Picture_10.jpeg) + +> **[그림 해설]** 메시 내부 수축 구(Interior shrinking ball) 기하 다이어그램. +> - 페이스 $f$의 법선 벡터 $\mathbf{n}^f$와 내부 접촉점 $q_1, q_2$ 사이의 구 반경 $r = \frac{\|q_1 - q_2\|^2}{2(\mathbf{n}^f \cdot (q_1 - q_2))}$ 계산 원리 도해. +> - 동일 기하 구조(예: 벽체, 기둥) 내 인접 페이스들은 유사한 내부 구 반경 $r$을 가지므로, 벽체 두께 측정 및 동일 구조 부품 클러스터링의 핵심 기하 특징으로 활용됨. + +sequently, the size of these of interior shrinking balls (red) in + +suggesting that adjacent faces within the same geometric structure should have similar radii. The mesh shrinking ball radius is derived as + +$$r = \frac{\|q_1 - q_2\|^2}{2(\mathbf{n}^f \cdot (q_1 - q_2))},$$ + +where r refers to the radius $r_i$ or $r_j$ , and $\mathbf{n}^f$ denotes the normal $\mathbf{n}_i$ or $\mathbf{n}_j$ of the respective faces $f_i$ or $f_j$ ; $q_1$ and $q_2$ are the tangent points on the faces. + +Planar segment matching. We define the feature vector $\mathbf{F}^{(\text{seg})}$ to quantify segment matching similarity, including: + +• Geometric homogeneity: Differences in area between geometrically similar segments are calculated as: + +$$\Delta A^{(seg)} = \frac{\left| area^{(c)} - area^{(t)} \right|}{area^{(t)}},$$ + +where $area^{(c)}$ and $area^{(t)}$ are the areas of the candidate and template segments, respectively. + +• Spatial distribution: Vertical distribution similarity is measured by comparing weighted average heights: + +$$\Delta H^{(seg)} = \left| \frac{\sum_{i=1}^{m'} z_i \cdot a_i}{area^{(c)}} - \frac{\sum_{j=1}^{m''} z_j \cdot a_j}{area^{(t)}} \right|.$$ + +where $z_i$ and $a_i$ denote the z-coordinate and area of each face $f_i$ in the candidate segment $P_k^{(c)}$ . $z_j$ and $a_j$ denote those in the template segment $P^{(t)}$ . + +- Spatial orientation: Similarity in vertical orientation is as- +- sessed between segments $P_k^{(c)}$ and $P^{(t)}$ [51]. + + Shape sphericity: Calculated using eigenvalues from triangle vertices of the segment $P_k^{(c)}$ and $P^{(t)}$ to evaluate similarity [59]. +- Photometric coherence: Color similarity is assessed using CIELAB [38] color distance and greenness [41] dif- + +**Protrusion matching.** For seed expansion, in addition to spatial and segment scale constraints, we introduce optional topology constraints based on adjacency to optimize user focus and simplify inspection. In urban scenes, small protrusions (e.g., cars, balconies, dormers) reduce global matching efficiency by increasing inspection workload and computation (e.g., matching cars globally takes 5s, whereas planar facades take only 0.4s, see Fig. 4b). Therefore, we set topology constraints as the default for practical efficiency by confining the search space to planar segments of the template support surface (e.g., limiting annotations to the current facade for facade installations). + +The feature vector $\mathbf{F^{(str)}}$ includes: + +• Spatial compactness: The compactness of a protrusion is quantified by considering its volume. We expect similar protrusions to have comparable values, defined as + +$$\Delta V^{(str)} = \left| \frac{vol^{(c)}}{vol_{box}^{(c)}} - \frac{vol^{(t)}}{vol_{box}^{(t)}} \right|,$$ + +where $vol^{(c)}$ and $vol_{hox}^{(c)}$ represent the volume of $f^{(c)}$ and its bounding box volume, respectively. $vol^{(t)}$ and $vol_{box}^{(t)}$ are the corresponding values for $f^{(t)}$ . + + Surface complexity: We assume complex 3D shapes decompose into multiple planar segments. Surface complexity similarity is measured by the ratio of the number of planar segments in the template and candidate protrusions, defined as + +$$\Delta N^{(str)} = \left(\frac{\max(n^{(t)}, n^{(c)})}{\min(n^{(t)}, n^{(c)})}\right)^{\mu},$$ + +where $n^{(t)}$ and $n^{(c)}$ respectively represent the number of planar segments for the template and candidate protrusions, and $\mu = \min(n^{(t)}, n^{(c)})$ . + +• Structural features: Measuring the similarity of protrusions involves comparing their structural features through eigenvalue analysis including linearity, planarity, and sphericity [59]. We determine similarity by the $\ell_1$ distance in the feature space, including differences in linearity $\Delta L^{(str)}$ , planarity $\Delta P^{(str)}$ , and sphericity $\Delta S^{(str)}$ . + +#### 6.2. Texture-based annotation + +**Gaussian mixture model (GMM).** $G_k$ denotes the GMM for the k-th channel, defined as + +$$G_k(S) = \sum_{m=1}^{M} \pi_{km} \mathcal{N}(x; \mu_{km}, \Sigma_{km}),$$ + +where S represents the superpixel $S_0$ or $S_j$ . x is a pixel sample point of S, and M is the number of components in GMM (M set to 5 in all experiments in this paper). $\mathcal{N}(s;\mu,\Sigma)$ denotes the multivariate normal distribution, with $\mu$ representing the mean for superpixels $S_0$ or $S_j$ , and $\Sigma$ denotes their respective covariance matrices. + +**Local color consistency.** For local color consistency, where $\rho_j = \Delta E_{00}(U_0,U_j)$ is the color distance (i.e., CIEDE2000 [38]) from the superpixel $S_j$ to its seed $S_0$ . To more accurately capture the intrinsic structure and variability within superpixels' color distributions, we employ a GMM to compute the average Lab color, represented by $U = \sum_{m=1}^M \pi_m \mu_m$ , where U represents $U_0$ or $U_j$ , with $\pi_m$ as the mixing weight and $\mu_m$ as the mean for the m-th Gaussian component in the Lab color space. Additionally, seed samples for $U_0$ are taken from its first-order neighborhood, whereas samples for $U_j$ come from its own pixels. + +Region-based template matching. The feature vector $\mathbf{F^{(reg)}}$ includes: + +• Shape index: To assess shape similarity, we use a shape index reflecting elongation or flatness, which is defined as: $r = \frac{\min(w,h)}{\max(w,h)}$ , where w and h represent the width and height of the object's bounding box, respectively. The similarity between regions is calculated as: $\Delta I^{(reg)} = |r_c - r_t|$ , where r represents the ratio $r_c$ of the candidate region or $r_t$ of the template region. + +• Shape regularity: We assess shape regularity to calculate the similarity between areas. Similar to structural matching, compactness is used to describe how well a shape fills its bounding box, defined as: + +$$\Delta A^{(reg)} = \left| \frac{area^{(c)}}{area^{(c)}_{box}} - \frac{area^{(t)}}{area^{(t)}_{box}} \right|,$$ + +where $area^{(c)}$ and $area^{(t)}$ represent the area of the candidate and template regions, respectively, and $area^{(c)}_{box}$ and $area^{(t)}_{box}$ are the areas of their bounding boxes. + +• Contextual features: Similar regions should have similar internal and external color distributions. We evaluate these differences using the Wasserstein distance, calculated as: $\Delta D^{(reg)} = \left|W(G_k(R_{in}^{(c)}),G_k(R_{out}^{(c)}))-W(G_k(R_{in}^{(t)}),G_k(R_{out}^{(t)}))\right|$ , where $R_{in}^{(c)}$ and $R_{out}^{(c)}$ denote the interior and exterior pixel collections of the candidate region, respectively, and $R_{in}^{(t)}$ and $R_{out}^{(t)}$ for the template region. The external region $R_{out}$ includes pixels covered but not selected during local expansion. + +Scalability. Our workflow is fully compatible with deep learning-based frameworks like Semantic-SAM [33] and Mask DINO [32]. Combining them demonstrates the potential to accelerate template generation through prompt-based segmentation at various granularities and refine template matching with instance/object detection. Additionally, our 2D paint canvas (see Fig. 5) converts texture segments into images that are compatible with these segmentation methods. This, combined with our annotated dataset, allows direct training on 3D textured surfaces, setting the stage for future improvements in efficiency and accuracy. + +#### 7. Details on benchmark results + +#### 7.1. Evaluation of interactive annotation + +**Evaluation metrics.** Traditional metrics, such as click counts to achieve specific Intersection over Union (IoU) or Average Precision (AP), are quantifiable but do not fully capture the true efficiency of the annotation process. The main shortcomings of these methods include: + +- Evaluation limitations: Relying solely on IoU or AP does not fully capture annotation comprehensiveness. For example, a 90% IoU may still require multiple boundary adjustments for accuracy. +- Interaction limitations: Click-based interactions alone cannot perfectly annotate boundaries, often requiring tools like lassos or polygons. Additionally, standardized click positions do not account for individual user variations, hindering realistic efficiency assessment. + +• Efficiency limitations: Average click counts do not reflect actual interaction efficiency due to varying user speeds. Measuring total annotation time provides a more accurate assessment of efficiency. + +To address these issues, we developed an evaluation system comprising Intersection over Union (IoU), Boundary IoU (BIoU), number of operations (Oper), annotation time (Time), and smart interaction ratio (SR). BIoU assesses boundary annotation accuracy [\[13,](#page-8-26) [20\]](#page-8-5). Oper counts mouse clicks and keyboard keystrokes. Time measures annotation duration in seconds. SR quantifies the frequency of nonmanual interactions (counting only click-based selections, excluding other operations). Our evaluation is based on user studies with u users across n test scenes, each with c categories. The average metrics are calculated as follows: + +- Evaluating a single scenario. For a given scenario s, annotated by u users across c categories, mIoU, mBIoU, mOper, mT ime, and mSR, can be obtained by averaging the IoU, mBIoU, mOper, mT ime, and mSR values across all users and categories. +- Evaluating multiple scenarios. To obtain M, B, O, T, and S, the averages of for multiple scenarios, we take the average of each scenario's mIoU, mBIoU, mOper, mT ime, mSR and then average these values: + +$$\overline{M} = \frac{1}{n} \sum_{s=1}^{n} \overline{mIoU}_{s} \qquad \overline{B} = \frac{1}{n} \sum_{s=1}^{n} \overline{mBIoU}_{s}$$ + +$$\overline{O} = \frac{1}{n} \sum_{s=1}^{n} \overline{mOper}_{s} \qquad \overline{T} = \frac{1}{n} \sum_{s=1}^{n} \overline{mTime}_{s}$$ + +$$\overline{S} = \frac{1}{n} \sum_{s=1}^{n} \overline{mSR}_{s}$$ + +In the user study, we recorded each user's annotation progress and interactions in real-time, requiring at least 95% scene completion based on mesh area or texture pixels. + +Comparisons. Tab. [5](#page-13-0) shows that our method outperforms segment-based annotation [\[19\]](#page-8-4) in object region and boundary quality. Across all four test scenarios, it significantly reduces both interaction counts and annotation time. We also provide additional qualitative analysis, as shown in Fig. [16](#page-14-0) and Fig. [17.](#page-14-1) + +From Tab. [6,](#page-18-0) our method has slightly lower M than GrabCut [\[50\]](#page-10-2), but achieves higher mIoUs in most scenarios and excels in boundary quality. While SimpleClick [\[37\]](#page-9-17) is more efficient in interaction count, our method outperforms others in most scenarios. Though slower than SAM [\[27\]](#page-8-9), our method still surpasses other methods in interaction time. The need for manual corrections enhances annotation quality without significant time cost, and our approach delivers more accurate boundaries with similar correction workloads compared to deep learning methods. We achieve this + + + +| Metric | Method | Cour. | Stre. | Park. | Harb. | +|------------|--------|-----------------------------|-------|---------------------|-------| +| | Seg | 89.1 | 94.0 | 92.1 | 91.1 | +| mIoUs(%) | Ours | 89.5 | 94.2 | 92.9 | 92.2 | +| | Seg. | 72.7 | 85.4 | 70.6 | 70.3 | +| mBIoUs(%) | Ours. | 72.4 | 84.5 | 71.9 | 71.0 | +| | Man. | 18154 | 1589 | 3559 | 3714 | +| mOpers | Seg. | 17645 | 1407 | 2529 | 2894 | +| | Ours | 13231 | 909 | 1797 | 1957 | +| | Man. | 11401.0 969.5 2146.5 2215.8 | | | | +| mT \times(s) | Seg. | 10441.3 757.0 1441.4 1623.7 | | | | +| | Ours | 9107.2 | | 498.0 1105.9 1257.5 | | +| mSRs(%) | Ours | 66.5 | 94.9 | 85.8 | 84.8 | + +Table 5. Performance evaluation of interactive mesh face annotation methods across four scenarios: Cour. (courtyard complex), Stre. (streets with vehicles), Park. (park with trees), Harb. (harbor with ships). Methods include Man. [\[52\]](#page-10-0) and Seg. [\[19\]](#page-8-4). Highest values are shown in bold. + +through: (1) Better quality from the user-defined template that enables pixel-level boundary control, outperforming deep learning-based clicks by approximately +3.5∼6.5% mIoU and +18.4∼19.5% boundary mIoU ( Tab. [4\)](#page-6-2), especially for regular shapes like windows in Fig. [11.](#page-7-0) (2) Higher efficiency offered by reusable, scale- and rotationinvariant templates, which reduces the interaction count by -18.7% compared to SAM (582 vs. SAM's 716) and annotation time by -23% compared to SimpleClick (663.3s vs. SimpleClick's 861.3s), benefiting repetitive structures ( Tab. [4\)](#page-6-2). Although SAM is slightly faster and SimpleClick requires fewer interactions, our intentional design using handcrafted templates instead of intensive smart clicks prioritizes higher-quality annotations while maintaining a similar total annotation time. + +For regular-shaped objects, interactive clicking is suboptimal. As shown in Fig. [18](#page-14-2) and Fig. [19,](#page-14-3) single clicks lack boundary precision, and multiple clicks do not significantly improve accuracy. Repetitive structures increase the annotation burden due to frequent clicking. Instead, users achieve high precision by drawing rectangles or polygons for elements like windows or doors. Our method enables efficient annotation of similar structures by creating a graphical template once. In summary, if manual corrections in semi-automatic annotations take as much or more time than fully manual annotations, the method loses its utility. Additional qualitative results from our annotation methods are presented in Fig. [20.](#page-15-0) + +Ablation studies on template matching. Our feature design is grounded in geometric priors (shape properties and + +![](SUM-Parts_2503.15300_images/_page_14_Figure_0.jpeg) + +Figure 16. Qualitative analysis of interactive mesh face annotations and their error maps (shown in red) for the courtyard complex. + +> **[그림 해설]** 중정형 대형 건물 블록(Courtyard complex)에 대한 대화형 메시 페이스 주석 및 오류 맵(빨간색 표시). +> - **순서**: Input $\to$ Segment-based (경계 오류 다수) $\to$ Ours (오류 최소화, 최고 88.4% mIoU 달성) $\to$ Manual Ground Truth. + +![](SUM-Parts_2503.15300_images/_page_14_Figure_2.jpeg) + +Figure 17. Qualitative analysis of interactive mesh face annotations and their error maps (shown in red) for the street with vehicles. + +> **[그림 해설]** 도로변 주차 차량(돌출물)에 대한 대화형 메시 페이스 주석 및 오류 맵(빨간색 표시). +> - **순서**: Input $\to$ Segment-based (차량 경계 오분류 다수) $\to$ Ours (돌출물 일괄 매칭으로 오류 최소화, 최고 97.0% mIoU 달성) $\to$ Manual Ground Truth. + +![](SUM-Parts_2503.15300_images/_page_14_Figure_4.jpeg) + +Figure 18. Qualitative analysis of interactive texture annotation results for the facade. + +> **[그림 해설]** 대규모 건물 외벽 파사드 텍스처 주석 정성 비교 (Input vs GrabCut vs Simclick vs SAM vs Ours vs Manual). +> - 수많은 아치형 창문(파란색)과 출입문(갈색), 외벽(노란색)의 반복 패턴을 제안 기법(Ours, 최고 90.7% mIoU)이 완벽한 정렬과 크기 일관성으로 복원. + +![](SUM-Parts_2503.15300_images/_page_14_Figure_6.jpeg) + +Figure 19. Qualitative analysis of interactive texture annotation results for the park. + +> **[그림 해설]** 공원 텍스처(잔디밭 vs 산책로) 주석 정성 비교 (Input vs GrabCut vs Simclick vs SAM vs Ours vs Manual). +> - SAM 및 Simclick은 끊어진 산책로 연결부를 누락하나, 제안 기법(Ours)은 산책로 네트워크(회색/청록색)와 잔디밭(연두색)을 수동 주석(Manual) 수준으로 정밀하게 분할. + +structural distribution), label-free operation, and computational efficiency, validated through hierarchical ablation studies (mIoU) as follows. For matching: (1) Planar segments (e.g., roofs, best 88.4% in Fig. [16\)](#page-14-0). Removing geometric homogeneity (-6.1%), spatial distribution (-13.8%), orientation (-13.3%), and shape sphericity (-1.8%) caused performance drops. (2) Protrusions (e.g., cars, best 97.0% in Fig. [17\)](#page-14-1). When spatial compactness (-2.6%), surface complexity (-1.2%), and structural features (-1.6%) were removed, precise matching suffered significantly. (3) Regions (e.g., windows, best 90.7% in Fig. [18\)](#page-14-2). Eliminating shape index (-5.2%), regularity (-38.1%), and contextual features (-20.3%) severely impaired boundary alignment and color consistency. These results highlight the essential role of each feature and their combined effectiveness, confirming our method's superior performance. + +### 7.2. Evaluation of semantic segmentation + +*1) Face labeling track.* Tab. [7](#page-19-0) provides a detailed comparison of results for all face-labeled classes. Due to class imbalance, most methods show better performance in categories with more samples and poorer performance in categories with fewer samples. We conducted qualitative analyses on all methods except PointNet for two scenarios, as shown in Fig. [21](#page-16-0) and Fig. [22.](#page-17-0) + +*2) Pixel labeling track.* Tab. [8](#page-19-1) provides a detailed comparison of results for all face-labeled and pixel-labeled classes. PointVector [\[17\]](#page-8-24) surpasses other methods in all categories, particularly with pixel labels. However, compared to the categories shared with Tab. [7,](#page-19-0) the IoU results for most methods have decreased. This is mainly because the three mesh sampling methods produce relatively uniform point clouds, failing to capture the geometric density variations inherent in adaptive meshes. Additionally, the increase in the number of classes has exacerbated the issue of class imbalance. We performed qualitative analyses for all methods in two scenarios, with global and zoomed-in views, as shown in Fig. [23](#page-20-0) and Fig. [24.](#page-21-0) + +![](SUM-Parts_2503.15300_images/_page_15_Figure_0.jpeg) + +Figure 20. Examples of part-level annotated semantic urban meshes are displayed from the first to the third column, showing textured meshes, face-based semantic meshes (13 classes), and texture-based semantic meshes (19 classes), respectively. + +> **[그림 해설]** 5개 대표 도시 씬에 대한 파트 레벨 주석 결과 비교 (5행 3열). +> - **1열 (Input)**: 원본 항공 텍스처 메시 (도심 블록, 광장, 대성당, 공원, 해안가). +> - **2열 (Face-based annotation)**: 13개 클래스 페이스 기반 주석 (지붕, 벽체, 바닥, 수목 등). +> - **3열 (Texture-based annotation)**: 19개 클래스 텍스처(픽셀) 기반 초정밀 주석 (창문, 출입문, 굴뚝, 차선, 보도, 잔디밭 등 미세 구조 완벽 포함). + +## 8. Comparison of related datasets + +Compare with SUM. Our proposed SUM Parts dataset extends beyond SUM's object-level labels [\[19\]](#page-8-4), offering three key benefits: (1) finer geometric analysis, such as evaluating heat loss at the window-level rather than at the building-scale; (2) support for part-aware tasks, e.g., drone navigation for precise delivery by localizing windows, doors, and rooftop solar panel planning; (3) seamless integration with urban digital twins and BIM workflows. + +Compare with KITTI-360. KITTI-360 [\[35\]](#page-9-0) focuses on street-view LiDAR-image fusion for autonomous driving, providing 37 Cityscapes-aligned classes, including roadaccessible static and dynamic objects (≥0.1m resolution) labeled via manual selection and trajectory-based matching. In contrast, SUM Parts addresses broader urban planning and sustainability challenges using oblique photogramme- + +![](SUM-Parts_2503.15300_images/_page_16_Figure_0.jpeg) + +Figure 21. Qualitative analysis of semantic segmentation and error maps in the face labeling track for all methods except PointNet [\[12\]](#page-8-21) in the first scenario. F c. and Sp. denote face-centered and superpixel sampling, respectively. + +> **[그림 해설]** Face labeling 트랙 시나리오 1(대형 굴뚝이 있는 공장/건물 단지)에 대한 11개 모델의 세그멘테이션 및 오류 맵(빨간색 표시). +> - **비교 모델**: RF_MRF, SUM_RF, PSSNet, PointNet++$^{Sp}$, SPG$^{Sp}$, SparseUNet$^{Fc}$, Randla-net$^{Fc}$, KPConv$^{Fc}$, PointNext$^{Fc}$, PointTransV3$^{Fc}$, PointVector$^{Fc}$. +> - PointVector$^{Fc}$가 대형 굴뚝(빨간색/주황색), 복잡한 옥상 구조물 및 외벽 경계에서 오류(빨간색 마킹)를 최소화하며 최고 성능 입증. + +![](SUM-Parts_2503.15300_images/_page_17_Figure_0.jpeg) + +Figure 22. Qualitative analysis of semantic segmentation and error maps in the face labeling track for all methods except PointNet [\[12\]](#page-8-21) in the second scenario. F c. and Sp. denote face-centered and superpixel sampling, respectively. + +> **[그림 해설]** Face labeling 트랙 시나리오 2(헬싱키 대성당 및 광장 주변)에 대한 11개 모델의 세그멘테이션 및 오류 맵(빨간색 표시). +> - 대성당 돔(자주색/보라), 계단, 지붕(회색), 외벽(노랑), 수목(초록), 광장 바닥(갈색) 분할 비교. +> - PointVector$^{Fc}$가 대성당 돔과 복잡한 지붕 구조물에서 가장 높은 정확도를 기록. + + + +| Metric | Method Fac1. | | | | | Fac2. Par1. Par2. Rod1. | Rod2. | +|------------|--------------|------|------|------|------|----------------------------------------|-------| +| | Gra. | 80.2 | 87.3 | 94.0 | 90.4 | 91.3∗ | 85.4 | +| | SAM | 81.0 | 86.4 | 85.3 | 86.4 | 86.5 | 80.7 | +| mIoUs(%) | Sip. | 73.7 | 77.5 | 87.9 | 86.2 | 84.0 | 79.1 | +| | Ours | 79.2 | 88.7 | 95.0 | 91.2 | 91.3 | 82.1 | +| | Gra. | 27.8 | 45.4 | 63.7 | 45.8 | 49.9 | 50.1 | +| | SAM | 24.7 | 33.9 | 25.8 | 23.3 | 34.1 | 37.0 | +| mBIoUs(%) | Sip. | 19.7 | 26.8 | 38.0 | 34.4 | 31.8 | 34.6 | +| | Ours | 28.1 | 50.5 | 67.8 | 48.6 | 50.5 | 50.6 | +| | Man. | 515 | 124 | 497 | 297 | 718 | 1764 | +| | Gra. | 717 | 156 | 462 | 243 | 960 | 1729 | +| | SAM | 715 | 319 | 363 | 319 | 1020 | 1684 | +| mOpers | Sip. | 297 | 119 | 77 | 78 | 400 | 539 | +| | Ours | 487 | 105 | 497 | 213 | 720 | 1468 | +| | Man. | | | | | 816.9 185.6 636.7 280.1 801.7 1941.4 | | +| | Gra. | | | | | 920.1 242.6 494.9 270.8 836.3 1920.6 | | +| mT \times(s) | SAM | | | | | 565.5 150.6 460.9 389.5 800.6 1476.5 | | +| | Sip. | | | | | 1128.5 338.9 197.7 230.8 1526.4 1745.5 | | +| | Ours | | | | | 631.1 155.8 565.9 189.1 767.9 1670.3 | | +| | Gra. | 7.1 | 11.8 | 59.3 | 66.5 | 9.9 | 26.7 | +| | SAM | 94.7 | 92.8 | 78.8 | 78.6 | 49.2 | 36.0 | +| mSRs(%) | Ours | 20.3 | 21.2 | 51.6 | 75.8 | 32.2 | 40.8 | + +Table 6. Performance evaluation of interactive texture annotation methods across six scenarios: Man. (manual), Gra. (GrabCut [\[50\]](#page-10-2)), SAM (Segment Anything [\[27\]](#page-8-9)), Sip. (SimpleClick [\[37\]](#page-9-17)), Fac1./Fac2. (facades 1 & 2), Par1./Par2. (parks 1 & 2), Rod1./Rod2. (roads 1 & 2). ∗GrabCut on Rod1. achieved 91.29%, slightly below our method's 91.31%. Highest values are in bold. + +try meshes. Key differences include: (1) Labeling granularity: SUM Parts offers both object- and part-level annotations (21 CityGML-aligned classes) for fine-grained urban infrastructure details. (2) Annotation tools: Our meshtexture semi-automatic selection tools (click, stroke, lasso) with 2D/3D template matching ensure efficient annotation. (3) Coverage: SUM Parts provides full-city coverage, annotating all static objects (≥0.5m resolution), including vehicle-inaccessible areas. Hence, SUM Parts complements KITTI-360 for broader urban applications. + + + +| | | terr. hveg. faca. wate. | | | car | | | | boat roof. chim. dorm. balc. roin. wall OA mAcc mIoU | | | | | | | +|----------------------------|------|-------------------------|------|------|-----------|---------------------|------|------|------------------------------------------------------|------|---------------------|-----|-----------|------|------| +| RF_MRF | | 81.6 86.6 | 81.3 | | 84.5 24.8 | 3.7 | 73.3 | 27.6 | 0.0 | 4.8 | 0.4 | 5.9 | 85.2 | 45.3 | 39.5 | +| SUM_RF | | 84.8 88.1 | 84.0 | | | 79.0 42.5 10.6 77.7 | | 42.4 | 3.5 | 22.2 | 4.7 | | 12.7 86.9 | 53.6 | 46.0 | +| PSSNet | | 80.7 90.5 | 85.2 | | | 64.2 52.6 13.0 78.1 | | 44.0 | 6.6 | 25.7 | 6.9 | | 16.6 86.3 | 56.4 | 47.0 | +| PoinNetSp. | 52.6 | 7.1 | 38.6 | 59.9 | 0.0 | 0.0 | 22.8 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 50.6 | 22.0 | 15.1 | +| PoinNet++Sp. | | 67.9 68.7 | 59.2 | | | 86.1 24.2 11.1 51.1 | | 24.9 | 0.0 | 0.0 | 3.3 | 1.1 | 69.0 | 46.9 | 33.1 | +| SPGSp. | | 53.4 55.3 | 62.5 | | | 40.5 27.4 13.1 64.3 | | 33.9 | 5.1 | 11.3 | 3.9 | 9.9 | 64.9 | 55.0 | 31.7 | +| SparseUNetF c. | | 88.6 91.7 | 88.6 | | | 76.7 75.6 14.6 82.3 | | 70.1 | 27.0 | | 49.0 28.0 33.9 90.3 | | | 71.7 | 60.5 | +| Randla-netF c. | | 86.7 92.3 | 81.6 | | | 87.1 82.9 41.2 71.6 | | 55.6 | 21.6 | | 27.6 19.0 21.1 86.7 | | | 76.3 | 57.4 | +| KPConvF c. | | 86.9 90.8 | 88.3 | | | 81.5 66.4 16.5 81.9 | | 66.7 | 16.1 | | 45.3 21.2 28.2 90.1 | | | 64.7 | 57.5 | +| PointNextF c. | | 91.0 95.0 | 90.4 | | | 81.6 91.2 17.9 83.1 | | 74.6 | 33.8 | | 56.0 30.0 39.3 91.8 | | | 77.2 | 65.3 | +| PointTransV3F c. 88.6 90.1 | | | 87.9 | | | 78.9 72.1 16.1 81.0 | | 66.2 | 21.4 | | 45.2 25.0 36.4 89.9 | | | 70.2 | 59.1 | +| PointVectorF c. | | 92.3 96.8 | 91.7 | | | 85.1 95.2 22.0 85.9 | | 82.6 | 47.9 | | 62.4 38.6 | | 40.0 93.1 | 80.7 | 70.0 | + +Table 7. Comparison of 3D semantic segmentation methods for face labeling using optimal sampling. Semantic categories: 'terr.' (terrain), 'hveg.' (high vegetation), 'faca.' (facade), 'wate.' (water), 'roof.', 'chim.' (chimney), 'dorm.' (dormer), 'balc.' (balcony), and 'roin.' (roof installation). F c. and Sp. denote face-centered and superpixel sampling, respectively. Results are presented as IoU (%), Overall Accuracy (OA %), mean Accuracy (mAcc %), and mean IoU (mIoU %). Highest values in IoU, OA, mAcc, and mIoU are highlighted in bold. + + + +| | hveg. | faca. | wate. | car | boa. | roof. | chim. | dorm. | balc. | roin. | wall | wind. | door | lveg. | impe. | road | roma. | cycl. | side. | OA | mAcc | mIoU | +|----------------------------------------------------------------------------------------------------|-------|------------------------------------------------|-------|-----|------|-------|-------|-------|-------|-------|---------------------------------------------------------------------------------------------------------------|-------|------|-------|-------|-----------------------------|-------|-------|-------|-------------------------|------|------| +| PoinNetSp. | | 0.5 13.3 16.5 0.0 | | | 2.1 | 7.9 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 8.5 | 0.0 | 0.0 | 0.0 | | 0.2 17.3 9.8 | | 2.6 | +| PoinNet++Sp. | | 72.7 47.5 86.4 34.9 12.4 52.4 28.1 0.0 | | | | | | | 5.3 | 5.6 | 0.4 | | | | | 13.0 5.0 42.4 31.2 14.6 9.5 | | 0.0 | | 7.3 55.4 35.2 24.7 | | | +| SPGSp. | | 58.2 50.8 18.4 24.1 2.7 60.4 39.9 3.1 13.6 4.4 | | | | | | | | | 10.5 | 2.4 | | | | 4.0 13.4 14.6 31.0 0.0 | | | | 1.7 12.1 51.5 34.5 19.2 | | | +| SparseUNetRd. | | | | | | | | | | | 88.8 70.0 5.9 51.6 2.5 79.8 55.0 12.3 45.4 22.6 31.5 32.0 12.3 15.2 43.8 44.6 5.2 | | | | | | | | | 0.6 35.8 72.9 45.1 34.5 | | | +| Randla-netSp. | | | | | | | | | | | 90.5 60.9 84.6 67.6 22.7 74.7 53.3 0.6 29.3 16.2 26.3 33.4 12.8 59.8 48.8 50.2 31.5 0.0 37.1 73.5 57.7 42.1 | | | | | | | | | | | | +| KPConvSp. | | | | | | | | | | | 84.0 68.5 81.7 68.6 21.8 78.2 66.4 25.0 41.8 29.6 31.5∗ 36.1 14.9 21.4 35.8 50.0 7.3 13.4 34.1 74.4 58.3 42.6 | | | | | | | | | | | | +| PointNextP o. | | | | | | | | | | | 90.1 66.2 87.9 68.1 16.3 74.5 59.7 14.9 35.6 19.1 31.0 33.2 13.7 55.5 51.4 55.5 29.0 6.9 40.0 76.0 57.6 44.7 | | | | | | | | | | | | +| PointTransV3Rd. 85.9 59.9 74.6 64.7 17.8 75.9 58.7 15.3 37.2 16.2 29.3 11.8 7.9 27.1 43.3 51.5 3.5 | | | | | | | | | | | | | | | | | | | | 7.2 33.4 70.6 54.1 38.0 | | | +| PointVectorSp. | | | | | | | | | | | 92.7 66.6 92.0 70.2 19.8 76.8 60.8 21.8 37.0 20.6 30.8 37.1 16.5 59.8 53.9 57.4 35.0 16.4 45.0 77.0 63.8 47.9 | | | | | | | | | | | | + +Table 8. Comparison of 3D semantic segmentation methods for pixel labeling using optimal sampling strategies: 'hveg.' (high vegetation), 'faca.' (facade surface), 'wate.' (water), 'roof.' (roof surface), 'chim.' (chimney), 'dorm.' (dormer), 'balc.' (balcony), 'roin.' (roof installation), 'wind.' (window), 'lveg.' (low vegetation), 'impe.' (impervious surfaces), 'roma.' (road marking), 'cycl.' (cycle lane), and 'side.' (sidewalk). Additionally, Sp. denotes superpixel sampling, Rd. for random sampling, and P o. for Poisson-disk sampling [\[14\]](#page-8-20). Results are presented as IoU (%), Overall Accuracy (OA %), mean Accuracy (mAcc %), and mean IoU (mIoU %). ∗KPConv's IoU for wall is 31.46%, slightly below SparseUnet's 31.47%. Highest values in IoU, OA, mAcc, and mIoU are highlighted in bold. + +![](SUM-Parts_2503.15300_images/_page_20_Figure_0.jpeg) + +Figure 23. Qualitative analysis of semantic segmentation and error maps in the pixel labeling track for all methods in the first scenario. Sp. denotes superpixel sampling, Rd. for random sampling, and P o. for Poisson-disk sampling [\[14\]](#page-8-20). The zoomed-in view direction is indicated in the input mesh image. + +> **[그림 해설]** Pixel labeling 트랙 시나리오 1(도시 주거/상업 블록 및 확대 파사드 뷰)에 대한 9개 딥러닝 모델의 세그멘테이션 및 오류 맵(빨간색 표시). +> - **비교 모델**: PointNet$^{Sp}$, PointNet++$^{Sp}$, SPG$^{Sp}$, SparseUNet$^{Rd}$, Randla-net$^{Sp}$, KPConv$^{Sp}$, PointNext$^{Po}$, PointTransV3$^{Rd}$, PointVector$^{Sp}$. +> - PointNet/PointNet++는 창문/미세 부품을 구분하지 못하고 외벽(노랑)으로 뭉개지는 반면, PointVector$^{Sp}$ 및 PointTransV3가 층별 창문 격자와 옥상 구조를 가장 정밀하게 분할. + +![](SUM-Parts_2503.15300_images/_page_21_Figure_0.jpeg) + +Figure 24. Qualitative analysis of semantic segmentation and error maps in the pixel labeling track for all methods in the second scenario. Sp. denotes superpixel sampling, Rd. for random sampling, and P o. for Poisson-disk sampling [\[14\]](#page-8-20). The zoomed-in view direction is indicated in the input mesh image. + +> **[그림 해설]** Pixel labeling 트랙 시나리오 2(항구 터미널 대형 크루즈선 부두 및 원형 교차로 라운드어바웃)에 대한 9개 모델의 세그멘테이션 및 오류 맵(빨간색 표시). +> - 대형 여객선/크루즈(남색), 터미널 건물, 원형 교차로 차선/화단/인도 분할 비교. +> - PointVector$^{Sp}$ 및 PointTransV3가 원형 교차로의 복잡한 차선/횡단보도 선형과 항구 수면 경계를 가장 정밀하게 분할. \ No newline at end of file diff --git a/docs/papers/md/SUM-Parts_2503.15300_images/_page_0_Figure_4.jpeg b/docs/papers/md/SUM-Parts_2503.15300_images/_page_0_Figure_4.jpeg new file mode 100644 index 0000000..a305b33 Binary files /dev/null and b/docs/papers/md/SUM-Parts_2503.15300_images/_page_0_Figure_4.jpeg differ diff --git a/docs/papers/md/SUM-Parts_2503.15300_images/_page_0_Picture_13.jpeg b/docs/papers/md/SUM-Parts_2503.15300_images/_page_0_Picture_13.jpeg new file mode 100644 index 0000000..e47ba1b Binary files /dev/null and b/docs/papers/md/SUM-Parts_2503.15300_images/_page_0_Picture_13.jpeg differ diff --git a/docs/papers/md/SUM-Parts_2503.15300_images/_page_11_Picture_10.jpeg b/docs/papers/md/SUM-Parts_2503.15300_images/_page_11_Picture_10.jpeg new file mode 100644 index 0000000..e58f4dc Binary files /dev/null and b/docs/papers/md/SUM-Parts_2503.15300_images/_page_11_Picture_10.jpeg differ diff --git a/docs/papers/md/SUM-Parts_2503.15300_images/_page_14_Figure_0.jpeg b/docs/papers/md/SUM-Parts_2503.15300_images/_page_14_Figure_0.jpeg new file mode 100644 index 0000000..16bc583 Binary files /dev/null and b/docs/papers/md/SUM-Parts_2503.15300_images/_page_14_Figure_0.jpeg differ diff --git a/docs/papers/md/SUM-Parts_2503.15300_images/_page_14_Figure_2.jpeg b/docs/papers/md/SUM-Parts_2503.15300_images/_page_14_Figure_2.jpeg new file mode 100644 index 0000000..7033140 Binary files /dev/null and b/docs/papers/md/SUM-Parts_2503.15300_images/_page_14_Figure_2.jpeg differ diff --git a/docs/papers/md/SUM-Parts_2503.15300_images/_page_14_Figure_4.jpeg b/docs/papers/md/SUM-Parts_2503.15300_images/_page_14_Figure_4.jpeg new file mode 100644 index 0000000..80ad25a Binary files /dev/null and b/docs/papers/md/SUM-Parts_2503.15300_images/_page_14_Figure_4.jpeg differ diff --git a/docs/papers/md/SUM-Parts_2503.15300_images/_page_14_Figure_6.jpeg b/docs/papers/md/SUM-Parts_2503.15300_images/_page_14_Figure_6.jpeg new file mode 100644 index 0000000..e152beb Binary files /dev/null and b/docs/papers/md/SUM-Parts_2503.15300_images/_page_14_Figure_6.jpeg differ diff --git a/docs/papers/md/SUM-Parts_2503.15300_images/_page_15_Figure_0.jpeg b/docs/papers/md/SUM-Parts_2503.15300_images/_page_15_Figure_0.jpeg new file mode 100644 index 0000000..6665b87 Binary files /dev/null and b/docs/papers/md/SUM-Parts_2503.15300_images/_page_15_Figure_0.jpeg differ diff --git a/docs/papers/md/SUM-Parts_2503.15300_images/_page_16_Figure_0.jpeg b/docs/papers/md/SUM-Parts_2503.15300_images/_page_16_Figure_0.jpeg new file mode 100644 index 0000000..5f76bdb Binary files /dev/null and b/docs/papers/md/SUM-Parts_2503.15300_images/_page_16_Figure_0.jpeg differ diff --git a/docs/papers/md/SUM-Parts_2503.15300_images/_page_17_Figure_0.jpeg b/docs/papers/md/SUM-Parts_2503.15300_images/_page_17_Figure_0.jpeg new file mode 100644 index 0000000..1c691c0 Binary files /dev/null and b/docs/papers/md/SUM-Parts_2503.15300_images/_page_17_Figure_0.jpeg differ diff --git a/docs/papers/md/SUM-Parts_2503.15300_images/_page_20_Figure_0.jpeg b/docs/papers/md/SUM-Parts_2503.15300_images/_page_20_Figure_0.jpeg new file mode 100644 index 0000000..81a8c80 Binary files /dev/null and b/docs/papers/md/SUM-Parts_2503.15300_images/_page_20_Figure_0.jpeg differ diff --git a/docs/papers/md/SUM-Parts_2503.15300_images/_page_21_Figure_0.jpeg b/docs/papers/md/SUM-Parts_2503.15300_images/_page_21_Figure_0.jpeg new file mode 100644 index 0000000..5c753e0 Binary files /dev/null and b/docs/papers/md/SUM-Parts_2503.15300_images/_page_21_Figure_0.jpeg differ diff --git a/docs/papers/md/SUM-Parts_2503.15300_images/_page_2_Figure_0.jpeg b/docs/papers/md/SUM-Parts_2503.15300_images/_page_2_Figure_0.jpeg new file mode 100644 index 0000000..79963f1 Binary files /dev/null and b/docs/papers/md/SUM-Parts_2503.15300_images/_page_2_Figure_0.jpeg differ diff --git a/docs/papers/md/SUM-Parts_2503.15300_images/_page_2_Figure_13.jpeg b/docs/papers/md/SUM-Parts_2503.15300_images/_page_2_Figure_13.jpeg new file mode 100644 index 0000000..2dba02a Binary files /dev/null and b/docs/papers/md/SUM-Parts_2503.15300_images/_page_2_Figure_13.jpeg differ diff --git a/docs/papers/md/SUM-Parts_2503.15300_images/_page_3_Figure_18.jpeg b/docs/papers/md/SUM-Parts_2503.15300_images/_page_3_Figure_18.jpeg new file mode 100644 index 0000000..5d23525 Binary files /dev/null and b/docs/papers/md/SUM-Parts_2503.15300_images/_page_3_Figure_18.jpeg differ diff --git a/docs/papers/md/SUM-Parts_2503.15300_images/_page_3_Figure_9.jpeg b/docs/papers/md/SUM-Parts_2503.15300_images/_page_3_Figure_9.jpeg new file mode 100644 index 0000000..e47a393 Binary files /dev/null and b/docs/papers/md/SUM-Parts_2503.15300_images/_page_3_Figure_9.jpeg differ diff --git a/docs/papers/md/SUM-Parts_2503.15300_images/_page_4_Figure_0.jpeg b/docs/papers/md/SUM-Parts_2503.15300_images/_page_4_Figure_0.jpeg new file mode 100644 index 0000000..33d8550 Binary files /dev/null and b/docs/papers/md/SUM-Parts_2503.15300_images/_page_4_Figure_0.jpeg differ diff --git a/docs/papers/md/SUM-Parts_2503.15300_images/_page_5_Figure_0.jpeg b/docs/papers/md/SUM-Parts_2503.15300_images/_page_5_Figure_0.jpeg new file mode 100644 index 0000000..ef846f6 Binary files /dev/null and b/docs/papers/md/SUM-Parts_2503.15300_images/_page_5_Figure_0.jpeg differ diff --git a/docs/papers/md/SUM-Parts_2503.15300_images/_page_5_Picture_2.jpeg b/docs/papers/md/SUM-Parts_2503.15300_images/_page_5_Picture_2.jpeg new file mode 100644 index 0000000..b815b72 Binary files /dev/null and b/docs/papers/md/SUM-Parts_2503.15300_images/_page_5_Picture_2.jpeg differ diff --git a/docs/papers/md/SUM-Parts_2503.15300_images/_page_6_Figure_2.jpeg b/docs/papers/md/SUM-Parts_2503.15300_images/_page_6_Figure_2.jpeg new file mode 100644 index 0000000..9efd13a Binary files /dev/null and b/docs/papers/md/SUM-Parts_2503.15300_images/_page_6_Figure_2.jpeg differ diff --git a/docs/papers/md/SUM-Parts_2503.15300_images/_page_6_Figure_8.jpeg b/docs/papers/md/SUM-Parts_2503.15300_images/_page_6_Figure_8.jpeg new file mode 100644 index 0000000..5a84fc2 Binary files /dev/null and b/docs/papers/md/SUM-Parts_2503.15300_images/_page_6_Figure_8.jpeg differ diff --git a/docs/papers/md/SUM-Parts_2503.15300_images/_page_7_Figure_0.jpeg b/docs/papers/md/SUM-Parts_2503.15300_images/_page_7_Figure_0.jpeg new file mode 100644 index 0000000..5cf3e02 Binary files /dev/null and b/docs/papers/md/SUM-Parts_2503.15300_images/_page_7_Figure_0.jpeg differ diff --git a/docs/papers/md/SUM-Parts_2503.15300_images/_page_7_Figure_11.jpeg b/docs/papers/md/SUM-Parts_2503.15300_images/_page_7_Figure_11.jpeg new file mode 100644 index 0000000..40cacf2 Binary files /dev/null and b/docs/papers/md/SUM-Parts_2503.15300_images/_page_7_Figure_11.jpeg differ diff --git a/docs/papers/md/SUM-Parts_2503.15300_images/_page_7_Figure_13.jpeg b/docs/papers/md/SUM-Parts_2503.15300_images/_page_7_Figure_13.jpeg new file mode 100644 index 0000000..a7514ee Binary files /dev/null and b/docs/papers/md/SUM-Parts_2503.15300_images/_page_7_Figure_13.jpeg differ diff --git a/docs/papers/md/SUM-Parts_2503.15300_images/_page_7_Figure_9.jpeg b/docs/papers/md/SUM-Parts_2503.15300_images/_page_7_Figure_9.jpeg new file mode 100644 index 0000000..2eec836 Binary files /dev/null and b/docs/papers/md/SUM-Parts_2503.15300_images/_page_7_Figure_9.jpeg differ diff --git a/docs/papers/md/SUM_2021_dataset.md b/docs/papers/md/SUM_2021_dataset.md new file mode 100644 index 0000000..4ca6bdd --- /dev/null +++ b/docs/papers/md/SUM_2021_dataset.md @@ -0,0 +1,522 @@ +#### SUM: A Benchmark Dataset of Semantic Urban Meshes + +Weixiao GAOa, ∗ , Liangliang Nan a , Bas Boom b , Hugo Ledoux a + +a3D Geoinformation Research Group, Faculty of Architecture and the Built Environment, Delft University of Technology, 2628 BL Delft, The Netherlands bCycloMedia Technology, Zaltbommel, The Netherlands + +## Abstract + +Recent developments in data acquisition technology allow us to collect 3D texture meshes quickly. Those can help us understand and analyse the urban environment, and as a consequence are useful for several applications like spatial analysis and urban planning. Semantic segmentation of texture meshes through deep learning methods can enhance this understanding, but it requires a lot of labelled data. The contributions of this work are threefold: (1) a new benchmark dataset of semantic urban meshes, (2) a novel semi-automatic annotation framework, and (3) an annotation tool for 3D meshes. In particular, our dataset covers about 4 km 2 in Helsinki (Finland), with six classes, and we estimate that we save about 600 hours of labelling work using our annotation framework, which includes initial segmentation and interactive refinement. We also compare the performance of several state-of-theart 3D semantic segmentation methods on the new benchmark dataset. Other researchers can use our results to train their networks: the dataset is publicly available, and the annotation tool is released as open-source. + +Keywords: Texture meshes; Urban scene understanding; Mesh annotation; Semantic segmentation; Over-segmentation; Benchmark dataset + +### 1. Introduction + +Understanding the urban environment from 3D data (e.g. point clouds and 3D meshes) is a long-standing goal in photogrammetry and computer vision [\[1,](#page-22-0) [2\]](#page-22-1). The fast recent developments in data acquisition technologies and processing pipelines have allowed us to collect a great number of datasets on our 3D urban environments. Prominent examples are Google Earth [\[3\]](#page-22-2), texture meshes covering entire cities (e.g. Helsinki [\[4\]](#page-22-3)), or point clouds covering entire countries (e.g., the Netherlands AHN [\[5\]](#page-22-4)). These datasets have attracted + +∗Corresponding author + +Email addresses: w.gao-1@tudelft.nl (Weixiao GAO), liangliang.nan@tudelft.nl (Liangliang Nan), bboom@cyclomedia.com (Bas Boom), h.ledoux@tudelft.nl (Hugo Ledoux) + +interest because of their potential in several applications, for instance, urban planning [\[6,](#page-22-5) [7\]](#page-22-6), positioning and navigation [\[8,](#page-22-7) [9,](#page-22-8) [10\]](#page-22-9), spatial analysis [\[11\]](#page-22-10), environmental analysis [\[12\]](#page-23-0), and urban fluid simulation [\[13\]](#page-23-1). + +To effectively understand the urban phenomena behind the data, a large amount of ground truth is typically required, especially when applying supervised learning-based techniques, such as a deep Convolutional Neural Network (CNN). The recent development of machine learning (especially deep learning) techniques has demonstrated promising performance in semantic segmentation of 3D point clouds [\[14,](#page-23-2) [15,](#page-23-3) [16\]](#page-23-4). Compared to point clouds, a surface representation (in the form of a 3D mesh, often with textures, see Figure [1](#page-1-0) and [2](#page-2-0) for an example) of the urban scene has multiple advantages: easy to acquire, compact storage, accurate, and with well-defined topological structures. + +![](SUM_2021_dataset_images/_page_1_Picture_2.jpeg) + +Figure 1: Part of the semantic urban mesh benchmark dataset shown as a texture mesh. + +> **[그림 해설]** SUM (Semantic Urban Mesh) 데이터셋의 헬싱키 도심 3D 항공 텍스처 메시 조감도. +> - **상단 전체 뷰**: 헬싱키 중앙역, 대성당, 항구, 도심 블록을 아우르는 대규모 3D 도시 텍스처 메시. +> - **하단 좌측 확대**: 도심 고층 빌딩과 공원 수목 영역의 정밀한 3D 기하 및 사진 텍스처 디테일. +> - **하단 우측 확대**: 해안가 교량, 도로, 주차된 차량, 선착장 보트 및 수면 영역의 상세 메시 표현. + +![](SUM_2021_dataset_images/_page_2_Figure_0.jpeg) + +Figure 2: Part of the semantic urban mesh benchmark dataset, showing the semantic classes (unclassified regions are in black). + +> **[그림 해설]** SUM 데이터셋의 6가지 시맨틱 클래스 레이블링 결과 조감도 및 세부 확대도. +> - **범례**: Terrain(지형/도로, 갈색), Building(건물, 노란색), Water(수면, 하늘색), High vegetation(교목/수목, 연두색), Vehicle(차량, 자주색), Boat(선박/보트, 남색), Unclassified(미분류, 검은색). +> - 상단 좌측 확대: 건물 벽체(노랑)와 바닥 지면(갈색), 인접 나무(연두색) 간의 정확한 3D 경계 분할. +> - 상단 우측 확대: 해안 부두 수면(하늘색), 도로 위 차량(자주색), 선착장에 정박된 보트(남색)의 정밀 분할 시각화. + +This means that 3D meshes have the potential to serve as input for scene understanding. As a consequence, there is an urgent demand for large-scale urban mesh datasets that can be used as ground truth for both training and evaluating the 3D semantic segmentation workflows. + +In this paper, we aim to establish a benchmark dataset of large-scale urban meshes reconstructed from aerial oblique images. To achieve this goal, we propose a semi-automatic mesh annotation framework that includes two components: (1) an automatic process to generate intermediate labels from the raw 3D mesh; (2) manual semantic refinement of those labels. For the intermediate label generation step, we have developed a semantic mesh segmentation method that classifies each triangle into a pre-defined object class. This semantic initialization allows us to achieve an overall accuracy of 93.0% in the classification of the triangle faces in our dataset, saving significant efforts for manually labelling. Then, in the semantic refinement step, a mesh annotation tool (which we have developed) is used to refine the semantic labels of the pre-labelled data (at the triangle and segment levels). + +We have used our proposed framework to generate a semantic-rich urban mesh dataset consisting of 19 million triangles and covering about 4 km2 with six object classes commonly found in an urban environment: terrain, highvegetation, building, water, vehicle, and boat (Figure [2](#page-2-0) shows an example from our dataset). With our semi-automatic annotation framework, generating the ground truth took only about 400 hours; we estimate that manually labelling the triangles would have taken more than 1000 hours. The contributions of our work are: + +- a semantic-rich urban mesh dataset of six classes of common urban objects with texture information; +- a semi-automatic mesh annotation framework consisting of two parts: a pipeline for semantic mesh segmentation and an annotation tool for semantic refinement; +- a comprehensive evaluation and comparison of the state-of-the-art semantic segmentation methods on the new dataset. + +The benchmark dataset is freely available, and the semantic mesh segmentation methods and the annotation software for 3D meshes are released as opensource[1](#page-3-0) . + +### 2. Related Work + +Urban datasets can be captured with different sensors and be reconstructed with different methods, and the resulting datasets will have different properties. Most benchmark urban datasets focus on point clouds, whereas our semantic urban benchmark dataset is based on textured triangular meshes. + +The input of the semantic labelling process can be raw or pre-labelled urban datasets such as the automatically generated results from over-segmentation or semantic segmentation (see Section [3.3\)](#page-8-0). Regardless of the input data, it still needs to be manually checked and annotated with a labelling tool, which involves selecting a correct semantic label from a predefined list for each triangle (or point, depending on the dataset) by users. In addition, some interactive approaches can make the labelling process semi-manual. However, unlike our proposed approach, the labelling work of most of the 3D benchmark data does not take full advantage of over-segmentation and semantic segmentation on 3D data, and interactive annotation in the 3D space. + +We present in this section an overview of the publicly available semantic 3D urban benchmark datasets categorised by sensors and reconstruction types (see Table [1\)](#page-4-0). More specifically, we elaborate on the quality, scale, and labelling strategy of the existing urban datasets regarding semantic segmentation. + +1 + + + +| Name | Platforms | Year | Data Type | Area a / Length Classes | Classes | Points / Triangles | RGB | Automatic Pre-labelling | Annotation | Time Cost (hours) | +|------------------------|--------------------|------|-----------------------|------------------------------------|---------|---------------------------------|-------------------------------|------------------------------------|--------------------------|----------------------------------| +| Oakland 3D [17] | MLS | 2009 | Point Cloud | 1.5~km | 5 | 1.6 M | No | No | 3D Manually | Not reported | +| Paris-rue-Madame [18] | MLS | 2014 | Point Cloud | 0.16~km | 17 | 20~M | No | 2D semantic
segmentation | 3D Semi-manually | Not reported | +| iQmulus [19] | MLS | 2015 | Point Cloud | $10 \ km$ | œ | 300~M | No | No | 2D Semi-manually | Not reported | +| Semantic3D [2] | TLS | 2017 | Point Cloud | | œ | 4000 M | Yes | No | 2D & 3D Semi-manually | Not reported | +| Paris-Lille-3D [20] | MLS | 2018 | Point Cloud | $1.94 \ km$ | 6 | 143 M | No | No | 3D Manually | Not reported | +| SemanticKITTI [21] | MLS | 2019 | Point Cloud | $39.2 \ km$ | 22 | 4549 M | No | No | 3D Manually | 1700 | +| Toronto-3D [22] | MLS | 2020 | Point Cloud | 1.0~km | œ | 78.3 M | Yes | No | 3D Manually | Not reported | +| ISPRS [23] | ALS | 2012 | Point Cloud | $0.1 \ km^2$ | 6 | 1.2 M | No | No | 3D Manually | Not reported | +| AHN3 [5] | ALS | 2019 | Point Cloud | $41,543 km^2$ | 4 | $415.43~B^{\rm b}$ | No | 3D semantic
segmentation | 3D Manually | Not reported | +| DublinCity [24] | ALS | 2019 | Point Cloud | $2.0 \text{ km}^2$ | 13 | 260~M | No | No | 3D Manually | 2500 | +| DALES [25] | ALS | 2020 | Point Cloud | $10.0~km^2$ | œ | 505.3 M | No | 3D semantic
segmentation | 3D Manually | Not reported | +| LASDU [26] | ALS | 2020 | Point Cloud | $1.02 \ km^2$ | 20 | 3.12 M | No | No | 3D Manually | Not reported | +| ETHZ RueMonge [27, 28] | Auto-mobile camera | 2014 | Mesh | 0.7~km | 6 | $1.8 M (lowres)^c$ | Yes (per vertex) d | 2D over-segmentation | 2D Semi-manually | 230 (701 frames) e | +| Campus3D [29] | UAV camera | 2020 | Point Cloud | $1.58 \ km^2$ | 14 | 937.1 M | Yes | No | 2D & 3D Manually | Not reported | +| SensatUrban [30] | UAV camera | 2020 | Point Cloud | $6 \text{ km}^2$ | 13 | 2847.1 M | Yes | No | 3D Manually | 009 | +| Swiss3DCities [31] | UAV camera | 2020 | Point Cloud | $2.7 \text{ km}^2$ | 5 | $226\ M$ | Yes | No | 3D Manually (on mesh) | 144 (1 M Triangles) f | +| Hessigheim 3D [32, 33] | UAV Lidar & camera | 2021 | Point Cloud &
Mesh | $0.19~km^2$ | 11 | $125.7\ M\ /\ 36.76\ M^{\rm g}$ | Yes $(texture)^h$ | No | 3D Manually i | Not reported | +| SUM-Helsinki (Ours) | Airplane camera | 2021 | Mesh | $4 \ km^2$ | 9 | 19~M | Yes (texture) h | 3D over-segmentation & 3D semantic | 3D Semi-manually | 400 | +| | | | | | | | | segmentation | | | + +a The area was measured in a 2D map. + +b The number of total points (i.e., 415.43 billion) is estimated. + +c The number of total points (i.e., 415.43 billion) is estimated. + +c The low-resolution meshes contain 1.8 million triangle faces, according to the publications. + +d An RGB colour was assigned to each triangle vertex. + +e The frames were from video sequences. + +e The interpolation triangles (16 tiles) from simplified mesh were labelled, which took around 6 to 12 hours per tile. + +g The number of LiDAR point size 12.7 million and the number of triangle faces is 36.76 million. + +h The colour of each triangle face corresponds to a parted of the texture image. + +e The LiDAR point clouds were manually annotated and the labels were transferred to the mesh. + +Table 1: Comparison of existing 3D urban benchmark datasets. + +### 2.1. Photogrammetric Products + +### 2.1.1. Dense Point Clouds + +The Campus3D [\[29\]](#page-24-6) is to our knowledge the first aerial point cloud benchmark. The coarse labelling is conducted in 2D projected images with three views, and the grained labels are refined in 3D with user-defined rotation angles. The dataset covers only the campus of the National University of Singapore and is thus not representative of a typical urban scene. + +SensatUrban [\[30\]](#page-24-7) is another example of the photogrammetric point clouds covering various urban landscapes in two cities of the UK. The semantic points are manually annotated via the off-the-shelf software tool CloudCompare [\[34\]](#page-25-0), and the overall annotation is reported to have taken around 600 hours. The dataset also contains several areas without points, especially for water surfaces and regions with dense objects. The leading causes are the Lambertion surface assumption during the image matching and the inadequate image overlapping rate during the flight. + +Similarly, the Swiss3DCities [\[31\]](#page-24-8) was recently released that covers three cities in Zurich but twice smaller than the SensatUrban. The annotation work was conducted on a simplified mesh in the software Blender [\[35\]](#page-25-1), and then the semantics were transferred to the mesh vertices, which are regarded as point clouds, via the nearest neighbour search. The mesh simplification may result in the loss of small-scale objects such as building dormers and chimneys, and the automatic transfer of the labels could have introduced errors in the ground truth. + +### 2.1.2. Triangle Meshes + +To the best of our knowledge, the ETHZ RueMonge 2014 [\[28\]](#page-24-5) is the first urban-related benchmark dataset available as surface meshes. The label for each triangle is obtained from projecting selected images that are manually labelled from over-segmented image sequences [\[27\]](#page-24-4). In fact, due to the error of multiview optimisation and the ambiguous object boundary within triangle faces, the datasets contain many misclassified labels, making them unsuitable for training and evaluating supervised-learning algorithms. + +Hessigheim 3D [\[32,](#page-24-9) [33\]](#page-24-10) is a small-scale semantic urban dataset consisting of highly dense LiDAR point clouds and high resolution texture meshes. Particularly, the mesh is generated from both LiDAR point cloud and oblique aerial images in a hybrid way. The labels of point clouds are manually annotated in CloudCompare [\[34\]](#page-25-0), and the labels of the mesh are transferred from the point clouds by computing the majority votes per triangle. However, if the mesh triangle has no corresponding points, some faces may remain unlabelled which resulted in about 40% unlabelled area. In addition, this dataset contains non-manifold vertices, which makes it difficult to use directly. + +## 2.2. LiDAR Point Clouds + +Unlike photogrammetric point clouds, LiDAR point clouds usually do not contain colour information. To annotate them properly, additional information is often required, e.g. images or 2D maps. LiDAR point cloud benchmark datasets are more common than photogrammetric ones. + +## 2.2.1. Street-view Datasets + +The Oakland 3D [\[17\]](#page-23-5) is one of the earliest mobile laser scanning (MLS) point cloud datasets, which was designed for the classification of outdoor scenes. It has five hand-labelled classes with 44 sub-classes, but without colour information and semantic categories like roof, canopy, or interior building block, which are typical for all street-view captured datasets. + +Compared to Oakland 3D, Paris-rue-Madame [\[18\]](#page-23-6) is a relatively smaller dataset which used the 2D semantic segmentation results for 3D annotation. Specifically, the point clouds were projected onto images to extract the objects hierarchically with several unsupervised segmentation and classification algorithms. + +Although the 2D pre-labelled generation is fully automatic, different semantic categories require different segmentation algorithms resulting in difficulties in the classification of multiple classes. + +The iQmulus dataset [\[19\]](#page-23-7) is a 10 km street dataset annotated based on projected images in the 2D space. Specifically, the user first needs to extract objects by editing the image with a polyline tool and then assigns labels to the extracted object regions. Some automatic functions are made for polyline editing in this framework, but the entire annotation pipeline is still complicated. + +Unlike other street view datasets, Semantic3D [\[2\]](#page-22-1) is a dataset consisting of terrestrial laser scanning (TLS) point clouds (the scanner is not moving and scans are made from only a few viewpoints). It has eight classes and colours were obtained by projecting the points onto the original images. There are two annotation methods: (1) annotating in 3D with an iterative model-fitting approach on manually selected points; (2) annotating in a 2D view by separate background from a drawn polygon in CloudCompare [\[34\]](#page-25-0). Although it covers many urban scenes and includes RGB information, the acquired objects are incomplete because of the limited viewpoints and occlusions. + +The other three typical MLS point cloud datasets that were manually labelled are Paris-Lille-3D [\[20\]](#page-23-8), SemanticKITTI [\[21\]](#page-23-9), and Toronto-3D [\[22\]](#page-23-10). + +### 2.2.2. Aerial-view Datasets + +As for ALS benchmark point clouds, representative datasets are ISPRS [\[23\]](#page-24-0), DublinCity [\[24\]](#page-24-1), and LASDU [\[26\]](#page-24-3) covering various scales of city landscapes and were annotated manually with off-the-shelf software. Instead of fully manual annotation, the Dayton Annotated LiDAR Earth Scan (DALES) [\[25\]](#page-24-2) used digital elevation models (DEM) to distinguish ground points with a certain threshold, the estimated normal to label the building points roughly, and satellite images to provide contextual information as references for annotators to check and label the rest of data. Similarly, the AHN3 dataset [\[5\]](#page-22-4) was semimanually labelled by different companies with off-the-shelf software. Besides, since the ALS measurement is conducted in the top view direction, unlike oblique aerial cameras, the obtained point clouds often miss facade information to a certain degree. + +# 3. The Semantic Urban Mesh Dataset + +### 3.1. Dataset Specification + +We have used Helsinki's 3D texture meshes as input and annotated them as a benchmark dataset of semantic urban meshes. The Helsinki's raw dataset covers about 12 km2 , and it was generated in 2017 from oblique aerial images that have about a 7.5 cm ground sampling distance (GSD) using an off-theshelf commercial software namely ContextCapture [\[36\]](#page-25-2). The source images have three colour channels (i.e., red, green, and blue) and are collected from an airplane with five cameras that have 80% length coverage and 60% side coverage. To recover the 3D water bodies that do not fulfil the Lambertian hypothesis, 2D vector maps and ortho-photos are used when performing the surface reconstruction. Furthermore, processing like aerial triangulation, dense image matching, and mesh surface reconstruction were all performed with ContextCapture. It should be noticed that the entire region of Helsinki is split into tiles, and each of them covers about 250 m2 [\[37\]](#page-25-3). As shown in Figure [3,](#page-8-1) we have selected the central region of Helsinki as the study area, which includes 64 tiles and covers about 4 km2 map area (8 km2 surface area) in total. + +## 3.2. Object Classes + +We define the semantic categories for urban meshes by the most common objects in the urban environment with unambiguous geometry and texture appearance. Moreover, each triangle face is assigned to a label of one of the six semantic classes. Ambiguous regions (which account for about 2.6% of the total mesh surface area), such as shadowed regions or distorted surfaces, are labelled as unclassified (see Figure [4\)](#page-8-2). The object classes we consider in the benchmark dataset are: + +- terrain: roads, bridges, grass fields, and impervious surfaces; +- building: houses,high-rises, monuments, and security booths; +- high vegetation: trees, shrubs, and bushes; +- water: rivers, sea, and pools; +- vehicle: cars, buses, and lorries; +- boat: boats, ships, freighters, and sailboats; +- unclassified: incomplete objects like buses and trains, distorted surfaces like tables, tents and facades, construction sites, underground walls. + +![](SUM_2021_dataset_images/_page_8_Figure_0.jpeg) + +Figure 3: Overview of the semantic urban mesh benchmark. Left: the texture meshes covering about 4 km2 map area. Right: the ground truth meshes. More views of the same scene (with different visualization styles) are shown in Figures [1](#page-1-0) and [2.](#page-2-0) + +> **[그림 해설]** $2000\,\text{m} \times 2000\,\text{m} (4\,\text{km}^2)$ 면적의 전체 헬싱키 도심 벤치마크 영역 정사투영 비교. +> - **좌측 (Texture mesh)**: 고해상도 항공 사진으로 텍스처 매핑된 원본 3D 도시 메시. +> - **우측 (Ground truth mesh)**: 6개 시맨틱 클래스(Terrain, Building, Water, High vegetation, Boat, Vehicle)로 완벽하게 구축된 정답 시맨틱 메시 지도. + +![](SUM_2021_dataset_images/_page_8_Figure_2.jpeg) + +Figure 4: Ambiguous regions are labelled as unclassified (in black). (a) Shadow region with texture. (b) Shadow region with semantic colour. (c) Distorted region with texture. (d) Distorted region with semantic colour. + +> **[그림 해설]** 기하 왜곡 및 심한 음영으로 인해 미분류(Unclassified, 검은색)로 처리된 모호 영역 예시. +> - **(a), (b)**: 좁은 건물 중정(안뜰, Courtyard) 내부에 짙게 드리운 그림자로 인해 텍스처 식별이 불가능한 영역을 검은색으로 마스킹. +> - **(c), (d)**: 건물 하단과 도로변의 가로수, 차량, 차양 구조가 복잡하게 뒤엉켜 메시 기하가 왜곡된 경계부를 검은색으로 처리하여 레이블 노이즈 방지. + +## 3.3. Semi-automatic Mesh Annotation + +Rather than manually labelling each triangle face of the raw meshes, we design a semi-automatic mesh labelling framework to accelerate the labelling + +![](SUM_2021_dataset_images/_page_9_Figure_0.jpeg) + +Figure 5: The pipeline of the labelling workflow. + +> **[그림 해설]** SUM 데이터셋의 반자동(Semi-automatic) 주석 파이프라인 흐름도. +> 1. **(a) Input data**: 텍스처 및 기하 메시 입력. +> 2. **(b) Over-segmentation**: 평면성·곡률·색상 유사도를 기반으로 작은 세그먼트 패치로 과분할. +> 3. **(c) Features $\to$ Random Forest**: 기하 및 방사학적 특징을 추출해 초기 분류기 학습 및 (d) Predict data 생성. +> 4. **(e) Annotation and refinement**: 전용 3D GUI 도구에서 작업자가 스마트 선택/라쏘 툴로 오류를 검수 및 수정. +> 5. **(f) Ground truth**: 최종 정답 데이터 완성 및 모델 재학습 데이터로 피드백. + +process. Figure [5](#page-9-0) shows the overall pipeline of our labelling workflow. + +Given the fact that urban environments consist of a large number of planar regions in the data, we opt to label the data at the segment level instead of individual triangle faces. Specifically, we over-segment the input meshes into a set of planar segments. These segments can enrich local contextual information for feature extraction and serve as the basic annotation unit to improve annotation efficiency. + +Instead of randomly choosing a mesh tile as input for annotation and refinement, which is insufficient for manual annotation progress, we favour picking a mesh tile that is more difficult to classify. Similar to active learning, we first compute the feature diversity (see Equation [1\)](#page-9-1) to optimally select a mesh tile containing a variety of classes and objects at different scales and complexity. The feature diversity Fm of tile m is computed as + + +$$F_m = \frac{\sum_{i=1}^{N_f} (f_i - \bar{f})^2}{N_f}$$ + (1) + +where fi represents each handcrafted feature which describe in Section [3.3.1,](#page-10-0) and ¯f is mean value of a Nf dimensional feature vector. To acquire the first ground truth data, we manually annotate the mesh (with segments) that is selected with the highest feature diversity. Then, we add the first labelled mesh into the training dataset for the supervised classification. Specifically, we use the segment-based features as input for the classifier, and the output is a prelabelled mesh dataset. Next, we use the mesh annotation tool to manually refine the pre-labelled mesh according to the feature diversity. Finally, the new refined mesh will be added to the training dataset to improve the automatic classification accuracy incrementally. + +### 3.3.1. Initial Segmentation + +To avoid redundant computations of numerous triangles, we first apply mesh over-segmentation (i.e., linear least-squares fitting of planes) based on region growing on the input data to group triangle faces into homogeneous regions [38]. Such grouped regions are beneficial for computing local contextual features. We then extract both geometric and radiometric features from those mesh segments as follows: + +- Eigen-based features are computed from the covariance matrix of the triangle vertices with respect to the average centre within each segment, which is beneficial for identifying urban objects with various surface distributions. The linearity = $(\lambda_1 - \lambda_2)/\lambda_1$ , sphericity = $\lambda_3/\lambda_1$ and change of curvature = $\lambda_3/(\lambda_1 + \lambda_2 + \lambda_3)$ are computed based on the three eigenvalues $\lambda_1 \geq \lambda_2 \geq \lambda_3 \geq 0$ . The local eigenvectors $\mathbf{n}_i$ and the unit normal vector $\mathbf{n}_z$ along Z-axis are used to compute the verticality $= 1 - |\mathbf{n}_i \cdot \mathbf{n}_z|$ [39]. Note that many eigen-based features have been studied in literature [39, 40, 41], and some of them were designed for and tested on LiDAR point clouds. These eigen-based features are mostly computed per point based on its spherical neighbourhood, which often contains noise and does not form a surface. Our chosen eigen-based features are defined on a segment representing the surface of a mesh, and thus they can capture non-local geometric properties of an object. Additionally, in this work, we have tested all eigen-based features from the literature [39], and we only present the ones that are effective for texture meshes. +- Elevation is divided into absolute elevation $z_a$ , relative elevation $z_r$ and multiscale elevations $z_m$ . Where $z_a$ is the average elevation of the segment; the relative elevation is computed as $z_r = z_a z_{r_{min}}$ ; the multiscale elevation [42, 43] $z_m = \sqrt{\frac{z_a z_{min}}{z_{max} z_{min}}}$ . And $z_{r_{min}}$ denotes the lowest elevation of the local largest ground segment computed within a cylindrical neighbourhood with 30 meters radius around the segment centre. $z_{min}$ and $z_{max}$ represent the local minimum and maximum elevation values of a cylindrical neighbourhood within the scale of 10 meters, 20 meters, and 40 meters. Such large cylindrical neighbourhoods allow to find the local ground considering the resilience to hilly environments, and the square root ensures that small relative height values (i.e., values smaller than 1 m) get a larger elevation attribute to enlarge elevation differences between small objects and the local ground (e.g., cars against the ground, boats against the water surfaces). More importantly, due to the influence of terrain fluctuations and various scales of urban objects, the elevation of these three categories can complement each other. +- Segment area is computed as $area(S_k) = \sum_{i=1}^{N} area(f_i)$ , where $f_i$ denotes a triangle of the segment $S_k$ , and N denotes the total number of triangles in $S_k$ . + +- Triangle density is defined as $density(S_k) = \frac{N}{area(S_k)}$ , which reveals the object complexity, especially for adaptive urban meshes. +- Interior radius of 3D medial axis transform (InMAT) [44, 45] of a segment $S_k$ is formulated as $r_k = \frac{\sum_{i=1}^M r_i}{M}$ , where M denotes the total number of triangle vertices of $S_k$ , and $r_i$ denotes the interior radius of the shrinking ball that touches the vertex $v_i$ within the segment $S_k$ . It is designed to distinguish objects with different scales. +- HSV colour-based features are derived from the RGB channel of the entire texture map. We use the HSV colour space since it can better differentiate different objects than RGB. We compute the average colour, the variance of the colour distribution of all pixels within each segment, and we further discretize it into a histogram that consists of 15 bins of the hue channel, five bins of the saturation channel, and five bins of the value channel. +- Greenness $a_g$ is used to classify objects that are similar to green vegetation. Specifically, it is computed according to the averaged RGB colour of each segment via $a_g = G 0.39 \cdot R 0.61 \cdot B$ [46]. + +All the above features are concatenated into a 44-dimensional feature vector used by our random forest (RF) classifier in the initial segmentation. + +#### 3.3.2. Annotation Tool for Refinement + +Because of the under-segmentation errors and the imperfect results of the semantic mesh segmentation process, we design a mesh annotation tool (see Figure 6) to manually correct the labelling errors. Our mesh annotation tool is developed based on the labelling tool of CGAL [47]. + +As shown in Table 2, it consists of three operation categories: view, selection, and annotation. The view operations provide essential functions for the user to manipulate the scene camera, such as translate, rotate, zoom, or set the new pivot for the scene. In addition, to use textures as a reference for labelling, we map texture and face colour with a certain degree of transparency, and we visualize the segment border to differentiate each segment. + +The selection operations allow the user to select or deselect either triangle faces (see Figure 7) or segments (see Figure 8) freely via a brush or a lasso. Specifically, the face selection operation is used to fix the under-segmentation errors and generate new segments, and the segment selection operation is to fix incorrect segment labels. + +We also allow the user to edit the selection of each individual segment with splitting functions (see Figure 9) and automatic extraction of the most planar region (see Figure 10). As for splitting, we first detect the potential planar and non-planar segments marked by user strokes, and then the non-planar one is split according to the vertex-to-plane distance. It allows generating candidate non-planar regions (with respect to the detected planar segment) for the user to edit, and it is useful to split a segment that covers large non-planar regions or contains more than one dominant planar area. To extract the most planar + +![](SUM_2021_dataset_images/_page_12_Figure_0.jpeg) + +Figure 6: The interface of our annotation tool for 3D texture meshes. + +> **[그림 해설]** 자체 개발한 3D 텍스처 메시 전용 주석 GUI 소프트웨어 'UrbanMeshAnnotator' 인터페이스. +> - **좌측 패널**: +> - Geometric Objects: 메시 타일 관리 및 뷰 모드 설정. +> - Annotation: 분류 확률(Probability) 및 세그먼트 면적(Segment Area) 필터 슬라이더, 6개 클래스 단축키 버튼. +> - Surface Mesh Selection: 삼각형/세그먼트 선택 모드, 라쏘(Lasso) 툴, 선택 영역 확장/축소 버튼. +> - Console: 실시간 폴리곤 면 및 세그먼트 처리 로그. +> - **우측 3D 뷰어**: 세그먼트 경계선(파란색)과 현재 선택된 대규모 광장 영역(빨간색 하이라이트). + + + +| Categories | Operations | Objects | | | +|------------|--------------------------|----------------------|--|--| +| | Translate | Camera | | | +| | Rotate | Camera | | | +| View | Zoom in / out | Camera | | | +| | Set pivot | Camera | | | +| | Multi-selection / Lasso | Triangles / Segments | | | +| | Expand / Reduce | Triangles / Segments | | | +| | Semantic selection | Segments | | | +| Selection | Split region | Segments | | | +| | Planar region extraction | Triangles | | | +| | Split mesh | Triangles | | | +| | Probability slider | Segments | | | +| Annotation | Segment area slider | Segments | | | +| | Progress bar | Triangles | | | +| | Switch semantic view | Triangles | | | +| | Labelling | Triangles / Segments | | | + +Table 2: Basic operations in our annotation tool. + +region, we apply the region growing algorithm [\[38\]](#page-25-4) within the selected segment to + +![](SUM_2021_dataset_images/_page_13_Figure_0.jpeg) + +Figure 7: An example of labelling by selecting triangles using the lasso tool (blue edges: segment boundaries). (a) Before selection. (b) Lasso selection result (in red). (c) The correct label has been assigned to the selected region. In this example, the label of the selected region has been changed from 'ground' to 'vehicle'. + +> **[그림 해설]** 라쏘(Lasso) 선택 툴을 이용한 미세 삼각형 단위 수동 레이블링 과정. +> - **(a) Before selection**: 도로변에 주차된 긴 차량/버스가 지면(ground/terrain, 갈색)으로 잘못 오분류된 상태. +> - **(b) Lasso selection**: 마우스 라쏘 툴로 해당 차량 영역의 삼각형 면들을 빨간색으로 정밀 선택. +> - **(c) Correction**: 선택 영역의 레이블을 'vehicle'(자주색)로 즉시 변경하여 수정 완료. + +![](SUM_2021_dataset_images/_page_13_Figure_2.jpeg) + +Figure 8: An example of segment labelling. (a) Part of a wall of the building was previously labelled as 'high vegetation' (in green). (b) Segment selection result (in red). (c) The label of the selected segment has been corrected with the new label 'building'. + +> **[그림 해설]** 세그먼트(패치) 단위 레이블 수정 작업 예시. +> - **(a)**: 건물 벽면의 대형 세그먼트가 녹색 식생(high vegetation)으로 오분류된 상태. +> - **(b)**: 해당 세그먼트를 한 번의 클릭으로 선택(빨간색 하이라이트). +> - **(c)**: 단축키를 통해 'building'(노란색) 레이블로 일괄 변경하여 벽면 전체를 신속히 교정. + +![](SUM_2021_dataset_images/_page_13_Figure_4.jpeg) + +Figure 9: An example splitting planar and non-planar regions. (a) The user draws a stroke (in red) across the border of the non-planar segment and the planar segment. (b) The detected non-planar segment has been split into two parts (i.e., a non-planar region shown in red and a planar segment shown in green). + +> **[그림 해설]** 스트로크(Stroke) 그리기를 통한 평면 및 비평면 결합 세그먼트 분리 기능. +> - **(a)**: 건물 벽면과 밀착된 가로수가 하나의 세그먼트로 뭉쳐진 부위에 사용자가 빨간색 선(stroke)을 긋는다. +> - **(b)**: 알고리즘이 선을 경계로 복잡한 비평면 수목 영역(빨간색)과 평평한 건물 벽면(초록색)으로 자동 분할 분리. + +automatically generate the candidate triangle faces with user-defined thresholds (i.e., the maximum distance to the plane, the maximum accepted angle, and the minimum region size). Such an operation allows the user to filter out some small bumpy regions of the selected segment. + +Besides, probability and area-based sliders and a progress bar are provided in the annotation panel to improve annotation efficiency and experience, + +![](SUM_2021_dataset_images/_page_14_Figure_0.jpeg) + +Figure 10: Editing an individual segment. (a) A segment is selected (highlighted in green) for splitting. (b) Automatic extraction of the most planar region (shown in red) within the selected segment according to user-defined thresholds. + +> **[그림 해설]** 단일 세그먼트 내 평면 영역 자동 추출(Planar extraction) 기능. +> - **(a)**: 광장 바닥의 복잡한 세그먼트(초록색) 선택. +> - **(b)**: 평면 오차 거리, 법선 각도, 최소 면적 임계값에 따라 가장 평탄한 주 평면 바닥(빨간색)만 자동 추출 분리하여 울퉁불퉁한 노이즈와 분리. + +respectively. Specifically, the probability slider is introduced for the user to visually inspect the segments that are most likely misclassified. Moreover, the user can further use it to inspect a specific class by switching the view to highlight a specific semantic class. The segment area slider is used to identify isolated tiny segments, which commonly appear as errors. The progress bar is used to indicate the estimated labelling progress during the annotation. After performing the selection, the user can easily assign the corresponding label to the selected area. + +### 4. Experiments + +### 4.1. Data Split + +To perform the semantic segmentation task, we randomly select 40 tiles from the annotated 64 tiles of Helsinki as training data, 12 tiles as test data, and 12 tiles as validation data (see Figure [11](#page-15-0) (a)). For each of the six semantic categories, we compute the total area in the training and test dataset to show the class distribution. As shown in Figure [11](#page-15-0) (b), some classes, like vehicles and boats, only account for less than 5% of the total area, while the building and terrain together comprise more than 70%. The unbalanced classes impose significant challenges for semantic segmentation based on supervised learning. + +### 4.2. Evaluation Metric + +Since the triangle faces in the meshes have different sizes, we compute the surface area for semantic evaluation instead of using the number of triangles. The performance of semantic mesh segmentation is measured in precision, recall, + +![](SUM_2021_dataset_images/_page_15_Figure_0.jpeg) + +Figure 11: Overview of the data used in our experiment. (a) The distribution of the training, test, and validation dataset. (b) Semantic categories of training (including validation dataset) and test dataset. + +> **[그림 해설]** 실험 데이터 분할 및 6개 클래스별 표면적 분포 그래프. +> - **(a) 데이터 분할 그리드 ($8 \times 8 = 64$개 타일)**: Train(빨간색), Validation(초록색), Test(파란색) 타일의 공간적 배치. +> - **(b) 클래스별 표면적 ($km^2$, Train 빨강 vs Test 파랑)**: +> - building: Train 약 $3.34\,\text{km}^2$, Test 약 $0.80\,\text{km}^2$ (가장 높은 비중). +> - terrain: Train 약 $1.51\,\text{km}^2$, Test 약 $0.37\,\text{km}^2$. +> - high vegetation: Train 약 $1.03\,\text{km}^2$, Test 약 $0.22\,\text{km}^2$. +> - water: Train 약 $0.51\,\text{km}^2$, Test 약 $0.06\,\text{km}^2$. +> - vehicle: Train 약 $0.07\,\text{km}^2$, Test 약 $0.01\,\text{km}^2$. +> - boat: Train 약 $0.02\,\text{km}^2$, Test 약 $0.03\,\text{km}^2$. + +F1 score, and intersection over union (IoU) for each object class. The evaluation of the whole test area is applied with overall accuracy (OA), mean per-class accuracy (mAcc), and mean per-class intersection over union (mIoU). + +### 4.3. Evaluation of Initial Segmentation + +We have implemented the semantic mesh segmentation and annotation tool in C++ using the open-source libraries include CGAL [\[47\]](#page-26-2), Easy3D [\[48\]](#page-26-3), and ETHZ random forest [\[49\]](#page-26-4). + +Our proposed pipeline for initial segmentation only takes a few input parameters, which are shown in Table [3.](#page-16-0) The over-segmentation is intended to find all planar regions in the model, for which we set the distance threshold to 0.5 meters. This threshold value specifies the minimum geometric features we would like the over-segmentation method to identify. In other words, the region growing-based over-segmentation method will not be able to distinguish two parallel planes with a distance smaller than this threshold. We set the angle threshold to 90 degrees, which is large enough to cope with high levels of noise (e.g., the distance value is small, but the angle between the triangle normal and the plane normal is large). Moreover, the minimum area is set to zero to allow planar segments of any arbitrary size. As for the random forest classifier, we set the parameters initially to those of Rouhani et al. [\[43\]](#page-25-9) followed by fine-tuning using the validation data. Specifically, using 100 trees is sufficient to guarantee the stability of the model, and using the depth of 30 is adequate to avoid over-fitting and under-fitting for training. + + + +| Method | Parameters | Value | +|----------------|-----------------------------------------------------|----------------------| +| Region Growing | Minimum area
Distance to plane
Accepted angle | 0 m2
0.5 m
90◦ | +| Random Forest | Number of trees
Maximum depth | 100
30 | + +Table 3: Parameters used in our approach. + +Rather than classifying about 19 million triangle faces (i.e., the entire dataset), we use 515,176 segments that are clustered during over-segmentation. Although both semantic segmentation and labelling refinement can benefit from mesh over-segmentation, the degree of the under-segmentation error cannot be avoided. Since our mesh over-segmentation does not intend to retrieve the individual objects and the purpose is to perform semantic segmentation, we measure the maximum achievable performance by calculating the IoU instead of using under-segmentation errors to evaluate it. The upper bound IoU of each class we could achieve for semantic segmentation is presented in Table [4,](#page-16-1) and the upper bound mean IoU (mIoU) over all classes is about 90.9% as shown in Table [5.](#page-17-0) In addition, the results of our experiment in Tables [4](#page-16-1) and [5](#page-17-0) are reported based on the average performance of ten times experiments with the same configuration. + + + +| Class | Precision (%) | Recall (%) | F1 scores (%) | IoU (%) | Upper
bound IoU
(%) | +|-----------------|---------------|------------|---------------|---------|---------------------------| +| Terrain | 87.7 | 94.3 | 90.9 | 83.3 | 93.9 | +| High Vegetation | 96.3 | 93.8 | 95.0 | 90.5 | 96.2 | +| Building | 94.6 | 97.7 | 96.1 | 92.5 | 99.0 | +| Water | 97.0 | 88.3 | 92.5 | 86.0 | 92.7 | +| Vehicle | 77.9 | 41.7 | 54.4 | 37.3 | 73.2 | +| Boat | 77.9 | 7.5 | 13.7 | 7.4 | 90.5 | + +Table 4: Overall evaluation of our method. The Upper bound IoU refers to the maximum achievable IoU in theory. + +For semantic segmentation, a detailed evaluation of each class is listed in Table [4,](#page-16-1) and we achieve about 93.0% overall accuracy and 66.2% mIoU as shown in Table [5.](#page-17-0) The qualitative evaluation of it is shown in Figure [12.](#page-18-0) As shown in Figure [12](#page-18-0) (e), most of the prediction errors occur at small-scale objects such as vehicles and boats due to fewer training samples and errors from oversegmentation. + + + +| Model | OA (%) | mAcc (%) | mIoU (%) | ∆mIoU (%) | +|-------------------------------|--------|----------|----------|-----------| +| Upper bound (Perfect) | 98.1 | 91.6 | 90.9 | —— | +| Ours (best) | 93.0 | 70.6 | 66.2 | 0.0 | +| Without sphericity | 93.0 | 70.5 | 66.1 | -0.1 | +| Without segment area | 92.9 | 70.5 | 66.0 | -0.2 | +| Without triangle density | 92.9 | 70.4 | 66.0 | -0.3 | +| Without variance HSV | 92.9 | 70.3 | 65.9 | -0.3 | +| Without absolute elevation | 93.0 | 70.2 | 65.9 | -0.3 | +| Without relative elevation | 92.9 | 70.3 | 65.8 | -0.4 | +| Without curvature | 92.9 | 70.2 | 65.8 | -0.4 | +| Without multiscale elevations | 92.8 | 69.8 | 65.1 | -1.1 | +| Without linearity | 91.8 | 66.6 | 62.0 | -4.2 | +| Without greenness | 91.9 | 66.6 | 61.9 | -4.3 | +| Without InMat | 91.6 | 66.4 | 61.6 | -4.6 | +| Without average HSV | 91.7 | 66.1 | 61.4 | -4.8 | +| Without verticality | 91.4 | 66.1 | 61.3 | -4.9 | +| Without HSV histogram bins | 91.5 | 66.0 | 61.1 | -5.1 | + +Table 5: Ablation study of the features in our approach. The Upper bound (Perfect) refers to the maximum achievable performance in theory. + +To better understand the relevance of the features, we measure the feature importance and perform ablation studies (see Table [5\)](#page-17-0). We can observe that the radiometric features (which account for 62.8%) are more important than geometric ones (which account for 37.2%). Moreover, after removing individual feature vectors, the performance will decline, indicating each feature contributes to the best results. + +![](SUM_2021_dataset_images/_page_18_Figure_0.jpeg) + +Figure 12: Part of our semantic segmentation results. The first column shows the input texture meshes; the second column shows the over-segmentation results; the third column shows the predicted semantic meshes; the fourth column shows the ground truth meshes; the last column shows the error maps (red: errors; green: correct labels). + +> **[그림 해설]** 5개 대표 도시 타일에 대한 시맨틱 분할 단계별 정성적 결과 (5행 5열). +> - **(a) Original**: 입력 원본 텍스처 메시. +> - **(b) Segments**: 과분할(Over-segmentation) 결과 (무지개색 패치). +> - **(c) Predictions**: 초기 Random Forest 분류기의 사전 예측 메시. +> - **(d) Truth**: 정제된 Ground Truth 시맨틱 메시 (Terrain: 갈색, Building: 노랑, Water: 하늘, High veg: 초록, Vehicle: 자주, Boat: 남색). +> - **(e) Error maps**: 예측과 정답 간 오류 맵 (초록색: 일치, 빨간색: 오류). 대부분의 영역이 정확하게 일치하며 건물 경계부와 소형 객체(보트, 차량)에 국소적 오류 집중. + +![](SUM_2021_dataset_images/_page_19_Figure_0.jpeg) + +Figure 13: Sampling point cloud from texture meshes. Our sampled points preserve both geometric and radiometric information of the original mesh. + +> **[그림 해설]** 3D 텍스처 메시로부터 딥러닝 입력용 컬러 포인트 클라우드를 샘플링하는 과정. +> - **(a) Texture mesh**: 도심 항구 선착장의 원본 3D 텍스처 메시. +> - **(b) Wireframe**: 메시의 삼각 폴리곤 와이어프레임 구조 (수면은 대형 삼각형, 복잡한 선박/건물은 조밀한 삼각형). +> - **(c) Sampled point cloud**: 몬테카를로 샘플링과 포아송 디스크 분포를 적용하여 밀도 약 $10\,\text{pts}/\text{m}^2$로 추출한 균일 컬러 포인트 클라우드. + +### 4.4. Evaluation of Competition Methods + +To the best of our knowledge, none of the state-of-the-art deep learning frameworks of 3D semantic segmentation can directly be used on large-scale texture meshes. Additionally, although the data structures of point clouds and meshes are different, the inherent properties of geometry in the 3D space of the urban environment are nearly identical. In other words, they can share the feature vectors within the same scenes. Consequently, we sample the mesh into coloured point clouds (see Figure [13\)](#page-19-0) with a density of about 10 pts/m2 as input for the competing deep learning methods. In particular, we use Montecarlo sampling [\[50\]](#page-26-5) to generate randomly uniform dense samples, and we further prune these samples according to Poisson distributions [\[51\]](#page-26-6) and assign the colour via searching the nearest neighbour from the textures. + +To evaluate and compare with the current state-of-the-art 3D deep learning methods that can be applied to a large-scale urban dataset, we select five representative approaches (i.e., PointNet [\[14\]](#page-23-2), PointNet++ [\[52\]](#page-26-7), SPG [\[15\]](#page-23-3), KPConv [\[16\]](#page-23-4), and RandLA-Net [\[53\]](#page-26-8)). We perform all the experiments on an NVIDIA GEFORCE GTX 1080Ti GPU. Note that these deep learning-based methods downsample the input point clouds significantly as a pre-processing step. In our experiments, the point sampling density is limited by the GPU memory, and increasing or decreasing the sampling density within a reasonable range may lead to slightly different performance. It should be noted that no matter how dense the input point clouds are, almost all state-of-the-art deep learning architectures (such as PointNet, PointNet++, RandLaNet, KPConv, and SPG, etc.) downsample the input point clouds significantly, and they are still able to learn effective features for classification. Besides, different deep learning-based point cloud classification frameworks exploit different strategies for downsampling the input points. In addition, we also compare with the joint RF-MRF [\[43\]](#page-25-9), which is the only competition method that directly takes the mesh as input and without using GPU for computation. + +The hyper-parameters of all the competing methods are tuned according to the validation data to achieve the best results we could acquire. Besides, the results of each competitive method (see Table [6\)](#page-20-0) are demonstrated in average performance based on ten times experiments with the same setting. From the comparison results, as shown in Table [6,](#page-20-0) we found that our baseline method + + + +| | Terrain | High
Vegeta-
tion | Building | Water | Vehicle | Boat | mIoU | OA | mAcc | mF1 | $t_{train}$ | +|-----------------|---------|-------------------------|----------|-------|---------|------|----------------|----------------|----------------|----------------|-------------| +| PointNet [14] | 56.3 | 14.9 | 66.7 | 83.8 | 0.0 | 0.0 | $36.9 \pm 2.3$ | $71.4 \pm 2.1$ | $46.1 \pm 2.6$ | $44.6 \pm 3.2$ | 1.8 | +| RandLaNet [53] | 38.9 | 59.6 | 81.5 | 27.7 | 22.0 | 2.1 | $38.6 \pm 4.6$ | $74.9 \pm 3.2$ | $53.3 \pm 5.1$ | $49.9 \pm 4.8$ | 10.8 | +| SPG [15] | 56.4 | 61.8 | 87.4 | 36.5 | 34.4 | 6.2 | $47.1 \pm 2.4$ | $79.0 \pm 2.8$ | $64.8 \pm 1.2$ | $59.6 \pm 1.9$ | 17.8 | +| PointNet++ [52] | 68.0 | 73.1 | 84.2 | 69.9 | 0.5 | 1.6 | $49.5 \pm 2.1$ | $85.5 \pm 0.9$ | $57.8 \pm 1.8$ | $57.1 \pm 1.7$ | 2.8 | +| RF-MRF [43] | 77.4 | 87.5 | 91.3 | 83.7 | 23.8 | 1.7 | $60.9 \pm 0.0$ | $91.2 \pm 0.0$ | $65.9 \pm 0.0$ | $68.1 \pm 0.0$ | 1.1 | +| KPConv [16] | 86.5 | 88.4 | 92.7 | 77.7 | 54.3 | 13.3 | $68.8 \pm 5.7$ | $93.3 \pm 1.5$ | $73.7 \pm 5.4$ | $76.7 \pm 5.8$ | 23.5 | +| Baseline | 83.3 | 90.5 | 92.5 | 86.0 | 37.3 | 7.4 | $66.2 \pm 0.0$ | $93.0 \pm 0.0$ | $70.6 \pm 0.0$ | $73.8 \pm 0.0$ | 1.2 | + +**Table 6:** Comparison of various semantic segmentation methods on the new benchmark dataset. The results reported in this table are per-class IoU (%), mean IoU (mIoU, %) $\pm$ standard deviation, Overall Accuracy (OA, %) $\pm$ standard deviation, mean class Accuracy (mAcc, %) $\pm$ standard deviation, mean F1 score (mF1, %) $\pm$ standard deviation, and the time cost of training ( $t_{train}$ , hours). The running times of SPG include both feature computation and graph construction, and RF-MRF and our baseline method include feature computation. We repeated the same experiment ten times and presented the mean performance. + +outperforms other methods except for KPConv. Specifically, our approach outperforms RF-MRF with a margin of 5.3% mIoU, and deep learning methods (not including KPConv) from 16.7% to 29.3% mIoU. Compared with the KPConv, the performance of our method is much more robust, which can be observed from Table 6 that the standard deviation of our method is close to zero (i.e., the standard deviation of mIoU of our method is about 0.024%). The reason is that in our method, we set 100 trees in the random forest to ensure the stability of the model, but in KPConv, the kernel point initialization strategy may not be able to select some parts of the point cloud, which leads to the instability of the results. Furthermore, compared with all deep learning pipelines, our method is conducted on a CPU and uses much less time for training (including feature computation). This can be explained by the fact that we have fewer input data (triangles versus points), and the time complexity of our handcrafted features computation is much lower than the features learned from deep learning. + +#### 4.5. Evaluation of Annotation Refinement + +Following the proposed framework, a total of 19,080,325 triangle faces have been labelled, which took around 400 working hours. Compared with a triangle-based manual approach, we estimate that our framework saved us more than 600 hours of manual labour. Specifically, we have measured the labelling speed with these two different approaches on the same mesh tile consisting of 309,445 triangle faces and 8,033 segments. It took around 17 hours for manual labelling based on triangle faces, while with our segment-based semi-automatic approach, it took only 6.5 hours. + +We also evaluate the performance of semantic segmentation with different amounts of input training data on our baseline approach with the intention of understanding the required amount of data to obtain decent results. Specifically, we use ten sets of different training areas with ten times experiments with the same configuration of each set, and we linearly interpolate the results as shown in Figure 14. From Figures 14a, 14b, and 14c, we can observe that our initial + +![](SUM_2021_dataset_images/_page_21_Figure_0.jpeg) + +Figure 14: Effect of the amount of training data on the performance of the initial segmentation method used in the semi-automatic annotation. We repeated the same experiment ten times for each set of training areas and presented the mean performance. + +> **[그림 해설]** 훈련 데이터 면적 비율(Training area %, 5%~100%)에 따른 초기 세그멘테이션 성능 및 안정성 평가 곡선 (10회 반복 평균). +> - **(a) mIoU (%)**: 5% 면적에서 약 58.5%로 시작하여 10~20% 면적만으로 66% 수준에 도달한 뒤 100%까지 65~66%로 안정적 수렴. +> - **(b) OA (%)**: 5% 면적(89.5%)에서 10% 면적(92%)으로 급상승한 후 20% 이상에서 약 92.5~93%로 유지. +> - **(c) Standard deviation (%)**: mIoU(검정) 및 OA(빨강)의 표준편차가 훈련 영역 40% 이상에서 0.02% 이하로 급감. 전체 영역의 단 10%(약 $0.325\,\text{km}^2$) 데이터만으로도 고품질 사전 레이블링이 가능함을 입증. + +segmentation method only requires about 10% (equal to about 0.325 km2 ) of the total training area to achieve acceptable and stable results. In other words, using a small amount of ground truth data, our framework can provide robust pre-labelled results and significantly reduce the manually labelling efforts. + +### 5. Conclusion + +We have developed a semi-automatic mesh annotation framework to generate a large-scale semantic urban mesh benchmark dataset covering about 4 km2 . In particular, we have first used a set of handcrafted features and a random forest classifier to generate the pre-labelled dataset, which saved us around 600 hours of manual labour. Then we have developed a mesh labelling tool that allows the users to interactively refining the labels at both the triangle face and the segment levels. We have further evaluated the current state-of-the-art semantic segmentation methods that can be applied to large-scale urban meshes, and as a result, we have found that our classification based on handcrafted features achieves 93.0% overall accuracy and 66.2% of mIoU. This outperforms the state-of-the-art machine learning and most deep learning-based methods that use point clouds as input. Despite this, there is still room for improvement, especially on the issues of imbalanced classes and object scalability. For future work, we plan to label more urban meshes of different cities and extend our Helsinki dataset to include parts of urban objects (such as roof, chimney, dormer, and facade). We will also investigate smart annotation operators (such as automatic boundary refinement and structure extraction), which involve more user interactivity and may help reduce further the manual labelling task. + +### 6. Acknowledgements + +We would like to thank EuroSDR for providing the funding for this project. The authors appreciate the people who have helped the project, especially Ziqian Ni for the development and the testing of the annotation platform, and Mels Smit and Charalampos Chatzidiakos for assisting with the annotation of the meshes. + +### References + +- [1] F. Matrone, A. Lingua, R. Pierdicca, E. S. Malinverni, M. Paolanti, E. Grilli, F. Remondino, A. Murtiyoso, T. Landes, A benchmark for large-scale heritage point cloud semantic segmentation, The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences XLIII-B2-2020 (2020) 1419–1426. [doi:10.5194/](https://doi.org/10.5194/isprs-archives-XLIII-B2-2020-1419-2020) [isprs-archives-XLIII-B2-2020-1419-2020](https://doi.org/10.5194/isprs-archives-XLIII-B2-2020-1419-2020). +- [2] T. Hackel, N. Savinov, L. Ladicky, J. D. Wegner, K. Schindler, M. Pollefeys, SEMANTIC3D.NET: A new large-scale point cloud classification benchmark, in: ISPRS Annals of the Photogrammetry, Remote Sensing and Spatial Information Sciences, Vol. IV-1-W1, 2017, pp. 91–98. +- [3] Google, 3D imagery in google earth, , accessed: 2021-01-16 (dec 2012). +- [4] C. of Helsinki, Helsinki's 3D city models, [https://www.hel.fi/helsinki/](https://www.hel.fi/helsinki/en/administration/information/general/3d) [en/administration/information/general/3d](https://www.hel.fi/helsinki/en/administration/information/general/3d), accessed: 2020-11-25 (Dec. 2019). +- [5] Actueel Hoogtebestand Nederland (AHN), , accessed: 2021-04-16 (2019). +- [6] C. Ran, The development of 3D city model and its applications in urban planning, in: 2011 19th International Conference on Geoinformatics, IEEE, 2011. [doi:10.1109/geoinformatics.2011.5981007](https://doi.org/10.1109/geoinformatics.2011.5981007). +- [7] K. Czy´nska, P. Rubinowicz, Application of 3D virtual city models in urban analyses of tall buildings: today practice and future challenges, Architecturae et Artibus 6 (1) (2014) 9–13. +- [8] C. Cappelle, M. E. El Najjar, F. Charpillet, D. Pomorski, Virtual 3D city model for navigation in urban areas, Journal of Intelligent & Robotic Systems 66 (3) (2012) 377–399. +- [9] S. Peyraud, D. B´etaille, S. Renault, M. Ortiz, F. Mougel, D. Meizel, F. Peyret, [About non-line-of-sight satellite detection and exclusion in](https://www.mdpi.com/1424-8220/13/1/829) [a 3D map-aided localization algorithm,](https://www.mdpi.com/1424-8220/13/1/829) Sensors 13 (1) (2013) 829–847. [doi:10.3390/s130100829](https://doi.org/10.3390/s130100829). URL +- [10] H. Li-Ta, G. Yanlei, K. Shunsuke, NLOS correction/exclusion for GNSS measurement using RAIM and city building models, Sensors 15 (7) (2015) 17329–17349. [doi:10.3390/s150717329](https://doi.org/10.3390/s150717329). +- [11] Y. Reda, Y. Mabrouk, K. Abdullah, K. Walid, HybVOR: A voronoibased 3D GIS approach for camera surveillance network placement, ISPRS International Journal of Geo-Information 4 (2) (2015) 754–782. [doi:](https://doi.org/10.3390/ijgi4020754) [10.3390/ijgi4020754](https://doi.org/10.3390/ijgi4020754). + +- [12] D. Yichuan, C. P. C. Jack, A. Chimay, A framework for 3D traffic noise mapping using data from BIM and GIS integration, Structure and Infrastructure Engineering 12 (10) (2016) 1267–1280. [doi:10.1080/](https://doi.org/10.1080/15732479.2015.1110603) [15732479.2015.1110603](https://doi.org/10.1080/15732479.2015.1110603). +- [13] C. Garc´ıa-S´anchez, D. Philips, C. Gorl´e, Quantifying inflow uncertainties for CFD simulations of the flow in downtown oklahoma city, Building and Environment 78 (2014) 118–129. [doi:10.1016/j.buildenv.2014.04.013](https://doi.org/10.1016/j.buildenv.2014.04.013). +- [14] C. R. Qi, H. Su, K. Mo, L. J. Guibas, Pointnet: Deep learning on point sets for 3D classification and segmentation, in: Proceedings of the IEEE conference on computer vision and pattern recognition, 2017, pp. 652–660. +- [15] L. Landrieu, M. Simonovsky, Large-scale point cloud semantic segmentation with superpoint graphs, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2018, pp. 4558–4567. +- [16] H. Thomas, C. R. Qi, J.-E. Deschaud, B. Marcotegui, F. Goulette, L. J. Guibas, Kpconv: Flexible and deformable convolution for point clouds, in: Proceedings of the IEEE International Conference on Computer Vision, 2019, pp. 6411–6420. +- [17] D. Munoz, J. A. Bagnell, N. Vandapel, M. Hebert, Contextual classification with functional max-margin markov networks, in: 2009 IEEE Conference on Computer Vision and Pattern Recognition, 2009, pp. 975–982. +- [18] A. Serna, B. Marcotegui, F. Goulette, J.-E. Deschaud, Paris-rue-Madame database: a 3D mobile laser scanner dataset for benchmarking urban detection, segmentation and classification methods, in: 4th International Conference on Pattern Recognition, Applications and Methods ICPRAM 2014, 2014. +- [19] B. Vallet, M. Br´edif, A. Serna, B. Marcotegui, N. Paparoditis, TerraMobilita/iQmulus urban point cloud analysis benchmark, Computers & Graphics 49 (2015) 126–133. +- [20] X. Roynard, J.-E. Deschaud, F. Goulette, Paris-Lille-3D: A large and highquality ground-truth urban point cloud dataset for automatic segmentation and classification, The International Journal of Robotics Research 37 (6) (2018) 545–557. +- [21] J. Behley, M. Garbade, A. Milioto, J. Quenzel, S. Behnke, C. Stachniss, J. Gall, SemanticKITTI: A dataset for semantic scene understanding of lidar sequences, in: Proceedings of the IEEE International Conference on Computer Vision, 2019, pp. 9297–9307. +- [22] W. Tan, N. Qin, L. Ma, Y. Li, J. Du, G. Cai, K. Yang, J. Li, Toronto-3D: A large-scale mobile lidar dataset for semantic segmentation of urban roadways, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, 2020, pp. 202–203. + +- [23] J. Niemeyer, F. Rottensteiner, U. Soergel, Contextual classification of lidar data and building object detection in urban areas, ISPRS journal of photogrammetry and remote sensing 87 (2014) 152–165. +- [24] S. Zolanvari, S. Ruano, A. Rana, A. Cummins, R. E. da Silva, M. Rahbar, A. Smolic, Dublincity: Annotated lidar point cloud and its applications, in: BMVC 30th British Machine Vision Conference, 2019. +- [25] N. Varney, V. K. Asari, Q. Graehling, Dales: A large-scale aerial lidar data set for semantic segmentation, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, 2020, pp. 186–187. +- [26] Z. Ye, Y. Xu, R. Huang, X. Tong, X. Li, X. Liu, K. Luan, L. Hoegner, U. Stilla, Lasdu: A large-scale aerial lidar dataset for semantic labeling in dense urban areas, ISPRS International Journal of Geo-Information 9 (7) (2020) 450. +- [27] G. J. Brostow, J. Fauqueur, R. Cipolla, Semantic object classes in video: A high-definition ground truth database, Pattern Recognition Letters 30 (2) (2009) 88–97. +- [28] H. Riemenschneider, A. B´odis-Szomor´u, J. Weissenberg, L. Van Gool, Learning where to classify in multi-view semantic segmentation, in: European Conference on Computer Vision, Springer, 2014, pp. 516–532. +- [29] X. Li, C. Li, Z. Tong, A. Lim, J. Yuan, Y. Wu, J. Tang, R. Huang, Campus3D: A photogrammetry point cloud benchmark for hierarchical understanding of outdoor scene, in: Proceedings of the 28th ACM International Conference on Multimedia, 2020, pp. 238–246. +- [30] Q. Hu, B. Yang, S. Khalid, W. Xiao, N. Trigoni, A. Markham, Towards semantic segmentation of urban-scale 3d point clouds: A dataset, benchmarks and challenges, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021, pp. 4977–4987. +- [31] G. Can, D. Mantegazza, G. Abbate, S. Chappuis, A. Giusti, Semantic segmentation on swiss3dcities: A benchmark study on aerial photogrammetric 3D pointcloud dataset, arXiv preprint arXiv:2012.12996 (2020). +- [32] D. Laupheimer, M. Shams Eddin, N. Haala, On the association of lidar point clouds and textured meshes for multi-modal semantic segmentation, ISPRS Annals of Photogrammetry, Remote Sensing & Spatial Information Sciences 5 (2) (2020). +- [33] M. K¨olle, D. Laupheimer, S. Schmohl, N. Haala, F. Rottensteiner, J. D. Wegner, H. Ledoux, The hessigheim 3d (h3d) benchmark on semantic segmentation of high-resolution 3d point clouds and textured meshes from + +- uav lidar and multi-view-stereo, ISPRS Open Journal of Photogrammetry and Remote Sensing 1 (2021) 100001. [doi:https://doi.org/10.1016/j.](https://doi.org/https://doi.org/10.1016/j.ophoto.2021.100001) [ophoto.2021.100001](https://doi.org/https://doi.org/10.1016/j.ophoto.2021.100001). +- [34] D. Girardeau-Montaut, CloudCompare, , accessed: 2021-01-16 (2016). +- [35] B. Foundation, Blender, , accessed: 2021-01- 16 (2002). +- [36] B. SYSTEMS, ContextCapture, [https://www.bentley.com/](https://www.bentley.com/zh/products/product-line/reality-modeling-software/contextcapture) [zh/products/product-line/reality-modeling-software/](https://www.bentley.com/zh/products/product-line/reality-modeling-software/contextcapture) [contextcapture](https://www.bentley.com/zh/products/product-line/reality-modeling-software/contextcapture), accessed: 2021-01-16 (2016). +- [37] KIGA-digi, The kalasatama digital twins project - the final report of the kira-digi pilot project, [https://www.hel.fi/hel2/tietokeskus/data/](https://www.hel.fi/hel2/tietokeskus/data/helsinki/kaupunginkanslia/3D-malli/Helsinki3D_Kalasatama_Digital_Twins_020519.pdf) [helsinki/kaupunginkanslia/3D-malli/Helsinki3D\\_Kalasatama\\_](https://www.hel.fi/hel2/tietokeskus/data/helsinki/kaupunginkanslia/3D-malli/Helsinki3D_Kalasatama_Digital_Twins_020519.pdf) [Digital\\_Twins\\_020519.pdf](https://www.hel.fi/hel2/tietokeskus/data/helsinki/kaupunginkanslia/3D-malli/Helsinki3D_Kalasatama_Digital_Twins_020519.pdf), accessed: 2020-11-25 (May 2019). +- [38] F. Lafarge, C. Mallet, Creating large-scale city models from 3d-point clouds: a robust approach with hybrid representation, International journal of computer vision 99 (1) (2012) 69–85. +- [39] T. Hackel, J. D. Wegner, K. Schindler, Fast semantic segmentation of 3D point clouds with strongly varying density, ISPRS annals of the photogrammetry, remote sensing and spatial information sciences 3 (2016) 177–184. +- [40] K. F. West, B. N. Webb, J. R. Lersch, S. Pothier, J. M. Triscari, A. E. Iverson, Context-driven automated target detection in 3d data, in: Automatic Target Recognition XIV, Vol. 5426, International Society for Optics and Photonics, 2004, pp. 133–143. +- [41] M. Weinmann, B. Jutzi, C. Mallet, Feature relevance assessment for the semantic interpretation of 3D point cloud data, ISPRS Annals of the Photogrammetry, Remote Sensing and Spatial Information Sciences 5 (W2) (2013) 1. +- [42] Y. Verdie, F. Lafarge, P. Alliez, LOD generation for urban scenes, ACM Transactions on Graphics 34 (3) (2015) 1–14. [doi:10.1145/2732527](https://doi.org/10.1145/2732527). +- [43] M. Rouhani, F. Lafarge, P. Alliez, Semantic segmentation of 3D textured meshes for urban scene analysis, ISPRS Journal of Photogrammetry and Remote Sensing 123 (2017) 124–139. [doi:10.1016/j.isprsjprs.2016.](https://doi.org/10.1016/j.isprsjprs.2016.12.001) [12.001](https://doi.org/10.1016/j.isprsjprs.2016.12.001). +- [44] J. Ma, S. W. Bae, S. Choi, 3D medial axis point approximation using nearest neighbors and the normal field, The Visual Computer 28 (1) (2012) 7–19. + +- [45] R. Peters, H. Ledoux, Robust approximation of the medial axis transform of lidar point clouds as a tool for visualisation, Computers & Geosciences 90 (2016) 123–133. +- [46] T. McKinnon, P. Hoff, Comparing rgb-based vegetation indices with NDVI for drone based agricultural sensing, Agribotix. Com 21 (17) (2017) 1–8. +- [47] The CGAL Project, [CGAL User and Reference Manual,](https://doc.cgal.org/5.1.1/Manual/packages.html) 5.1.1 Edition, CGAL Editorial Board, 2020. URL +- [48] L. Nan, Easy3d: a lightweight, easy-to-use, and efficient c++ library for processing and rendering 3D data, [https://github.com/LiangliangNan/](https://github.com/LiangliangNan/Easy3D) [Easy3D](https://github.com/LiangliangNan/Easy3D) (2018). +- [49] S. Walk, Ethz random forest, [https://prs.igp.ethz.ch/research/](https://prs.igp.ethz.ch/research/Source_code_and_datasets.html) [Source\\_code\\_and\\_datasets.html](https://prs.igp.ethz.ch/research/Source_code_and_datasets.html), accessed: 2020-11-25 (2014). +- [50] P. Cignoni, C. Rocchini, R. Scopigno, Metro: measuring error on simplified surfaces, in: Computer graphics forum, Vol. 17, Wiley Online Library, 1998, pp. 167–174. +- [51] M. Corsini, P. Cignoni, R. Scopigno, Efficient and flexible sampling with blue noise properties of triangular meshes, IEEE transactions on visualization and computer graphics 18 (6) (2012) 914–924. +- [52] C. R. Qi, L. Yi, H. Su, L. J. Guibas, Pointnet++: Deep hierarchical feature learning on point sets in a metric space, Advances in neural information processing systems 30 (2017) 5099–5108. +- [53] Q. Hu, B. Yang, L. Xie, S. Rosa, Y. Guo, Z. Wang, N. Trigoni, A. Markham, Randla-net: Efficient semantic segmentation of large-scale point clouds, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020, pp. 11108–11117. \ No newline at end of file diff --git a/docs/papers/md/SUM_2021_dataset_images/_page_12_Figure_0.jpeg b/docs/papers/md/SUM_2021_dataset_images/_page_12_Figure_0.jpeg new file mode 100644 index 0000000..109ef2b Binary files /dev/null and b/docs/papers/md/SUM_2021_dataset_images/_page_12_Figure_0.jpeg differ diff --git a/docs/papers/md/SUM_2021_dataset_images/_page_13_Figure_0.jpeg b/docs/papers/md/SUM_2021_dataset_images/_page_13_Figure_0.jpeg new file mode 100644 index 0000000..8a7ce2e Binary files /dev/null and b/docs/papers/md/SUM_2021_dataset_images/_page_13_Figure_0.jpeg differ diff --git a/docs/papers/md/SUM_2021_dataset_images/_page_13_Figure_2.jpeg b/docs/papers/md/SUM_2021_dataset_images/_page_13_Figure_2.jpeg new file mode 100644 index 0000000..0d4eb0f Binary files /dev/null and b/docs/papers/md/SUM_2021_dataset_images/_page_13_Figure_2.jpeg differ diff --git a/docs/papers/md/SUM_2021_dataset_images/_page_13_Figure_4.jpeg b/docs/papers/md/SUM_2021_dataset_images/_page_13_Figure_4.jpeg new file mode 100644 index 0000000..e99ef2f Binary files /dev/null and b/docs/papers/md/SUM_2021_dataset_images/_page_13_Figure_4.jpeg differ diff --git a/docs/papers/md/SUM_2021_dataset_images/_page_14_Figure_0.jpeg b/docs/papers/md/SUM_2021_dataset_images/_page_14_Figure_0.jpeg new file mode 100644 index 0000000..fded551 Binary files /dev/null and b/docs/papers/md/SUM_2021_dataset_images/_page_14_Figure_0.jpeg differ diff --git a/docs/papers/md/SUM_2021_dataset_images/_page_15_Figure_0.jpeg b/docs/papers/md/SUM_2021_dataset_images/_page_15_Figure_0.jpeg new file mode 100644 index 0000000..a631076 Binary files /dev/null and b/docs/papers/md/SUM_2021_dataset_images/_page_15_Figure_0.jpeg differ diff --git a/docs/papers/md/SUM_2021_dataset_images/_page_18_Figure_0.jpeg b/docs/papers/md/SUM_2021_dataset_images/_page_18_Figure_0.jpeg new file mode 100644 index 0000000..e67ab8b Binary files /dev/null and b/docs/papers/md/SUM_2021_dataset_images/_page_18_Figure_0.jpeg differ diff --git a/docs/papers/md/SUM_2021_dataset_images/_page_19_Figure_0.jpeg b/docs/papers/md/SUM_2021_dataset_images/_page_19_Figure_0.jpeg new file mode 100644 index 0000000..b23b4e1 Binary files /dev/null and b/docs/papers/md/SUM_2021_dataset_images/_page_19_Figure_0.jpeg differ diff --git a/docs/papers/md/SUM_2021_dataset_images/_page_1_Picture_2.jpeg b/docs/papers/md/SUM_2021_dataset_images/_page_1_Picture_2.jpeg new file mode 100644 index 0000000..0362b01 Binary files /dev/null and b/docs/papers/md/SUM_2021_dataset_images/_page_1_Picture_2.jpeg differ diff --git a/docs/papers/md/SUM_2021_dataset_images/_page_21_Figure_0.jpeg b/docs/papers/md/SUM_2021_dataset_images/_page_21_Figure_0.jpeg new file mode 100644 index 0000000..941a748 Binary files /dev/null and b/docs/papers/md/SUM_2021_dataset_images/_page_21_Figure_0.jpeg differ diff --git a/docs/papers/md/SUM_2021_dataset_images/_page_2_Figure_0.jpeg b/docs/papers/md/SUM_2021_dataset_images/_page_2_Figure_0.jpeg new file mode 100644 index 0000000..1adecb2 Binary files /dev/null and b/docs/papers/md/SUM_2021_dataset_images/_page_2_Figure_0.jpeg differ diff --git a/docs/papers/md/SUM_2021_dataset_images/_page_8_Figure_0.jpeg b/docs/papers/md/SUM_2021_dataset_images/_page_8_Figure_0.jpeg new file mode 100644 index 0000000..2b9d1e3 Binary files /dev/null and b/docs/papers/md/SUM_2021_dataset_images/_page_8_Figure_0.jpeg differ diff --git a/docs/papers/md/SUM_2021_dataset_images/_page_8_Figure_2.jpeg b/docs/papers/md/SUM_2021_dataset_images/_page_8_Figure_2.jpeg new file mode 100644 index 0000000..254facc Binary files /dev/null and b/docs/papers/md/SUM_2021_dataset_images/_page_8_Figure_2.jpeg differ diff --git a/docs/papers/md/SUM_2021_dataset_images/_page_9_Figure_0.jpeg b/docs/papers/md/SUM_2021_dataset_images/_page_9_Figure_0.jpeg new file mode 100644 index 0000000..a858c9a Binary files /dev/null and b/docs/papers/md/SUM_2021_dataset_images/_page_9_Figure_0.jpeg differ diff --git a/docs/sum-parts-explained.html b/docs/sum-parts-explained.html new file mode 100644 index 0000000..2597957 --- /dev/null +++ b/docs/sum-parts-explained.html @@ -0,0 +1,965 @@ +SUM Parts 해설 + + + + + + +
+ +
해설 · 2026-08-24
+

SUM Parts 해설

+

+ 도시 3D 메시를 부분 단위로 분할하는 벤치마크가 무엇이고, + 우리 과제에 어떻게 쓰이는가. +

+ +
+
+ + + + + + + + +
+
+ + +
+ +

한 줄로

+ +

+ SUM Parts는 데이터셋이다. 모델이 아니다. + 새 신경망을 만든 논문이 아니라, 기존 모델들을 채점할 새 시험지를 만든 논문이다. + CVPR 2025, TU Delft. +

+ +
+ 가장 흔한 오해 +

+ "SUM Parts를 쓴다" = 데이터로 모델을 학습시킨다는 뜻이다. + 실제로 예측을 내놓는 건 PointVector 같은 별개 모델이고, + 그건 다른 팀이 2023년에 발표한 것이다. +

+
+ +

먼저 알아야 할 용어

+ +
+
메시 (mesh)
+
삼각형 면을 이어 붙여 만든 3D 표면. 드론 사진측량 결과물이 보통 이 형태다. + 각 면에 사진 텍스처가 입혀진다.
+ +
포인트 클라우드
+
면 없이 점만 있는 3D 데이터. 신경망 대부분이 이걸 입력으로 받는다. + 메시를 쓰려면 표면에서 점을 뽑아 변환한다.
+ +
시맨틱
세그멘테이션
+
모든 점(또는 픽셀)에 "이건 건물, 이건 나무" 하고 이름을 붙이는 일. + 물체 하나를 통째로 찾는 객체 검출과 다르다.
+ +
part-level
+
SUM Parts의 핵심. 건물을 통으로 "건물"이라 하지 않고 + 외벽·지붕·굴뚝·발코니로 쪼갠다. 이전 SUM(2021)은 객체 단위였다.
+ +
IoU
+
정답과 예측이 겹치는 정도. 겹친 것 ÷ 합친 것. + 클래스별로 구하고 평균 낸 것이 mIoU다.
+
+ +

무엇을 제공하나

+ +
+ + + + + + + + + + +
제공내용
✓ 데이터셋2.5 km² 도시 텍스처 메시, 21개 클래스, 라벨 완비
✓ 벤치마크 점수표기존 모델 13종을 태워본 결과 — 어느 모델이 좋은지
✓ 학습 코드PointNeXt 번들 (모델 구현 + 학습 스크립트)
✗ 학습된 가중치없다. 직접 학습해야 한다
✗ 어노테이션 도구비공개. 저자가 유료 서비스로 판매 중
✗ 채점 스크립트TODO. test 세트 채점은 저자에게 요청
+
+ +
+ + + + + + + + + + + + + + + + + + + + + + + + +
+ + diff --git a/docs/walkthrough.md b/docs/walkthrough.md new file mode 100644 index 0000000..8c0488f --- /dev/null +++ b/docs/walkthrough.md @@ -0,0 +1,55 @@ +# 논문 마크다운 파일 이미지 Vision 분석 기반 해설 주석 작업 완료 보고서 + +6개 논문 마크다운 파일에 포함된 **총 86개 이미지**에 대해 Vision 모델을 활용한 시각 정보 분석을 수행하고, `ANNOTATE_PROMPT.md` 가이드라인에 맞춘 한국어 상세 해설(`> **[그림 해설]** ...`) 주석 삽입을 100% 완료하였습니다. + +--- + +## 1. 파일별 작업 현황 및 검증 결과 + +| 논문 파일명 | 이미지 수 | 삽입된 주석 수 | 검증 상태 | +| :--- | :---: | :---: | :---: | +| [PointNet_1612.00593.md](file:///d:/MYCLAUDE_PROJECT/sum-parts-test/docs/papers/md/PointNet_1612.00593.md) | 24 | 24 | **완료 (100%)** | +| [PointNet++_1706.02413.md](file:///d:/MYCLAUDE_PROJECT/sum-parts-test/docs/papers/md/PointNet++_1706.02413.md) | 10 | 10 | **완료 (100%)** | +| [PointNeXt_2206.04670.md](file:///d:/MYCLAUDE_PROJECT/sum-parts-test/docs/papers/md/PointNeXt_2206.04670.md) | 5 | 5 | **완료 (100%)** | +| [PointVector_2205.10528.md](file:///d:/MYCLAUDE_PROJECT/sum-parts-test/docs/papers/md/PointVector_2205.10528.md) | 8 | 8 | **완료 (100%)** | +| [SUM_2021_dataset.md](file:///d:/MYCLAUDE_PROJECT/sum-parts-test/docs/papers/md/SUM_2021_dataset.md) | 14 | 14 | **완료 (100%)** | +| [SUM-Parts_2503.15300.md](file:///d:/MYCLAUDE_PROJECT/sum-parts-test/docs/papers/md/SUM-Parts_2503.15300.md) | 25 | 25 | **완료 (100%)** | +| **합계 (Total)** | **86** | **86** | **100% 일치** | + +--- + +## 2. 주요 주석 작성 내용 요약 + +### 1) PointNet (`PointNet_1612.00593.md`, 24개) +- **아키텍처/데이터 흐름**: $N \times 3$ 입력 포인트 $\to$ T-Net(3x3 정렬) $\to$ $64$-dim MLP $\to$ Feature Transform(64x64 정렬) $\to$ $1024$-dim MLP $\to$ Max Pooling 전역 특징 $\to$ Classification Head ($k$개 클래스 점수) 및 Segmentation Head ($N \times m$ 점별 레이블 점수) 연결 구조 상세 서술. +- **정량 그래프 및 표**: 포인트 누락(Point Drop) 강건성, 가우시안 노이즈/아웃라이어 저항성, 시간 및 공간 복잡도(FLOPs/파라미터 수) 차트 수치 복원. +- **정성 시각화**: Semantic3D, ShapeNet Part 세그멘테이션 결과 및 Critical Points(핵심 골격점) / Upper Bound Shapes(상한 바운딩 형상) 시각화 분석. + +### 2) PointNet++ (`PointNet++_1706.02413.md`, 10개) +- **핵심 모듈**: Set Abstraction 계층(Sampling $\to$ Grouping $\to$ PointNet), MSG(Multi-scale grouping) 및 MRG(Multi-resolution grouping) 밀도 적응형 아키텍처. +- **업샘플링 & 디코더**: Feature Propagation(거리 역가중치 $k$-NN 보간 + Skip connection 결합 + Unit PointNet). +- **실험 및 정성 비교**: ScanNet 실내 씬 3D 시맨틱 세그멘테이션, 비유클리드 SHREC 다양체 메시(포즈 변형 말/사람 모델) 등 시각적 비교 분석. + +### 3) PointNeXt (`PointNeXt_2206.04670.md`, 5개) +- **현대화 아키텍처**: PointNet++ 기반 모델 스케일링, 인버티드 잔차 MLP 블록(InvResMLP, Inverted Residual with Depthwise/Pointwise Separable MLP), 대칭적 U-Net 인코더-디코더 계층 구조. +- **정성 결과**: S3DIS 6-fold 교차검증 실내 씬(Area 5 등) 및 ShapeNetPart 16개 카테고리 3D 객체 파트 분할 결과 시각 분석. + +### 4) PointVector (`PointVector_2205.10528.md`, 8개) +- **이방성(Anisotropy) 벡터 표현**: 스칼라 특징에 회전각($\alpha, \beta$)을 부여하여 $c \times 3$ 차원의 3D 공간 벡터로 확장하는 VPSA(Vector-oriented Point Set Abstraction) 모듈 및 2단계 3차원 회전 기하($Rot_z Rot_x$) 흐름 상세 설명. +- **구조 및 씬 비교**: PointVector 분류/세그멘테이션 U-Net 구조 및 S3DIS 실내 시맨틱 분할에서 PointNeXt 대비 돌출 기둥/복도 빔/출입구 코너 오분류 개선 정성 분석. + +### 5) SUM 2021 Dataset (`SUM_2021_dataset.md`, 14개) +- **헬싱키 $4\,\text{km}^2$ 3D 도시 텍스처 메시**: 6개 클래스(Terrain, Building, Water, High vegetation, Vehicle, Boat) 시맨틱 벤치마크. +- **반자동 주석 파이프라인**: 과분할(Over-segmentation) $\to$ Random Forest 초기 분류 $\to$ 전용 3D GUI 도구(UrbanMeshAnnotator)에서의 라쏘/스트로크/평면 추출 기반 정제 워크플로우. +- **포인트 샘플링 및 데이터 분석**: 몬테카를로/포아송 디스크 샘플링($10\,\text{pts}/\text{m}^2$) 및 훈련 데이터량(10% 영역만으로 66% mIoU 달성) 민감도 분석. + +### 6) SUM-Parts (`SUM-Parts_2503.15300.md`, 25개) +- **부품 레벨(Part-level) 시맨틱 메시**: 13개 클래스 페이스 트랙 및 19/21개 클래스 텍스처(픽셀) 트랙(창문, 출입문, 굴뚝, 차선 마킹, 보도, 잔디밭 등) 정밀 분할. +- **대화형 주석 알고리즘**: 메시 내부 수축 구(Interior shrinking ball) 반경 계산, 3D/2D 구조 인식 템플릿 일괄 매칭, 슈퍼픽셀 국소 확장 및 GrabCut 그래프 컷 정밀화. +- **정량 및 정성 비교**: 11개 3D 딥러닝 모델(PointVector, PointTransV3, PointNext, KPConv 등)의 페이스/픽셀 트랙 비교 및 오류 맵(빨간색 표시) 정밀 분석. + +--- + +## 3. 원본 문서 무결성 준수 +- 원본 캡션 텍스트, 본문 수식/인용 링크, 마크다운 앵커 태그(``)를 단 하나도 손상시키지 않고 오직 캡션 직하단에 `> **[그림 해설]**` 인용 블록만을 정확히 삽입하였습니다. +- LLM이 마크다운 파일 전체를 읽을 때 텍스트와 이미지 시각 정보를 온전히 학습할 수 있도록 모든 수치, 텐서 차원, 모델 명칭, 색상 범례를 구체적으로 수록하였습니다. diff --git a/scripts/convert_papers.sh b/scripts/convert_papers.sh new file mode 100644 index 0000000..5a9f713 --- /dev/null +++ b/scripts/convert_papers.sh @@ -0,0 +1,87 @@ +#!/usr/bin/env bash +# Convert the reference papers to Markdown with the local doc2md tool +# +# Two things had to line up to make this work: +# +# 1. Call it from bash, not PowerShell. PowerShell prepends a BOM when piping +# to a native process, has no `<` redirection, and PS 5.1 lacks +# StandardInputEncoding to turn the BOM off. (main.py now reads +# utf-8-sig, so the BOM is tolerated - but bash avoids the issue entirely.) +# +# 2. The tool's venv had a CPU-only torch (2.12.0+cpu). marker-pdf picks its +# device from torch.cuda.is_available(), so it silently ran on CPU and +# stalled. Replaced with 2.5.1+cu121; no code change was needed. +# +# Runs one paper at a time - they share a single GPU. +set -uo pipefail + +TOOL="/d/Teknom/jjangoo/tools/doc.convert.doc2md" +PAPERS="/d/MYCLAUDE_PROJECT/sum-parts-test/docs/papers" +OUT="$PAPERS/md" +WIN_PAPERS="D:/MYCLAUDE_PROJECT/sum-parts-test/docs/papers" +WIN_OUT="D:/MYCLAUDE_PROJECT/sum-parts-test/docs/papers/md" + +mkdir -p "$OUT" +cd "$TOOL" || exit 1 + +for pdf in "$PAPERS"/*.pdf; do + name=$(basename "$pdf" .pdf) + + if [ -f "$OUT/$name.md" ]; then + echo "=== $name (이미 변환됨, 건너뜀) ===" + continue + fi + + echo "=== $name ===" + printf '%s' "{\"file\":\"$WIN_PAPERS/$name.pdf\",\"outputDir\":\"$WIN_OUT\"}" > /tmp/doc2md_req.json + + start=$(date +%s) + TORCH_DEVICE=cuda ./.venv/Scripts/python.exe main.py < /tmp/doc2md_req.json \ + > "/tmp/doc2md_$name.out" 2>"/tmp/doc2md_$name.err" + rc=$? + elapsed=$(( $(date +%s) - start )) + + if [ $rc -ne 0 ]; then + echo " FAILED rc=$rc (${elapsed}s)" + tail -5 "/tmp/doc2md_$name.err" + continue + fi + + # the result line is JSON on stdout + python3 - "/tmp/doc2md_$name.out" "$elapsed" <<'PY' +import json, sys +from pathlib import Path + +txt = Path(sys.argv[1]).read_text(encoding="utf-8", errors="replace") +elapsed = sys.argv[2] +for line in txt.splitlines(): + line = line.strip() + if not line.startswith("{"): + continue + try: + d = json.loads(line) + except Exception: + continue + if d.get("type") == "result": + o = d["output"] + md = Path(o["file"]) + size = md.stat().st_size / 1024 if md.exists() else 0 + print(f" ok {elapsed}s {size:,.0f} KB " + f"images={len(o.get('images') or [])} " + f"pages={len(o.get('pages') or [])} " + f"diagrams={o.get('hasDiagrams')}") + elif d.get("type") == "error": + print(f" ERROR {d.get('code')}: {d.get('message')}") +PY +done + +echo +echo "=== 결과 ===" +for md in "$OUT"/*.md; do + [ -f "$md" ] || continue + n=$(basename "$md") + kb=$(( $(stat -c%s "$md") / 1024 )) + heads=$(grep -cE '^#{1,3} ' "$md") + rows=$(grep -c '^|' "$md") + echo " $(printf '%-34s' "$n") ${kb}KB 헤딩 ${heads} 표행 ${rows}" +done