sum-parts-test/docs/papers/md/PointNet_1612.00593.md

PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation

Charles R. Qi* Hao Su* Kaichun Mo Leonidas J. Guibas Stanford University

Abstract

Point cloud is an important type of geometric data structure. Due to its irregular format, most researchers transform such data to regular 3D voxel grids or collections of images. This, however, renders data unnecessarily voluminous and causes issues. In this paper, we design a novel type of neural network that directly consumes point clouds, which well respects the permutation invariance of points in the input. Our network, named PointNet, provides a unified architecture for applications ranging from object classification, part segmentation, to scene semantic parsing. Though simple, PointNet is highly efficient and effective. Empirically, it shows strong performance on par or even better than state of the art. Theoretically, we provide analysis towards understanding of what the network has learnt and why the network is robust with respect to input perturbation and corruption.

1. Introduction

In this paper we explore deep learning architectures capable of reasoning about 3D geometric data such as point clouds or meshes. Typical convolutional architectures require highly regular input data formats, like those of image grids or 3D voxels, in order to perform weight sharing and other kernel optimizations. Since point clouds or meshes are not in a regular format, most researchers typically transform such data to regular 3D voxel grids or collections of images (e.g, views) before feeding them to a deep net architecture. This data representation transformation, however, renders the resulting data unnecessarily voluminous — while also introducing quantization artifacts that can obscure natural invariances of the data.

For this reason we focus on a different input representation for 3D geometry using simply point clouds – and name our resulting deep nets PointNets. Point clouds are simple and unified structures that avoid the combinatorial irregularities and complexities of meshes, and thus are easier to learn from. The PointNet, however,

Figure 1. Applications of PointNet. We propose a novel deep net architecture that consumes raw point cloud (set of points) without voxelization or rendering. It is a unified architecture that learns both global and local point features, providing a simple, efficient and effective approach for a number of 3D recognition tasks.

[그림 해설] PointNet이 다루는 세 가지 핵심 3D 태스크를 요약한 다이어그램.

• Classification(분류): 컵(mug), 테이블(table), 자동차(car) 등 단일 3D 포인트 클라우드 입력을 받아 객체의 전체 클래스 레이블을 판별.
• Part Segmentation(부품 분할): 램프, 비행기(동체·날개·엔진), 탁자(상판·다리) 등 단일 객체 내 각 부품 영역을 포인트 단위로 분할하여 색상별로 구분.
• Semantic Segmentation(시맨틱 분할): 실내 공간 전체 포인트 클라우드에서 바닥, 벽, 의자, 테이블 등 복합 환경의 각 구성 요소를 포인트별 시맨틱 클래스로 분류.

still has to respect the fact that a point cloud is just a set of points and therefore invariant to permutations of its members, necessitating certain symmetrizations in the net computation. Further invariances to rigid motions also need to be considered.

Our PointNet is a unified architecture that directly takes point clouds as input and outputs either class labels for the entire input or per point segment/part labels for each point of the input. The basic architecture of our network is surprisingly simple as in the initial stages each point is processed identically and independently. In the basic setting each point is represented by just its three coordinates (x, y, z). Additional dimensions may be added by computing normals and other local or global features.

Key to our approach is the use of a single symmetric function, max pooling. Effectively the network learns a set of optimization functions/criteria that select interesting or informative points of the point cloud and encode the reason for their selection. The final fully connected layers of the network aggregate these learnt optimal values into the global descriptor for the entire shape as mentioned above (shape classification) or are used to predict per point labels (shape segmentation).

Our input format is easy to apply rigid or affine transformations to, as each point transforms independently. Thus we can add a data-dependent spatial transformer network that attempts to canonicalize the data before the PointNet processes them, so as to further improve the results.

^* indicates equal contributions.

We provide both a theoretical analysis and an experimental evaluation of our approach. We show that our network can approximate any set function that is continuous. More interestingly, it turns out that our network learns to summarize an input point cloud by a sparse set of key points, which roughly corresponds to the skeleton of objects according to visualization. The theoretical analysis provides an understanding why our PointNet is highly robust to small perturbation of input points as well as to corruption through point insertion (outliers) or deletion (missing data).

On a number of benchmark datasets ranging from shape classification, part segmentation to scene segmentation, we experimentally compare our PointNet with state-ofthe-art approaches based upon multi-view and volumetric representations. Under a unified architecture, not only is our PointNet much faster in speed, but it also exhibits strong performance on par or even better than state of the art.

The key contributions of our work are as follows:

• We design a novel deep net architecture suitable for consuming unordered point sets in 3D;
• We show how such a net can be trained to perform 3D shape classification, shape part segmentation and scene semantic parsing tasks;
• We provide thorough empirical and theoretical analysis on the stability and efficiency of our method;
• We illustrate the 3D features computed by the selected neurons in the net and develop intuitive explanations for its performance.

The problem of processing unordered sets by neural nets is a very general and fundamental problem – we expect that our ideas can be transferred to other domains as well.

2. Related Work

Point Cloud Features Most existing features for point cloud are handcrafted towards specific tasks. Point features often encode certain statistical properties of points and are designed to be invariant to certain transformations, which are typically classified as intrinsic [2, 24, 3] or extrinsic [20, 19, 14, 10, 5]. They can also be categorized as local features and global features. For a specific task, it is not trivial to find the optimal feature combination.

Deep Learning on 3D Data 3D data has multiple popular representations, leading to various approaches for learning. Volumetric CNNs: [28, 17, 18] are the pioneers applying 3D convolutional neural networks on voxelized shapes. However, volumetric representation is constrained by its resolution due to data sparsity and computation cost of 3D convolution. FPNN [13] and Vote3D [26] proposed special methods to deal with the sparsity problem; however, their operations are still on sparse volumes, it's challenging for them to process very large point clouds. Multiview CNNs: [23, 18] have tried to render 3D point cloud or shapes into 2D images and then apply 2D conv nets to classify them. With well engineered image CNNs, this line of methods have achieved dominating performance on shape classification and retrieval tasks [21]. However, it's nontrivial to extend them to scene understanding or other 3D tasks such as point classification and shape completion. Spectral CNNs: Some latest works [4, 16] use spectral CNNs on meshes. However, these methods are currently constrained on manifold meshes such as organic objects and it's not obvious how to extend them to non-isometric shapes such as furniture. Feature-based DNNs: [6, 8] firstly convert the 3D data into a vector, by extracting traditional shape features and then use a fully connected net to classify the shape. We think they are constrained by the representation power of the features extracted.

Deep Learning on Unordered Sets From a data structure point of view, a point cloud is an unordered set of vectors. While most works in deep learning focus on regular input representations like sequences (in speech and language processing), images and volumes (video or 3D data), not much work has been done in deep learning on point sets.

One recent work from Oriol Vinyals et al [25] looks into this problem. They use a read-process-write network with attention mechanism to consume unordered input sets and show that their network has the ability to sort numbers. However, since their work focuses on generic sets and NLP applications, there lacks the role of geometry in the sets.

3. Problem Statement

We design a deep learning framework that directly consumes unordered point sets as inputs. A point cloud is represented as a set of 3D points {Pⁱ | i = 1, ..., n}, where each point Pⁱ is a vector of its (x, y, z) coordinate plus extra feature channels such as color, normal etc. For simplicity and clarity, unless otherwise noted, we only use the (x, y, z) coordinate as our point's channels.

For the object classification task, the input point cloud is either directly sampled from a shape or pre-segmented from a scene point cloud. Our proposed deep network outputs k scores for all the k candidate classes. For semantic segmentation, the input can be a single object for part region segmentation, or a sub-volume from a 3D scene for object region segmentation. Our model will output n × m scores for each of the n points and each of the m semantic subcategories.

[그림 해설] 'Classification Network' 텍스트 배너. 아래 Figure 2 다이어그램의 상단 파란색 블록에 해당하는 객체 분류 네트워크 영역을 지칭한다.

Figure 2. PointNet Architecture. The classification network takes n points as input, applies input and feature transformations, and then aggregates point features by max pooling. The output is classification scores for k classes. The segmentation network is an extension to the classification net. It concatenates global and local features and outputs per point scores. "mlp" stands for multi-layer perceptron, numbers in bracket are layer sizes. Batchnorm is used for all layers with ReLU. Dropout layers are used for the last mlp in classification net.

[그림 해설] PointNet의 전체 신경망 구조도.

• Classification Network (상단 파란색 영역):
  • 입력: n \times 3 점 좌표.
  • Input Transform: T-Net을 통해 3 \times 3 변환 행렬을 예측하여 입력 좌표 공간을 정렬 \to n \times 3.
  • Shared MLP (64, 64): 각 점을 독립적으로 $64$차원 공간으로 매핑 \to n \times 64.
  • Feature Transform: T-Net을 통해 64 \times 64 특징 변환 행렬을 예측하고 직교 정규화(L_{reg})를 적용하여 특징 공간 정렬 \to n \times 64.
  • Shared MLP (64, 128, 1024): 점별 특징을 1024차원으로 확장 \to n \times 1024.
  • Max Pool: 대칭 함수(Symmetric Function)로 점 순서 불변성을 확보하며 전체 $n$개 점의 최댓값을 집계 $\to 1024$차원 Global Feature 생성.
  • MLP (512, 256, k): 완전연결 레이어와 Dropout을 거쳐 $k$개 클래스 분류 점수(output scores) 출력.
• Segmentation Network (하단 노란색 영역):
  • $n \times 64$의 로컬 점 특징과 1024차원의 글로벌 특징 벡터를 결합(concatenation)하여 n \times 1088 크기의 통합 특징 구성.
  • Shared MLP (512, 256, 128) \to n \times 128.
  • Shared MLP (128, m) \to n \times m 출력 점수를 계산하여 $n$개의 각 점마다 $m$개 시맨틱/부품 범주 점수를 도출.

4. Deep Learning on Point Sets

The architecture of our network (Sec 4.2) is inspired by the properties of point sets in \mathbb{R}^n (Sec 4.1).

4.1. Properties of Point Sets in \mathbb{R}^n

Our input is a subset of points from an Euclidean space. It has three main properties:

• Unordered. Unlike pixel arrays in images or voxel arrays in volumetric grids, point cloud is a set of points without specific order. In other words, a network that consumes N 3D point sets needs to be invariant to N! permutations of the input set in data feeding order.
• Interaction among points. The points are from a space with a distance metric. It means that points are not isolated, and neighboring points form a meaningful subset. Therefore, the model needs to be able to capture local structures from nearby points, and the combinatorial interactions among local structures.
• Invariance under transformations. As a geometric object, the learned representation of the point set should be invariant to certain transformations. For example, rotating and translating points all together should not modify the global point cloud category nor the segmentation of the points.

4.2. PointNet Architecture

Our full network architecture is visualized in Fig 2, where the classification network and the segmentation network share a great portion of structures. Please read the caption of Fig 2 for the pipeline.

Our network has three key modules: the max pooling layer as a symmetric function to aggregate information from

all the points, a local and global information combination structure, and two joint alignment networks that align both input points and point features.

We will discuss our reason behind these design choices in separate paragraphs below.

Symmetry Function for Unordered Input In order to make a model invariant to input permutation, three strategies exist: 1) sort input into a canonical order; 2) treat the input as a sequence to train an RNN, but augment the training data by all kinds of permutations; 3) use a simple symmetric function to aggregate the information from each point. Here, a symmetric function takes n vectors as input and outputs a new vector that is invariant to the input order. For example, + and * operators are symmetric binary functions.

While sorting sounds like a simple solution, in high dimensional space there in fact does not exist an ordering that is stable w.r.t. point perturbations in the general sense. This can be easily shown by contradiction. If such an ordering strategy exists, it defines a bijection map between a high-dimensional space and a 1d real line. It is not hard to see, to require an ordering to be stable w.r.t point perturbations is equivalent to requiring that this map preserves spatial proximity as the dimension reduces, a task that cannot be achieved in the general case. Therefore, sorting does not fully resolve the ordering issue, and it's hard for a network to learn a consistent mapping from input to output as the ordering issue persists. As shown in experiments (Fig 5), we find that applying a MLP directly on the sorted point set performs poorly, though slightly better than directly processing an unsorted input.

The idea to use RNN considers the point set as a sequential signal and hopes that by training the RNN

with randomly permuted sequences, the RNN will become invariant to input order. However in "OrderMatters" [25] the authors have shown that order does matter and cannot be totally omitted. While RNN has relatively good robustness to input ordering for sequences with small length (dozens), it's hard to scale to thousands of input elements, which is the common size for point sets. Empirically, we have also shown that model based on RNN does not perform as well as our proposed method (Fig 5).

Our idea is to approximate a general function defined on a point set by applying a symmetric function on transformed elements in the set:

f(\lbrace x_1, \dots, x_n \rbrace) \approx g(h(x_1), \dots, h(x_n)), \tag{1}

where

f: 2^{\mathbb{R}^N} \to \mathbb{R}

, h: \mathbb{R}^N \to \mathbb{R}^K and g: \mathbb{R}^K \times \cdots \times \mathbb{R}^K \to \mathbb{R} is a symmetric function.

Empirically, our basic module is very simple: we approximate h by a multi-layer perceptron network and g by a composition of a single variable function and a max pooling function. This is found to work well by experiments. Through a collection of h, we can learn a number of f's to capture different properties of the set.

While our key module seems simple, it has interesting properties (see Sec 5.3) and can achieve strong performace (see Sec 5.1) in a few different applications. Due to the simplicity of our module, we are also able to provide theoretical analysis as in Sec 4.3.

Local and Global Information Aggregation The output from the above section forms a vector [f_1,\ldots,f_K] , which is a global signature of the input set. We can easily train a SVM or multi-layer perceptron classifier on the shape global features for classification. However, point segmentation requires a combination of local and global knowledge. We can achieve this by a simple yet highly effective manner.

Our solution can be seen in Fig 2 (Segmentation Network). After computing the global point cloud feature vector, we feed it back to per point features by concatenating the global feature with each of the point features. Then we extract new per point features based on the combined point features - this time the per point feature is aware of both the local and global information.

With this modification our network is able to predict per point quantities that rely on both local geometry and global semantics. For example we can accurately predict per-point normals (fig in supplementary), validating that the network is able to summarize information from the point's local neighborhood. In experiment session, we also show that our model can achieve state-of-the-art performance on shape part segmentation and scene segmentation. Joint Alignment Network The semantic labeling of a point cloud has to be invariant if the point cloud undergoes certain geometric transformations, such as rigid transformation. We therefore expect that the learnt representation by our point set is invariant to these transformations.

A natural solution is to align all input set to a canonical space before feature extraction. Jaderberg et al. [9] introduces the idea of spatial transformer to align 2D images through sampling and interpolation, achieved by a specifically tailored layer implemented on GPU.

Our input form of point clouds allows us to achieve this goal in a much simpler way compared with [9]. We do not need to invent any new layers and no alias is introduced as in the image case. We predict an affine transformation matrix by a mini-network (T-net in Fig 2) and directly apply this transformation to the coordinates of input points. The mininetwork itself resembles the big network and is composed by basic modules of point independent feature extraction, max pooling and fully connected layers. More details about the T-net are in the supplementary.

This idea can be further extended to the alignment of feature space, as well. We can insert another alignment network on point features and predict a feature transformation matrix to align features from different input point clouds. However, transformation matrix in the feature space has much higher dimension than the spatial transform matrix, which greatly increases the difficulty of optimization. We therefore add a regularization term to our softmax training loss. We constrain the feature transformation matrix to be close to orthogonal matrix:

L_{reg} = ||I - AA^T||_F^2, (2)

where A is the feature alignment matrix predicted by a mini-network. An orthogonal transformation will not lose information in the input, thus is desired. We find that by adding the regularization term, the optimization becomes more stable and our model achieves better performance.

4.3. Theoretical Analysis

Universal approximation We first show the universal approximation ability of our neural network to continuous set functions. By the continuity of set functions, intuitively, a small perturbation to the input point set should not greatly change the function values, such as classification or segmentation scores.

Formally, let \mathcal{X} = \{S : S \subseteq [0,1]^m \text{ and } |S| = n\}, f : \mathcal{X} \to \mathbb{R} is a continuous set function on \mathcal{X} w.r.t to Hausdorff distance d_H(\cdot,\cdot) , i.e., \forall \epsilon > 0, \exists \delta > 0 , for any S, S' \in \mathcal{X} , if d_H(S,S') < \delta , then |f(S) - f(S')| < \epsilon . Our theorem says that f can be arbitrarily approximated by our network given enough neurons at the max pooling layer, i.e., K in (1) is sufficiently large.

Figure 3. Qualitative results for part segmentation. We visualize the CAD part segmentation results across all 16 object categories. We show both results for partial simulated Kinect scans (left block) and complete ShapeNet CAD models (right block).

[그림 해설] 16개 카테고리에 대한 3D 파트 분할(Part Segmentation) 정성적 결과 시각화.

• 좌측 (Partial Inputs, 불완전 입력): 가상 Kinect 스캔으로 생성된 한쪽 면만 스캔되고 결손이 있는 포인트 클라우드에 대한 결과(table, motorbike, car, airplane, mug, lamp, guitar, chair 8종). 결손 및 가림이 있는 상태에서도 바퀴, 손잡이, 날개 등의 부품이 정확한 색상으로 분할됨.
• 우측 (Complete Inputs, 완전 입력): ShapeNet 3D CAD 모델의 완전한 포인트 클라우드에 대한 결과(bag, knife, cap, skateboard, pistol, rocket, earphone, laptop 8종). 칼날과 손잡이, 모자 챙과 본체, 노트북 모니터와 본체 등 복잡한 기하학적 세부 부품이 정밀하게 구분됨.

Theorem 1. Suppose f: \mathcal{X} \to \mathbb{R} is a continuous set function w.r.t Hausdorff distance d_H(\cdot, \cdot) . \forall \epsilon > 0 , \exists a continuous function h and a symmetric function g(x_1, \ldots, x_n) = \gamma \circ MAX , such that for any S \in \mathcal{X} ,

\left| f(S) - \gamma \left( \max_{x_i \in S} \{h(x_i)\} \right) \right| < \epsilon

where x_1, \ldots, x_n is the full list of elements in S ordered arbitrarily, \gamma is a continuous function, and MAX is a vector max operator that takes n vectors as input and returns a new vector of the element-wise maximum.

The proof to this theorem can be found in our supplementary material. The key idea is that in the worst case the network can learn to convert a point cloud into a volumetric representation, by partitioning the space into equal-sized voxels. In practice, however, the network learns a much smarter strategy to probe the space, as we shall see in point function visualizations.

Bottleneck dimension and stability Theoretically and experimentally we find that the expressiveness of our network is strongly affected by the dimension of the max pooling layer, i.e., K in (1). Here we provide an analysis, which also reveals properties related to the stability of our model.

We define \mathbf{u} = \max_{x_i \in S} \{h(x_i)\} to be the sub-network of f which maps a point set in [0,1]^m to a K-dimensional vector. The following theorem tells us that small corruptions or extra noise points in the input set are not likely to change the output of our network:

Theorem 2. Suppose \mathbf{u}: \mathcal{X} \to \mathbb{R}^K such that \mathbf{u} = \max_{x \in S} \{h(x_i)\} and f = \gamma \circ \mathbf{u} . Then,

(a)

\forall S, \exists C_S, \mathcal{N}_S \subseteq \mathcal{X}, f(T) = f(S) \text{ if } C_S \subseteq T \subseteq \mathcal{N}_S;

(b)

|\mathcal{C}_S| \leq K

	input	#views	accuracy	accuracy
			avg. class	overall
SPH [11]	mesh	-	68.2	-
3DShapeNets [28]	volume	1	77.3	84.7
VoxNet [17]	volume	12	83.0	85.9
Subvolume [18]	volume	20	86.0	89.2
LFD [28]	image	10	75.5	-
MVCNN [23]	image	80	90.1	-
Ours baseline	point	-	72.6	77.4
Ours PointNet	point	1	86.2	89.2

Table 1. Classification results on ModelNet40. Our net achieves state-of-the-art among deep nets on 3D input.

We explain the implications of the theorem. (a) says that f(S) is unchanged up to the input corruption if all points in \mathcal{C}_S are preserved; it is also unchanged with extra noise points up to \mathcal{N}_S . (b) says that \mathcal{C}_S only contains a bounded number of points, determined by K in (1). In other words, f(S) is in fact totally determined by a finite subset \mathcal{C}_S \subseteq S of less or equal to K elements. We therefore call \mathcal{C}_S the critical point set of S and K the bottleneck dimension of f.

Combined with the continuity of h, this explains the robustness of our model w.r.t point perturbation, corruption and extra noise points. The robustness is gained in analogy to the sparsity principle in machine learning models. Intuitively, our network learns to summarize a shape by a sparse set of key points. In experiment section we see that the key points form the skeleton of an object.

5. Experiment

Experiments are divided into four parts. First, we show PointNets can be applied to multiple 3D recognition tasks (Sec 5.1). Second, we provide detailed experiments to validate our network design (Sec 5.2). At last we visualize what the network learns (Sec 5.3) and analyze time and space complexity (Sec 5.4).

5.1. Applications

In this section we show how our network can be trained to perform 3D object classification, object part segmentation and semantic scene segmentation ¹. Even though we are working on a brand new data representation (point sets), we are able to achieve comparable or even better performance on benchmarks for several tasks.

3D Object Classification Our network learns global point cloud feature that can be used for object classification. We evaluate our model on the ModelNet40 [28] shape classification benchmark. There are 12,311 CAD models from 40 man-made object categories, split into 9,843 for

^<sup>1More application examples such as correspondence and point cloud based CAD model retrieval are included in supplementary material.

	mean	aero	bag	cap	car	chair	ear	guitar knife		lamp	laptop	motor		mug pistol	rocket	skate	table
							phone									board
# shapes		2690	76	55	898	3758	69	787	392	1547	451	202	184	283	66	152	5271
Wu [27]	-	63.2	-	-	-	73.5	-	-	-	74.4	-	-	-	-	-	-	74.8
Yi [29]	81.4	81.0	78.4	77.7	75.7	87.6	61.9	92.0	85.4	82.5	95.7	70.6		91.9 85.9	53.1	69.8	75.3
3DCNN	79.4	75.1	72.8	73.3	70.0	87.2	63.5	88.4	79.6	74.4	93.9	58.7		91.8 76.4	51.2	65.3	77.1
Ours	83.7	83.4	78.7	82.5	74.9	89.6	73.0	91.5	85.9	80.8	95.3	65.2		93.0 81.2	57.9	72.8	80.6

Table 2. Segmentation results on ShapeNet part dataset. Metric is mIoU(%) on points. We compare with two traditional methods [27] and [29] and a 3D fully convolutional network baseline proposed by us. Our PointNet method achieved the state-of-the-art in mIoU.

training and 2,468 for testing. While previous methods focus on volumetric and mult-view image representations, we are the first to directly work on raw point cloud.

We uniformly sample 1024 points on mesh faces according to face area and normalize them into a unit sphere. During training we augment the point cloud on-the-fly by randomly rotating the object along the up-axis and jitter the position of each points by a Gaussian noise with zero mean and 0.02 standard deviation.

In Table 1, we compare our model with previous works as well as our baseline using MLP on traditional features extracted from point cloud (point density, D2, shape contour etc.). Our model achieved state-of-the-art performance among methods based on 3D input (volumetric and point cloud). With only fully connected layers and max pooling, our net gains a strong lead in inference speed and can be easily parallelized in CPU as well. There is still a small gap between our method and multi-view based method (MVCNN [23]), which we think is due to the loss of fine geometry details that can be captured by rendered images.

3D Object Part Segmentation Part segmentation is a challenging fine-grained 3D recognition task. Given a 3D scan or a mesh model, the task is to assign part category label (e.g. chair leg, cup handle) to each point or face.

We evaluate on ShapeNet part data set from [29], which contains 16,881 shapes from 16 categories, annotated with 50 parts in total. Most object categories are labeled with two to five parts. Ground truth annotations are labeled on sampled points on the shapes.

We formulate part segmentation as a per-point classification problem. Evaluation metric is mIoU on points. For each shape S of category C, to calculate the shape's mIoU: For each part type in category C, compute IoU between groundtruth and prediction. If the union of groundtruth and prediction points is empty, then count part IoU as 1. Then we average IoUs for all part types in category C to get mIoU for that shape. To calculate mIoU for the category, we take average of mIoUs for all shapes in that category.

In this section, we compare our segmentation version PointNet (a modified version of Fig 2, Segmentation Network) with two traditional methods [27] and [29] that both take advantage of point-wise geometry features and correspondences between shapes, as well as our own 3D CNN baseline. See supplementary for the detailed modifications and network architecture for the 3D CNN.

In Table 2, we report per-category and mean IoU(%) scores. We observe a 2.3% mean IoU improvement and our net beats the baseline methods in most categories.

We also perform experiments on simulated Kinect scans to test the robustness of these methods. For every CAD model in the ShapeNet part data set, we use Blensor Kinect Simulator [7] to generate incomplete point clouds from six random viewpoints. We train our PointNet on the complete shapes and partial scans with the same network architecture and training setting. Results show that we lose only 5.3% mean IoU. In Fig 3, we present qualitative results on both complete and partial data. One can see that though partial data is fairly challenging, our predictions are reasonable.

Semantic Segmentation in Scenes Our network on part segmentation can be easily extended to semantic scene segmentation, where point labels become semantic object classes instead of object part labels.

We experiment on the Stanford 3D semantic parsing data set [1]. The dataset contains 3D scans from Matterport scanners in 6 areas including 271 rooms. Each point in the scan is annotated with one of the semantic labels from 13 categories (chair, table, floor, wall etc. plus clutter).

To prepare training data, we firstly split points by room, and then sample rooms into blocks with area 1m by 1m. We train our segmentation version of PointNet to predict

	mean IoU	overall accuracy
Ours baseline	20.12	53.19
Ours PointNet	47.71	78.62

Table 3. Results on semantic segmentation in scenes. Metric is average IoU over 13 classes (structural and furniture elements plus clutter) and classification accuracy calculated on points.

	table	chair	sofa	board	mean
# instance	455	1363	55	137
Armeni et al. [1]	46.02	16.15	6.78	3.91	18.22
Ours	46.67	33.80	4.76	11.72	24.24

Table 4. Results on 3D object detection in scenes. Metric is average precision with threshold IoU 0.5 computed in 3D volumes.

Figure 4. Qualitative results for semantic segmentation. Top row is input point cloud with color. Bottom row is output semantic segmentation result (on points) displayed in the same camera viewpoint as input.

[그림 해설] Stanford 3D Semantic Parsing 데이터셋 실내 환경 3개 씬(오피스 2곳, 회의실 1곳)에 대한 Semantic Segmentation 결과.

• 상단 (Input): RGB 컬러 정보가 포함된 원본 실내 공간 3D 포인트 클라우드 입력.
• 하단 (Output): PointNet이 예측한 포인트별 시맨틱 레이블.
• 색상 대응: 바닥(파란색), 벽(하늘색/청록색), 천장(초록색), 테이블(보라색), 의자(빨간색), 보드(회색), 책장/도어(노란색/연두색) 등으로 객체 및 실내 구조물이 명확히 분리됨.

per point class in each block. Each point is represented by a 9-dim vector of XYZ, RGB and normalized location as to the room (from 0 to 1). At training time, we randomly sample 4096 points in each block on-the-fly. At test time, we test on all the points. We follow the same protocol as [1] to use k-fold strategy for train and test.

We compare our method with a baseline using handcrafted point features. The baseline extracts the same 9 dim local features and three additional ones: local point density, local curvature and normal. We use standard MLP as the classifier. Results are shown in Table 3, where our PointNet method significantly outperforms the baseline method. In Fig 4, we show qualitative segmentation results. Our network is able to output smooth predictions and is robust to missing points and occlusions.

Based on the semantic segmentation output from our network, we further build a 3D object detection system using connected component for object proposal (see supplementary for details). We compare with previous stateof-the-art method in Table 4. The previous method is based on a sliding shape method (with CRF post processing) with SVMs trained on local geometric features and global room context feature in voxel grids. Our method outperforms it by a large margin on the furniture categories reported.

5.2. Architecture Design Analysis

In this section we validate our design choices by control experiments. We also show the effects of our network's hyperparameters.

Comparison with Alternative Order-invariant Methods

As mentioned in Sec 4.2, there are at least three options for consuming unordered set inputs. We use the ModelNet40 shape classification problem as a test bed for comparisons of those options, the following two control experiment will also use this task.

The baselines (illustrated in Fig 5) we compared with include multi-layer perceptron on unsorted and sorted

rnn rnn rnn MLP
cell cell cell
MLP MLP MLP
(1,2,3) (2,3,4) (1,3,1) sequential model
sorted
(1,2,3) (1,2,3) MLP (1,3,1) (2,3,4) MLP
MLP MLP (1,3,1) (2,3,4)
MLP sorting symmetry function

Figure 5. Three approaches to achieve order invariance. Multilayer perceptron (MLP) applied on points consists of 5 hidden layers with neuron sizes 64,64,64,128,1024, all points share a single copy of MLP. The MLP close to the output consists of two layers with sizes 512,256.

points as n×3 arrays, RNN model that considers input point as a sequence, and a model based on symmetry functions. The symmetry operation we experimented include max pooling, average pooling and an attention based weighted sum. The attention method is similar to that in [25], where a scalar score is predicted from each point feature, then the score is normalized across points by computing a softmax. The weighted sum is then computed on the normalized scores and the point features. As shown in Fig 5, maxpooling operation achieves the best performance by a large winning margin, which validates our choice.

Effectiveness of Input and Feature Transformations In

Table 5 we demonstrate the positive effects of our input and feature transformations (for alignment). It's interesting to see that the most basic architecture already achieves quite reasonable results. Using input transformation gives a 0.8% performance boost. The regularization loss is necessary for the higher dimension transform to work. By combining both transformations and the regularization term, we achieve the best performance.

Robustness Test We show our PointNet, while simple and effective, is robust to various kinds of input corruptions. We use the same architecture as in Fig 5's max pooling network. Input points are normalized into a unit sphere. Results are in Fig 6.

As to missing points, when there are 50% points missing, the accuracy only drops by 2.4% and 3.8% w.r.t. furthest and random input sampling. Our net is also robust to outlier

Transform	accuracy
none	87.1
input (3x3)	87.9
feature (64x64)	86.9
feature (64x64) + reg.	87.4
both	89.2

Table 5. Effects of input feature transforms. Metric is overall classification accuracy on ModelNet40 test set.

Figure 6. PointNet robustness test. The metric is overall classification accuracy on ModelNet40 test set. Left: Delete points. Furthest means the original 1024 points are sampled with furthest sampling. Middle: Insertion. Outliers uniformly scattered in the unit sphere. Right: Perturbation. Add Gaussian noise to each point independently.

[그림 해설] 포인트 결손, 이상치, 노이즈에 대한 PointNet의 강건성(Robustness) 평가 그래프 (ModelNet40 테스트셋 기준).

• 좌측 (Missing data ratio, 데이터 결손): 점을 무작위(Random, 빨간 사각) 또는 최원점 샘플링(Furthest, 파란 원)으로 삭제했을 때의 정확도 변화. 50%의 점이 누락되어도 정확도는 약 86% 수준을 유지하며, 75% 누락 시에도 74~81%로 유지되다가 90% 이상 누락 시 급격히 하락.
• 중앙 (Outlier ratio, 이상치 비율): 공간 내 임의의 노이즈 점 추가 시 정확도 변화. XYZ 좌표만 사용한 경우(파란 사각)와 밀도 정보(XYZ+\text{density}, 갈색 다이아몬드)를 함께 사용한 경우 모두 이상치 비율 30%까지 70% 이상의 정확도를 견고하게 유지.
• 우측 (Perturbation noise std, 점 섭동 노이즈): 점 좌표에 가우시안 노이즈(표준편차 0 \sim 0.1)를 가했을 때의 정확도 변화. 표준편차 0.05까지 약 80% 이상을 유지하다가 0.1에 도달하면 약 30%로 감소.

points, if it has seen those during training. We evaluate two models: one trained on points with (x, y, z) coordinates; the other on (x, y, z) plus point density. The net has more than 80% accuracy even when 20% of the points are outliers. Fig 6 right shows the net is robust to point perturbations.

5.3. Visualizing PointNet

In Fig 7, we visualize critical point sets C^S and upperbound shapes N^S (as discussed in Thm 2) for some sample shapes S. The point sets between the two shapes will give exactly the same global shape feature f(S).

We can see clearly from Fig 7 that the critical point sets CS, those contributed to the max pooled feature, summarizes the skeleton of the shape. The upper-bound shapes N^S illustrates the largest possible point cloud that give the same global shape feature f(S) as the input point cloud S. C^S and N^S reflect the robustness of PointNet, meaning that losing some non-critical points does not change the global shape signature f(S) at all.

The N^S is constructed by forwarding all the points in a edge-length-2 cube through the network and select points p whose point function values (h1(p), h2(p), · · · , hK(p)) are no larger than the global shape descriptor.

Figure 7. Critical points and upper bound shape. While critical points jointly determine the global shape feature for a given shape, any point cloud that falls between the critical points set and the upper bound shape gives exactly the same feature. We color-code all figures to show the depth information.

[그림 해설] PointNet이 학습한 임계 포인트 세트(\mathcal{C}_S)와 상한 형상(\mathcal{N}_S)의 시각화 (테이블, 권총, 램프, 스탠드 4종, 깊이에 따라 무지개색 코딩).

• 1행 (Original Shape, S): 원본 입력 포인트 클라우드.
• 2행 (Critical Point Sets, \mathcal{C}_S): Max Pooling 레이어의 1024개 글로벌 특징을 결정짓는 핵심 포인트들만 추출한 서브셋. 물체의 외곽 윤곽(스켈레톤) 형태를 띄며, 입력 포인트의 극히 일부만으로 구성됨.
• 3행 (Upper-bound Shapes, \mathcal{N}_S): 글로벌 특징 벡터 $\mathbf{u}$의 출력을 변화시키지 않으면서 최대로 추가할 수 있는 포인트들의 점유 영역. 원본 형태를 둘러싼 두꺼운 볼륨 형태를 형성하여, 노이즈나 추가 점이 이 영역 내에 존재해도 네트워크 출력이 불변함을 입증.

5.4. Time and Space Complexity Analysis

Table 6 summarizes space (number of parameters in the network) and time (floating-point operations/sample) complexity of our classification PointNet. We also compare PointNet to a representative set of volumetric and multiview based architectures in previous works.

While MVCNN [23] and Subvolume (3D CNN) [18] achieve high performance, PointNet is orders more efficient in computational cost (measured in FLOPs/sample: 141x and 8x more efficient, respectively). Besides, PointNet is much more space efficient than MVCNN in terms of #param in the network (17x less parameters). Moreover, PointNet is much more scalable – it's space and time complexity is O(N) – linear in the number of input points. However, since convolution dominates computing time, multi-view method's time complexity grows squarely on image resolution and volumetric convolution based method grows cubically with the volume size.

Empirically, PointNet is able to process more than one million points per second for point cloud classification (around 1K objects/second) or semantic segmentation (around 2 rooms/second) with a 1080X GPU on Tensor-Flow, showing great potential for real-time applications.

	#params	FLOPs/sample
PointNet (vanilla)	0.8M	148M
PointNet	3.5M	440M
Subvolume [18]	16.6M	3633M
MVCNN [23]	60.0M	62057M

Table 6. Time and space complexity of deep architectures for 3D data classification. PointNet (vanilla) is the classification PointNet without input and feature transformations. FLOP stands for floating-point operation. The "M" stands for million. Subvolume and MVCNN used pooling on input data from multiple rotations or views, without which they have much inferior performance.

6. Conclusion

In this work, we propose a novel deep neural network PointNet that directly consumes point cloud. Our network provides a unified approach to a number of 3D recognition tasks including object classification, part segmentation and semantic segmentation, while obtaining on par or better results than state of the arts on standard benchmarks. We also provide theoretical analysis and visualizations towards understanding of our network.

Acknowledgement. The authors gratefully acknowledge the support of a Samsung GRO grant, ONR MURI N00014- 13-1-0341 grant, NSF grant IIS-1528025, a Google Focused Research Award, a gift from the Adobe corporation and hardware donations by NVIDIA.

References

• [1] I. Armeni, O. Sener, A. R. Zamir, H. Jiang, I. Brilakis, M. Fischer, and S. Savarese. 3d semantic parsing of large-scale indoor spaces. In Proceedings of the IEEE International Conference on Computer Vision and Pattern Recognition, 2016. 6, 7

• [2] M. Aubry, U. Schlickewei, and D. Cremers. The wave kernel signature: A quantum mechanical approach to shape analysis. In Computer Vision Workshops (ICCV Workshops), 2011 IEEE International Conference on, pages 1626–1633. IEEE, 2011. 2

• [3] M. M. Bronstein and I. Kokkinos. Scale-invariant heat kernel signatures for non-rigid shape recognition. In Computer Vision and Pattern Recognition (CVPR), 2010 IEEE Conference on, pages 1704–1711. IEEE, 2010. 2

• [4] J. Bruna, W. Zaremba, A. Szlam, and Y. LeCun. Spectral networks and locally connected networks on graphs. arXiv preprint arXiv:1312.6203, 2013. 2

• [5] D.-Y. Chen, X.-P. Tian, Y.-T. Shen, and M. Ouhyoung. On visual similarity based 3d model retrieval. In Computer graphics forum, volume 22, pages 223–232. Wiley Online Library, 2003. 2

• [6] Y. Fang, J. Xie, G. Dai, M. Wang, F. Zhu, T. Xu, and E. Wong. 3d deep shape descriptor. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 2319–2328, 2015. 2

• [7] M. Gschwandtner, R. Kwitt, A. Uhl, and W. Pree. BlenSor: Blender Sensor Simulation Toolbox Advances in Visual Computing. volume 6939 of Lecture Notes in Computer Science, chapter 20, pages 199–208. Springer Berlin / Heidelberg, Berlin, Heidelberg, 2011. 6

• [8] K. Guo, D. Zou, and X. Chen. 3d mesh labeling via deep convolutional neural networks. ACM Transactions on Graphics (TOG), 35(1):3, 2015. 2

• [9] M. Jaderberg, K. Simonyan, A. Zisserman, et al. Spatial transformer networks. In NIPS 2015. 4

• [10] A. E. Johnson and M. Hebert. Using spin images for efficient object recognition in cluttered 3d scenes. IEEE Transactions on pattern analysis and machine intelligence, 21(5):433– 449, 1999. 2

• [11] M. Kazhdan, T. Funkhouser, and S. Rusinkiewicz. Rotation invariant spherical harmonic representation of 3 d shape descriptors. In Symposium on geometry processing, volume 6, pages 156–164, 2003. 5

• [12] Y. LeCun, L. Bottou, Y. Bengio, and P. Haffner. Gradientbased learning applied to document recognition. Proceedings of the IEEE, 86(11):2278–2324, 1998. 13

• [13] Y. Li, S. Pirk, H. Su, C. R. Qi, and L. J. Guibas. Fpnn: Field probing neural networks for 3d data. arXiv preprint arXiv:1605.06240, 2016. 2

• [14] H. Ling and D. W. Jacobs. Shape classification using the inner-distance. IEEE transactions on pattern analysis and machine intelligence, 29(2):286–299, 2007. 2

• [15] L. v. d. Maaten and G. Hinton. Visualizing data using t-sne. Journal of Machine Learning Research, 9(Nov):2579–2605, 2008. 15

• [16] J. Masci, D. Boscaini, M. Bronstein, and P. Vandergheynst. Geodesic convolutional neural networks on riemannian manifolds. In Proceedings of the IEEE International Conference on Computer Vision Workshops, pages 37–45, 2015. 2

• [17] D. Maturana and S. Scherer. Voxnet: A 3d convolutional neural network for real-time object recognition. In IEEE/RSJ International Conference on Intelligent Robots and Systems, September 2015. 2, 5, 10, 11

• [18] C. R. Qi, H. Su, M. Nießner, A. Dai, M. Yan, and L. Guibas. Volumetric and multi-view cnns for object classification on 3d data. In Proc. Computer Vision and Pattern Recognition (CVPR), IEEE, 2016. 2, 5, 8

• [19] R. B. Rusu, N. Blodow, and M. Beetz. Fast point feature histograms (fpfh) for 3d registration. In Robotics and Automation, 2009. ICRA'09. IEEE International Conference on, pages 3212–3217. IEEE, 2009. 2

• [20] R. B. Rusu, N. Blodow, Z. C. Marton, and M. Beetz. Aligning point cloud views using persistent feature histograms. In 2008 IEEE/RSJ International Conference on Intelligent Robots and Systems, pages 3384–3391. IEEE, 2008. 2

• [21] M. Savva, F. Yu, H. Su, M. Aono, B. Chen, D. Cohen-Or, W. Deng, H. Su, S. Bai, X. Bai, et al. Shrec16 track largescale 3d shape retrieval from shapenet core55. 2

• [22] P. Y. Simard, D. Steinkraus, and J. C. Platt. Best practices for convolutional neural networks applied to visual document analysis. In ICDAR, volume 3, pages 958–962, 2003. 13

• [23] H. Su, S. Maji, E. Kalogerakis, and E. G. Learned-Miller. Multi-view convolutional neural networks for 3d shape recognition. In Proc. ICCV, to appear, 2015. 2, 5, 6, 8

• [24] J. Sun, M. Ovsjanikov, and L. Guibas. A concise and provably informative multi-scale signature based on heat diffusion. In Computer graphics forum, volume 28, pages 1383–1392. Wiley Online Library, 2009. 2

• [25] O. Vinyals, S. Bengio, and M. Kudlur. Order matters: Sequence to sequence for sets. arXiv preprint arXiv:1511.06391, 2015. 2, 4, 7

• [26] D. Z. Wang and I. Posner. Voting for voting in online point cloud object detection. Proceedings of the Robotics: Science and Systems, Rome, Italy, 1317, 2015. 2

• [27] Z. Wu, R. Shou, Y. Wang, and X. Liu. Interactive shape cosegmentation via label propagation. Computers & Graphics, 38:248–254, 2014. 6, 10

• [28] Z. Wu, S. Song, A. Khosla, F. Yu, L. Zhang, X. Tang, and J. Xiao. 3d shapenets: A deep representation for volumetric shapes. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 1912–1920, 2015. 2, 5, 11

• [29] L. Yi, V. G. Kim, D. Ceylan, I.-C. Shen, M. Yan, H. Su, C. Lu, Q. Huang, A. Sheffer, and L. Guibas. A scalable active framework for region annotation in 3d shape collections. SIGGRAPH Asia, 2016. 6, 10, 18

Supplementary

A. Overview

This document provides additional quantitative results, technical details and more qualitative test examples to the main paper.

In Sec B we extend the robustness test to compare PointNet with VoxNet on incomplete input. In Sec C we provide more details on neural network architectures, training parameters and in Sec D we describe our detection pipeline in scenes. Then Sec E illustrates more applications of PointNet, while Sec F shows more analysis experiments. Sec G provides a proof for our theory on PointNet. At last, we show more visualization results in Sec H.

B. Comparison between PointNet and VoxNet (Sec 5.2)

We extend the experiments in Sec 5.2 Robustness Test to compare PointNet and VoxNet [17] (a representative architecture for volumetric representation) on robustness to missing data in the input point cloud. Both networks are trained on the same train test split with 1024 number of points as input. For VoxNet we voxelize the point cloud to 32 × 32 × 32 occupancy grids and augment the training data by random rotation around up-axis and jittering.

At test time, input points are randomly dropped out by a certain ratio. As VoxNet is sensitive to rotations, its prediction uses average scores from 12 viewpoints of a point cloud. As shown in Fig 8, we see that our PointNet is much more robust to missing points. VoxNet's accuracy dramatically drops when half of the input points are missing, from 86.3% to 46.0% with a 40.3% difference, while our PointNet only has a 3.7% performance drop. This can be explained by the theoretical analysis and explanation of our PointNet – it is learning to use a collection of critical points to summarize the shape, thus it is very robust to missing data.

C. Network Architecture and Training Details (Sec 5.1)

PointNet Classification Network As the basic architecture is already illustrated in the main paper, here we provides more details on the joint alignment/transformation network and training parameters.

The first transformation network is a mini-PointNet that takes raw point cloud as input and regresses to a 3 × 3 matrix. It's composed of a shared MLP(64, 128, 1024) network (with layer output sizes 64, 128, 1024) on each point, a max pooling across points and two fully connected layers with output sizes 512, 256. The output matrix is initialized as an identity matrix. All layers, except the last one, include ReLU and batch normalization. The second

Figure 8. PointNet v.s. VoxNet [17] on incomplete input data. Metric is overall classification accurcacy on ModelNet40 test set. Note that VoxNet is using 12 viewpoints averaging while PointNet is using only one view of the point cloud. Evidently PointNet presents much stronger robustness to missing points.

[그림 해설] 데이터 결손율(Missing Data Ratio, 0~1.0)에 따른 PointNet(파란 원)과 3D 복셀 기반 VoxNet(빨간 사각)의 분류 정확도(Accuracy %) 비교 그래프.

• 데이터 결손이 0일 때 두 모델 모두 약 87%의 정확도로 시작함.
• 데이터 결손율이 50%(0.5)일 때 PointNet은 약 84%의 높은 정확도를 유지하는 반면, VoxNet은 46% 수준으로 급락함.
• 결손율 75%(0.75)에서도 PointNet은 약 74%를 기록하나 VoxNet은 18%로 추락하여, 불완전/부분 스캔 데이터에 대한 PointNet의 압도적인 강건성을 실증함.

transformation network has the same architecture as the first one except that the output is a 64 × 64 matrix. The matrix is also initialized as an identity. A regularization loss (with weight 0.001) is added to the softmax classification loss to make the matrix close to orthogonal.

We use dropout with keep ratio 0.7 on the last fully connected layer, whose output dimension 256, before class score prediction. The decay rate for batch normalization starts with 0.5 and is gradually increased to 0.99. We use adam optimizer with initial learning rate 0.001, momentum 0.9 and batch size 32. The learning rate is divided by 2 every 20 epochs. Training on ModelNet takes 3-6 hours to converge with TensorFlow and a GTX1080 GPU.

PointNet Segmentation Network The segmentation network is an extension to the classification PointNet. Local point features (the output after the second transformation network) and global feature (output of the max pooling) are concatenated for each point. No dropout is used for segmentation network. Training parameters are the same as the classification network.

As to the task of shape part segmentation, we made a few modifications to the basic segmentation network architecture (Fig 2 in main paper) in order to achieve best performance, as illustrated in Fig 9. We add a one-hot vector indicating the class of the input and concatenate it with the max pooling layer's output. We also increase neurons in some layers and add skip links to collect local point features in different layers and concatenate them to form point feature input to the segmentation network.

While [27] and [29] deal with each object category independently, due to the lack of training data for some categories (the total number of shapes for all the categories in the data set are shown in the first line), we train our PointNet across categories (but with one-hot vector input to indicate category). To allow fair comparison, when testing

Figure 9. Network architecture for part segmentation. T1 and T2 are alignment/transformation networks for input points and features. FC is fully connected layer operating on each point. MLP is multi-layer perceptron on each point. One-hot is a vector of size 16 indicating category of the input shape.

[그림 해설] ShapeNet Part Segmentation을 위한 심층 PointNet 세그멘테이션 네트워크 구조 다이어그램.

• 입력 n \times 3 \to T1(Spatial Transform) \to n \times 3 \to FC(64) \to n \times 64 \to FC(128) \to n \times 128 \to FC(128) \to n \times 128 \to T2(Feature Transform) \to n \times 128 \to FC(512) \to n \times 512 \to FC(2048) \to n \times 2048 \to Max Pooling $\to 2048$차원 Global Feature.
• 다중 계층 특징 결합: 각 단계의 점별 특징(64 + 128 + 128 + 128 + 512 = 960), 2048차원 글로벌 특징, 그리고 객체 카테고리를 나타내는 one-hot 벡터를 모두 결합하여 n \times 3024 통합 특징 벡터 구성.
• 최종 MLP(256, 256, 128)을 거쳐 n \times 50 파트 예측 점수(part scores) 출력.

Figure 10. Baseline 3D CNN segmentation network. The network is fully convolutional and predicts part scores for each voxel.

[그림 해설] 파트 분할 성능 비교를 위한 3D 복셀 기반 CNN(Voxel-CNN) 베이스라인 아키텍처 다이어그램.

• 32 \times 32 \times 32 크기의 3D 복셀 그리드 입력을 받음.
• 인코더: 32개 필터(커널 5, stride 1) 컨볼루션 4회 \to 32개 필터(커널 3, stride 1) 컨볼루션 1회를 거쳐 형상 특징 추출.
• 디코더: 1개 크기의 특징을 스킵 연결(Skip Connection) 및 64 필터 \to 64 필터 \to 50 필터(커널 1, stride 1) 컨볼루션을 거쳐 32 \times 32 \times 32 복셀 공간에 대해 각 복셀의 파트 카테고리를 예측(in-category prediction).

these two models, we only predict part labels for the given specific object category.

As to semantic segmentation task, we used the architecture as in Fig 2 in the main paper.

It takes around six to twelve hours to train the model on ShapeNet part dataset and around half a day to train on the Stanford semantic parsing dataset.

Baseline 3D CNN Segmentation Network In ShapeNet part segmentation experiment, we compare our proposed segmentation version PointNet to two traditional methods as well as a 3D volumetric CNN network baseline. In Fig 10, we show the baseline 3D volumetric CNN network we use. We generalize the well-known 3D CNN architectures, such as VoxNet [17] and 3DShapeNets [28] to a fully convolutional 3D CNN segmentation network.

For a given point cloud, we first convert it to the volumetric representation as a occupancy grid with resolution 32 \times 32 \times 32 . Then, five 3D convolution operations each with 32 output channels and stride of 1 are sequentially applied to extract features. The receptive field is 19 for each voxel. Finally, a sequence of 3D convolutional layers with kernel size 1 \times 1 \times 1 is appended to the computed feature map to predict segmentation label for each voxel. ReLU and

batch normalization are used for all layers except the last one. The network is trained across categories, however, in order to compare with other baseline methods where object category is given, we only consider output scores in the given object category.

D. Details on Detection Pipeline (Sec 5.1)

We build a simple 3D object detection system based on the semantic segmentation results and our object classification PointNet.

We use connected component with segmentation scores to get object proposals in scenes. Starting from a random point in the scene, we find its predicted label and use BFS to search nearby points with the same label, with a search radius of 0.2 meter. If the resulted cluster has more than 200 points (assuming a 4096 point sample in a 1m by 1m area), the cluster's bounding box is marked as one object proposal. For each proposed object, it's detection score is computed as the average point score for that category. Before evaluation, proposals with extremely small areas/volumes are pruned. For tables, chairs and sofas, the bounding boxes are extended to the floor in case the legs are separated with the seat/surface.

We observe that in some rooms such as auditoriums lots of objects (e.g. chairs) are close to each other, where connected component would fail to correctly segment out individual ones. Therefore we leverage our classification network and uses sliding shape method to alleviate the problem for the chair class. We train a binary classification network for each category and use the classifier for sliding window detection. The resulted boxes are pruned by non-maximum suppression. The proposed boxes from connected component and sliding shapes are combined for final evaluation.

In Fig 11, we show the precision-recall curves for object detection. We trained six models, where each one of them is trained on five areas and tested on the left area. At test phase, each model is tested on the area it has never seen. The test results for all six areas are aggregated for the PR curve generation.

E. More Applications (Sec 5.1)

Model Retrieval from Point Cloud Our PointNet learns a global shape signature for every given input point cloud. We expect geometrically similar shapes have similar global signature. In this section, we test our conjecture on the shape retrieval application. To be more specific, for every given query shape from ModelNet test split, we compute its global signature (output of the layer before the score prediction layer) given by our classification PointNet and retrieve similar shapes in the train split by nearest neighbor search. Results are shown in Fig 12.

Figure 11. Precision-recall curves for object detection in 3D point cloud. We evaluated on all six areas for four categories: table, chair, sofa and board. IoU threshold is 0.5 in volume.

[그림 해설] 3D 씬 객체 검출(Object Detection)에서 주요 4개 카테고리에 대한 Precision-Recall(정밀도-재현율) PR 곡선.

• table (상단 좌측): Recall 0.5 부근까지 Precision 0.7~0.8 이상을 유지하다가 점진적으로 하강 (최종 Recall 약 0.68).
• chair (상단 우측): Precision이 0.8에서 출발하여 Recall 0.65 부근(Precision 약 0.2)까지 완만하게 선형 하강.
• sofa (하단 좌측): 초기 급격한 하강 후 낮은 Precision 영역에서 Recall 0.24 부근까지 형성.
• board (하단 우측): 초기 Precision 약 0.8까지 상승 후 Recall 0.2 부근에서 급격히 감소.

Figure 12. Model retrieval from point cloud. For every given point cloud, we retrieve the top-5 similar shapes from the ModelNet test split. From top to bottom rows, we show examples of chair, plant, nightstand and bathtub queries. Retrieved results that are in wrong category are marked by red boxes.

[그림 해설] 불완전 쿼리 포인트 클라우드에 대한 PointNet 글로벌 특징 기반 Top-5 CAD 모델 검색(3D Shape Retrieval) 결과.

• 4개 쿼리(좌측: 불완전 스캔 의자, 식물, 캐비닛/수납장, 세면대)에 대해 가장 유사한 CAD 모델 5개를 우측에 순서대로 배열.
• 1~3행의 의자, 식물, 가구는 모두 올바른 카테고리의 유사 형상이 완벽히 검색됨.
• 4행(세면대 쿼리)의 경우 1번째, 3번째, 5번째는 세면대(sink, 빨간 사각 박스)로 올바르게 검색되었으나, 2번째와 4번째는 외형이 유사한 욕조(bathtub)가 검색된 검색 실패/혼동 사례를 표시.

Shape Correspondence In this section, we show that point features learnt by PointNet can be potentially used to compute shape correspondences. Given two shapes, we compute the correspondence between their critical point sets CS's by matching the pairs of points that activate the same dimensions in the global features. Fig 13 and Fig 14 show the detected shape correspondence between two similar chairs and tables.

F. More Architecture Analysis (Sec 5.2)

Effects of Bottleneck Dimension and Number of Input Points Here we show our model's performance change with regard to the size of the first max layer output as well as the number of input points. In Fig 15 we see that performance grows as we increase the number of points however it saturates at around 1K points. The max layer size plays an important role, increasing the layer size from

Figure 13. Shape correspondence between two chairs. For the clarity of the visualization, we only show 20 randomly picked correspondence pairs.

[그림 해설] 서로 다른 두 개의 의자 포인트 클라우드(빨간색, 파란색) 간의 기하학적 부품 대응점(Shape Correspondence) 시각화.

• 무작위로 선택된 20개 대응 쌍을 색상 직선으로 연결.
• 등받이 상단, 좌판 모서리, 의자 다리 끝단 등 구조적으로 일치하는 부품 위치끼리 평행하게 연결되어 PointNet 특징이 의미론적 형상 대응을 정확히 학습했음을 입증.

Figure 14. Shape correspondence between two tables. For the clarity of the visualization, we only show 20 randomly picked correspondence pairs.

[그림 해설] 형태가 서로 다른 두 개의 테이블 포인트 클라우드(빨간색 직사각형 테이블, 파란색 원형/타원형 테이블) 간 형상 대응 시각화.

• 무작위 20개 포인트 쌍이 상판의 둘레, 모서리, 테이블 다리 하단부 등 대응하는 구조적 위치로 정확히 매핑되어 연결선을 형성.

64 to 1024 results in a 2−4% performance gain. It indicates that we need enough point feature functions to cover the 3D space in order to discriminate different shapes.

It's worth notice that even with 64 points as input (obtained from furthest point sampling on meshes), our network can achieve decent performance.

Figure 15. Effects of bottleneck size and number of input points. The metric is overall classification accuracy on Model-Net40 test set.

[그림 해설] 병목 차원(Bottleneck size, 64~1024) 및 입력 포인트 수(#points: 64, 128, 512, 1024, 2048)에 따른 ModelNet40 테스트 분류 정확도(Accuracy %) 변화 그래프.

• X축: Bottleneck size (0, 200, 400, 600, 800, 1000). Y축: Accuracy % (81% ~ 88%).
• 입력 포인트 수가 64개(하늘색 사각)일 때는 정확도 82~84.5% 수준이나, 1024개(초록 삼각) 및 2048개(주황 원)로 증가하면 87% 이상으로 향상됨.
• 병목 크기 256~512 이상에서 성능이 포화(87.3% 수준)에 도달하며, 1024 차원에서 최대 성능을 기록함.

MNIST Digit Classification While we focus on 3D point cloud learning, a sanity check experiment is to apply our network on a 2D point clouds - pixel sets.

To convert an MNIST image into a 2D point set we threshold pixel values and add the pixel (represented as a point with (x, y) coordinate in the image) with values larger than 128 to the set. We use a set size of 256. If there are more than 256 pixels int he set, we randomly sub-sample it; if there are less, we pad the set with the one of the pixels in the set (due to our max operation, which point to use for the padding will not affect outcome).

As seen in Table 7, we compare with a few baselines including multi-layer perceptron that considers input image as an ordered vector, a RNN that consider input as sequence from pixel (0,0) to pixel (27,27), and a vanilla version CNN. While the best performing model on MNIST is still well engineered CNNs (achieving less than 0.3% error rate), it's interesting to see that our PointNet model can achieve reasonable performance by considering image as a 2D point set.

	input	error (%)
Multi-layer perceptron [22]	vector	1.60
LeNet5 [12]	image	0.80
Ours PointNet	point set	0.78

Table 7. MNIST classification results. We compare with vanilla versions of other deep architectures to show that our network based on point sets input is achieving reasonable performance on this traditional task.

Normal Estimation In segmentation version of PointNet, local point features and global feature are concatenated in order to provide context to local points. However, it's unclear whether the context is learnt through this concatenation. In this experiment, we validate our design by showing that our segmentation network can be trained to predict point normals, a local geometric property that is determined by a point's neighborhood.

We train a modified version of our segmentation Point-Net in a supervised manner to regress to the groundtruth point normals. We just change the last layer of our segmentation PointNet to predict normal vector for each point. We use absolute value of cosine distance as loss.

Fig. 16 compares our PointNet normal prediction results (the left columns) to the ground-truth normals computed from the mesh (the right columns). We observe a reasonable normal reconstruction. Our predictions are more smooth and continuous than the ground-truth which includes flipped normal directions in some region.

Segmentation Robustness As discussed in Sec 5.2 and Sec B, our PointNet is less sensitive to data corruption and missing points for classification tasks since the global shape feature is extracted from a collection of critical points from the given input point cloud. In this section, we show that the robustness holds for segmentation tasks too. The per-point part labels are predicted based on the combination of perpoint features and the learnt global shape feature. In Fig 17,

Figure 16. PointNet normal reconstrution results. In this figure, we show the reconstructed normals for all the points in some sample point clouds and the ground-truth normals computed on the mesh.

[그림 해설] PointNet을 이용한 포인트별 표면 법선 벡터(Normal Vector) 재구성 결과 비교.

• 3개 객체 샘플(의자 등받이/좌판, 비행기, 변기)에 대해 PointNet이 예측한 법선 벡터(Prediction, 좌측)와 3D 메시에서 계산된 실제 정답 법선 벡터(Ground-truth, 우측)를 파란색 선분으로 시각화.
• 각 점의 국소 기하 구조(평면, 곡면, 경계선)에 수직인 법선 방향이 GT와 거의 완벽하게 일치하게 재구성됨.

we illustrate the segmentation results for the given input point clouds S (the left-most column), the critical point sets C^S (the middle column) and the upper-bound shapes NS.

Network Generalizability to Unseen Shape Categories In Fig 18, we visualize the critical point sets and the upperbound shapes for new shapes from unseen categories (face, house, rabbit, teapot) that are not present in ModelNet or ShapeNet. It shows that the learnt per-point functions are generalizable. However, since we train mostly on manmade objects with lots of planar structures, the reconstructed upper-bound shape in novel categories also contain more planar surfaces.

G. Proof of Theorem (Sec 4.3)

Let

\mathcal{X} = \{S : S \subseteq [0, 1] \text{ and } |S| = n\}.

f : X → R is a continuous function on X w.r.t to Hausdorff distance dH(·, ·) if the following condition is satisfied:

\forall \epsilon > 0, \exists \delta > 0

, for any S, S' \in \mathcal{X} , if d_H(S, S') < \delta , then |f(S) - f(S')| < \epsilon .

We show that f can be approximated arbitrarily by composing a symmetric function and a continuous function.

Input Point Cloud Critical Point Sets Upper-bound Shapes

Figure 17. The consistency of segmentation results. We illustrate the segmentation results for some sample given point clouds S, their critical point sets \mathcal{C}_S and upper-bound shapes \mathcal{N}_S . We observe that the shape family between the \mathcal{C}_S and \mathcal{N}_S share a consistent segmentation results.

[그림 해설] 형상 집합군에서의 파트 세그멘테이션 일관성 시각화 (탁자, 머그컵, 자동차 3개 예시).

• 1열 (Input Point Cloud, S): 원본 포인트 클라우드와 예측된 파트 분할 결과 (색상별 부품 분할).
• 2열 (Critical Point Sets, \mathcal{C}_S): 해당 형상의 임계 포인트들만 남긴 서브셋에서도 원본과 동일한 부품 경계와 분할 결과가 유지됨.
• 3열 (Upper-bound Shapes, \mathcal{N}_S): 포인트가 두껍게 확장된 상한 형상에서도 동일한 파트 분할 레이블이 완벽하게 일관성을 유지하며 보존됨을 확인.

Figure 18. The critical point sets and the upper-bound shapes for unseen objects. We visualize the critical point sets and the upper-bound shapes for teapot, bunny, hand and human body, which are not in the ModelNet or ShapeNet shape repository to test the generalizability of the learnt per-point functions of our PointNet on other unseen objects. The images are color-coded to reflect the depth information.

[그림 해설] 훈련 세트(ModelNet/ShapeNet)에 포함되지 않은 새로운 미학습 객체 4종(주전자 Teapot, 토끼 Stanford Bunny, 손 Hand, 인체 Human body)에 대한 일반화 성능 평가.

• 1행 (Original Shape): 원본 객체 포인트 클라우드 (깊이 정보에 따른 무지개색 코딩).
• 2행 (Critical Point Sets): 임계 포인트 세트가 주전자 주구/손잡이, 토끼 귀/발, 손가락 끝/관절, 인체 사지 등 핵심 외곽 스켈레톤 구조를 정확히 포착.
• 3행 (Upper-bound Shapes): 점유 상한 형상 또한 각 객체의 전체 볼륨 윤곽을 충실하게 감싸며 모델의 뛰어난 일반화 능력을 입증.

Theorem 1. Suppose f: \mathcal{X} \to \mathbb{R} is a continuous set function w.r.t Hausdorff distance d_H(\cdot,\cdot) . \forall \epsilon > 0 , \exists a continuous function h and a symmetric function g(x_1, \dots, x_n) = \gamma \circ MAX , where \gamma is a continuous function, MAX is a vector max operator that takes n vectors as input and returns a new vector of the element-wise maximum, such that for any S \in \mathcal{X} ,

|f(S) - \gamma(MAX(h(x_1), \dots, h(x_n)))| < \epsilon

where x_1, \ldots, x_n are the elements of S extracted in certain

order,

Proof. By the continuity of f, we take \delta_{\epsilon} so that |f(S) - f(S')| < \epsilon for any S, S' \in \mathcal{X} if d_H(S, S') < \delta_{\epsilon} .

Define K = \lceil 1/\delta_{\epsilon} \rceil , which split [0,1] into K intervals evenly and define an auxiliary function that maps a point to the left end of the interval it lies in:

\sigma(x) = \frac{\lfloor Kx \rfloor}{K}

Let \tilde{S} = {\sigma(x) : x \in S} , then

|f(S) - f(\tilde{S})| < \epsilon

because d_H(S, \tilde{S}) < 1/K \le \delta_{\epsilon} .

Let h_k(x) = e^{-d(x, \lfloor \frac{k-1}{K}, \frac{k}{K} \rfloor)} be a soft indicator function where d(x, I) is the point to set (interval) distance. Let \mathbf{h}(x) = [h_1(x); \dots; h_K(x)] , then \mathbf{h} : \mathbb{R} \to \mathbb{R}^K .

Let v_j(x_1, \ldots, x_n) = \max\{\hat{h}_j(x_1), \ldots, \hat{h}_j(x_n)\} , indicating the occupancy of the j-th interval by points in S. Let \mathbf{v} = [v_1; \ldots; v_K] , then \mathbf{v} : \underbrace{\mathbb{R} \times \ldots \times \mathbb{R}}_n \to \{0, 1\}^K

is a symmetric function, indicating the occupancy of each interval by points in S.

Define \tau:\{0,1\}^K\to\mathcal{X} as \tau(v)=\{\frac{k-1}{K}:v_k\geq 1\} , which maps the occupancy vector to a set which contains the left end of each occupied interval. It is easy to show:

\tau(\mathbf{v}(x_1,\ldots,x_n)) \equiv \tilde{S}

where x_1, \ldots, x_n are the elements of S extracted in certain order

Let \gamma:\mathbb{R}^K\to\mathbb{R} be a continuous function such that \gamma(\mathbf{v})=f(\tau(\mathbf{v})) for v\in\{0,1\}^K . Then,

|\gamma(\mathbf{v}(x_1,\ldots,x_n)) - f(S)|

=|f(\tau(\mathbf{v}(x_1,\ldots,x_n))) - f(S)| < \epsilon

Note that \gamma(\mathbf{v}(x_1,\ldots,x_n)) can be rewritten as follows:

\gamma(\mathbf{v}(x_1,\ldots,x_n)) = \gamma(\mathbf{MAX}(\mathbf{h}(x_1),\ldots,\mathbf{h}(x_n))) = (\gamma \circ \mathbf{MAX})(\mathbf{h}(x_1),\ldots,\mathbf{h}(x_n))

Obviously \gamma \circ MAX is a symmetric function. \square

Next we give the proof of Theorem 2. We define \mathbf{u} = \underset{x_i \in S}{\operatorname{MAX}} \{h(x_i)\} to be the sub-network of f which maps a point set in [0,1]^m to a K-dimensional vector. The following theorem tells us that small corruptions or extra noise points in the input set is not likely to change the output of our network:

Theorem 2. Suppose \mathbf{u}: \mathcal{X} \to \mathbb{R}^K such that \mathbf{u} = \max_{x \in S} \{h(x_i)\} and f = \gamma \circ \mathbf{u} . Then,

(a) ∀S, ∃ CS, N^S ⊆ X , f(T) = f(S) if C^S ⊆ T ⊆ NS*;*

(b) |CS| ≤ K

Proof. Obviously, ∀S ∈ X , f(S) is determined by u(S). So we only need to prove that ∀S, ∃ CS, N^S ⊆ X , f(T) = f(S)if C^S ⊆ T ⊆ NS.

For the jth dimension as the output of u, there exists at least one x^j ∈ X such that h^j (x^j ) = u^j , where h^j is the jth dimension of the output vector from h. Take C^S as the union of all x^j for j = 1, . . . , K. Then, C^S satisfies the above condition.

Adding any additional points x such that h(x) ≤ u(S) at all dimensions to C^S does not change u, hence f. Therefore, T^S can be obtained adding the union of all such points to NS.

Figure 19. Point function visualization. For each per-point function h, we calculate the values h(p) for all the points p in a cube of diameter two located at the origin, which spatially covers the unit sphere to which our input shapes are normalized when training our PointNet. In this figure, we visualize all the points p that give h(p) > 0.5 with function values color-coded by the brightness of the voxel. We randomly pick 15 point functions and visualize the activation regions for them.

[그림 해설] PointNet 내부의 per-point 함수 h 중 무작위로 선택된 15개 뉴런의 3D 공간 활성화 영역(h(p) > 0.5) 시각화.

• 원점을 중심으로 한 3D 큐브 공간 내에서 각 뉴런이 활성화되는 3D 볼륨(회색 음영 영역)을 3행 5열 격자로 배열.
• 특정 뉴런은 구의 상단 반구, 모서리 단면, 쐐기형 슬라이스, 중심부 국소 구체 등 다양한 형태의 기하학적 3D 공간 필터(공간 분할 함수) 역할을 수행함을 확인.

H. More Visualizations

Classification Visualization We use t-SNE[15] to embed point cloud global signature (1024-dim) from our classification PointNet into a 2D space. Fig 20 shows the embedding space of ModelNet 40 test split shapes. Similar shapes are clustered together according to their semantic categories.

Segmentation Visualization We present more segmentation results on both complete CAD models and simulated Kinect partial scans. We also visualize failure cases with error analysis. Fig 21 and Fig 22 show more segmentation results generated on complete CAD models and their simulated Kinect scans. Fig 23 illustrates some failure cases. Please read the caption for the error analysis.

Scene Semantic Parsing Visualization We give a visualization of semantic parsing in Fig 24 where we show input point cloud, prediction and ground truth for both semantic segmentation and object detection for two office rooms and one conference room. The area and the rooms are unseen in the training set.

Point Function Visualization Our classification Point-Net computes K (we take K = 1024 in this visualization) dimension point features for each point and aggregates all the per-point local features via a max pooling layer into a single K-dim vector, which forms the global shape descriptor.

To gain more insights on what the learnt per-point functions h's detect, we visualize the points pi's that give high per-point function value f(pi) in Fig 19. This visualization clearly shows that different point functions learn to detect for points in different regions with various shapes scattered in the whole space.

Figure 20. 2D embedding of learnt shape global features. We use t-SNE technique to visualize the learnt global shape features for the shapes in ModelNet40 test split.

[그림 해설] ModelNet40 테스트셋 전체 형상들에 대해 PointNet이 학습한 1024차원 글로벌 형상 특징 벡터의 2D t-SNE 임베딩 시각화.

• 비행기, 의자, 테이블, 자동차, 병, 화분 등 동일 범주 및 유사 기하 구조를 가진 3D 객체들이 2차원 공간 상에서 밀집된 클러스터를 형성하고 상호 분리됨을 보여줌.

Figure 21. PointNet segmentation results on complete CAD models.

[그림 해설] 완전한 ShapeNet 3D CAD 모델에 대한 파트 분할 결과 (총 16개 카테고리, 카테고리당 3개씩 총 48개 객체 시각화).

• 좌측: airplane, bag, cap, car, chair, earphone, guitar, knife.
• 우측: rocket, pistol, table, skateboard, motorbike, mug, laptop, lamp.
• 세부 부품(비행기 엔진, 가방 손잡이, 자동차 바퀴/유리, 의자 등받이/다리, 헤드폰 밴드/패드, 기타 넥/바디 등)이 선명한 고유 색상으로 정확하게 분할됨.

Figure 22. PointNet segmentation results on simulated Kinect scans.

[그림 해설] 시뮬레이션된 Kinect 스캔(단면 결손 및 가림이 존재하는 부분 스캔 데이터)에 대한 파트 분할 결과 (16개 카테고리별 각 3개씩 시각화).

• 심한 점 결손과 가림 현상 속에서도 비행기 날개, 자동차 프레임, 의자 좌판, 모자 챙 등 부품 영역을 견고하게 인식하여 분할함.

Figure 23. PointNet segmentation failure cases. In this figure, we summarize six types of common errors in our segmentation application. The prediction and the ground-truth segmentations are given in the first and second columns, while the difference maps are computed and shown in the third columns. The red dots correspond to the wrongly labeled points in the given point clouds. (a) illustrates the most common failure cases: the points on the boundary are wrongly labeled. In the examples, the label predictions for the points near the intersections between the table/chair legs and the tops are not accurate. However, most segmentation algorithms suffer from this error. (b) shows the errors on exotic shapes. For examples, the chandelier and the airplane shown in the figure are very rare in the data set. (c) shows that small parts can be overwritten by nearby large parts. For example, the jet engines for airplanes (yellow in the figure) are mistakenly classified as body (green) or the plane wing (purple). (d) shows the error caused by the inherent ambiguity of shape parts. For example, the two bottoms of the two tables in the figure are classified as table legs and table bases (category other in [29]), while ground-truth segmentation is the opposite. (e) illustrates the error introduced by the incompleteness of the partial scans. For the two caps in the figure, almost half of the point clouds are missing. (f) shows the failure cases when some object categories have too less training data to cover enough variety. There are only 54 bags and 39 caps in the whole dataset for the two categories shown here.

[그림 해설] PointNet 세그멘테이션의 6가지 주요 실패/오류 유형 분석 ((a)~(f) 각 유형별로 1열: Prediction, 2열: Ground-truth, 3열: 빨간 점으로 표시된 Difference map).

• (a) Boundary Error (경계 오류): 탁자/의자 상판과 다리 결합 부위 등 경계면 포인트의 오분류 (가장 흔한 오류).
• (b) Exotic Shapes (특이 형상): 샹들리에나 특이한 형태의 비행기 등 훈련 세트에 드문 희귀 형상에서의 오류.
• (c) Overwriting Small Parts (소형 부품 덮어쓰기): 제트 엔진(노란색) 같은 작은 부품이 인접한 큰 부품(동체/날개) 레이블로 덮어쓰여지는 현상.
• (d) Inherent Part Ambiguity (부품 정의의 모호성): 테이블 하단 지지 구조를 다리(leg)로 보는지 받침대(base)로 보는지에 대한 GT와의 정의 불일치.
• (e) Incomplete Scans (불완전 스캔): 모자 포인트 클라우드의 절반 가까이가 누락되어 형상 전체 맥락 파악 실패.
• (f) Lack of Training Data (데이터 부족): 가방(54개), 모자(39개) 등 훈련 데이터 수가 극히 적은 클래스에서 다양성 부족으로 인한 실패.

Figure 24. Examples of semantic segmentation and object detection. First row is input point cloud, where walls and ceiling are hided for clarity. Second and third rows are prediction and ground-truth of semantic segmentation on points, where points belonging to different semantic regions are colored differently (chairs in red, tables in purple, sofa in orange, board in gray, bookcase in green, floors in blue, windows in violet, beam in yellow, column in magenta, doors in khaki and clutters in black). The last two rows are object detection with bounding boxes, where predicted boxes are from connected components based on semantic segmentation prediction.

[그림 해설] 실내 대규모 3D 씬(3개 방/오피스)에 대한 시맨틱 분할 및 3D 바운딩 박스 객체 검출(Object Detection) 결과 (5개 행으로 구성).

• 1행 (Input Point Cloud): 천장과 벽을 일부 제거하여 내부를 가시화한 실내 포인트 클라우드 입력.
• 2행 (pred - Semantic Segmentation): PointNet의 포인트별 시맨틱 클래스 예측 결과 (의자: 빨강, 테이블: 보라, 책장: 초록, 바닥: 파랑, 빔: 노랑, 소파: 주황 등).
• 3행 (GT - Semantic Segmentation): 실제 정답 시맨틱 분할.
• 4행 (pred - Object Detection): 시맨틱 분할의 연결 요소(Connected Components)를 기반으로 추출된 3D 바운딩 박스 객체 검출 결과.
• 5행 (GT - Object Detection): 실제 정답 3D 바운딩 박스. 예측 박스가 정답 박스의 위치와 크기를 거의 정확히 추정함.