Multiway Non-rigid Point Cloud Registration via Learned Functional Map Synchronization

We present SyNoRiM, a novel way to jointly register Multiple Non-Rigid shapes by Synchronizing the maps that relate learned functions defined on the point clouds. Even though the ability to process non-rigid shapes is critical in various applications ranging from computer animation to 3D digitization, the literature still lacks a robust and flexible framework to match and align a collection of real, noisy scans observed under occlusions. Given a set of such point clouds, our method first computes the pairwise correspondences parameterized via functional maps. We simultaneously learn potentially non-orthogonal basis functions to effectively regularize the deformations, while handling the occlusions in an elegant way. To maximally benefit from the multi-way information provided by the inferred pairwise deformation fields, we synchronize the pairwise functional maps into a cycle-consistent whole thanks to our novel and principled optimization formulation. We demonstrate via extensive experiments that our method achieves a state-of-the-art performance in registration accuracy, while being flexible and efficient as we handle both non-rigid and multi-body cases in a unified framework and avoid the costly optimization over point-wise permutations by the use of basis function maps.

Method

The input to our method is a set of point clouds $\{ \mathbf{X}_k \in \mathbb{R}^{N_k \times 3} \}$ and the output of our method contains all the pairwise per-point 3D flow vectors $\{ \mathbf{F}_{kl} \in \mathbb{R}^{N_k \times 3} \}$. The flow vectors naturally induce the non-rigid warp field from $\mathbf{X}_k$ to $\mathbf{X}_l$ as $ \mathcal{W}_k (\mathbf{X}_k) = \mathbf{X}_k + \mathbf{F}_{kl} $, optimally aligning the given point cloud pairs by deforming the source $\mathbf{X}_k$ onto the target $\mathbf{X}_l$. For multiple input clouds we additionally encourage the cyclic consistency of the flow estimates, i.e.: $ \mathcal{W}_{k_1} \circ \mathcal{W}_{k_2} \circ ... \circ \mathcal{W}_{k_p} \approx \mathcal{I} $ for all cycles in the densely-connected graph formed by the input set.

During training, our method is supervised in a pairwise fashion. We first ① use a sparse-convolution-based neural network $\varphi_{\mathrm{desc}}$ (i.e., $\mathfrak{g}_\mathrm{pd}$) to establish putative correspondences between each point cloud pair. We then ② estimate a set of basis functions $\{ \mathbf{\Phi}_k \in \mathbb{R}^{N_k \times M} \}$ for each point cloud using another network $\varphi_{\mathrm{basis}}$. By jointly modeling the correspondences and the bases in a robust way, the initial functional map $\mathbf{C}_{kl}^0$ is obtained before being refined with $\varphi_{\mathrm{refine}}$ that yields pairwise flow estimates.

During test time with multiple inputs, we estimate the map set $\{\mathbf{C}_{kl}^0\}$ for all pairs. ③ The maps are subsequently synchronized to optimize for cycle consistency among the inputs. Finally, ④ 3D flows are estimated from the optimized functional maps $\{\mathbf{C}_{kl}^\star\}$ as our final output, using the same procedure as done in the pairwise setting. The registered point cloud is a fusion of all initial point clouds warped by the estimated flows. For more details, please read our technical report.

Results

Some demonstrative registration results are shown below, all rendered as raw point clouds. For each animation the points from all other sources are warped to the current view as target. We use datasets from CAPE, DeformingThings4D, DeepDeform, and SAPIEN.

We also provide extensive quantitative experiments and ablations in our main paper. Our results on FAUST online challenge can also be viewed here.

Dataset

Our dataset is heavily based on published datasets. Although our modification is license-free, the original data providers may require you to accept different terms & conditions before being allowed to use them. Be sure to double-check that before downloading. If you find any legal issues, please let us know immediately.

Citation

If you find our work interesting, please consider citing us:

@article{huang2021multiway,
  title={Multiway Non-rigid Point Cloud Registration via Learned Functional Map Synchronization},
  author={Huang, Jiahui and Birdal, Tolga and Gojcic, Zan and Guibas, Leonidas J and Hu, Shi-Min},
  journal={arXiv preprint arXiv:2111.12878},
  year={2021}
}

Acknowledgements

We thank all the reviewers for their thoughtful comments and constructive suggestions. This paper was supported by National Key R&D Program of China (project No. 2021ZD0112902), a Vannevar Bush faculty fellowship, ARL grant W911NF2120104, NSF grant IIS-1763268, and a gift from the Autodesk Corporation.