PointRegGPT: Boosting 3D Point Cloud Registration Using Generative Point-Cloud Pairs for Training
Authors: 
Suyi Chen, Hao Xu, Haipeng Li, Kunming Luo, Guanghui Liu, Chi-Wing Fu, Ping Tan, Shuaicheng LiuAuthors Info & Claims
Pages 272 - 289
Abstract
Data plays a crucial role in training learning-based methods for 3D point cloud registration. However, the real-world dataset is expensive to build, while rendering-based synthetic data suffers from domain gaps. In this work, we present PointRegGPT, boosting 3D Point cloud Registration using Generative Point-cloud pairs for Training. Given a single depth map, we first apply a random camera motion to re-project it into a target depth map. Converting them to point clouds gives a training pair. To enhance the data realism, we formulate a generative model as a depth inpainting diffusion to process the target depth map with the re-projected source depth map as the condition. Also, we design a depth correction module to alleviate artifacts caused by point penetration during the re-projection. To our knowledge, this is the first generative approach that explores realistic data generation for indoor 3D point cloud registration. When equipped with our approach, several recent algorithms can improve their performance significantly and achieve SOTA consistently on two common benchmarks. The code and dataset will be released on https://github.com/Chen-Suyi/PointRegGPT.
References
[1]
Ao, S., Hu, Q., Wang, H., Xu, K., Guo, Y.: BUFFER: balancing accuracy, efficiency, and generalizability in point cloud registration. In: CVPR, pp. 1255–1264 (2023)
[2]
Ao, S., Hu, Q., Yang, B., Markham, A., Guo, Y.: SpinNet: learning a general surface descriptor for 3D point cloud registration. In: CVPR, pp. 11753–11762 (2021)
[3]
Aoki, Y., Goforth, H., Srivatsan, R.A., Lucey, S.: PointNetLK: robust & efficient point cloud registration using PointNet. In: CVPR, pp. 7163–7172 (2019)
[4]
Bai, X., et al.: PointDSC: robust point cloud registration using deep spatial consistency. In: CVPR, pp. 15859–15869 (2021)
[5]
j Bai, X., Luo, Z., Zhou, L., Fu, H., Quan, L., Tai, C.L.: D3Feat: joint learning of dense detection and description of 3D local features. In: CVPR, pp. 6359–6367 (2020)
[6]
Besl PJ and McKay ND A method for registration of 3D shapes IEEE TPAMI 1992 14 2 239-256
[7]
Chen, G., Wang, M., Yuan, L., Yang, Y., Yue, Y.: Rethinking point cloud registration as masking and reconstruction. In: ICCV, pp. 17717–17727 (2023)
[8]
Chen, S., Xu, H., Li, R., Liu, G., Fu, C.W., Liu, S.: SIRA-PCR: sim-to-real adaptation for 3D point cloud registration. In: ICCV, pp. 14394–14405 (2023)
[9]
Choy, C., Dong, W., Koltun, V.: Deep global registration. In: CVPR, pp. 2514–2523 (2020)
[10]
Choy, C., Park, J., Koltun, V.: Fully convolutional geometric features. In: ICCV, pp. 8958–8966 (2019)
[11]
Dang, Z., Salzmann, M.: AutoSynth: learning to generate 3D training data for object point cloud registration. In: ICCV, pp. 9009–9019 (2023)
[12]
Dhariwal, P., Nichol, A.: Diffusion models beat GANs on image synthesis. In: NeurIPS, vol. 34, pp. 8780–8794 (2021)
[13]
Fischler MA and Bolles RC Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography Commun. ACM 1981 24 6 381-395
[14]
Fu, K., Liu, S., Luo, X., Wang, M.: Robust point cloud registration framework based on deep graph matching. In: CVPR, pp. 8893–8902 (2021)
[15]
Gojcic, Z., Zhou, C., Wegner, J.D., Wieser, A.: The perfect match: 3D point cloud matching with smoothed densities. In: CVPR, pp. 5545–5554 (2019)
[16]
Hatem, A., Qian, Y., Wang, Y.: Point-TTA: test-time adaptation for point cloud registration using multitask meta-auxiliary learning. In: ICCV, pp. 16494–16504 (2023)
[17]
Ho, J., Jain, A., Abbeel, P.: Denoising diffusion probabilistic models. In: NeurIPS, vol. 33, pp. 6840–6851 (2020)
[18]
Ho, J., Salimans, T.: Classifier-free diffusion guidance. arXiv preprint arXiv:2207.12598 (2022)
[19]
Horache, S., Deschaud, J.E., Goulette, F.: 3D point cloud registration with multi-scale architecture and unsupervised transfer learning. In: 3DV, pp. 1351–1361. IEEE (2021)
[20]
Hsu, C., Sun, C., Chen, H.: Moving in a 360 world: Synthesizing panoramic parallaxes from a single panorama. CoRR abs/2106.10859 (2021). https://arxiv.org/abs/2106.10859
[21]
Huang, S., Gojcic, Z., Usvyatsov, M., Wieser, A., Schindler, K.: PREDATOR: registration of 3D point clouds with low overlap. arXiv:2011.13005 (2020)
[22]
Huang, X., Mei, G., Zhang, J.: Feature-metric registration: a fast semi-supervised approach for robust point cloud registration without correspondences. In: CVPR, pp. 11366–11374 (2020)
[23]
Jabri, A., Fleet, D., Chen, T.: Scalable adaptive computation for iterative generation. arXiv preprint arXiv:2212.11972 (2022)
[24]
Jiang, H., Dang, Z., Wei, Z., Xie, J., Yang, J., Salzmann, M.: Robust outlier rejection for 3D registration with variational Bayes. In: CVPR, pp. 1148–1157 (2023)
[25]
Karras, T., Aittala, M., Aila, T., Laine, S.: Elucidating the design space of diffusion-based generative models. In: NeurIPS (2022)
[26]
Lei, J., Tang, J., Jia, K.: RGBD2: generative scene synthesis via incremental view inpainting using RGBD diffusion models. In: CVPR, pp. 8422–8434 (2023)
[27]
Li H et al. DMHomo: learning homography with diffusion models ACM TOG 2024 43 3 1-16
[28]
Li J, Zhang C, Xu Z, Zhou H, and Zhang C Vedaldi A, Bischof H, Brox T, and Frahm J-M Iterative distance-aware similarity matrix convolution with mutual-supervised point elimination for efficient point cloud registration Computer Vision – ECCV 2020 2020 Cham Springer 378-394
[29]
Li, Y., Harada, T.: Lepard: learning partial point cloud matching in rigid and deformable scenes. In: CVPR (2022)
[30]
Liu, J., Wang, G., Liu, Z., Jiang, C., Pollefeys, M., Wang, H.: RegFormer: an efficient projection-aware transformer network for large-scale point cloud registration. In: ICCV, pp. 8451–8460 (2023)
[31]
Liu, Q., Zhu, H., Zhou, Y., Li, H., Chang, S., Guo, M.: Density-invariant features for distant point cloud registration. In: ICCV, pp. 18215–18225 (2023)
[32]
Liu, R., Wu, R., Van Hoorick, B., Tokmakov, P., Zakharov, S., Vondrick, C.: Zero-1-to-3: zero-shot one image to 3D object. In: CVPR, pp. 9298–9309 (2023)
[33]
Liu, X., et al.: More control for free! image synthesis with semantic diffusion guidance. In: WACV, pp. 289–299 (2023)
[34]
Luo, Z., Gustafsson, F.K., Zhao, Z., Sjölund, J., Schön, T.B.: Image restoration with mean-reverting stochastic differential equations. arXiv preprint arXiv:2301.11699 (2023)
[35]
Mei, G., et al.: Unsupervised deep probabilistic approach for partial point cloud registration. In: CVPR, pp. 13611–13620 (2023)
[36]
Pais, G.D., Ramalingam, S., Govindu, V.M., Nascimento, J.C., Chellappa, R., Miraldo, P.: 3DRegNet: a deep neural network for 3D point registration. In: CVPR, pp. 7193–7203 (2020)
[37]
Poiesi, F., Boscaini, D.: Learning general and distinctive 3D local deep descriptors for point cloud registration. IEEE TPAMI (2022)
[38]
Pomerleau F, Liu M, Colas F, and Siegwart R Challenging data sets for point cloud registration algorithms IJRR 2012 31 14 1705-1711
[39]
Qin, Z., Yu, H., Wang, C., Guo, Y., Peng, Y., Xu, K.: Geometric transformer for fast and robust point cloud registration. In: CVPR, pp. 11143–11152 (2022)
[40]
Rombach, R., Blattmann, A., Lorenz, D., Esser, P., Ommer, B.: High-resolution image synthesis with latent diffusion models. In: CVPR, pp. 10684–10695 (2022)
[41]
Rusinkiewicz S A symmetric objective function for LiDAR ACM TOG 2019 38 4 1-7
[42]
Rusinkiewicz, S., Levoy, M.: Efficient variants of the ICP algorithm. In: International Conference on 3-D Digital Imaging and Modeling (3DIM), pp. 145–152 (2001)
[43]
Rusu, R.B., Blodow, N., Beetz, M.: Fast point feature histograms (FPFH) for 3D registration. In: ICRA, pp. 3212–3217 (2009)
[44]
Saharia, C., Ho, J., Chan, W., Salimans, T., Fleet, D.J., Norouzi, M.: Image super-resolution via iterative refinement. IEEE TPAMI (2022)
[45]
Segal, A., Haehnel, D., Thrun, S.: Generalized-LiDAR. In: Robotics: Science and Systems, vol. 2, p. 435 (2009)
[46]
Sohl-Dickstein, J., Weiss, E., Maheswaranathan, N., Ganguli, S.: Deep unsupervised learning using nonequilibrium thermodynamics. In: ICML, pp. 2256–2265. PMLR (2015)
[47]
Song, J., Meng, C., Ermon, S.: Denoising diffusion implicit models. arXiv preprint arXiv:2010.02502 (2020)
[48]
Song, Y., Shen, L., Xing, L., Ermon, S.: Solving inverse problems in medical imaging with score-based generative models. arXiv preprint arXiv:2111.08005 (2021)
[49]
Tevet, G., Raab, S., Gordon, B., Shafir, Y., Cohen-Or, D., Bermano, A.H.: Human motion diffusion model. arXiv preprint arXiv:2209.14916 (2022)
[50]
Wang, H., Liu, Y., Dong, Z., Wang, W.: You only hypothesize once: point cloud registration with rotation-equivariant descriptors. In: IEEE TMM, pp. 1630–1641 (2022)
[51]
Wang, H., et al.: RoReg: pairwise point cloud registration with oriented descriptors and local rotations. IEEE TPAMI (2023)
[52]
Wang, Y., Yu, J., Zhang, J.: Zero-shot image restoration using denoising diffusion null-space model. arXiv preprint arXiv:2212.00490 (2022)
[53]
Wang, Y., Solomon, J.M.: Deep closest point: learning representations for point cloud registration. In: ICCV, pp. 3523–3532 (2019)
[54]
Wang, Y., Solomon, J.M.: PRNet: self-supervised learning for partial-to-partial registration. In: NeurIPS, pp. 8814–8826 (2019)
[55]
Xu, H., Li, H., Wang, Y., Liu, S., Fu, C.W.: HandBooster: boosting 3D hand-mesh reconstruction by conditional synthesis and sampling of hand-object interactions. In: CVPR, pp. 10159–10169 (2024)
[56]
Xu, H., Liu, S., Wang, G., Liu, G., Zeng, B.: OMNet: learning overlapping mask for partial-to-partial point cloud registration. In: ICCV (2021)
[57]
Xu, H., Ye, N., Liu, G., Zeng, B., Liu, S.: FINet: dual branches feature interaction for partial-to-partial point cloud registration. In: AAAI, vol. 36, pp. 2848–2856 (2022)
[58]
Yang, F., Guo, L., Chen, Z., Tao, W.: One-inlier is first: towards efficient position encoding for point cloud registration. In: NeurIPS, vol. 35, pp. 6982–6995 (2022)
[59]
Yang, J., Li, H., Jia, Y.: Go-ICP: Solving 3D registration efficiently and globally optimally. In: CVPR, pp. 1457–1464 (2013)
[60]
Yang, Z., Li, H., Hong, M., Zeng, B., Liu, S.: Single image rolling shutter removal with diffusion models. arXiv preprint arXiv:2407.02906 (2024)
[61]
Yew, Z.J., Lee, G.H.: RPM-Net: robust point matching using learned features. In: CVPR, pp. 11824–11833 (2020)
[62]
Yew, Z.J., Lee, G.H.: REGTR: end-to-end point cloud correspondences with transformers. In: CVPR, pp. 6677–6686 (2022)
[63]
Yu H, Li F, Saleh M, Busam B, and Ilic S CoFiNet: Reliable coarse-to-fine correspondences for robust point cloud registration NeurIPS 2021 34 23872-23884
[64]
Yu, J., Ren, L., Zhang, Y., Zhou, W., Lin, L., Dai, G.: PEAL: Prior-embedded explicit attention learning for low-overlap point cloud registration. In: CVPR. pp. 17702–17711 (2023)
[65]
Zeng, A., Song, S., Nießner, M., Fisher, M., Xiao, J., Funkhouser, T.: 3DMatch: Learning local geometric descriptors from RGB-D reconstructions. In: CVPR. pp. 1802–1811 (2017)
[66]
Zhang, X., Yang, J., Zhang, S., Zhang, Y.: 3D registration with maximal cliques. In: CVPR. pp. 17745–17754 (June 2023)
[67]
Zhou, T., Li, H., Wang, Z., Luo, A., Zhang, C.L., Li, J., Zeng, B., Liu, S.: RecDiffusion: Rectangling for image stitching with diffusion models. In: CVPR. pp. 2692–2701 (2024)
Index Terms
- PointRegGPT: Boosting 3D Point Cloud Registration Using Generative Point-Cloud Pairs for Training
Index terms have been assigned to the content through auto-classification.
Recommendations
Accurate depth image generation via overfit training of point cloud registration using local frame sets
AbstractAccurate three-dimensional perception is a fundamental task in various computer vision applications. Recently, commercial RGB-depth (RGB-D) cameras have been widely adopted as single-view depth-sensing devices owing to their efficient ...
Highlights- We propose a method for a high-quality real-world depth dataset using only RGB-D stream dataset.
Tomographic surface reconstruction from point cloud
Inspired by computed tomography (CT), this paper presents a novel surface reconstruction algorithm, tomographic surface reconstruction, to reconstruct a surface mesh from a point cloud equipped with oriented normals. In the process of scanning a real ...
Registration of Structured Light Camera Point Cloud Data with CT Images
Intelligent Robotics and ApplicationsAbstractWith the advancement of structured-light cameras, surgical robots equipped with such cameras have been utilized for lesion localization during surgeries. Achieving precise registration between CT images and point cloud data remains a challenge. ...
Comments
Please enable JavaScript to view thecomments powered by Disqus.Information & Contributors
Information
Published In
Sep 2024
576 pages
ISBN:978-3-031-72982-9
DOI:10.1007/978-3-031-72983-6
© The Author(s), under exclusive license to Springer Nature Switzerland AG 2025.
Publisher
Springer-Verlag
Berlin, Heidelberg
Publication History
Published: 29 October 2024
Author Tags
Qualifiers
- Article
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 0Total Downloads
- Downloads (Last 12 months)0
- Downloads (Last 6 weeks)0
Reflects downloads up to 05 Jan 2025