PDE-Based 3D Surface Reconstruction from Multi-View 2D Images
<p>The pipeline of the proposed PDE-based 3D surface reconstruction method from multi-view images.</p> "> Figure 2
<p>(<b>a</b>) Scene setting; (<b>b</b>) rendered multi-view 2D images.</p> "> Figure 3
<p>(<b>a</b>) Reconstructed cylinder point cloud with the Meshroom algorithm; (<b>b</b>) magnified view of the reconstructed point cloud in (<b>a</b>); (<b>c</b>) magnified view of the reconstructed point cloud in <a href="#mathematics-10-00542-f004" class="html-fig">Figure 4</a>c with Colmap algorithm.</p> "> Figure 4
<p>(<b>a</b>) Input to Colmap: multi-view 2D images; (<b>b</b>) 3D point cloud reconstruction from multi-view 2D images; (<b>c</b>) reconstructed 3D point cloud.</p> "> Figure 5
<p>(<b>a</b>) Cylinder in the cylindrical coordinate system; (<b>b</b>) parameterizing point cloud of cylinder shape.</p> "> Figure 6
<p>(<b>a</b>) Fitting plane to the point clouds; (<b>b</b>) projecting points to the projecting plane (u, v plane).</p> "> Figure 7
<p>(<b>a</b>) Reconstructed 3D point cloud of a cylinder shape from multi-view 2D images; (<b>b</b>) reconstructed PDE surface using a single PDE model with 16 variables; (<b>c</b>) reconstructed PDE surface using a single PDE model with 64 variables; (<b>d</b>) reconstructed PDE surface using two PDE models with 16 variables; (<b>e</b>) segmented point cloud.</p> "> Figure 8
<p>(<b>a</b>) Point set of a bowl; (<b>b</b>) surface reconstructed using Poisson; (<b>c</b>) PDE-based surface using single 16-variables PDE model; (<b>d</b>) PDE-based surface using single 64-variables PDE model.</p> "> Figure 9
<p>(<b>a</b>) The ground truth of a bench surface; (<b>b</b>) point set of a bench surface; (<b>c</b>) surface reconstructed using Poisson; (<b>d</b>) PDE-based surface using a single 16-variables PDE model; and (<b>e</b>) PDE-based surface using a single 64-variables PDE model.</p> "> Figure 10
<p>(<b>a</b>) The ground truth of a slide surface; (<b>b</b>) point set of a slide surface; (<b>c</b>) surface reconstructed using Poisson after segmentation; (<b>d</b>) PDE-based surface using a single 16-variable PDE model; (<b>e</b>) PDE-based surface using a single 64-variable PDE model.</p> "> Figure 11
<p>(<b>a</b>) The point cloud of a hat; (<b>b</b>) segmented 2 subsets; (<b>c</b>) reconstructed PDE-based surface using 2 PDE patches defined by the 64-variables PDE model; (<b>d</b>) segmented 3 subsets; (<b>e</b>) reconstructed PDE-based surface using 3 PDE patches defined by the 64-variable PDE mode.</p> "> Figure 12
<p>(<b>a</b>) The point cloud of a truck; (<b>b</b>) segmented subsets; (<b>c</b>) reconstructed PDE-based surface.</p> ">
Abstract
:1. Introduction
2. Related Work
3. PDE Model
4. PDE-Based Surface Reconstruction from Multi-View Images
4.1. Multi-View Images Generation
4.2. Point Cloud Reconstruction from Multi-View Images
4.3. PDE-Based Surface Reconstruction from Point Clouds
4.3.1. Segmentation and Parameterization of Point Cloud
4.3.2. Fitting PDE Model to Point Cloud
- 1
- PDE model with 16 variables
- 2
- PDE model with 64 variables
5. Empirical Results
6. Conclusions and Future Work
Author Contributions
Funding
Institutional Review Board Statement
Informed Consent Statement
Data Availability Statement
Conflicts of Interest
References
- Nocerino, E.; Stathopoulou, E.K.; Rigon, S.; Remondino, F. Surface Reconstruction Assessment in Photogrammetric Applications. Sensors 2020, 20, 5863. [Google Scholar] [CrossRef]
- Nicolae, C.; Nocerino, E.; Menna, F.; Remondino, F. Photogrammetry Applied to Problematic Artefacts. Int. Arch. Photogramm. Remote Sens. Spat. Inf. Sci. 2014, 40, 451. [Google Scholar] [CrossRef] [Green Version]
- Toschi, I.; Ramos, M.M.; Nocerino, E.; Menna, F.; Remondino, F.; Moe, K.; Poli, D.; Legat, K.; Fassi, F. Oblique photogrammetry supporting 3D urban reconstruction of complex scenarios. Int. Arch. Photogramm. Remote Sens. Spat. Inf. Sci. 2017, 8, 519–526. [Google Scholar] [CrossRef] [Green Version]
- Zeng, G.; Paris, S.; Quan, L.; Sillion, F.X. Progressive Surface Reconstruction from Images Using a Local Prior. In Proceedings of the Tenth IEEE International Conference on Computer Vision (ICCV’05), Beijing, China, 17–21 October 2005; Volume 1, p. 9. [Google Scholar]
- Lhuillier, M.; Quan, L. A Quasi-Dense Approach to Surface Reconstruction from Uncalibrated Iages. IEEE Trans. Pattern Anal. Mach. Intell. 2005, 27, 418–433. [Google Scholar] [CrossRef] [Green Version]
- Maiti, A.; Chakravarty, D. Performance Analysis of Different Surface Reconstruction Algorithms for 3D Reconstruction of Outdoor Objects from Their Digital Images. SpringerPlus 2016, 5, 932. [Google Scholar] [CrossRef] [Green Version]
- Deepika, S.; Sanjeev, K.; Gulati, T.R. NURBS-Based Geometric Inverse Reconstruction of Free-Form Shaped Objects. JKSU-Comput. Inf. Sci. 2014, 29, 116–133. [Google Scholar]
- Zhao, C.; Mohr, R. Relative 3D Regularized B-Spline Surface Reconstruction through Image Sequences. In European Conference on Computer Vision; Springer: Berlin/Heidelberg, Germany, 1994; pp. 417–426. [Google Scholar]
- Zhao, C.; Mohr, R. Global Three-Dimensional Surface Reconstruction from Occluding Contours. Comput. Vis. Image Underst. 1996, 64, 62–96. [Google Scholar] [CrossRef] [Green Version]
- Han, X.-F.; Laga, H.; Bennamoun, M. Image-Based 3D Object Reconstruction: State-of-the-Art and Trends in the Deep Learning Era. IEEE Trans. Pattern Anal. Mach. Intell. 2021, 43, 1578–1604. [Google Scholar] [CrossRef] [Green Version]
- Ozyesil, O.; Voroninski, V.; Basri, R.; Singer, A. A survey of structure from motion. Acta Numer. 2007, 26, 305–364. [Google Scholar] [CrossRef]
- Heinly, J.; Schonberger, J.L.; Dunn, E.; Frahm, J.-M. Reconstructing the World* in Six Days* (as Captured by the Yahoo 100 Million Image Dataset). In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Boston, MA, USA, 7–12 June 2015; pp. 3287–3295. [Google Scholar]
- Schonberger, J.L.; Frahm, J.-M. Structure-from-Motion Revisited. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA, 27–30 June 2016; pp. 4104–4113. [Google Scholar]
- Snavely, N.; Seitz, S.M.; Szeliski, R. Photo Tourism: Exploring Photo Collections in 3D. In ACM Siggraph 2006 Papers; ACM: New York, NY, USA, 2006; pp. 835–846. Available online: http://phototour.cs.washington.edu/Photo_Tourism.pdf (accessed on 1 July 2006).
- Furukawa, Y.; Ponce, J. Accurate, Dense, and Robust Multi-View Stereopsis (PMVS). In Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, Minneapolis, MN, USA, 17–22 June 2007; Volume 2. [Google Scholar]
- Bailer, C.; Finckh, M.; Lensch, H.P. Scale Robust Multi View Stereo. In Proceedings of the European Conference on Computer Vision, Florence, Italy, 7–13 October 2012; pp. 398–411. [Google Scholar]
- Shan, Q.; Curless, B.; Furukawa, Y.; Hernandez, C.; Seitz, S.M. Occluding Contours for Multi-View Stereo. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Columbus, OH, USA, 23–28 June 2014; pp. 4002–4009. [Google Scholar]
- Schönberger, J.L.; Zheng, E.; Frahm, J.-M.; Pollefeys, M. Pixelwise View Selection for Unstructured Multi-View Stereo. In Proceedings of the European Conference on Computer Vision, Glasgow, UK, 23–28 August 2016; Springer: Berlin/Heidelberg, Germany, 2016; pp. 501–518. [Google Scholar]
- Wang, S.; Xia, Y.; You, L.; Zhang, J. Reconstruction of Curve Networks from Unorganized Spatial Points. J. Univers. Comput. Sci. 2020, 26, 1265–1280. [Google Scholar] [CrossRef]
- Berger, M.; Tagliasacchi, A.; Seversky, L.M.; Alliez, P.; Guennebaud, G.; Levine, J.A.; Sharf, A.; Silva, C.T. A Survey of Surface Reconstruction from Point Clouds. Comput. Graph. Forum 2017, 36, 301–329. [Google Scholar] [CrossRef] [Green Version]
- Lei, J.; Sridhar, S.; Guerrero, P.; Sung, M.; Mitra, N.; Guibas, L.J. Pix2surf: Learning Parametric 3d Surface Models of Objects from Images. In Proceedings of the European Conference on Computer Vision, Glasgow, UK, 23–28 August 2020; Springer: Berlin/Heidelberg, Germany, 2020; pp. 121–138. [Google Scholar]
- Kato, H.; Ushiku, Y.; Harada, T. Neural 3d Mesh Renderer. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA, 18–23 June 2018; pp. 3907–3916. [Google Scholar]
- Sun, X.; Wu, J.; Zhang, X.; Zhang, Z.; Zhang, C.; Xue, T.; Tenenbaum, J.B.; Freeman, W.T. Pix3d: Dataset and Methods for Single-Image 3d Shape Modeling. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA, 18–23 June 2018; pp. 2974–2983. [Google Scholar]
- Wu, J.; Wang, Y.; Xue, T.; Sun, X.; Freeman, B.; Tenenbaum, J. Marrnet: 3d Shape Reconstruction via 2.5 d Sketches. arXiv 2017, arXiv:1711.03129. [Google Scholar]
- Fu, K.; Peng, J.; He, Q.; Zhang, H. Single image 3D object reconstruction based on deep learning: A review. Multimed. Tools Appl. 2021, 80, 463–498. [Google Scholar] [CrossRef]
- Mandikal, P.; Navaneet, K.L.; Agarwal, M.; Babu, R.V. 3D-LMNet: Latent Embedding Matching for Accurate and Diverse 3D Point Cloud Reconstruction from a Single Image. arXiv 2018, arXiv:1807.07796. [Google Scholar]
- Navaneet, K.L.; Mathew, A.; Kashyap, S.; Hung, W.-C.; Jampani, V.; Babu, R.V. From Image Collections to Point Clouds with Self-Supervised Shape and Pose Networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA, 18–23 June 2018; pp. 1132–1140. [Google Scholar]
- Mandikal, P.; KL, N.; Venkatesh Babu, R. 3D-PSRNet: Part Segmented 3D Point Cloud Reconstruction from a Single Image. In Proceedings of the European Conference on Computer Vision (ECCV) Workshops, Munich, Germany, 8–14 September 2018. [Google Scholar]
- Lu, Q.; Xiao, M.; Lu, Y.; Yuan, X.; Yu, Y. Attention-Based Dense Point Cloud Reconstruction from a Single Image. IEEE Access 2019, 7, 137420–137431. [Google Scholar] [CrossRef]
- Yi, H.; Wei, Z.; Ding, M.; Zhang, R.; Chen, Y.; Wang, G.; Tai, Y.-W. Pyramid Multi-View Stereo Net with Self-Adaptive View Aggregation. In Proceedings of the European Conference on Computer Vision, Salt Lake City, UT, USA, 18–23 June 2018; Springer: Berlin/Heidelberg, Germany, 2020; pp. 766–782. [Google Scholar]
- Zaheer, A.; Rashid, M.; Riaz, M.A.; Khan, S. Single-View Reconstruction using orthogonal line-pairs. Comput. Vis. Image Underst. 2018, 172, 107–123. [Google Scholar] [CrossRef]
- Zou, Q. A PDE Model for Smooth Surface Reconstruction from 2D Parallel Slices. IEEE Signal Processing Lett. 2020, 27, 1015–1019. [Google Scholar] [CrossRef]
- Duan, Y.; Yang, L.; Qin, H.; Samaras, D. Shape Reconstruction from 3D and 2D Data Using PDE-Based Deformable Surfaces. In Proceedings of the European Conference on Computer Vision, Prague, Czech Republic, 11–14 May 2004; Springer: Berlin/Heidelberg, Germany, 2004; pp. 238–251. [Google Scholar]
- Othman, M.N.M.; Yusoff, Y.; Haron, H.; You, L. An Overview of Surface Reconstruction Using Partial Differential Equation (PDE). In Proceedings of the IOP Conference Series: Materials Science and Engineering; IOP Publishing: Bristol, UK, 2019; Volume 551, p. 012054. [Google Scholar] [CrossRef] [Green Version]
- Zhu, Z.; Chaudhry, E.; Wang, S.; Xia, Y.; Iglesias, A.; You, L.; Zhang, J.J. Shape Reconstruction from Point Clouds Using Closed Form Solution of a Fourth-Order Partial Differential Equation. In Proceedings of the International Conference on Computational Science, Krakow, Poland, 16–18 June 2021; Springer: Berlin/Heidelberg, Germany, 2021; pp. 207–220. [Google Scholar]
- Bianco, S.; Ciocca, G.; Marelli, D. Evaluating the Performance of Structure from Motion Pipelines. J. Imaging 2018, 4, 98. [Google Scholar] [CrossRef] [Green Version]
- Wu, C. Towards Linear-Time Incremental Structure from Motion. In Proceedings of the 2013 International Conference on 3D Vision-3DV 2013, Seattle, WA, USA, 29 June–1 July 2013; pp. 127–134. [Google Scholar]
- Grilli, E.; Menna, F.; Remondino, F. A Review of Point Clouds Segmentation and Classification Algorithms. Int. Arch. Photogramm. Remote Sens. Spat. Inf. Sci. 2017, 42, 330–344. [Google Scholar] [CrossRef] [Green Version]
- He, Y.; Yu, H.; Liu, X.; Yang, Z.; Sun, W.; Wang, Y.; Fu, Q.; Zou, Y.; Mian, A. Deep Learning Based 3D Segmentation: A Survey. arXiv 2021, arXiv:2103.05423. [Google Scholar]
- Choi, G.P.T.; Liu, Y.; Lui, L.M. Free-Boundary Conformal Parameterization of Point Clouds. J. Sci. Comput. 2022, 90, 14. [Google Scholar] [CrossRef]
- Azariadis, P.N. Parameterization of clouds of unorganized points using dynamic base surfaces. Comput. Aided Des. 2004, 36, 607–623. [Google Scholar] [CrossRef]
- Ma, W.; Kruth, J.-P. Parameterization of randomly measured points for least squares fitting of B-spline curves and surfaces. Comput. Aided Des. 1995, 27, 663–675. [Google Scholar] [CrossRef]
- Park, I.K.; Yun, D., II; Lee, S.U. Constructing NURBS surface model from scattered and unorganized range data. In Proceedings of the Second International Conference on 3-D Digital Imaging and Modeling (Cat. No.PR00062), Ottawa, ON, Canada, 8 October 1999; pp. 312–320. [Google Scholar]
Methods Errors | Ground Truth to Polygon Surface | Ground Truth to PDE-Based Surface with 16 Variables | Ground Truth to PDE-Based Surface with 64 Variables |
---|---|---|---|
Mean distance | 0.039 | 0.031 | 0.029 |
Standard deviation | 0.016 | 0.021 | 0.015 |
Publisher’s Note: MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations. |
© 2022 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/).
Share and Cite
Zhu, Z.; Iglesias, A.; Zhou, L.; You, L.; Zhang, J. PDE-Based 3D Surface Reconstruction from Multi-View 2D Images. Mathematics 2022, 10, 542. https://doi.org/10.3390/math10040542
Zhu Z, Iglesias A, Zhou L, You L, Zhang J. PDE-Based 3D Surface Reconstruction from Multi-View 2D Images. Mathematics. 2022; 10(4):542. https://doi.org/10.3390/math10040542
Chicago/Turabian StyleZhu, Zaiping, Andres Iglesias, Liqi Zhou, Lihua You, and Jianjun Zhang. 2022. "PDE-Based 3D Surface Reconstruction from Multi-View 2D Images" Mathematics 10, no. 4: 542. https://doi.org/10.3390/math10040542
APA StyleZhu, Z., Iglesias, A., Zhou, L., You, L., & Zhang, J. (2022). PDE-Based 3D Surface Reconstruction from Multi-View 2D Images. Mathematics, 10(4), 542. https://doi.org/10.3390/math10040542