[go: up one dir, main page]
More Web Proxy on the site http://driver.im/ Skip to main content
Springer Nature Link
Log in

3D Scene Reconstruction from Multiple Spherical Stereo Pairs

  • Published:
International Journal of Computer Vision Aims and scope Submit manuscript

Abstract

We propose a 3D environment modelling method using multiple pairs of high-resolution spherical images. Spherical images of a scene are captured using a rotating line scan camera. Reconstruction is based on stereo image pairs with a vertical displacement between camera views. A 3D mesh model for each pair of spherical images is reconstructed by stereo matching. For accurate surface reconstruction, we propose a PDE-based disparity estimation method which produces continuous depth fields with sharp depth discontinuities even in occluded and highly textured regions. A full environment model is constructed by fusion of partial reconstruction from spherical stereo pairs at multiple widely spaced locations. To avoid camera calibration steps for all camera locations, we calculate 3D rigid transforms between capture points using feature matching and register all meshes into a unified coordinate system. Finally a complete 3D model of the environment is generated by selecting the most reliable observations among overlapped surface measurements considering surface visibility, orientation and distance from the camera. We analyse the characteristics and behaviour of errors for spherical stereo imaging. Performance of the proposed algorithm is evaluated against ground-truth from the Middlebury stereo test bed and LIDAR scans. Results are also compared with conventional structure-from-motion algorithms. The final composite model is rendered from a wide range of viewpoints with high quality textures.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
£29.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price includes VAT (United Kingdom)

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15
Fig. 16
Fig. 17
Fig. 18
Fig. 19
Fig. 20
Fig. 21
Fig. 22

Similar content being viewed by others

Explore related subjects

Discover the latest articles, news and stories from top researchers in related subjects.

Notes

  1. City group, http://city.csail.mit.edu/

  2. Pointgrey, http://www.ptgrey.com/

  3. Spheron, http://www.spheron.com/en/spheron-cgi/products/spherocam-hdr.html

  4. Middlebury multi-view, http://vision.middlebury.edu/mview/

  5. Middlebury stereo, http://vision.middlebury.edu/stereo/

References

  • Agarwal, S., Snavely, N., Simon, I., Seitz, S., & Szeliski, R. (2009). Building rome in a day. In Proceedings of ICCV, pp. 72–79.

  • Aiger, D., Mitra, N., & Cohen-Or, D. (2008). 4-points congruent sets for robust surface registration. In Proceedings of SIGGRAPH, pp. 1–10.

  • Akbarzadeh, A., Frahm, J.-M., Mordohai, P., Clipp, B., Engels, C., Gallup, D., Merrell, P., Phelps, M., Sinha, S., Talton, B., Wang, L., Yang, Q., Stewenius, H., Yang, R., Welch, G., Towles, H., Nister, D., & Pollefeys, M. (2006). Towards urban 3d reconstruction from video. In Proceedings of 3DPVT, pp. 1–8.

  • Alvarez, L., Deriche, R., Papadopoulo, T., & Sánchez, J. (2007). Symmetrical dense optical flow estimation with oclussions detection. International Journal of Computer Vision, 75(3), 371–385.

    Article  Google Scholar 

  • Alvarez, L., Deriche, R., Sánchez, J., & Weickert, J. (2002). Dense disparity map estimation respecting image discontinuities: A pde and scale-space based approach. Journal of Visual Communication and Image Representation, 13(1), 3–21.

    Article  Google Scholar 

  • Anguelov, D., Dulong, C., Filip, D., Frueh, C., Lafon, S., Lyon, R., et al. (2010). Google street view: Capturing the world at street level. IEEE Computer, 43(6), 32–38.

    Google Scholar 

  • Asai, T., Kanbara, M., & Yokoya, N. (2005). 3d modeling of outdoor environments by integrating omnidirectional range and color images. In Proceedings of 3DIM, pp. 447–454.

  • Banno, A., & Ikeuchi, K. (2009). Disparity map refinement and 3d surface smoothing via directed anisotropic diffusion. In Proceedings of 3DIM.

  • Banno, A., & Ikeuchi, K. (2010). Omnidirectional texturing based on robust 3d registration through euclidean reconstruction from two spherical images. Computer Vision and Image Understanding, 114(4), 491–499.

    Article  Google Scholar 

  • Bay, H., Ess, A., Tuytelaars, T., & Gool, L. (2008). Surf: Speeded up robust features. Computer Vision and Image Understanding, 110, 346–359.

    Article  Google Scholar 

  • Ben-Ari, R., & Sochen, N. (2007). Variational stereo vision with sharp discontinuities and occlusion handling. In Proceedings of ICCV, pp. 1–7.

  • Benosman, R., & Devars, J. (1998). Panoramic stereovision sensor. In Proceedings of ICPR, pp. 767–769.

  • Besl, P., & McKay, N. (1992). A method for registration of 3-d shapes. IEEE Transactions on Pattern Analysis and Machine Intelligence, 14(2), 239–256.

    Article  Google Scholar 

  • Brox, T., Bruhn, A., Papenberg, N., & Weickert, J. (2004). High accuracy optical flow estimation based on a theory for warping. In Proceedings of ECCV, pp. 25–36.

  • Burt, P. J. (1981). Fast filter transforms for image processing. Computer Vision, Graphics and Image Processing, 6, 20–51.

    Google Scholar 

  • Chen, S. (1995). Quicktime vr—An image based approach to virtual environment navigation. In Proceedings of SIGGRAPH, pp. 29–38.

  • Chen, Y., & Medioni, G. (1992). Object modeling by registration of multiple range images. Image and Vision Computing, 10(3), 145–155.

    Google Scholar 

  • Cornelis, N., Leibe, B., Cornelis, K., & Gool, L. (2008). 3d urban scene modeling integrating recognition and reconstruction. International Journal of Computer Vision, 78(2), 121–141.

    Article  Google Scholar 

  • Dellaert, F., Seitz, S., Thorpe, C., & Thrun, S. (2000). Structure from motion without correspondence. In Proceedings of CVPR.

  • Desouza, G., & Kak, A. (2002). Vision for mobile robot navigation: A survey. IEEE Transactions on Pattern Analysis and Machine Intelligence, 24(2), 237–267.

    Article  Google Scholar 

  • Feldman, D., & Weinshall, D. (2005). Realtime ibr with omnidirectional crossed-slits projection. In Proceedings of ICCV, pp. 839–845.

  • Fischler, M., & Bolles, R. (1982). Random sample consensus: A paradigm for model fitting with applications to image analysis and automated cartography. Communication of the ACM, 24, 381–395.

    Article  MathSciNet  Google Scholar 

  • Fisher, R. (2007). Registration and fusion of range images. Cvonline, http://homepages.inf.ed.ac.uk/rbf/CVonline/LOCAL_COPIES/FISHER/REGIS/regis.html.

  • Frahm, J.-M., Fite-Georgel, P., Gallup, D., Johnson, T., Raguram, R., Wu, C., Jen, Y.-H., Dunn, E., Clipp, B., Lazebnik, S., & Pollefeys, M. (2010). Building rome on a cloudless day. In Proceedings of ECCV, pp. 368–381.

  • Furukawa, Y., Curless, B., Seitz, S., & Szeliski, R. (2009). Manhattan-world stereo. In Proceedings of CVPR.

  • Furukawa, Y., Curless, B., Seitz, S., & Szeliski, R. (2010). Towards internet-scale multi-view stereo. In Proceedings of CVPR.

  • Furukawa, Y., & Ponce, J. (2010). Accurate, dense, and robust multiview stereopsis. IEEE Transactions on Pattern Analysis and Machine Intelligence, 32(8), 1362–1376.

    Article  Google Scholar 

  • Gargallo, P., & Sturm, P. (1988). Bayesian 3d modeling from images using multiple depth maps. In Proceedings of CVPR, pp. 885–891.

  • Geman, S., & McClure, D. (1985). Bayesian image analysis: An application to single photon emission tomography. In Proceedings of Statistical Computation Section, pp. 12–18.

  • Goesele, M., Snavely, N., Curless, B., Hoppe, H., & Seitz, S.M. (2007). Multi-view stereo for community photo collections. In Proceedings of ICCV, pp. 368–381.

  • Granger, S., Pennec, X., & Roche, X. (2001). Rigid point-surface registration using oriented points and an em variant of icp for computer guided oral implantology. In Proceedings of MICCAI, pp. 752–761.

  • Haala, N., & Kada, M. (2005). Panoramic scenes for texture mapping of 3d city models. In Proceedings of PanoPhot.

  • Hilton, A. (2005). Scene modelling from sparse 3d data. Image and Vision Computing, 23(10), 900–920.

    Article  Google Scholar 

  • Hilton, A., Stoddart, A., Illingworth, J., & Windeatt, T. (1998). Implicit surface based geometric fusion. Computer Vision and Image Understanding, 69(3), 273–291.

    Article  Google Scholar 

  • Hirschmüller, H., & Scharstein, D. (2008). Evaluation of stereo matching costs on images with radiometric differences. IEEE Transactions on Pattern Analysis and Machine Intelligence, 31(9), 1582–1599.

    Google Scholar 

  • Ince, S., & Konrad, J. (2008). Occlusion-aware optical flow estimation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 17, 1443–1451.

    MathSciNet  Google Scholar 

  • Johnson, C. (1988). Numerical solution of partial differential equations by the finite element method. Cambridge: Cambridge University Press.

    Google Scholar 

  • Kang, S., & Szeliski, R. (1997). 3-d scene data recovery using omnidirectional multibaseline stereo. International Journal of Computer Vision, 25(2), 167–183.

    Article  Google Scholar 

  • Kazhdan, M., Bolitho, M., & Hoppe, H. (2006). Poisson surface reconstruction. In Proceedings of SGP, pp. 61–70.

  • Kim, H., & Hilton, A. (2009). 3d environment modelling using spherical stereo imaging. In Proceedings of 3DIM.

  • Kim, H., & Hilton, A. (2010). 3d modelling of static environments using multiple spherical stereo. In Proceedings of RMLE workshop in ECCV.

  • Kim, H., & Sohn, K. (2003a). Hierarchical depth estimation for image synthesis in mixed reality. In Proceedings of SPIE Electronic Imaging, pp. 544–553.

  • Kim, H., Sohn, K. (2003b). Hierarchical disparity estimation with energy-based regularization. In Proceedings of ICIP, pp. 373–376.

  • Klaus, A., Sormann, M., & Karner, K. (2006). Segment-based stereo matching using belief propagation and a self-adapting dissimilarity measure. In Proceedings of ICPR.

  • Kolmogorov, V., & Zabih, R. (2001). Computing visual correspondence with occlusions using graph cuts. In Proceedings of ICCV.

  • Lemmens, M. (2007). Airborne lidar sensor. GIM International, 21(2), 13–17.

    Google Scholar 

  • Lhuillier, M. (2008). Automatic scene structure and camera motion using a catadioptric system. Computer Vision and Image Understanding, 109(2), 186–203.

    Article  Google Scholar 

  • Li, S. (2006). Real-time spherical stereo. In Proceedings of ICPR, pp. 1046–1049.

  • Mathias, M., Martinovic, A., Weissenberg, J., & Gool, L. J. V. (2011). Procedural 3d building reconstruction using shape grammars and detectors. In Proceedings of 3DIMPVT, pp. 304–311.

  • Merrell, P., Akbarzadeh, A., Wang, L., Mordohai, P., Frahm, J.-M., Yang, R., et al. (2007). Real-time visibility-based fusion of depth maps. In Proceedings of ICCV.

  • Micusik, B., & Kosecka, J. (2009). Piecewise planar city 3d modeling from street view panoramic sequences. In Proceedings of CVPR, pp. 2906–2912.

  • Micusik, B., Martinec, D., & Pajdla, T. (2004). 3d metric reconstruction from uncalibrated omnidirectional images. In Proceedings of ACCV.

  • Min, D., & Sohn, K. (2008). Cost aggregation and occlusion handling with wls in stereo matching. IEEE Transactions on Image Processing, 17(8), 1431–1442.

    Article  MathSciNet  Google Scholar 

  • Nagel, H., & Enkelmann, W. (1986). An investigation of smoothness constraints for the estimation of displacements vector fields from image sequences. IEEE Transactions on Pattern Analysis and Machine Intelligence, 8, 565–593.

    Article  Google Scholar 

  • Nayar, S.K., & Karmarkar, A. (2000). 360 x 360 mosaics. In Proceedings of CVPR, pp. 2388–2388.

  • Pollefeys, M., Koch, R., Vergauwen, M., & Gool, L. (2000). Automated reconstruction of 3d scenes from sequences of images. ISPRS Journal of Photogrammetry and Remote Sensing, 55(4), 251–267.

    Article  Google Scholar 

  • Pollefeys, M., Nistér, D., Frahm, J., Akbarzadeh, A., Mordohai, P., Clipp, B., et al. (2008). Detailed real-time urban 3d reconstruction from video. International Journal of Computer Vision, 78(2), 143–167.

    Google Scholar 

  • Rusinkiewicz, S., & Levoy, M. (2001). Efficient variants of the icp algorithm. In Proceedings of 3DIM, pp. 145–152.

  • Salman, N., & Yvinec, M. (2009). Surface reconstruction from multi-view stereo. In Proceedings of ACCV.

  • Scharstein, D., & Szeliski, R. (2002). A taxonomy and evaluation of dense two-frame stereo correspondence algorithms. International Journal of Computer Vision, 47(1), 7–42.

    Article  MATH  Google Scholar 

  • Simon, L., Teboul, O., Koutsourakis, P., & Paragios, N. (2011). Random exploration of the procedural space for single-view 3d modeling of buildings. International Journal of Computer Vision, 93(2), 253–271.

    Google Scholar 

  • Sizintsev, M. (2008). Hierarchical stereo with thin structures and transparency. In Proceedings of CRV, pp. 97–104.

  • Slesareva, N., Bruhn, A., & Weickert, J. (2005). Optic flow goes stereo: A variational method for estimating discontinuity- preserving dense disparity maps. In Proceedings of DAGM, pp. 33–40.

  • Snavely, N., Seitz, S., & Szeliski, R. (2006). Photo tourism: Exploring photo collections in 3d. In Proceedings of ACM SIGGRAPH, pp. 835–846.

  • Snavely, N., Seitz, S., & Szeliski, R. (2008). Modeling the world from internet photo collections. International Journal of Computer Vision, 80(2), 189–210.

    Article  Google Scholar 

  • Soucy, M., & Laurendeau, D. (1995). A general surface approach to the integration of a set of range views. IEEE Transactions on Pattern Analysis and Machine Intelligence, 17(4), 344–358.

    Google Scholar 

  • Strecha, C., Fransens, R., & Gool, L. J. V. (2004). Wide-baseline stereo from multiple views: A probabilistic account. In Proceedings of CVPR, pp. 552–559.

  • Strecha, C., Hansen, W., Gool, L., Fua, P., & Thoennessen, U. (2008). On benchmarking camera calibration and multi-view stereo for high resolution imagery. In Proceedings of CVPR, pp. 1–8.

  • Sun, D., Roth, S., Lewis, J., & Black, M. (2008). Learning optical flow. In Proceedings of ECCV, pp. 83–97.

  • Sun, J., Zheng, N., & Shum, H. (2003). Stereo matching using belief propagation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 25(7), 787–800.

    Article  Google Scholar 

  • Szeliski, R., & Scharstein, D. (2004). Sampling the disparity space image. IEEE Transactions on Pattern Analysis and Machine Intelligence, 26(3), 419–425.

    Article  Google Scholar 

  • Teller, S., Antone, M., Bodnar, Z., Bosse, M., Coorg, S., Jethwa, M., et al. (2003). Calibrated, registered images of an extended urban area. International Journal of Computer Vision, 53(1), 93–107.

    Article  Google Scholar 

  • Tighe, J., Feldman, J., & Lazebnik, S. (2010). SuperParsing: Scalable Nonparametric Image Parsing with Superpixels. In Proceedings of ECCV.

  • Turk, G., & Levoy, M. (1994). Zippered polygon meshes from range images. In Proceedings of SIGGRAPH, pp. 311–318.

  • Vergauwen, M., & Gool, L. (2006). Web-based 3d reconstruction service. Machine Vision Applications, 17, 411–426.

    Article  Google Scholar 

  • Vu, H., Keriven, R., Labatut, P., & Pons, J. (2009). Towards high-resolution large-scale multi-view stereo. In Proceedings of CVPR, pp. 1430–1437.

  • Weickert, J. (1997). A review of nonlinear diffusion filtering. Lecture Notes in Computer Science, 1252, 3–28.

    Google Scholar 

  • Williams, J., & Bennamoun, M. (2001). Simultaneous registration of multiple corresponding point sets. Computer Vision and Image Understanding, 81(1), 117–142.

    Article  MATH  Google Scholar 

  • Yang, Q., Wang, L., Yang, R., Stewénius, H., & Nistér, D. (2008). Stereo matching with color-weighted correlation, hierarchical belief propagation and occlusion handling. IEEE Transactions on Pattern Analysis and Machine Intelligence, 31(3), 492–504.

    Article  Google Scholar 

  • Yuille, A., & Poggio, T. (1984). A generalized ordering constraint for stereo correspondence. MIT A.I. Memo 777.

  • Zimmer, H., Bruhn, A., Valgaerts, L., Breuß, M., Weickert, J., Rosenhahn, B., & Seidel, H. (2008). Pde-based anisotropic disparity-driven stereo vision. In Proceedings of VMV, pp. 263–272.

  • Zomet, A., Feldman, D., Peleg, S., & Weinshall, D. (2003). Mosaicing new views: The crossed-slits projection. IEEE Transactions on Pattern Analysis and Machine Intelligence, 25(6), 741–754.

    Article  Google Scholar 

Download references

Acknowledgments

This research was executed with the financial support of the EU FP7 project i3DPost, UK TSB project SyMMM and EU ICT FP7 project IMPART.

Author information

Authors and Affiliations

  1. Centre for Vision, Speech and Signal Processing, University of Surrey, Guildford, Surrey, GU2 7XH, UK

    Hansung Kim & Adrian Hilton

Authors
  1. Hansung Kim
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Adrian Hilton
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Hansung Kim.

Electronic supplementary material

Below is the link to the electronic supplementary material.

Supplementary material 1 (mov 28970 KB)

Rights and permissions

Reprints and permissions

About this article

Cite this article

Kim, H., Hilton, A. 3D Scene Reconstruction from Multiple Spherical Stereo Pairs. Int J Comput Vis 104, 94–116 (2013). https://doi.org/10.1007/s11263-013-0616-1

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11263-013-0616-1

Keywords

Search

Navigation