[go: up one dir, main page]
More Web Proxy on the site http://driver.im/
Skip to main content
Springer Nature Link
Log in

A Kendall Shape Space Approach to 3D Shape Estimation from 2D Landmarks

  • Conference paper
  • First Online:
Computer Vision – ECCV 2022 (ECCV 2022)

Part of the book series: Lecture Notes in Computer Science ((LNCS,volume 13662))

Included in the following conference series:

Abstract

3D shapes provide substantially more information than 2D images. However, the acquisition of 3D shapes is sometimes very difficult or even impossible in comparison with acquiring 2D images, making it necessary to derive the 3D shape from 2D images. Although this is, in general, a mathematically ill-posed problem, it might be solved by constraining the problem formulation using prior information. Here, we present a new approach based on Kendall’s shape space to reconstruct 3D shapes from single monocular 2D images. The work is motivated by an application to study the feeding behavior of the basking shark, an endangered species whose massive size and mobility render 3D shape data nearly impossible to obtain, hampering understanding of their feeding behaviors and ecology. 2D images of these animals in feeding position, however, are readily available. We compare our approach with state-of-the-art shape-based approaches, both on human stick models and on shark head skeletons. Using a small set of training shapes, we show that the Kendall shape space approach is substantially more robust than previous methods and results in plausible shapes. This is essential for the motivating application in which specimens are rare and therefore only few training shapes are available.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution

Subscribe and save

Springer+ Basic
£29.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Chapter
GBP 19.95
Price includes VAT (United Kingdom)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
GBP 79.50
Price includes VAT (United Kingdom)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
GBP 99.99
Price includes VAT (United Kingdom)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

Notes

  1. 1.

    Carnegie Mellon University - Graphics Lab - motion capture library http://mocap.cs.cmu.edu/.

  2. 2.

    Source: https://www.amustard.com/library/albums/1/AMU_12_300611_0014.jpg.

  3. 3.

    Source: https://www.manxbaskingsharkwatch.org/shark-files/biology-behaviour/feeding.

References

  1. Agudo, A., Agapito, L., Calvo, B., Montiel, J.M.: Good vibrations: a modal analysis approach for sequential non-rigid structure from motion. In: Proceedings of IEEE Computer Vision and Pattern Recognition, pp. 1558–1565 (2014)

    Google Scholar 

  2. Akhter, I., Black, M.J.: Pose-conditioned joint angle limits for 3D human pose reconstruction. In: Proceedings of IEEE Computer Vision and Pattern Recognition, pp. 1446–1455 (2015)

    Google Scholar 

  3. Ambellan, F., Hanik, M., von Tycowicz, C.: Morphomatics: geometric morphometrics in non-Euclidean shape spaces (2021). https://doi.org/10.12752/8544

  4. Ambellan, F., Lamecker, H., von Tycowicz, C., Zachow, S.: Statistical shape models: understanding and mastering variation in anatomy. In: Rea, P.M. (ed.) Biomedical Visualisation. AEMB, vol. 1156, pp. 67–84. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-19385-0_5

    Chapter  Google Scholar 

  5. Boumal, N., Mishra, B., Absil, P.A., Sepulchre, R.: ManOPT, a MATLAB toolbox for optimization on manifolds. J. Mach. Learn. Res. 15(1), 1455–1459 (2014)

    MATH  Google Scholar 

  6. Bregler, C., Hertzmann, A., Biermann, H.: Recovering non-rigid 3D shape from image streams. In: Proceedings of IEEE Computer Vision and Pattern Recognition, vol. 2, pp. 690–696. IEEE (2000)

    Google Scholar 

  7. Cao, C., Weng, Y., Lin, S., Zhou, K.: 3D shape regression for real-time facial animation. ACM Trans. Graph. (TOG) 32(4), 1–10 (2013)

    Article  MATH  Google Scholar 

  8. Chakraborty, R., Bouza, J., Manton, J., Vemuri, B.C.: Manifoldnet: a deep neural network for manifold-valued data with applications. IEEE Trans. Pattern Anal. Mach. Intell. (2020)

    Google Scholar 

  9. Cootes, T.F., Taylor, C.J., Cooper, D.H., Graham, J.: Active shape models-their training and application. Comput. Vis. Image Underst. 61(1), 38–59 (1995)

    Article  Google Scholar 

  10. Fletcher, P.T., Lu, C., Pizer, S.M., Joshi, S.: Principal geodesic analysis for the study of nonlinear statistics of shape. IEEE Trans. Med. Imaging 23(8), 995–1005 (2004)

    Article  Google Scholar 

  11. Friji, R., Drira, H., Chaieb, F., Kurtek, S., Kchok, H.: KshapeNet: Riemannian network on Kendall shape space for skeleton based action recognition. arXiv preprint arXiv:2011.12004 (2020)

  12. Frostig, R., Johnson, M.J., Leary, C.: Compiling machine learning programs via high-level tracing. Syst. Mach. Learn. 4(9) (2018)

    Google Scholar 

  13. Heimann, T., Meinzer, H.P.: Statistical shape models for 3D medical image segmentation: a review. Med. Image Anal. 13(4), 543–563 (2009)

    Article  Google Scholar 

  14. Hejrati, M., Ramanan, D.: Analyzing 3D objects in cluttered images. In: Advances in Neural Information Processing Systems, vol. 25 (2012)

    Google Scholar 

  15. Howe, N.R.: Silhouette lookup for automatic pose tracking. In: 2004 Conference on Computer Vision and Pattern Recognition Workshop, pp. 15–22. IEEE (2004)

    Google Scholar 

  16. Jiang, H., Liu, H., Tan, P., Zhang, G., Bao, H.: 3D reconstruction of dynamic scenes with multiple handheld cameras. In: Fitzgibbon, A., Lazebnik, S., Perona, P., Sato, Y., Schmid, C. (eds.) ECCV 2012. LNCS, vol. 7573, pp. 601–615. Springer, Heidelberg (2012). https://doi.org/10.1007/978-3-642-33709-3_43

    Chapter  Google Scholar 

  17. Kendall, D.G.: Shape manifolds, procrustean metrics, and complex projective spaces. Bull. Lond. Math. Soc. 16(2), 81–121 (1984)

    Article  MathSciNet  MATH  Google Scholar 

  18. Kendall, D.G., Barden, D., Carne, T.K., Le, H.: Shape and Shape Theory. Wiley, Hoboken (2009)

    Google Scholar 

  19. Larsen, E.S., Mordohai, P., Pollefeys, M., Fuchs, H.: Temporally consistent reconstruction from multiple video streams using enhanced belief propagation. In: 2007 IEEE 11th International Conference on Computer Vision, pp. 1–8. IEEE (2007)

    Google Scholar 

  20. Lloyd, S.: Least squares quantization in PCM. IEEE Trans. Inf. Theory 28(2), 129–137 (1982)

    Article  MathSciNet  MATH  Google Scholar 

  21. Mehta, D., et al.: VNect: real-time 3D human pose estimation with a single RGB camera. ACM Trans. Graph. (TOG) 36(4), 1–14 (2017)

    Article  Google Scholar 

  22. Michalkiewicz, M., Parisot, S., Tsogkas, S., Baktashmotlagh, M., Eriksson, A., Belilovsky, E.: Few-shot single-view 3-D object reconstruction with compositional priors. In: Vedaldi, A., Bischof, H., Brox, T., Frahm, J.-M. (eds.) ECCV 2020. LNCS, vol. 12370, pp. 614–630. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-58595-2_37

    Chapter  Google Scholar 

  23. Mori, G., Malik, J.: Recovering 3D human body configurations using shape contexts. IEEE Trans. Pattern Anal. Mach. Intell. 28(7), 1052–1062 (2006)

    Article  Google Scholar 

  24. Mustafa, A., Kim, H., Guillemaut, J.Y., Hilton, A.: General dynamic scene reconstruction from multiple view video. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 900–908 (2015)

    Google Scholar 

  25. Nava-Yazdani, E., Hege, H.C., Sullivan, T.J., von Tycowicz, C.: Geodesic analysis in Kendall’s shape space with epidemiological applications. J. Math. Imaging Vision 62(4), 549–559 (2020)

    Article  MathSciNet  MATH  Google Scholar 

  26. Park, S., Lee, M., Kwak, N.: Procrustean regression networks: learning 3D structure of non-rigid objects from 2D annotations. In: Vedaldi, A., Bischof, H., Brox, T., Frahm, J.-M. (eds.) ECCV 2020. LNCS, vol. 12374, pp. 1–18. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-58526-6_1

    Chapter  Google Scholar 

  27. Paskin, M., Dean, M., Baum, D., von Tycowicz, C.: A Kendall shape space approach to 3D shape estimation from 2D landmarks - source code and data (2022). https://doi.org/10.12752/8730

  28. Pennec, X.: Barycentric subspace analysis on manifolds. Ann. Stat. 46(6A), 2711–2746 (2018)

    Article  MathSciNet  MATH  Google Scholar 

  29. Plänkers, R., Fua, P.: Tracking and modeling people in video sequences. Comput. Vis. Image Underst. 81(3), 285–302 (2001)

    Article  MATH  Google Scholar 

  30. Ramakrishna, V., Kanade, T., Sheikh, Y.: Reconstructing 3D human pose from 2D image landmarks. In: Fitzgibbon, A., Lazebnik, S., Perona, P., Sato, Y., Schmid, C. (eds.) ECCV 2012. LNCS, vol. 7575, pp. 573–586. Springer, Heidelberg (2012). https://doi.org/10.1007/978-3-642-33765-9_41

    Chapter  Google Scholar 

  31. Sanderson, S.L., Roberts, E., Lineburg, J., Brooks, H.: Fish mouths as engineering structures for vortical cross-step filtration. Nat. Commun. 7(1), 1–9 (2016)

    Article  Google Scholar 

  32. Sanzari, M., Ntouskos, V., Pirri, F.: Bayesian image based 3D pose estimation. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) ECCV 2016. LNCS, vol. 9912, pp. 566–582. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-46484-8_34

    Chapter  Google Scholar 

  33. Sims, D.W.: Sieving a living: a review of the biology, ecology and conservation status of the plankton-feeding basking shark Cetorhinus maximus. Adv. Mar. Biol. 54, 171–220 (2008)

    Article  Google Scholar 

  34. Tomasi, C., Kanade, T.: Shape and motion from image streams under orthography: a factorization method. Int. J. Comput. Vis. 9(2), 137–154 (1992)

    Article  Google Scholar 

  35. Wang, C., Wang, Y., Lin, Z., Yuille, A.L., Gao, W.: Robust estimation of 3d human poses from a single image. In: Proceedings of IEEE Computer Vision and Pattern Recognition, pp. 2361–2368 (2014)

    Google Scholar 

  36. Wegner, N.C.: Elasmobranch gill structure. In: Fish Physiology, vol. 34, pp. 101–151. Elsevier, Amsterdam (2015)

    Google Scholar 

  37. Zhou, X., Leonardos, S., Hu, X., Daniilidis, K.: 3d shape estimation from 2d landmarks: A convex relaxation approach. In: Proc. IEEE Comput. Vis. Pattern Recognit. pp. 4447–4455 (2015)

    Google Scholar 

  38. Zia, M.Z., Stark, M., Schiele, B., Schindler, K.: Detailed 3D representations for object recognition and modeling. IEEE Trans. Pattern Anal. Mach. Intell. 35(11), 2608–2623 (2013)

    Article  Google Scholar 

Download references

Acknowledgments

We are grateful for the funding by Deutsche Forschungsgemeinschaft (DFG) through Germany’s Excellence Strategy – The Berlin Mathematics Research Center MATH+ (EXC-2046/1, project ID: 390685689) and by Bundesministerium für Bildung und Forschung (BMBF) through BIFOLD – The Berlin Institute for the Foundations of Learning and Data (ref. 01IS18025A and ref. 01IS18037A). The basking shark work was funded in part by an HFSP Program Grant (RGP0010-2020 to M.N.D.). We also want to thank the British Museum of Natural History and Zoological Museum, University of Copenhagen for providing basking shark specimens, Pepijn Kamminga, James Maclaine, Allison Luger and Henrik Lauridsen for help acquiring CT data, and Alex Mustard (Underwater Photography) and Maura Mitchell (Manx Basking Shark Watch) for granting us permission to use their underwater images of basking sharks.

Author information

Authors and Affiliations

  1. Zuse Institute Berlin, Berlin, Germany

    Martha Paskin, Daniel Baum & Christoph von Tycowicz

  2. City University of Hong Kong, Hong Kong, China

    Mason N. Dean

  3. Freie Universität Berlin, Berlin, Germany

    Christoph von Tycowicz

Authors
  1. Martha Paskin
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Daniel Baum
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Mason N. Dean
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Christoph von Tycowicz
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Martha Paskin .

Editor information

Editors and Affiliations

  1. Tel Aviv University, Tel Aviv, Israel

    Shai Avidan

  2. University College London, London, UK

    Gabriel Brostow

  3. Google AI, Accra, Ghana

    Moustapha Cissé

  4. University of Catania, Catania, Italy

    Giovanni Maria Farinella

  5. Facebook (United States), Menlo Park, CA, USA

    Tal Hassner

Rights and permissions

Reprints and permissions

Copyright information

© 2022 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Paskin, M., Baum, D., Dean, M.N., von Tycowicz, C. (2022). A Kendall Shape Space Approach to 3D Shape Estimation from 2D Landmarks. In: Avidan, S., Brostow, G., Cissé, M., Farinella, G.M., Hassner, T. (eds) Computer Vision – ECCV 2022. ECCV 2022. Lecture Notes in Computer Science, vol 13662. Springer, Cham. https://doi.org/10.1007/978-3-031-20086-1_21

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-20086-1_21

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-20085-4

  • Online ISBN: 978-3-031-20086-1

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation