Abstract
We introduce a new problem of retrieving 3D models that are deformable to a given query shape and present a novel deep deformation-aware embedding to solve this retrieval task. 3D model retrieval is a fundamental operation for recovering a clean and complete 3D model from a noisy and partial 3D scan. However, given a finite collection of 3D shapes, even the closest model to a query may not be satisfactory. This motivates us to apply 3D model deformation techniques to adapt the retrieved model so as to better fit the query. Yet, certain restrictions are enforced in most 3D deformation techniques to preserve important features of the original model that prevent a perfect fitting of the deformed model to the query. This gap between the deformed model and the query induces asymmetric relationships among the models, which cannot be handled by typical metric learning techniques. Thus, to retrieve the best models for fitting, we propose a novel deep embedding approach that learns the asymmetric relationships by leveraging location-dependent egocentric distance fields. We also propose two strategies for training the embedding network. We demonstrate that both of these approaches outperform other baselines in our experiments with both synthetic and real data. Our project page can be found at deformscan2cad.github.io.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
Notes
- 1.
This is not exactly the same with the property of metrics, identity of indiscernibles, meaning the two-way identity (\(e_\mathcal {D}(\mathbf {s}, \mathbf {t}) = 0 \Leftrightarrow \mathbf {s} = \mathbf {t}\)). We cannot guarantee that \(e_\mathcal {D}(\mathbf {s}, \mathbf {t}) = 0 \Rightarrow \mathbf {s} = \mathbf {t}\) from our definition of \(e_\mathcal {D}\). Nevertheless, this is not necessary in the retrieval problem.
- 2.
Due to space restrictions we present results of Image-to-CAD in our supplementary material.
References
Achlioptas, P., Diamanti, O., Mitliagkas, I., Guibas, L.J.: Learning representations and generative models for 3D point clouds. In: ICML (2018)
Ahmed, A., Shervashidze, N., Narayanamurthy, S., Josifovski, V., Smola, A.J.: Distributed large-scale natural graph factorization. In: WWW (2013)
Aldrovandi, R., Pereira, J.: An Introduction to Geometrical Physics. World Scientific (1995)
Arandjelović, R., Gronat, P., Torii, A., Pajdla, T., Sivic, J.: NetVLAD: CNN architecture for weakly supervised place recognition. In: CVPR (2016)
Avetisyan, A., Dahnert, M., Dai, A., Savva, M., Chang, A.X., Nießner, M.: Scan2CAD: learning cad model alignment in RGB-D scans. In: CVPR (2019)
Avetisyan, A., Dai, A., Nießner, M.: End-to-end cad model retrieval and 9DoF alignment in 3D scans. In: ICCV (2019)
Bellet, A., Habrard, A., Sebban, M.: A survey on metric learning for feature vectors and structured data (2013)
Buldygin, V., et al.: Metric Characterization of Random Variables and Random Processes. American Mathematical Society (2000)
Bylow, E., Sturm, J., Kerl, C., Kahl, F., Cremers, D.: Real-time camera tracking and 3D reconstruction using signed distance functions. In: RSS (2013)
Chang, A.X., et al.: Shapenet: an information-rich 3D model repository (2015)
Chechik, G., Sharma, V., Shalit, U., Bengio, S.: Large scale online learning of image similarity through ranking. J. Mach. Learn. Res. 11, 1109–1135 (2010)
Chen, L., Lian, X.: Efficient similarity search in nonmetric spaces with local constant embedding. IEEE Trans. Knowl. Data Eng. 20(3), 321–336 (2008)
Chopra, S., Hadsell, R., LeCun, Y.: Learning a similarity metric discriminatively, with application to face verification. In: CVPR (2005)
Cortes, C., Vapnik, V.: Support-vector networks. Mach. Learn. 20(3), 273–297 (1995)
Dahnert, M., Dai, A., Guibas, L., Nießner, M.: Joint embedding of 3D scan and cad objects. In: ICCV (2019)
Dai, A., Nießner, M., Zollhöfer, M., Izadi, S., Theobalt, C.: BundleFusion: real-time globally consistent 3D reconstruction using on-the-fly surface reintegration. In: ACM SIGGRAPH (2017)
Dai, A., Ruizhongtai Qi, C., Nießner, M.: Shape completion using 3D-encoder-predictor CNNs and shape synthesis. In: CVPR (2017)
Deng, H., Birdal, T., Ilic, S.: PPFNet: global context aware local features for robust 3D point matching. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 195–205 (2018)
Dong, Y., Chawla, N.V., Swami, A.: metapath2vec: scalable representation learning for heterogeneous networks. In: KDD (2017)
Fan, H., Su, H., Guibas, L.J.: A point set generation network for 3D object reconstruction from a single image. In: CVPR (2016)
G, V.K.B., Carneiro, G., Reid, I.: Learning local image descriptors with deep siamese and triplet convolutional networks by minimizing global loss functions. In: CVPR (2016)
Garcia, N., Vogiatzis, G.: Learning non-metric visual similarity for image retrieval. Image Vis. Comput. 82, 18–25 (2019)
Garland, M., Heckbert, P.S.: Simplifying surfaces with color and texture using quadric error metrics. In: Visualization (1998)
Groueix, T., Fisher, M., Kim, V.G., Russell, B.C., Aubry, M.: Deep self-supervised cycle-consistent deformation for few-shot shape segmentation. In: Eurographics Symposium on Geometry Processing (2019)
Grover, A., Leskovec, J.: node2vec: scalable feature learning for networks. In: KDD (2016)
Hadsell, R., Chopra, S., LeCun, Y.: Dimensionality reduction by learning an invariant mapping. In: CVPR (2006)
Han, X., Leung, T., Jia, Y., Sukthankar, R., Berg, A.C.: MatchNet: unifying feature and metric learning for patch-based matching. In: CVPR (2015)
Hanocka, R., Fish, N., Wang, Z., Giryes, R., Fleishman, S., Cohen-Or, D.: ALIGNet: partial-shape agnostic alignment via unsupervised learning. ACM Trans. Graph. 38(1), 1–14 (2018)
Hinton, G.E., Roweis, S.T.: Stochastic neighbor embedding. In: NIPS (2003)
Huang, J., Dai, A., Guibas, L.J., Nießner, M.: 3Dlite: towards commodity 3D scanning for content creation. In: ACM SIGGRAPH Asia (2017)
Huang, J., Su, H., Guibas, L.: Robust watertight manifold surface generation method for shapenet models (2018)
Igarashi, T., Moscovich, T., Hughes, J.F.: As-rigid-as-possible shape manipulation. In: ACM SIGGRAPH (2005)
Jack, D., et al.: Learning free-Form deformations for 3D object reconstruction. In: ICCV (2018)
Joshi, P., Meyer, M., DeRose, T., Green, B., Sanocki, T.: Harmonic coordinates for character articulation. In: ACM SIGGRAPH (2007)
Ju, T., Schaefer, S., Warren, J.: Mean value coordinates for closed triangular meshes. In: ACM SIGGRAPH (2005)
Kraevoy, V., Sheffer, A., Shamir, A., Cohen-Or, D.: Non-homogeneous resizing of complex models. In: ACM SIGGRAPH Asia (2006)
Kulis, B., et al.: Metric learning: a survey. Found. Trends® Mach. Learn. 5(4), 287–364 (2013)
Kurenkov, A., et al.: DeformNet: free-form deformation network for 3D shape reconstruction from a single image. In: WACV (2018)
Li, Y., Dai, A., Guibas, L., Nießner, M.: Database-assisted object retrieval for real-time 3D reconstruction. In: Eurographics (2015)
Lipman, Y., Sorkine, O., Cohen-Or, D., Levin, D., Rossi, C., Seidel, H.P.: Differential coordinates for interactive mesh editing. In: Shape Modeling Applications (2004)
Lipman, Y., Levin, D., Cohen-Or, D.: Green coordinates. In: ACM SIGGRAPH (2008)
Lipman, Y., Sorkine, O., Levin, D., Cohen-Or, D.: Linear rotation-invariant coordinates for meshes. In: ACM SIGGRAPH (2005)
Liu, E.Y., Guo, Z., Zhang, X., Jojic, V., Wang, W.: Metric learning from relative comparisons by minimizing squared residual. In: ICDM (2012)
van der Maaten, L., Hinton, G.: Visualizing data using t-SNE. J. Mach. Learn. Res. 9, 2579–2605 (2008)
Mahalanobis, P.C.: On the generalized distance in statistics. In: Proceedings of the National Institute of Science. National Institute of Science of India (1936)
Morozov, S., Babenko, A.: Non-metric similarity graphs for maximum inner product search. In: Advances in Neural Information Processing Systems (2018)
Newcombe, R.A., et al.: KinectFusion: real-time dense surface mapping and tracking. In: ISMAR (2011)
Perozzi, B., Al-Rfou, R., Skiena, S.: Deepwalk: online learning of social representations. In: KDD (2013)
Qi, C.R., Su, H., Mo, K., Guibas, L.J.: PointNet: deep learning on point sets for 3D classification and segmentation. In: CVPR (2017)
Schroff, F., Kalenichenko, D., Philbin, J.: FaceNet: a unified embedding for face recognition and clustering. In: CVPR (2015)
Sederberg, T.W., Parry, S.R.: Free-form deformation of solid geometric models. In: ACM SIGGRAPH (1986)
Simo-Serra, E., Trulls, E., Ferraz, L., Kokkinos, I., Fua, P., Moreno-Noguer, F.: Discriminative learning of deep convolutional feature point descriptors. In: ICCV (2015)
Skopal, T.: On fast non-metric similarity search by metric access methods. In: Ioannidis, Y., et al. (eds.) EDBT 2006. LNCS, vol. 3896, pp. 718–736. Springer, Heidelberg (2006). https://doi.org/10.1007/11687238_43
Skopal, T., Bustos, B.: On nonmetric similarity search problems in complex domains. ACM Comput. Surv. (CSUR) 43(4), 1–50 (2011)
Skopal, T., Lokoč, J.: NM-tree: flexible approximate similarity search in metric and non-metric spaces. In: Bhowmick, S.S., Küng, J., Wagner, R. (eds.) DEXA 2008. LNCS, vol. 5181, pp. 312–325. Springer, Heidelberg (2008). https://doi.org/10.1007/978-3-540-85654-2_30
Sorkine, O., Alexa, M.: As-rigid-as-possible surface modeling. In: Eurographics Symposium on Geometry Processing (2007)
Sorkine, O., Cohen-Or, D., Lipman, Y., Alexa, M., Rössl, C., Seidel, H.P.: Laplacian surface editing. In: Eurographics Symposium on Geometry Processing (2004)
Stratasys: GrabCAD community. https://grabcad.com/library
Tan, X., Chen, S., Li, J., Zhou, Z.H.: Learning non-metric partial similarity based on maximal margin criterion. In: 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR 2006), vol. 1. IEEE (2006)
Tian, Y., Fan, B., Wu, F.: L2-Net: deep learning of discriminative patch descriptor in euclidean space. In: CVPR (2017)
Trimble: 3D warehouse. https://3dwarehouse.sketchup.com/
TurboSquid: TurboSquid. https://www.turbosquid.com/
Wang, W., Ceylan, D., Mech, R., Neumann, U.: 3DN: 3D deformation network. In: CVPR (2019)
Weber, O., Ben-Chen, M., Gotsman, C.: Complex barycentric coordinates with applications to planar shape deformation. In: Eurographics (2009)
Whelan, T., Leutenegger, S., Salas-Moreno, R.F., Glocker, B., Davison, A.J.: ElasticFusion: dense slam without a pose graph. Robot.: Sci. Syst. (2011)
Hamilton, W.L., Ying, R., Leskovec, J.: Representation learning on graphs: methods and applications. IEEE Data Eng. Bull. (2017)
Yang, G., Huang, X., Hao, Z., Liu, M.Y., Belongie, S., Hariharan, B.: Pointflow: 3D point cloud generation with continuous normalizing flows. In: ICCV (2019)
Yifan, W., Aigerman, N., Kim, V., Chaudhuri, S., Sorkine-Hornung, O.: Neural cages for detail-preserving 3D deformations (2019)
Yumer, E., Mitra, N.J.: Learning semantic deformation flows with 3D convolutional networks. In: ECCV (2016)
Acknowledgements
This work is supported by a Google AR/VR University Research Award, a Vannevar Bush Faculty Fellowship, a grant from the Stanford SAIL Toyota Research Center, and gifts from the Adobe Corporation and the Dassault Foundation.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
1 Electronic supplementary material
Below is the link to the electronic supplementary material.
Rights and permissions
Copyright information
© 2020 Springer Nature Switzerland AG
About this paper
Cite this paper
Uy, M.A., Huang, J., Sung, M., Birdal, T., Guibas, L. (2020). Deformation-Aware 3D Model Embedding and Retrieval. In: Vedaldi, A., Bischof, H., Brox, T., Frahm, JM. (eds) Computer Vision – ECCV 2020. ECCV 2020. Lecture Notes in Computer Science(), vol 12352. Springer, Cham. https://doi.org/10.1007/978-3-030-58571-6_24
Download citation
DOI: https://doi.org/10.1007/978-3-030-58571-6_24
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-58570-9
Online ISBN: 978-3-030-58571-6
eBook Packages: Computer ScienceComputer Science (R0)