[go: up one dir, main page]
More Web Proxy on the site http://driver.im/ skip to main content
research-article

A similarity measure for material appearance

Published: 12 July 2019 Publication History

Abstract

We present a model to measure the similarity in appearance between different materials, which correlates with human similarity judgments. We first create a database of 9,000 rendered images depicting objects with varying materials, shape and illumination. We then gather data on perceived similarity from crowdsourced experiments; our analysis of over 114,840 answers suggests that indeed a shared perception of appearance similarity exists. We feed this data to a deep learning architecture with a novel loss function, which learns a feature space for materials that correlates with such perceived appearance similarity. Our evaluation shows that our model outperforms existing metrics. Last, we demonstrate several applications enabled by our metric, including appearance-based search for material suggestions, database visualization, clustering and summarization, and gamut mapping.

Supplementary Material

ZIP File (repository.zip)
Code for the paper "A Similarity Measure for Material Appearance" presented in SIGGRAPH 2019 and published in ACM Transactions on Graphics (TOG).
The code is also available via GitHub: https://github.com/mlagunas/material-appearance-similarity

References

[1]
Edward H Adelson. 2001. On seeing stuff: the perception of materials by humans and machines. In Human vision and electronic imaging VI, Vol. 4299. International Society for Optics and Photonics, 1--13.
[2]
Sameer Agarwal, Josh Wills, Lawrence Cayton, Gert Lanckriet, David Kriegman, and Serge Belongie. 2007. Generalized non-metric multidimensional scaling. In Conference on artifical intelligence and statistics. 11--18.
[3]
Barton L Anderson. 2011. Visual perception of materials and surfaces. Current biology 21, 24 (2011), R978--R983.
[4]
Amit Banerjee and Rajesh N Dave. 2004. Validating clusters using the Hopkins statistic. In 2004 IEEE International conference on fuzzy systems (IEEE Cat. No. 04CH37542), Vol. 1. IEEE, 149--153.
[5]
Sean Bell, Paul Upchurch, Noah Snavely, and Kavita Bala. 2013. Opensurfaces: A richly annotated catalog of surface appearance. ACM Transactions on graphics (TOG) 32, 4 (2013), 111.
[6]
Sean Bell, Paul Upchurch, Noah Snavely, and Kavita Bala. 2015. Material recognition in the wild with the materials in context database. In Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR). 3479--3487.
[7]
A. C. Chadwick and R. W. Kentridge. 2015. The perception of gloss: A review. Vision research 109 (2015), 221 -- 235.
[8]
Kang Chen, Kun Xu, Yizhou Yu, Tian-Yi Wang, and Shi-Min Hu. 2015. Magic decorator: automatic material suggestion for indoor digital scenes. ACM Transactions on graphics (TOG) 34, 6 (2015), 232.
[9]
De Cheng, Yihong Gong, Sanping Zhou, Jinjun Wang, and Nanning Zheng. 2016. Person re-identification by multi-channel parts-based cnn with improved triplet loss function. In Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR). 1335--1344.
[10]
D. W. Cunningham, C. Wallraven, R. W. Fleming, and W. Strasser. 2007. Perceptual Reparameterization of Material Properties. In Proceedings of the third Eurographics conference on computational aesthetics in graphics, visualization and imaging (Computational Aesthetics'07). 89--96.
[11]
Kristin J. Dana, Bram van Ginneken, Shree K. Nayar, and Jan J. Koenderink. 1999. Reflectance and Texture of Real-world Surfaces. ACM Transactions on graphics (TOG) 18, 1 (Jan. 1999), 1--34.
[12]
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. 2009. Imagenet: A large-scale hierarchical image database. In Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR). Ieee, 248--255.
[13]
Katja Doerschner, Roland W. Fleming, Ozgur Yilmaz, Paul R. Schrater, Bruce Hartung, and Daniel Kersten. 2011. Visual Motion and the Perception of Surface Material. Current biology 21, 23 (2011), 2010 -- 2016.
[14]
Julie Dorsey, Holly Rushmeier, and François Sillion. 2010. Digital modeling of material appearance. Elsevier.
[15]
Jonathan Dupuy and Wenzel Jakob. 2018. An Adaptive Parameterization for Efficient Material Acquisition and Rendering. ACM Transactions on graphics (TOG).
[16]
Jirí Filip and Radomír Vávra. 2014. Template-based sampling of anisotropic BRDFs. In Computer graphics forum, Vol. 33. Wiley Online Library, 91--99.
[17]
Roland W. Fleming. 2014. Visual perception of materials and their properties. Vision research 94 (2014), 62 -- 75.
[18]
Roland W Fleming. 2017. Material perception. Annual review of vision science 3 (2017), 365--388.
[19]
Roland W Fleming, Ron O Dror, and Edward H Adelson. 2003. Real-world illumination and the perception of surface reflectance properties. Journal of vision 3, 5 (2003), 3--3.
[20]
Roland W. Fleming, Shin'ya Nishida, and Karl R. Gegenfurtner. 2015. Perception of material properties. Vision research 115 (2015), 157 -- 162.
[21]
Adria Fores, James Ferwerda, and Jinwei Gu. 2012. Toward a perceptually based metric for BRDF modeling. In Color and imaging conference, Vol. 2012. Society for Imaging Science and Technology, 142--148.
[22]
Jiyang Gao and Ram Nevatia. 2018. Revisiting Temporal Modeling for Video-based Person ReID. arXiv preprint arXiv:1805.02104 (2018).
[23]
Elena Garces, Aseem Agarwala, Diego Gutierrez, and Aaron Hertzmann. 2014. A Similarity Measure for Illustration Style. ACM transactions on graphics (TOG, proc. SIGGRAPH) 33, 4 (2014).
[24]
Stamatios Georgoulis, Vincent Vanweddingen, Marc Proesmans, and Luc Van Gool. 2017. Material Classification under Natural Illumination Using Reflectance Maps. In IEEE Winter conference on applications of computer vision (WACV). IEEE, 244--253.
[25]
Ioannis Gkioulekas, Bruce Walter, Edward H Adelson, Kavita Bala, and Todd Zickler. 2015. On the appearance of translucent edges. In Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR). 5528--5536.
[26]
Ioannis Gkioulekas, Bei Xiao, Shuang Zhao, Edward H Adelson, Todd Zickler, and Kavita Bala. 2013. Understanding the role of phase function in translucent appearance. ACM Transactions on graphics (TOG) 32, 5 (2013), 147.
[27]
Vlastimil Havran, Jiri Filip, and Karol Myszkowski. 2016. Perceptually Motivated BRDF Comparison using Single Image. Computer graphics forum (2016).
[28]
Eric Hayman, Barbara Caputo, Mario Fritz, and Jan-Olof Eklundh. 2004. On the significance of real-world conditions for material classification. In European conference on computer vision (ECCV). Springer, 253--266.
[29]
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR). 770--778.
[30]
Yun-Xian Ho, Michael S Landy, and Laurence T Maloney. 2006. How direction of illumination affects visually perceived surface roughness. Journal of vision 6, 5 (2006), 8--8.
[31]
Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger. 2017. Densely Connected Convolutional Networks. In Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR), Vol. 1. 3.
[32]
Wenzel Jakob. 2010. Mitsuba renderer. http://www.mitsuba-renderer.org.
[33]
Adrian Jarabo, Hongzhi Wu, Julie Dorsey, Holly Rushmeier, and Diego Gutierrez. 2014. Effects of Approximate Filtering on the Appearance of Bidirectional Texture Functions. IEEE Transactions on visualization and computer graphics 20, 6 (2014).
[34]
Maurice Kendall and Jean D. Gibbons. 1990. Rank Correlation Methods (5 ed.). A Charles Griffin Title.
[35]
M G Kendall and B Babington-Smith. 1940. On the Method of Paired Comparisons. Biometrica 31 (1940), 324--345.
[36]
Kihwan Kim, Jinwei Gu, Stephen Tyree, Pavlo Molchanov, Matthias Nießner, and Jan Kautz. 2017. A lightweight approach for on-the-fly reflectance estimation. In Proceedings of the international conference on computer vision (ICCV). 20--28.
[37]
Diederik P Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014).
[38]
Jaroslav Křivánek, James A. Ferwerda, and Kavita Bala. 2010. Effects of global illumination approximations on material appearance. ACM transactions on graphics (TOG, proc. SIGGRAPH) 29, 4, Article 112 (July 2010), 112:1--112:10 pages.
[39]
Manuel Lagunas, Elena Garces, and Diego Gutierrez. 2018. Learning icons appearance similarity. Multimedia tools and applications (2018), 1--19.
[40]
Thomas Leung and Jitendra Malik. 2001. Representing and recognizing the visual appearance of materials using three-dimensional textons. International journal of computer vision 43, 1 (2001), 29--44.
[41]
Stephen Lombardi and Ko Nishino. 2012. Reflectance and natural illumination from a single image. In European conference on computer vision (ECCV). Springer, 582--595.
[42]
Feng Lu, Xiaowu Chen, Imari Sato, and Yoichi Sato. 2018. SymPS: BRDF symmetry guided photometric stereo for shape and light source estimation. IEEE transactions on pattern analysis and machine intelligence 40, 1 (2018), 221--234.
[43]
Zhaoliang Lun, Evangelos Kalogerakis, and Alla Sheffer. 2015. Elements of style: learning perceptual shape style similarity. ACM Transactions on graphics (TOG) 34, 4 (2015), 84.
[44]
Laurence T Maloney and David H Brainard. 2010. Color and material perception: Achievements and challenges. Journal of vision 10, 9 (2010), 19--19.
[45]
Rafał Mantiuk, Scott Daly, and Louis Kerofsky. 2008. Display adaptive tone mapping. In ACM Transactions on graphics (TOG), Vol. 27. ACM, 68.
[46]
Wojciech Matusik, Boris Ajdin, Jinwei Gu, Jason Lawrence, Hendrik P. A. Lensch, Fabio Pellacini, and Szymon Rusinkiewicz. 2009. Printing Spatially-Varying Reflectance. ACM transactions on graphics (TOG, proc. SIGGRAPH Asia) 28, 5 (Dec. 2009).
[47]
Wojciech Matusik, Hanspeter Pfister, Matt Brand, and Leonard McMillan. 2003. A Data-Driven Reflectance Model. ACM Transactions on graphics (TOG) 22, 3 (July 2003), 759--769.
[48]
Brian McFee and Gert Lanckriet. 2011. Learning Multi-modal Similarity. Journal of machine learning research 12 (2011), 491--523.
[49]
Leland McInnes and John Healy. 2018. Umap: Uniform manifold approximation and projection for dimension reduction. arXiv preprint arXiv:1802.03426 (2018).
[50]
M Mylo, M Giesel, Q Zaidi, M Hullin, and R Klein. 2017. Appearance bending: A perceptual editing paradigm for data-driven material models. Vision, modeling and visualization. The Eurographics association (2017).
[51]
Addy Ngan, Frédo Durand, and Wojciech Matusik. 2005. Experimental Analysis of BRDF Models. In Eurographics Symposium on Rendering (2005). The Eurographics association.
[52]
Addy Ngan, Frédo Durand, and Wojciech Matusik. 2006. Image-driven Navigation of Analytical BRDF Models. In Rendering Techniques. 399--407.
[53]
Fabio Pellacini, James A. Ferwerda, and Donald P. Greenberg. 2000. Toward a Psychophysically-based Light Reflection Model for Image Synthesis. In Proceedings of the 27th annual conference on computer graphics and interactive techniques (SIGGRAPH '00). 55--64.
[54]
Thiago Pereira and Szymon Rusinkiewicz. 2012. Gamut mapping spatially varying reflectance with an improved BRDF similarity metric. In Computer graphics forum, Vol. 31. Wiley Online Library, 1557--1566.
[55]
Sashank J Reddi, Satyen Kale, and Sanjiv Kumar. 2019. On the convergence of adam and beyond. arXiv preprint arXiv:1904.09237 (2019).
[56]
Michael Rubinstein, Diego Gutierrez, Olga Sorkine, and Ariel Shamir. 2010. A Comparative Study of Image Retargeting. ACM transactions on graphics (TOG, proc. SIGGRAPH Asia) 29, 6 (2010), 160:1--160:10.
[57]
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. 2015. Imagenet large scale visual recognition challenge. International journal of computer vision 115, 3 (2015), 211--252.
[58]
Florian Schroff, Dmitry Kalenichenko, and James Philbin. 2015. Facenet: A unified embedding for face recognition and clustering. In Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR). 815--823.
[59]
M. Schultz and T. Joachims. 2003. Learning a Distance Metric from Relative Comparisons. In Advances in neural information processing systems.
[60]
Gabriel Schwartz and Ko Nishino. 2016. Material Recognition from Local Appearance in Global Context. arXiv preprint arXiv:1611.09394 (2016).
[61]
Gabriel Schwartz and Ko Nishino. 2018. Recognizing Material Properties from Images. arXiv preprint arXiv:1801.03127 (2018).
[62]
Ana Serrano, Diego Gutierrez, Karol Myszkowski, Hans-Peter Seidel, and Belen Masia. 2016. An Intuitive Control Space for Material Appearance. ACM Transactions on graphics (TOG) 35, 6, Article 186 (Nov. 2016), 186:1--186:12 pages.
[63]
Lavanya Sharan, Ruth Rosenholtz, and Edward Adelson. 2009. Material perception: What can you see in a brief glance? Journal of vision 9, 8 (2009), 784--784.
[64]
Ali Sharif Razavian, Hossein Azizpour, Josephine Sullivan, and Stefan Carlsson. 2014. CNN features off-the-shelf: an astounding baseline for recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR) workshops. 806--813.
[65]
Karen Simonyan and Andrew Zisserman. 2014. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556 (2014).
[66]
Cyril Soler, Kartic Subr, and Derek Nowrouzezahrai. 2018. A Versatile Parameterization for Measured Material Manifolds. In Computer graphics forum, Vol. 37. Wiley Online Library, 135--144.
[67]
Tiancheng Sun, Henrik Wann Jensen, and Ravi Ramamoorthi. 2018. Connecting Measured BRDFs to Analytic BRDFs by Data-driven Diffuse-specular Separation. In ACM transactions on graphics (TOG, proc. SIGGRAPH Asia). Article 273, 273:1--273:15 pages.
[68]
Tiancheng Sun, Ana Serrano, Diego Gutierrez, and Belen Masia. 2017. Attribute-preserving Gamut Mapping of Measured BRDFs. Computer graphics forum 36, 4 (July 2017).
[69]
Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jon Shlens, and Zbigniew Wojna. 2016. Rethinking the inception architecture for computer vision. In Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR). 2818--2826.
[70]
Omer Tamuz, Ce Liu, Serge Belongie, Ohad Shamir, and Adam Tauman Kalai. 2011. Adaptively Learning the Crowd Kernel. In Proceedings of the 28th international conference on machine learning (ICML'11). 673--680.
[71]
William Thompson, Roland Fleming, Sarah Creem-Regehr, and Jeanine Kelly Stefanucci. 2011. Visual Perception from a Computer Graphics Perspective (1st ed.). A. K. Peters, Ltd., Natick, MA, USA.
[72]
Jan Jaap R Van Assen, Pascal Barla, and Roland W Fleming. 2018. Visual features in the perception of liquids. Current biology 28, 3 (2018), 452--458.
[73]
Laurens Van Der Maaten and Kilian Weinberger. 2012. Stochastic triplet embedding. In Machine learning for signal processing (MLSP), 2012 IEEE International Workshop on. IEEE, 1--6.
[74]
Peter Vangorp, Pascal Barla, and Roland W Fleming. 2017. The perception of hazy gloss. Journal of vision 17, 5 (2017), 19--19.
[75]
Peter Vangorp, Jurgen Laurijssen, and Philip Dutré. 2007. The Influence of Shape on the Perception of Material Reflectance. ACM Transactions on graphics (TOG) 26, 3, Article 77 (July 2007).
[76]
Raquel Vidaurre, Dan Casas, Elena Garces, and Jorge Lopez-Moreno. 2019. BRDF Estimation of Complex Materials with Nested Learning. In IEEE Winter conference on applications of computer vision (WACV).
[77]
Zhou Wang, Alan C Bovik, Hamid R Sheikh, Eero P Simoncelli, et al. 2004. Image quality assessment: from error visibility to structural similarity. IEEE transactions on image processing 13, 4 (2004), 600--612.
[78]
Peter Welinder, Steve Branson, Pietro Perona, and Serge J Belongie. 2010. The multidimensional wisdom of crowds. In Advances in neural information processing systems. 2424--2432.
[79]
Josh Wills, Sameer Agarwal, David Kriegman, and Serge Belongie. 2009. Toward a Perceptual Space for Gloss. ACM Transactions on graphics (TOG) 28, 4, Article 103 (Sept. 2009), 103:1--103:15 pages.
[80]
Jason Yosinski, Jeff Clune, Yoshua Bengio, and Hod Lipson. 2014. How transferable are features in deep neural networks?. In Advances in neural information processing systems. 3320--3328.
[81]
Richard Zhang, Phillip Isola, Alexei A. Efros, Eli Shechtman, and Oliver Wang. 2018. The Unreasonable Effectiveness of Deep Features as a Perceptual Metric. In Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR). IEEE Computer Society, 586--595.
[82]
Károly Zsolnai-Fehér, Peter Wonka, and Michael Wimmer. 2018. Gaussian Material Synthesis. ACM Transactions on graphics (TOG) 37, 4, Article 76 (July 2018), 76:1--76:14 pages.

Cited By

View all
  • (2024)Material Attribute Estimation as Part of Telecommunication Augmented Reality, Virtual Reality, and Mixed Reality System: Systematic ReviewElectronics10.3390/electronics1313247313:13(2473)Online publication date: 25-Jun-2024
  • (2024)Not All the Same: Understanding and Informing Similarity Estimation in Tile-Based Video GamesProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642077(1-23)Online publication date: 11-May-2024
  • (2024)Predicting Perceived Gloss: Do Weak Labels Suffice?Computer Graphics Forum10.1111/cgf.1503743:2Online publication date: 27-Apr-2024
  • Show More Cited By

Recommendations

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Transactions on Graphics
ACM Transactions on Graphics  Volume 38, Issue 4
August 2019
1480 pages
ISSN:0730-0301
EISSN:1557-7368
DOI:10.1145/3306346
Issue’s Table of Contents
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 12 July 2019
Published in TOG Volume 38, Issue 4

Permissions

Request permissions for this article.

Check for updates

Author Tags

  1. material appearance
  2. neural networks
  3. physically based material perception

Qualifiers

  • Research-article

Funding Sources

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • Downloads (Last 12 months)74
  • Downloads (Last 6 weeks)8
Reflects downloads up to 09 Dec 2024

Other Metrics

Citations

Cited By

View all
  • (2024)Material Attribute Estimation as Part of Telecommunication Augmented Reality, Virtual Reality, and Mixed Reality System: Systematic ReviewElectronics10.3390/electronics1313247313:13(2473)Online publication date: 25-Jun-2024
  • (2024)Not All the Same: Understanding and Informing Similarity Estimation in Tile-Based Video GamesProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642077(1-23)Online publication date: 11-May-2024
  • (2024)Predicting Perceived Gloss: Do Weak Labels Suffice?Computer Graphics Forum10.1111/cgf.1503743:2Online publication date: 27-Apr-2024
  • (2024)Studying the Effect of Material and Geometry on Perceptual Outdoor IlluminationIEEE Transactions on Visualization and Computer Graphics10.1109/TVCG.2023.334756030:9(6468-6480)Online publication date: Sep-2024
  • (2024)A Unification Between Deep-Learning Vision, Compartmental Dynamical Thermodynamics, and Robotic Manipulation for a Circular EconomyIEEE Access10.1109/ACCESS.2024.349375512(173502-173516)Online publication date: 2024
  • (2023)materialmodifier: An R package of photo editing effects for material perception researchBehavior Research Methods10.3758/s13428-023-02116-256:3(2657-2674)Online publication date: 10-May-2023
  • (2023)The Visual Language of FabricsACM Transactions on Graphics10.1145/359239142:4(1-15)Online publication date: 26-Jul-2023
  • (2023)In‐the‐wild Material Appearance Editing using Perceptual AttributesComputer Graphics Forum10.1111/cgf.1476542:2(333-345)Online publication date: 23-May-2023
  • (2023)How Will It Drape Like? Capturing Fabric Mechanics from Depth ImagesComputer Graphics Forum10.1111/cgf.1475042:2(149-160)Online publication date: 23-May-2023
  • (2023)One-shot recognition of any material anywhere using contrastive learning with physics-based rendering2023 IEEE/CVF International Conference on Computer Vision (ICCV)10.1109/ICCV51070.2023.02150(23467-23476)Online publication date: 1-Oct-2023
  • Show More Cited By

View Options

Login options

Full Access

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Media

Figures

Other

Tables

Share

Share

Share this Publication link

Share on social media