Learning 3D functionality representations
Article No.: 6, Pages 1 - 5
Abstract
A central goal of computer graphics is to provide tools for designing and simulating real or imagined artifacts. An understanding of functionality is important in enabling such modeling tools. Given that the majority of man-made artifacts are designed to serve a certain function, the functionality of objects is often reflected by their geometry, the way that they are organized in an environment, and their interaction with other objects or agents. Thus, in recent years, a variety of methods in shape analysis have been developed to extract functional information about objects and scenes from these different types of cues.
In this course, we discuss recent developments involving functionality analysis of 3D shapes and scenes. We provide a summary of the state-of-the-art in this area, including a discussion of key ideas and an organized review of the relevant literatures. More specifically, we first present a general definition of functionality from which we derive criteria for classifying the body of prior work. This definition facilitates a comparative view of methods for functionality analysis. Moreover, we connect these methods to recent advances in deep learning, computer vision and robotics. Finally, we discuss a variety of application areas, and outline current challenges and directions for future work.
References
[1]
Akizuki S., Aoki Y.: Tactile logging for understanding plausible tool use based on human demonstration. In BMVC Workshop on Vision for Interaction and Behaviour Understanding (VIBE) (2018).
[2]
Andries M., Dehban A., Santos-Victor J.: Automatic generation of object shapes with desired affordances using voxelgrid representation. Frontiers in Neurorobotics 14 (2020), 22.
[3]
Bar-Aviv E., Rivlin E.: Functional 3d object classification using simulation of embodied agent. pp. 307--316.
[4]
Fish N., Averkiou M., van Kaick O., Sorkine-Hornung O., Cohen-Or D., Mitra N. J.: Meta-representation of shape families. ACM Trans. on Graph (SIGGRAPH) 33, 4 (2014), 34:1--11.
[5]
Fu Q., Fu H., Yan H., Zhou B., Chen X., Li X.: Human-centric metrics for indoor scene assessment and synthesis. Graphical Models 110 (2020), 101073.
[6]
Fisher M., Ritchie D., Savva M., Funkhouser T., Hanrahan P.: Example-based synthesis of 3D object arrangements. ACM Trans. on Graph (SIGGRAPH Asia) 31, 6 (2012), 135:1--11.
[7]
Fisher M., Savva M., Hanrahan P.: Characterizing structural relationships in scenes using graph kernels. ACM Trans. on Graph (SIGGRAPH) 30, 4 (2011), 34:1--12.
[8]
Fisher M., Savva M., Li Y., Hanrahan P., Niessner M.: Activity-centric scene synthesis for functional 3D scene modeling. ACM Trans. on Graph (SIGGRAPH Asia) 34, 6 (2015), 179:1--13.
[9]
Gelfand N., Guibas L. J.: Shape segmentation using local slip-page analysis. pp. 214--223.
[10]
Grabner H., Gall J., Van Gool L.: What makes a chair a chair? In Proc. Conf. on Computer Vision and Pattern Recognition (CVPR) (2011), IEEE, pp. 1529--1536.
[11]
Hu R., Li W., Kaick O. V., Shamir A., Zhang H., Huang H.: Learning to predict part mobility from a single static snapshot. ACM Trans. on Graph (SIGGRAPH Asia) 36, 6 (2017), 217:1--13.
[12]
Hu R., Savva M., van Kaick O.: Functionality representations and applications for shape analysis. Computer Graphics Forum (Eurographics State-of-the-art Report) 37, 2 (2018), 603--624.
[13]
Hu R., van Kaick O., Wu B., Huang H., Shamir A., Zhang H.: Learning how objects function via co-analysis of interactions. ACM Trans. on Graph (SIGGRAPH) 35, 4 (2016), 47:1--13.
[14]
Hu R., Yan Z., Zhang J., van Kaick O., Shamir A., Zhang H., Huang H.: Predictive and generative neural networks for object functionality. ACM Trans. on Graph (SIGGRAPH) 37, 4 (2018), 151:1--151:13.
[15]
Hu R., Zhu C., van Kaick O., Liu L., Shamir A., Zhang H.: Interaction context (ICON): Towards a geometric functionality descriptor. ACM Trans. on Graph (SIGGRAPH) 34, 4 (2015), 83:1--12.
[16]
Jiang Y., Koppula H., Saxena A.: Hallucinated humans as the hidden context for labeling 3D scenes. In Proc. Conf. on Computer Vision and Pattern Recognition (CVPR) (2013), IEEE, pp. 2993--3000.
[17]
Kim V. G., Chaudhuri S., Guibas L., Funkhouser T.: Shape2pose: Human-centric shape analysis. ACM Trans. on Graph (SIGGRAPH) 33, 4 (2014), 120:1--12.
[18]
Krs V., Mech R., Gaillard M., Carr N., Benes B.: PICO: procedural iterative constrained optimizer for geometric modeling. IEEE Trans. Visualization & Computer Graphics early access (2020).
[19]
Kim D. I., Sukhatme G. S.: Semantic labeling of 3D point clouds with object affordance for robot manipulation. In Int. Conf. Robotics and Automation (ICRA) (2014), pp. 5578--5584.
[20]
Kokic M., Stork J. A., Haustein J. A., Kragic D.: Affordance detection for task-specific grasping using deep learning. In Proc. Conf. on Humanoid Robotics (2017), pp. 91--98.
[21]
Lee K. H., Choi M. G., Lee J.: Motion patches: building blocks for virtual environments annotated with motion data. ACM Trans. on Graph (SIGGRAPH) 25, 3 (2006), 898--906.
[22]
Li X., Liu S., Kim K., Wang X., Yang M., Kautz J.: Putting humans in a scene: Learning affordance in 3D indoor environments. In Proc. Conf. on Computer Vision and Pattern Recognition (CVPR) (2019), pp. 12360--12368.
[23]
Laga H., Mortara M., Spagnuolo M.: Geometry and context for semantic correspondences and functionality recognition in man-made 3D shapes. ACM Trans. on Graph 32, 5 (2013), 150:1--16.
[24]
Li Y., Schomaker L., Kasaei S. H.: Learning to grasp 3d objects using deep residual u-nets. arXiv preprint arXiv:2002.03892 (2020).
[25]
Li X., Wang H., Yi L., Guibas L. J., Abbott A. L., Song S.: Category-level articulated object pose estimation. In Proc. Conf. on Computer Vision and Pattern Recognition (CVPR) (2020).
[26]
Monszpart A., Guerrero P., Ceylan D., Yumer E., Mitra N. J.: iMapper: interaction-guided scene mapping from monocular videos. ACM Trans. on Graph (SIGGRAPH) 38, 4 (2019), 92:1--92:15.
[27]
Ma R., Li H., Zou C., Liao Z., Tong X., Zhang H.: Action-driven 3D indoor scene evolution. ACM Trans. on Graph (SIGGRAPH Asia) 35, 6 (2016), 173:1--13.
[28]
Merrell P., Schkufza E., Li Z., Agrawala M., Koltun V.: Interactive furniture layout using interior design guidelines. ACM Trans. on Graph (SIGGRAPH) 30, 4 (2011), 87:1--10.
[29]
Myers A., Teo C. L., Fermüller C., Aloimonos Y.: Affordance detection of tool parts from geometric features. In Robotics and Automation (ICRA), 2015 IEEE International Conference on (2015), IEEE, pp. 1374--1381.
[30]
Mitra N. J., Yang Y.-L., Yan D.-M., Li W., Agrawala M.: Illustrating how mechanical assemblies work. ACM Trans. on Graph (SIGGRAPH) 29, 4 (2010), 58:1--12.
[31]
Mao A., Zhang H., Xie Z., Yu M., Liu Y., He Y.: Automatic sitting pose generation for ergonomic ratings of chairs. IEEE Trans. Visualization & Computer Graphics early access (2019).
[32]
Pirk S., Krs V., Hu K., Rajasekaran S. D., Kang H., Yoshiyasu Y., Benes B., Guibas L. J.: Understanding and exploiting object interaction landscapes. ACM Trans. on Graph 36, 3 (2017), 31:1--14.
[33]
Pechuk M., Soldea O., Rivlin E.: Learning function-based object classification from 3D imagery. Computer Vision and Image Understanding 110, 2 (2008), 173--191.
[34]
Ruiz E., Mayol-Cuevas W. W.: Scalable real-time and one-shot multiple-affordance detection. In ICRA Workshop on Computational Models of Affordance in Robotics (2019).
[35]
Savva M., Chang A. X., Hanrahan P., Fisher M., Niessner M.: SceneGrok: Inferring action maps in 3D environments. ACM Trans. on Graph (SIGGRAPH Asia) 33, 6 (2014), 212:1--10.
[36]
Savva M., Chang A. X., Hanrahan P., Fisher M., Niessner M.: PiGraphs: Learning Interaction Snapshots from Observations. ACM Trans. on Graph (SIGGRAPH) 35, 4 (2016), 139:1--12.
[37]
Starke S., Zhang H., Komura T., Saito J.: Neural state machine for character-scene interactions. ACM Trans. on Graph (SIGGRAPH Asia) 38, 6 (2019), 209:1--209:14.
[38]
Wang H., Liang W., Yu L.-F.: Transferring objects: Joint inference of container and human pose. In Proc. Conf. on Computer Vision and Pattern Recognition (CVPR) (2017), pp. 2933--2941.
[39]
Wang X., Zhou B., Shi Y., Chen X., Zhao Q., Xu K.: Shape2Motion: joint analysis of motion parts and attributes from 3D shapes. In Proc. Conf. on Computer Vision and Pattern Recognition (CVPR) (2019).
[40]
Xu M., Li M., Xu W., Deng Z., Yang Y., Zhou K.: Interactive mechanism modeling from multi-view images. ACM Trans. on Graph (SIGGRAPH Asia) 35, 6 (2016), to appear.
[41]
Xiang F., Qin Y., Mo K., Xia Y., Zhu H., Liu F., Liu M., Jiang H., Yuan Y., Wang H., Yi L., Chang A. X., Guibas L. J., Su H.: SAPIEN: A simulated part-based interactive environment. In Proc. Conf. on Computer Vision and Pattern Recognition (CVPR) (2020).
[42]
Xu K., Stewart J., Fiume E.: Constraint-based automatic placement for scene composition. In Proc. Graphics Interface (2002), vol. 2.
[43]
Yi L., Huang H., Liu D., Kalogerakis E., Su H., Guibas L.: Deep part induction from articulated object pairs. ACM Trans. on Graph (SIGGRAPH Asia) 37, 6 (2018), 209:1--209:15.
[44]
Yan Z., Hu R., Yan X., Chen L., van Kaick O., Zhang H., Huang H.: RPM-Net: recurrent prediction of motion and parts from point cloud. ACM Trans. on Graph (SIGGRAPH Asia) 38, 6 (2019), 240:1--240:15.
[45]
Yumer M. E., Kara L. B.: Co-constrained handles for deformation in shape collections. ACM Trans. on Graph (SIGGRAPH Asia) 33, 6 (2014), 187:1--11.
[46]
Yu L.-F., Yeung S. K., Tang C.-K., Terzopoulos D., Chan T. F., Osher S.: Make it home: automatic optimization of furniture arrangement. ACM Trans. on Graph (SIGGRAPH) 30, 4 (2011), 86:1--12.
[47]
Zhao X., Choi M. G., Komura T.: Character-object interaction retrieval using the interaction bisector surface. Computer Graphics Forum (Eurographics) 36, 2 (2017), 119--129.
[48]
Zheng Y., Cohen-Or D., Mitra N. J.: Smart variations: Functional substructures for part compatibility. Computer Graphics Forum (Eurographics) 32, 2pt2 (2013), 195--204.
[49]
Zhao X., Hu R., Guerrero P., Mitra N., Komura T.: Relationship templates for creating scene variations. ACM Trans. on Graph (SIGGRAPH Asia) 35, 6 (2016), to appear.
[50]
Zhang Y., Hassan M., Neumann H., Black M. J., Tang S.: Generating 3D people in scenes without people. In Proc. Conf. on Computer Vision and Pattern Recognition (CVPR) (2020).
[51]
Zhu Y., Jiang C., Zhao Y., Terzopoulos D., Zhu S.-C.: Inferring forces and learning human utilities from videos. In Proc. Conf. on Computer Vision and Pattern Recognition (CVPR) (2016), IEEE, pp. 3823--3833.
[52]
Zheng Y., Liu H., Dorsey J., Mitra N. J.: Ergonomics-inspired reshaping and exploration of collections of models. IEEE Trans. Visualization & Computer Graphics 22, 6 (2016), 1732--1744.
[53]
Zhao X., Wang H., Komura T.: Indexing 3D scenes using the interaction bisector surface. ACM Trans. on Graph 33, 3 (2014), 22:1--14.
[54]
Zhu Y., Zhao Y., Chun Zhu S.: Understanding tools: Task-oriented object modeling, learning and recognition. In Proc. Conf. on Computer Vision and Pattern Recognition (CVPR) (2015), pp. 2855--2864.
Recommendations
SIGGRAPH Asia 2016: course notes directions in shape analysis towards functionality
SA '16: SIGGRAPH ASIA 2016 CoursesThe majority of man-made objects are designed to serve a certain function, and this is often reflected by the geometry of the objects, or the way that they are used or organized in an environment. In recent years, many efforts in shape analysis have ...
Learning how objects function via co-analysis of interactions
We introduce a co-analysis method which learns a functionality model for an object category, e.g., strollers or backpacks. Like previous works on functionality, we analyze object-to-object interactions and intra-object properties and relations. ...
Discrete scale axis representations for 3D geometry
SIGGRAPH '10: ACM SIGGRAPH 2010 papersThis paper addresses the fundamental problem of computing stable medial representations of 3D shapes. We propose a spatially adaptive classification of geometric features that yields a robust algorithm for generating medial representations at different ...
Comments
Please enable JavaScript to view thecomments powered by Disqus.Information & Contributors
Information
Published In
November 2020
842 pages
ISBN:9781450381123
DOI:10.1145/3415263
Copyright © 2020 Owner/Author.
Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for third-party components of this work must be honored. For all other uses, contact the Owner/Author.
Sponsors
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 04 December 2020
Check for updates
Author Tags
Qualifiers
- Article
Conference
SA '20
Sponsor:
Acceptance Rates
Overall Acceptance Rate 178 of 869 submissions, 20%
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 222Total Downloads
- Downloads (Last 12 months)23
- Downloads (Last 6 weeks)4
Reflects downloads up to 14 Dec 2024
Other Metrics
Citations
View Options
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign in