Abstract
In this paper, we present our approach towards designing and implementing a virtual 3D sound sculpting interface that creates audiovisual results using hand motions in real time. In the interface “Virtual Pottery,” we use the metaphor of pottery creation in order to adopt the natural hand motions to 3D spatial sculpting. Users can create their own pottery pieces by changing the position of their hands in real time, and also generate 3D sound sculptures based on pre-existing rules of music composition. The interface of Virtual Pottery can be categorized by shape design and camera sensing type. This paper describes how we developed the two versions of Virtual Pottery and implemented the technical aspects of the interfaces. Additionally, we investigate the ways of translating hand motions into musical sound. The accuracy of the detection of hand motions is crucial for translating natural hand motions into virtual reality. According to the results of preliminary evaluations, the accuracy of both motion-capture tracking system and portable depth sensing camera is as high as the actual data. We carried out user studies, which took into account information about the two exhibitions along with the various ages of users. Overall, Virtual Pottery serves as a bridge between the virtual environment and traditional art practices, with the consequence that it can lead to the cultivation of the deep potential of virtual musical instruments and future art education programs.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
Beyer and Meier M (2011) “Music interfaces for novice users: composing music on a public display with hand gestures,” in Proceedings of the International Conference on New Interfaces for Musical Expression (NIME 2011)
Blake A, Isard M (1994) “3D position, attitude and shape input using video tracking of hands and lips,” in Proceedings of the 21st annual conference on computer graphics and interactive techniques (SIGGRAPH '94). ACM Press, New York, pp 185–192
BlobScanner. Retrieved from http://code.google.com/p/BlobScanner/
Cani MP, Alexis Angelidis A (2006) “Towards virtual clay.” in proceedings of ACM SIGGRAPH 2006 courses. USA, New York, pp 67–83
Da Fontoura Costa L, et al. (2000) Shape analysis and classification: theory and practice. CRC Press
Han Y, Han B (2012) “Virtual Pottery: an interactive audio-visual installation,” in Proceedings of 12th international conferences on New Interfaces on Musical Expression (NIME 2012)
Holz C, Wilson A (2011) “Data miming: Inferring spatial object descriptions from human gesture,” in Proceedings of the 2011 annual conference on Human factors in computing systems (ACM CHI ’11), pp. 811–820
Ip HHS, Kwong B, Law KCK (2005) “BodyMusic: A novel framework design for body-driven music composition,” in Proceedings of the 2005 ACM SIGCHI international conference on advances in computer entertainment technology. ACM Press, New York, pp 342–345
Ip HHS, Kwong B, Law KCK (2005) “Cyber composer: hand gesture-driven intelligent music composition and generation,” in Proceedings of 11th international multimedia modeling conference, pp.46–52
Kameyama K (1997) “Virtual clay modeling system,” in Proceedings of the ACM Symposium on Virtual Reality Software and Technology, pp 197–200
Knopf GK, Igwe PC (2005) Deformable mesh for virtual shape sculpting. Robotics and Computer-Integrated Manufacturing 21:302–311
Korida K, Nishino H, Utsumiya K (1997) “An interactive 3D interface for a virtual ceramic art work environment,” in Proceedings of International Conference on Virtual Systems and Multimedia (VSMM '97), pp. 227–234. IEEE Computer Society
L’Artisan Electronique. Retrieved from http://www.creativeapplications.net/openframeworks/lartisan-electronique-openframeworks/.
Lee J, Han G, Choi S (2008) Haptic Pottery modeling using circular sector element method. Springer Lecture Notes in Computer Science (LNCS) 5024:668–674
Levin G, Lieberman Z (2005) “Sounds from shapes: audiovisual performance with hand silhouette contours in the manual input sessions,” in Proceedings of the 2005 international conference on New Interfaces for Musical Expression (NIME 2005), pp. 115–120
McDonnell KT, Qin H, Wlodarczyk RA (2001) “Virtual clay: a real-time sculpting system with haptic toolkits,” in Proceedings of the 2001 symposium of Interactive 3D graphics, pp.179–190, New York, USA
Mulder and Fels S (1998) “Sound sculpting: manipulating sound through virtual sculpting,” in Proceedings of Western Computer Graphics Symposium
Processing. Retrieved from http://processing.org/
Rigid Body toolkit. Retrieved from http://www.naturalpoint.com/OptiTrack/products/rigidbody-toolkit/
UDP library. Retrieved from http://ubaa.net/shared/processing/udp/
Vinayak, Murugappan S, Piya C, Ramani K (2012) “Handy-Potter: Rapid 3D Shape Exploration through Natural Hand Motions,” in Proceedings of the ASME 2012 International Design Engineering Technical Conferences & Computers and Information in Engineering Conference (IDETC/CIE 2012), August 12–15
Young D (2002) “The Hyperbow controller: Real-time dynamics measurement of violin performance,” in Proceedings of the 2002 conference on New Instruments for Musical Expression (NIME 2002)
Acknowledgement
These works have been supported by TransLab, University of California, Santa Barbara (UCSB), under the guidance of Prof. Marcos Novak.
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Han, Y.C., Han, Bj. Virtual pottery: a virtual 3D audiovisual interface using natural hand motions. Multimed Tools Appl 73, 917–933 (2014). https://doi.org/10.1007/s11042-013-1382-3
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11042-013-1382-3