[go: up one dir, main page]
More Web Proxy on the site http://driver.im/ skip to main content
10.1145/3462244.3479951acmconferencesArticle/Chapter ViewAbstractPublication Pagesicmi-mlmiConference Proceedingsconference-collections
short-paper

Enhancing Ultrasound Haptics with Parametric Audio Effects

Published: 18 October 2021 Publication History

Abstract

Ultrasound haptic devices can create parametric audio as well as contactless haptic feedback. We investigate if multimodal output from these devices can influence the perception of haptic feedback. We used a magnitude estimation experiment to evaluate perceived roughness of an ultrasound haptic pattern. We found that white noise audio from the haptics device increased perceived roughness and pure tones did not, and that lower rendering frequencies may increase perceived roughness. Our results show that multimodal output has the potential to expand the range of sensations that can be presented by an ultrasound haptic device, paving the way to richer mid-air haptic interfaces.

Supplementary Material

MP4 File (ICMI21-fp1037.mp4)
This video presents an ICMI 2021 paper on the perception of ultrasound haptics with parametric audio effects. In this presentation I give a quick introduction to ultrasound haptics technology and how state-of-the-art devices currently create haptic patterns. I then discuss the challenges of using this technology to render tactile sensations like roughness and texture. One of the novel aspects of this work is that we used an ultrasound haptics device to simultaneously create audio and haptic output. I conclude with an overview of an experiment that investigated if white noise sound effects could affect the tactile perception of roughness.

References

[1]
Damien Ablart, William Frier, Hannah Limerick, Orestis Georgiou, and Marianna Obrist. 2019. Using Ultrasonic Mid-air Haptic Patterns in Multi-Modal User Experiences. In 2019 IEEE International Symposium on Haptic, Audio and Visual Environments and Games (HAVE). 1–6. https://doi.org/10.1109/HAVE.2019.8920969
[2]
Damien Ablart, Carlos Velasco, and Marianna Obrist. 2017. Integrating Mid-Air Haptics into Movie Experiences. In Proceedings of the 2017 ACM International Conference on Interactive Experiences for TV and Online Video - TVX ’17. ACM, 77–84. https://doi.org/10.1145/3077548.3077551
[3]
David Beattie, William Frier, Orestis Georgiou, Benjamin Long, and Damien Ablart. 2020. Incorporating the Perception of Visual Roughness into the Design of Mid-Air Haptic Textures. In ACM Symposium on Applied Perception 2020(SAP ’20). ACM, Article 4, 10 pages. https://doi.org/10.1145/3385955.3407927
[4]
David Beattie, Orestis Georgiou, Adam Harwood, Rory Clark, Benjamin Long, and Tom Carter. 2019. Mid-Air Haptic Textures from Graphics. In Proceedings of the IEEE World Haptics Conference 2019 - Demos.
[5]
H. O. Berktay. 1965. Parametric amplification by the use of acoustic non-linearities and some possible applications. Journal of Sound and Vibration 2, 4 (1965), 462–470. https://doi.org/10.1016/0022-460X(65)90123-9
[6]
Thomas Carter, Sue Ann Seah, Benjamin Long, Bruce Drinkwater, and Sriram Subramanian. 2013. UltraHaptics: Multi-Point Mid-Air Haptic Feedback for Touch Surfaces. In Proceedings of the 26th Symposium on User Interface Software and Technology - UIST ’13. ACM, 505–514. https://doi.org/10.1145/2501988.2502018
[7]
Patricia I. Cornelio-Martinez, Silvana De Pirro, Chi Thanh Vi, and Sriram Subramanian. 2017. Agency in Mid-air Interfaces. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems - CHI ’17. ACM, 2426–2439. https://doi.org/10.1145/3025453.3025457
[8]
Heather Culbertson and Katherine J Kuchenbecker. 2017. Ungrounded haptic augmented reality system for displaying roughness and friction. IEEE/ASME Transactions on Mechatronics 22, 4 (2017), 1839–1849.
[9]
Brygida Dzidek, William Frier, Adam Harwood, and Richard Hayden. 2018. Design and Evaluation of Mid-Air Haptic Interactions in an Augmented Reality Environment. In Proceedings of EuroHaptics 2018 in LNCS 10894 - EuroHaptics ’18. Springer, 489–499. https://doi.org/10.1007/978-3-319-93399-3_42
[10]
Marc O. Ernst and Martin S. Banks. 2002. Humans integrate visual and haptic information in a statistically optimal fashion. Nature 415, 6870 (2002), 429–433. https://doi.org/10.1038/415429a
[11]
Euan Freeman. 2021. User Study Data for ”Enhancing Ultrasound Haptics with Parametric Audio Effects”. https://doi.org/10.5281/zenodo.5144878
[12]
Euan Freeman, Ross Anderson, Julie Williamson, Graham Wilson, and Stephen Brewster. 2017. Textured Surfaces for Ultrasound Haptic Displays. In Proceedings of the 19th ACM International Conference on Multimodal Interaction - ICMI ’17 Demos. ACM. https://doi.org/10.1145/3136755.3143020
[13]
Euan Freeman, Stephen Brewster, and Vuokko Lantz. 2014. Tactile Feedback for Above-Device Gesture Interfaces: Adding Touch to Touchless Interactions. In Proceedings of the 16th International Conference on Multimodal Interaction - ICMI ’14. ACM, 419–426. https://doi.org/10.1145/2663204.2663280
[14]
Euan Freeman, Dong-Bach Vo, and Stephen Brewster. 2019. HaptiGlow: Helping Users Position their Hands for Better Mid-Air Gestures and Ultrasound Haptic Feedback. In Proceedings of the IEEE World Haptics Conference 2019. IEEE, 289–294. https://doi.org/10.1109/WHC.2019.8816092
[15]
Euan Freeman and Graham Wilson. 2021. Perception of Ultrasound Haptic Focal Point Motion. In Proceedings of the 23rd ACM International Conference on Multimodal Interaction - ICMI ’21. ACM. https://doi.org/10.1145/3462244.3479950
[16]
Euan Freeman, Graham Wilson, Dong-Bach Vo, Alexander Ng, Ioannis Politis, and Stephen Brewster. 2017. Multimodal Feedback in HCI: Haptics, Non-Speech Audio, and Their Applications. In The Handbook of Multimodal-Multisensor Interfaces: Foundations, User Modeling and Common Modality Combinations.
[17]
William Frier, Damien Ablart, Jamie Chilles, Benjamin Long, Marcello Giordano, Marianna Obrist, and Sriram Subramanian. 2018. Using Spatiotemporal Modulation to Draw Tactile Patterns in Mid-air. In Proc. of EuroHaptics 2018.
[18]
William Frier, Dario Pittera, Damien Ablart, Marianna Obrist, and Sriram Subramanian. 2019. Sampling Strategy for Ultrasonic Mid-Air Haptics. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems Proceedings - CHI ’19. ACM, Paper 121. https://doi.org/10.1145/3290605.3300351
[19]
Steve Guest, Caroline Catmur, Donna Lloyd, and Charles Spence. 2002. Audiotactile interactions in roughness perception. Experimental Brain Research 146, 2 (2002), 161–171. https://doi.org/10.1007/s00221-002-1164-z
[20]
Daniel Hajas, Dario Pittera, Antony Nasce, Orestis Georgiou, and Marianna Obrist. 2020. Mid-Air Haptic Rendering of 2D Geometric Shapes with a Dynamic Tactile Pointer. IEEE Transactions on Haptics 13, 1 (2020), 1–12. https://doi.org/10.1109/TOH.2020.2966445
[21]
Ryuji Hirayama, Diego Martinez Plasencia, Nobuyuki Masuda, and Sriram Subramanian. 2019. A volumetric display for visual, tactile and audio presentation using acoustic trapping. Nature 575(2019). https://doi.org/10.1038/s41586-019-1739-5
[22]
Thomas Howard, Gerard Gallagher, Anatole Lécuyer, Claudio Pacchierotti, and Maud Marchal. 2019. Investigating the Recognition of Local Shapes Using Mid-air Ultrasound Haptics. Proceedings of the 2019 IEEE World Haptics Conference - WHC ’19 (2019), 503–508. https://doi.org/10.1109/WHC.2019.8816127
[23]
Takayuki Iwamoto, Mari Tatezono, and Hiroyuki Shinoda. 2008. Non-contact method for producing tactile sensation using airborne ultrasound. In Proceedings of EuroHaptics 2008. Springer. https://doi.org/10.1007/978-3-540-69057-3_64
[24]
Lynette A. Jones and Hong Z. Tan. 2013. Application of Psychophysical Techniques to Haptic Research. IEEE Transactions on Haptics 6, 3 (2013), 268–284. https://doi.org/10.1109/TOH.2012.74
[25]
Eric R. Kandel, James H. Schwartz, Thomas M. Jessell, Steven A. Siegelbaum, and A. J. Hudspeth. 2012. Touch. In Principles of Neural Science(5th ed.). McGraw-Hill Publishing, Chapter 23, 498–529.
[26]
Seung-Chan Kim, Ki-Uk Kyung, and Dong-Soo Kwon. 2007. The Effect of Sound on Haptic Perception. Second Joint EuroHaptics Conference and Symposium on Haptic Interfaces for Virtual Environment and Teleoperator Systems - WHC ’07 (2007). https://doi.org/10.1109/WHC.2007.110
[27]
Roberta L. Klatzky and Susan J. Lederman. 2010. Multisensory Texture Perception. In Multisensory Object Perception in the Primate Brain. Springer, Chapter 12, 211–230. https://doi.org/10.1007/978-1-4419-5615-6_12
[28]
Georgios Korres and Mohamad Eid. 2016. Haptogram: Ultrasonic Point-Cloud Tactile Stimulation. IEEE Access 4(2016), 7758 – 7769. https://doi.org/10.1109/ACCESS.2016.2608835
[29]
Hannah Limerick, Richard Hayden, David Beattie, Orestis Georgiou, and Jörg Müller. 2019. User Engagement for Mid-Air Haptic Interactions with Digital Signage. In Proceedings of the 8th ACM International Symposium on Pervasive Displays. ACM, Article 15. https://doi.org/10.1145/3321335.3324944
[30]
Benjamin Long, Sue Ann Seah, Tom Carter, and Sriram Subramanian. 2014. Rendering Volumetric Haptic Shapes in Mid-Air using Ultrasound. ACM Transactions on Graphics 33, 6 (2014), Article 181. https://doi.org/10.1145/2661229.2661257
[31]
Yasutoshi Makino, Yoshikazu Furuyama, Seki Inoue, and Hiroyuki Shinoda. 2016. HaptoClone (Haptic-Optical Clone) for Mutual Tele-Environment by Real-time 3D Image Transfer with Midair Force Feedback. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems - CHI ’16. ACM, 1980–1990. https://doi.org/10.1145/2858036.2858481
[32]
Jonatan Martinez, Daniel Griffiths, Valerio Biscione, Orestis Georgiou, and Tom Carter. 2018. Touchless Haptic Feedback for Supernatural VR Experiences. Proceedings of the 25th IEEE Conference on Virtual Reality and 3D User Interfaces (2018), 629–630. https://doi.org/10.1109/VR.2018.8446522
[33]
Marilyn McGee, Philip Gray, and Stephen Brewster. 2001. Feeling rough: multimodal perception of virtual roughness. In Proceedings of Eurohaptics ’01. 29–33.
[34]
Rafael Morales González, Asier Marzo, Euan Freeman, William Frier, and Orestis Georgiou. 2021. UltraPower: Powering Tangible & Wearable Devices with Focused Ultrasound. In Proceedings of the 15th International Conference on Tangible, Embedded, and Embodied Interaction. ACM. https://doi.org/10.1145/3430524.3440620
[35]
Marianna Obrist, Sue Ann Seah, and Sriram Subramanian. 2013. Talking about Tactile Experiences. In Proc. of the SIGCHI Conference on Human Factors in Computing Systems. ACM, 1659–1668. https://doi.org/10.1145/2470654.2466220
[36]
Marianna Obrist, Sriram Subramanian, Elia Gatti, Benjamin Long, and Thomas Carter. 2015. Emotions Mediated Through Mid-Air Haptics. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems - CHI ’15. ACM, 2053–2062. https://doi.org/10.1145/2702123.2702361
[37]
Yoichi Ochiai, Takayuki Hoshi, and Ippei Suzuki. 2017. Holographic Whisper: Rendering Audible Sound Spots in Three-dimensional Space by Focusing Ultrasonic Waves. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems - CHI ’17. ACM, 4314–4325. https://doi.org/10.1145/3025453.3025989
[38]
Dessislava Peeva, Bridget Baird, Ozgur Izmirli, and Donald Blevins. 2004. Haptic and sound correlations: Pitch, loudness and texture. In Proceedings. Eighth International Conference on Information Visualisation, 2004. IV 2004. IEEE, 659–664.
[39]
Diego Martinez Plasencia, Ryuji Hirayama, Roberto Montano-Murillo, and Sriram Subramanian. 2020. GS-PAT: High-Speed Multi-Point Sound-Fields for Phased Arrays of Transducers. ACM Transactions on Graphics 39, 4, Article 138(2020), 12 pages. https://doi.org/10.1145/3386569.3392492
[40]
F. Joseph Pompei. 2002. Sound from ultrasound: the parametric array as an audible sound source. PhD Thesis. Massachusetts Institute of Technology.
[41]
D. Purves, G. J. Augustine, D. Fitzpatrick, W. C. Hall, A.-S. LaMantia, J. O. McNamara, and S. M. Williams (Eds.). 2004. Neuroscience (3rded.). Sinauer Associates, Inc.https://doi.org/10.1212/01.WNL.0000154473.43364.47
[42]
Ismo Rakkolainen, Euan Freeman, Antti Sand, Roope Raisamo, and Stephen Brewster. 2020. A Survey of Mid-Air Ultrasound Haptics and Its Applications. IEEE Transactions on Haptics(2020). https://doi.org/10.1109/TOH.2020.3018754
[43]
Sonja Rümelin, Thomas Gabler, and Jesper Bellenbaum. 2017. Clicks are in the Air: How to Support the Interaction with Floating Objects through Ultrasonic Feedback. In Proceedings of AutomotiveUI ’17. ACM, 103–108. https://doi.org/10.1145/3122986.3123010
[44]
Paul Strohmeier and Kasper Hornbæk. 2017. Generating haptic textures with a vibrotactile actuator. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems. 4994–5005.
[45]
Yuika Suzuki and Jiro Gyoba. 2009. Effects of Sounds on Tactile Roughness Depend on the Congruency between Modalities. In Proceedings of the IEEE World Haptics Conference. IEEE, 150–153. https://doi.org/10.1109/WHC.2009.4810857
[46]
Yuika Suzuki, Jiro Gyoba, and Shuichi Sakamoto. 2008. Selective effects of auditory stimuli on tactile roughness perception. Brain Research 1242(2008), 87–94. https://doi.org/10.1016/j.brainres.2008.06.104
[47]
Ryoko Takahashi, Keisuke Hasegawa, and Hiroyuki Shinoda. 2018. Lateral Modulation of Midair Ultrasound Focus for Intensified Vibrotactile Stimuli. In Proceedings of EuroHaptics 2018 in LNCS 10894 - EuroHaptics ’18. Springer International Publishing, 276–288. https://doi.org/10.1007/978-3-319-93399-3_25
[48]
Ryoko Takahashi, Keisuke Hasegawa, and Hiroyuki Shinoda. 2019. Tactile Stimulation by Repetitive Lateral Movement of Midair Ultrasound Focus. IEEE Transactions on Haptics(2019). https://doi.org/10.1109/TOH.2019.2946136
[49]
Ryoko Takahashi, Keisuke Hasegawa, and Hiroyuki Shinoda. 2020. Tactile Stimulation by Repetitive Lateral Movement of Midair Ultrasound Focus. IEEE Transactions on Haptics 13, 2 (2020), 334–342. https://doi.org/10.1109/TOH.2019.2946136
[50]
Graham Wilson, Tom Carter, Sriram Subramanian, and Stephen Brewster. 2014. Perception of Ultrasonic Haptic Feedback on the Hand: Localisation and Apparent Motion. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems - CHI ’14. ACM, 1133–1142. https://doi.org/10.1145/2556288.2557033
[51]
Jeffrey M Yau, Jonathon B Olenczak, John F Dammann, and Sliman J Bensmaia. 2009. Temporal frequency channels are linked across audition and touch. Current biology 19, 7 (2009), 561–566.
[52]
Masahide Yoneyama, Jun‐ichiroh Fujimoto, Yu Kawamo, and Shoichi Sasabe. 1983. The audio spotlight: An application of nonlinear interaction of sound waves to a new type of loudspeaker design. The Journal of the Acoustical Society of America 73, 5 (1983), 1532–1536. https://doi.org/10.1121/1.389414

Cited By

View all
  • (2024)ScentHaptics: Augmenting the Haptic Experiences of Digital Mid-Air Textiles with ScentProceedings of the 26th International Conference on Multimodal Interaction10.1145/3678957.3685715(47-56)Online publication date: 4-Nov-2024
  • (2024)An Examination of Ultrasound Mid-air Haptics for Enhanced Material and Temperature Perception in Virtual EnvironmentsProceedings of the ACM on Human-Computer Interaction10.1145/36764888:MHCI(1-21)Online publication date: 24-Sep-2024
  • (2024)Survey on Haptic Feedback through Sensory Illusions in Interactive SystemsACM Computing Surveys10.1145/364835356:8(1-39)Online publication date: 10-Apr-2024
  • Show More Cited By

Recommendations

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences
ICMI '21: Proceedings of the 2021 International Conference on Multimodal Interaction
October 2021
876 pages
ISBN:9781450384810
DOI:10.1145/3462244
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 18 October 2021

Permissions

Request permissions for this article.

Check for updates

Author Tags

  1. audio-haptic illusion
  2. multisensory perception
  3. ultrasound haptics;

Qualifiers

  • Short-paper
  • Research
  • Refereed limited

Funding Sources

Conference

ICMI '21
Sponsor:
ICMI '21: INTERNATIONAL CONFERENCE ON MULTIMODAL INTERACTION
October 18 - 22, 2021
QC, Montréal, Canada

Acceptance Rates

Overall Acceptance Rate 453 of 1,080 submissions, 42%

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • Downloads (Last 12 months)65
  • Downloads (Last 6 weeks)9
Reflects downloads up to 02 Mar 2025

Other Metrics

Citations

Cited By

View all
  • (2024)ScentHaptics: Augmenting the Haptic Experiences of Digital Mid-Air Textiles with ScentProceedings of the 26th International Conference on Multimodal Interaction10.1145/3678957.3685715(47-56)Online publication date: 4-Nov-2024
  • (2024)An Examination of Ultrasound Mid-air Haptics for Enhanced Material and Temperature Perception in Virtual EnvironmentsProceedings of the ACM on Human-Computer Interaction10.1145/36764888:MHCI(1-21)Online publication date: 24-Sep-2024
  • (2024)Survey on Haptic Feedback through Sensory Illusions in Interactive SystemsACM Computing Surveys10.1145/364835356:8(1-39)Online publication date: 10-Apr-2024
  • (2024)FabSound: Audio-Tactile and Affective Fabric Experiences Through Mid-air HapticsProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642533(1-17)Online publication date: 11-May-2024
  • (2024)It Sounds Cool: Exploring Sonification of Mid-Air Haptic Textures Exploration on Texture Judgments, Body Perception, and Motor BehaviourIEEE Transactions on Haptics10.1109/TOH.2023.332049217:2(237-248)Online publication date: Apr-2024
  • (2024)Exploring the Range of Softness Perception Presented by Spatiotemporal Modulation in Mid-Air Ultrasound Haptic DisplaysHaptics: Understanding Touch; Technology and Systems; Applications and Interaction10.1007/978-3-031-70061-3_8(90-102)Online publication date: 3-Nov-2024
  • (2024)Evaluating Touchless Haptics for Interaction with Virtual ObjectsIntelligent Technologies for Interactive Entertainment10.1007/978-3-031-55722-4_15(210-221)Online publication date: 23-Mar-2024
  • (2023)Does It Par-?: Investigating the Relationship Between Mid-Air Haptics and Visual Representations of Surface TexturesIEEE Transactions on Haptics10.1109/TOH.2023.327295116:4(561-566)Online publication date: 4-May-2023
  • (2023)Development of Compact Actuator for Direct Measurement of Skin Displacement in Ultrasonic HapticsSensing Technology10.1007/978-3-031-29871-4_27(267-273)Online publication date: 9-Apr-2023
  • (2022)Multimodal Interaction with Mid-Air HapticsUltrasound Mid-Air Haptics for Touchless Interfaces10.1007/978-3-031-04043-6_7(185-205)Online publication date: 17-Sep-2022
  • Show More Cited By

View Options

Login options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

HTML Format

View this article in HTML Format.

HTML Format

Figures

Tables

Media

Share

Share

Share this Publication link

Share on social media