More Web Proxy on the site http://driver.im/

research-article

Predicting Perceptual Haptic Attributes of Textured Surface from Tactile Data Based on Deep CNN-LSTM Network

Authors:

Mudassir Ibrahim Awan,

Seokhee JeonAuthors Info & Claims

VRST '23: Proceedings of the 29th ACM Symposium on Virtual Reality Software and Technology

Article No.: 33, Pages 1 - 9

https://doi.org/10.1145/3611659.3615714

Published: 09 October 2023 Publication History

Abstract

This paper introduces a framework to predict multi-dimensional haptic attribute values that humans use to recognize the material by using the physical tactile signals (acceleration) generated when a textured surface is stroked. To this end, two spaces are established: a haptic attribute space and a physical signal space. A five-dimensional haptic attribute space is established through human adjective rating experiments with the 25 real texture samples. The physical space is constructed using tool-based interaction data from the same 25 samples. A mapping is modeled between the aforementioned spaces using a newly designed CNN-LSTM deep learning network. Finally, a prediction algorithm is implemented that takes acceleration data and returns coordinates in the haptic attribute space. A quantitative evaluation was conducted to inspect the reliability of the algorithm on unseen textures, showing that the model outperformed other similar models.

References

[1]

Arsen Abdulali and Seokhee Jeon. 2016. Data-driven modeling of anisotropic haptic textures: Data segmentation and interpolation. In International Conference on Human Haptic Sensing and Touch Enabled Computer Applications. Springer, Cham, 228–239.

[2]

Mudassir Ibrahim Awan, Tatyana Ogay, Waseem Hassan, Dongbeom Ko, Sungjoo Kang, and Seokhee Jeon. 2023. Model-Mediated Teleoperation for Remote Haptic Texture Sharing: Initial Study of Online Texture Modeling and Rendering. In 2023 IEEE International Conference on Robotics and Automation (ICRA). IEEE, New York City, United States, 12457–12463.

[3]

Heather Culbertson, Joseph M Romano, Pablo Castillo, Max Mintz, and Katherine J Kuchenbecker. 2012. Refined methods for creating realistic haptic virtual textures from tool-mediated contact acceleration data. In 2012 IEEE Haptics Symposium (HAPTICS). IEEE, New York City, United States, 385–391.

[4]

Heather Culbertson, Juliette Unwin, and Katherine J Kuchenbecker. 2014. Modeling and rendering realistic textures from unconstrained tool-surface interactions. IEEE transactions on haptics 7, 3 (2014), 381–393.

[5]

Siyuan Dong, Wenzhen Yuan, and Edward H Adelson. 2017. Improved gelsight tactile sensor for measuring geometry and slip. In 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). IEEE, New York City, United States, 137–144.

Digital Library

[6]

Subhash Arun Dwivedi, Amit Attry, Darshan Parekh, and Kanika Singla. 2021. Analysis and forecasting of Time-Series data using S-ARIMA, CNN and LSTM. In 2021 International Conference on Computing, Communication, and Intelligent Systems (ICCCIS). IEEE, New York City, United States, 131–136.

[7]

Ruihan Gao, Tasbolat Taunyazov, Zhiping Lin, and Yan Wu. 2020. Supervised autoencoder joint learning on heterogeneous tactile sensory data: Improving material classification performance. In 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). IEEE, New York City, United States, 10907–10913.

Digital Library

[8]

Yang Gao, Lisa Anne Hendricks, Katherine J Kuchenbecker, and Trevor Darrell. 2016. Deep learning for tactile understanding from visual and haptic data. In 2016 IEEE International Conference on Robotics and Automation (ICRA). IEEE, New York City, United States, 536–543.

Digital Library

[9]

Rebeen Ali Hamad, Longzhi Yang, Wai Lok Woo, and Bo Wei. 2020. Joint learning of temporal models to handle imbalanced data for human activity recognition. Applied Sciences 10, 15 (2020), 5293.

[10]

Waseem Hassan, Arsen Abdulali, and Seokhee Jeon. 2017. Perceptual thresholds for haptic texture discrimination. In 2017 14th International Conference on Ubiquitous Robots and Ambient Intelligence (URAI). IEEE, New York City, United States, 293–298.

[11]

Waseem Hassan, Arsen Abdulali, and Seokhee Jeon. 2019. Authoring new haptic textures based on interpolation of real textures in affective space. IEEE Transactions on Industrial Electronics 67, 1 (2019), 667–676.

[12]

Waseem Hassan and Seokhee Jeon. 2016. Evaluating differences between bare-handed and tool-based interaction in perceptual space. In 2016 IEEE Haptics Symposium (HAPTICS). IEEE, New York City, United States, 185–191.

[13]

Waseem Hassan, Joolekha Bibi Joolee, and Seokhee Jeon. 2023. Establishing haptic texture attribute space and predicting haptic attributes from image features using 1D-CNN. Scientific Reports 13, 1 (2023), 11684.

[14]

Negin Heravi, Wenzhen Yuan, Allison M Okamura, and Jeannette Bohg. 2020. Learning an action-conditional model for haptic texture generation. In 2020 IEEE International Conference on Robotics and Automation (ICRA). IEEE, New York City, United States, 11088–11095.

[15]

Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural computation 9, 8 (1997), 1735–1780.

Digital Library

[16]

Mark Hollins, Sliman Bensmaïa, Kristie Karlof, and Forrest Young. 2000. Individual differences in perceptual space for tactile textures: Evidence from multidimensional scaling. Perception & Psychophysics 62, 8 (2000), 1534–1544.

[17]

Inwook Hwang and Seungmoon Choi. 2010. Perceptual space and adjective rating of sinusoidal vibrations perceived via mobile device. In 2010 IEEE Haptics Symposium. IEEE, New York City, United States, 1–8.

Digital Library

[18]

Joolekha Bibi Joolee and Seokhee Jeon. 2021. Data-Driven Haptic Texture Modeling and Rendering Based on Deep Spatio-Temporal Networks. IEEE Transactions on Haptics 15, 1 (2021), 62–67.

[19]

Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. 2017. Imagenet classification with deep convolutional neural networks. Commun. ACM 60, 6 (2017), 84–90.

Digital Library

[20]

Olcay Kursun and Ahmad Patooghy. 2020. An embedded system for collection and real-time classification of a tactile dataset. IEEE Access 8 (2020), 97462–97473.

[21]

Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. 1998. Gradient-based learning applied to document recognition. Proc. IEEE 86, 11 (1998), 2278–2324.

[22]

Bruno Monteiro Rocha Lima, Thiago Eustaquio Alves de Oliveira, and Vinicius Prado da Fonseca. 2021. Classification of Textures using a Tactile-Enabled Finger in Dynamic Exploration Tasks. In 2021 IEEE Sensors. IEEE, New York City, United States, 1–4.

[23]

Timo Markert, Sebastian Matich, Elias Hoerner, Andreas Theissler, and Martin Atzmueller. 2021. Fingertip 6-axis force/torque sensing for texture recognition in robotic manipulation. In 2021 26th IEEE International Conference on Emerging Technologies and Factory Automation (ETFA). IEEE, New York City, United States, 1–8.

Digital Library

[24]

Sunung Mun, Hojin Lee, and Seungmoon Choi. 2019. Perceptual space of regular homogeneous haptic textures rendered using electrovibration. In 2019 IEEE World Haptics Conference (WHC). IEEE, New York City, United States, 7–12.

[25]

Shogo Okamoto, Hikaru Nagano, and Yoji Yamada. 2012. Psychophysical dimensions of tactile perception of textures. IEEE Transactions on Haptics 6, 1 (2012), 81–93.

Digital Library

[26]

Benjamin A Richardson and Katherine J Kuchenbecker. 2020. Learning to predict perceptual distributions of haptic adjectives. Frontiers in Neurorobotics 13 (2020), 116.

[27]

Benjamin A Richardson, Yasemin Vardar, Christian Wallraven, and Katherine J Kuchenbecker. 2022. Learning to Feel Textures: Predicting Perceptual Similarities From Unconstrained Finger-Surface Interactions. IEEE Transactions on Haptics 15, 4 (2022), 705–717.

[28]

Karen Simonyan and Andrew Zisserman. 2015. Very Deep Convolutional Networks for Large-Scale Image Recognition. arxiv:1409.1556 [cs.CV]

[29]

Mervyn Stone. 1974. Cross-validatory choice and assessment of statistical predictions. Journal of the royal statistical society: Series B (Methodological) 36, 2 (1974), 111–133.

[30]

Matti Strese, Clemens Schuwerk, Albert Iepure, and Eckehard Steinbach. 2016. Multimodal feature-based surface material classification. IEEE transactions on haptics 10, 2 (2016), 226–239.

[31]

Getu Tadele Taye, Han-Jeong Hwang, and Ki Moo Lim. 2020. Application of a convolutional neural network for predicting the occurrence of ventricular tachyarrhythmia using heart rate variability features. Scientific reports 10, 1 (2020), 1–7.

[32]

Cuili Yang, Xinxin Zhu, Zohaib Ahmad, Lei Wang, and Junfei Qiao. 2018. Design of incremental echo state network using leave-one-out cross-validation. IEEE Access 6 (2018), 74874–74884.

[33]

Jun Yang, Jingbin Qu, Qiang Mi, and Qing Li. 2020. A CNN-LSTM model for tailings dam risk prediction. IEEE Access 8 (2020), 206491–206502.

[34]

Yongjae Yoo, Inwook Hwang, and Seungmoon Choi. 2013. Consonance of vibrotactile chords. IEEE transactions on haptics 7, 1 (2013), 3–13.

[35]

Yongjae Yoo, Jaebong Lee, Jongman Seo, Eunhwa Lee, Jeongseok Lee, Yudong Bae, Daekwang Jung, and Seungmoon Choi. 2016. Large-scale survey on adjectival representation of vibrotactile stimuli. In Proc. HAPTICS. IEEE, New York City, United States, 393–395.

[36]

Masaaki Yoshida. 1968. Dimensions of tactual impressions (1). Japanese Psychological Research 10, 3 (1968), 123–137.

[37]

Haitian Zheng, Lu Fang, Mengqi Ji, Matti Strese, Yigitcan Özer, and Eckehard Steinbach. 2016. Deep learning for surface material classification using haptic and visual information. IEEE Transactions on Multimedia 18, 12 (2016), 2407–2416.

Digital Library

Cited By

Chen DDing YWu HJia QZeng HWei LHua CLiu JSong A(2025)Visually Impaired People Learning Virtual Textures Through Multimodal Feedback Combining Vibrotactile and VoiceIEEE Transactions on Neural Systems and Rehabilitation Engineering10.1109/TNSRE.2025.352804833(453-465)Online publication date: 2025
https://doi.org/10.1109/TNSRE.2025.3528048

Index Terms

Predicting Perceptual Haptic Attributes of Textured Surface from Tactile Data Based on Deep CNN-LSTM Network

Recommendations

Adaptation to Force in the Haptic Rendering of Virtual Environments
EuroHaptics '08: Proceedings of the 6th international conference on Haptics: Perception, Devices and Scenarios

Realistic haptic rendering is one of the most challenging issues in the field of virtual reality. The intent is for the user to experience the same kinesthetic sensations in the virtual realm as they would in the real world. Therefore, we need to know ...
Tactile brush: drawing on skin with a tactile grid display
CHI '11: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems

Tactile Brush is an algorithm that produces smooth, two-dimensional tactile moving strokes with varying frequency, intensity, velocity and direction of motion. The design of the algorithm is derived from the results of psychophysical investigations of ...
The Effect of Texture on Perceiving Haptic Stiffness
ICBAKE '11: Proceedings of the 2011 International Conference on Biometrics and Kansei Engineering

The sense of touch is essential to perform discriminating tasks in virtual reality applications or virtual training systems. To discriminate between different virtual objects, a human operator needs to clearly perceive the stiffness of each object. ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences

VRST '23: Proceedings of the 29th ACM Symposium on Virtual Reality Software and Technology

October 2023

542 pages

ISBN:9798400703287

DOI:10.1145/3611659

Copyright © 2023 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 09 October 2023

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed limited

Funding Sources

Ministry of Science and ICT Korea under the ITRC support program

Conference

VRST 2023

Sponsor:

VRST 2023: 29th ACM Symposium on Virtual Reality Software and Technology

October 9 - 11, 2023

Christchurch, New Zealand

Acceptance Rates

Overall Acceptance Rate 66 of 254 submissions, 26%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

1
Total Citations
View Citations
345
Total Downloads

Downloads (Last 12 months)181
Downloads (Last 6 weeks)9

Reflects downloads up to 03 Mar 2025

Other Metrics

View Author Metrics

Citations

Cited By

Chen DDing YWu HJia QZeng HWei LHua CLiu JSong A(2025)Visually Impaired People Learning Virtual Textures Through Multimodal Feedback Combining Vibrotactile and VoiceIEEE Transactions on Neural Systems and Rehabilitation Engineering10.1109/TNSRE.2025.352804833(453-465)Online publication date: 2025
https://doi.org/10.1109/TNSRE.2025.3528048

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

HTML Format

View this article in HTML Format.

Figures

Tables

Media

View Table of Conten