Predicting Perceptual Haptic Attributes of Textured Surface from Tactile Data Based on Deep CNN-LSTM Network
Article No.: 33, Pages 1 - 9
Abstract
This paper introduces a framework to predict multi-dimensional haptic attribute values that humans use to recognize the material by using the physical tactile signals (acceleration) generated when a textured surface is stroked. To this end, two spaces are established: a haptic attribute space and a physical signal space. A five-dimensional haptic attribute space is established through human adjective rating experiments with the 25 real texture samples. The physical space is constructed using tool-based interaction data from the same 25 samples. A mapping is modeled between the aforementioned spaces using a newly designed CNN-LSTM deep learning network. Finally, a prediction algorithm is implemented that takes acceleration data and returns coordinates in the haptic attribute space. A quantitative evaluation was conducted to inspect the reliability of the algorithm on unseen textures, showing that the model outperformed other similar models.
References
[1]
Arsen Abdulali and Seokhee Jeon. 2016. Data-driven modeling of anisotropic haptic textures: Data segmentation and interpolation. In International Conference on Human Haptic Sensing and Touch Enabled Computer Applications. Springer, Cham, 228–239.
[2]
Mudassir Ibrahim Awan, Tatyana Ogay, Waseem Hassan, Dongbeom Ko, Sungjoo Kang, and Seokhee Jeon. 2023. Model-Mediated Teleoperation for Remote Haptic Texture Sharing: Initial Study of Online Texture Modeling and Rendering. In 2023 IEEE International Conference on Robotics and Automation (ICRA). IEEE, New York City, United States, 12457–12463.
[3]
Heather Culbertson, Joseph M Romano, Pablo Castillo, Max Mintz, and Katherine J Kuchenbecker. 2012. Refined methods for creating realistic haptic virtual textures from tool-mediated contact acceleration data. In 2012 IEEE Haptics Symposium (HAPTICS). IEEE, New York City, United States, 385–391.
[4]
Heather Culbertson, Juliette Unwin, and Katherine J Kuchenbecker. 2014. Modeling and rendering realistic textures from unconstrained tool-surface interactions. IEEE transactions on haptics 7, 3 (2014), 381–393.
[5]
Siyuan Dong, Wenzhen Yuan, and Edward H Adelson. 2017. Improved gelsight tactile sensor for measuring geometry and slip. In 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). IEEE, New York City, United States, 137–144.
[6]
Subhash Arun Dwivedi, Amit Attry, Darshan Parekh, and Kanika Singla. 2021. Analysis and forecasting of Time-Series data using S-ARIMA, CNN and LSTM. In 2021 International Conference on Computing, Communication, and Intelligent Systems (ICCCIS). IEEE, New York City, United States, 131–136.
[7]
Ruihan Gao, Tasbolat Taunyazov, Zhiping Lin, and Yan Wu. 2020. Supervised autoencoder joint learning on heterogeneous tactile sensory data: Improving material classification performance. In 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). IEEE, New York City, United States, 10907–10913.
[8]
Yang Gao, Lisa Anne Hendricks, Katherine J Kuchenbecker, and Trevor Darrell. 2016. Deep learning for tactile understanding from visual and haptic data. In 2016 IEEE International Conference on Robotics and Automation (ICRA). IEEE, New York City, United States, 536–543.
[9]
Rebeen Ali Hamad, Longzhi Yang, Wai Lok Woo, and Bo Wei. 2020. Joint learning of temporal models to handle imbalanced data for human activity recognition. Applied Sciences 10, 15 (2020), 5293.
[10]
Waseem Hassan, Arsen Abdulali, and Seokhee Jeon. 2017. Perceptual thresholds for haptic texture discrimination. In 2017 14th International Conference on Ubiquitous Robots and Ambient Intelligence (URAI). IEEE, New York City, United States, 293–298.
[11]
Waseem Hassan, Arsen Abdulali, and Seokhee Jeon. 2019. Authoring new haptic textures based on interpolation of real textures in affective space. IEEE Transactions on Industrial Electronics 67, 1 (2019), 667–676.
[12]
Waseem Hassan and Seokhee Jeon. 2016. Evaluating differences between bare-handed and tool-based interaction in perceptual space. In 2016 IEEE Haptics Symposium (HAPTICS). IEEE, New York City, United States, 185–191.
[13]
Waseem Hassan, Joolekha Bibi Joolee, and Seokhee Jeon. 2023. Establishing haptic texture attribute space and predicting haptic attributes from image features using 1D-CNN. Scientific Reports 13, 1 (2023), 11684.
[14]
Negin Heravi, Wenzhen Yuan, Allison M Okamura, and Jeannette Bohg. 2020. Learning an action-conditional model for haptic texture generation. In 2020 IEEE International Conference on Robotics and Automation (ICRA). IEEE, New York City, United States, 11088–11095.
[15]
Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural computation 9, 8 (1997), 1735–1780.
[16]
Mark Hollins, Sliman Bensmaïa, Kristie Karlof, and Forrest Young. 2000. Individual differences in perceptual space for tactile textures: Evidence from multidimensional scaling. Perception & Psychophysics 62, 8 (2000), 1534–1544.
[17]
Inwook Hwang and Seungmoon Choi. 2010. Perceptual space and adjective rating of sinusoidal vibrations perceived via mobile device. In 2010 IEEE Haptics Symposium. IEEE, New York City, United States, 1–8.
[18]
Joolekha Bibi Joolee and Seokhee Jeon. 2021. Data-Driven Haptic Texture Modeling and Rendering Based on Deep Spatio-Temporal Networks. IEEE Transactions on Haptics 15, 1 (2021), 62–67.
[19]
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. 2017. Imagenet classification with deep convolutional neural networks. Commun. ACM 60, 6 (2017), 84–90.
[20]
Olcay Kursun and Ahmad Patooghy. 2020. An embedded system for collection and real-time classification of a tactile dataset. IEEE Access 8 (2020), 97462–97473.
[21]
Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. 1998. Gradient-based learning applied to document recognition. Proc. IEEE 86, 11 (1998), 2278–2324.
[22]
Bruno Monteiro Rocha Lima, Thiago Eustaquio Alves de Oliveira, and Vinicius Prado da Fonseca. 2021. Classification of Textures using a Tactile-Enabled Finger in Dynamic Exploration Tasks. In 2021 IEEE Sensors. IEEE, New York City, United States, 1–4.
[23]
Timo Markert, Sebastian Matich, Elias Hoerner, Andreas Theissler, and Martin Atzmueller. 2021. Fingertip 6-axis force/torque sensing for texture recognition in robotic manipulation. In 2021 26th IEEE International Conference on Emerging Technologies and Factory Automation (ETFA). IEEE, New York City, United States, 1–8.
[24]
Sunung Mun, Hojin Lee, and Seungmoon Choi. 2019. Perceptual space of regular homogeneous haptic textures rendered using electrovibration. In 2019 IEEE World Haptics Conference (WHC). IEEE, New York City, United States, 7–12.
[25]
Shogo Okamoto, Hikaru Nagano, and Yoji Yamada. 2012. Psychophysical dimensions of tactile perception of textures. IEEE Transactions on Haptics 6, 1 (2012), 81–93.
[26]
Benjamin A Richardson and Katherine J Kuchenbecker. 2020. Learning to predict perceptual distributions of haptic adjectives. Frontiers in Neurorobotics 13 (2020), 116.
[27]
Benjamin A Richardson, Yasemin Vardar, Christian Wallraven, and Katherine J Kuchenbecker. 2022. Learning to Feel Textures: Predicting Perceptual Similarities From Unconstrained Finger-Surface Interactions. IEEE Transactions on Haptics 15, 4 (2022), 705–717.
[28]
Karen Simonyan and Andrew Zisserman. 2015. Very Deep Convolutional Networks for Large-Scale Image Recognition. arxiv:1409.1556 [cs.CV]
[29]
Mervyn Stone. 1974. Cross-validatory choice and assessment of statistical predictions. Journal of the royal statistical society: Series B (Methodological) 36, 2 (1974), 111–133.
[30]
Matti Strese, Clemens Schuwerk, Albert Iepure, and Eckehard Steinbach. 2016. Multimodal feature-based surface material classification. IEEE transactions on haptics 10, 2 (2016), 226–239.
[31]
Getu Tadele Taye, Han-Jeong Hwang, and Ki Moo Lim. 2020. Application of a convolutional neural network for predicting the occurrence of ventricular tachyarrhythmia using heart rate variability features. Scientific reports 10, 1 (2020), 1–7.
[32]
Cuili Yang, Xinxin Zhu, Zohaib Ahmad, Lei Wang, and Junfei Qiao. 2018. Design of incremental echo state network using leave-one-out cross-validation. IEEE Access 6 (2018), 74874–74884.
[33]
Jun Yang, Jingbin Qu, Qiang Mi, and Qing Li. 2020. A CNN-LSTM model for tailings dam risk prediction. IEEE Access 8 (2020), 206491–206502.
[34]
Yongjae Yoo, Inwook Hwang, and Seungmoon Choi. 2013. Consonance of vibrotactile chords. IEEE transactions on haptics 7, 1 (2013), 3–13.
[35]
Yongjae Yoo, Jaebong Lee, Jongman Seo, Eunhwa Lee, Jeongseok Lee, Yudong Bae, Daekwang Jung, and Seungmoon Choi. 2016. Large-scale survey on adjectival representation of vibrotactile stimuli. In Proc. HAPTICS. IEEE, New York City, United States, 393–395.
[36]
Masaaki Yoshida. 1968. Dimensions of tactual impressions (1). Japanese Psychological Research 10, 3 (1968), 123–137.
[37]
Haitian Zheng, Lu Fang, Mengqi Ji, Matti Strese, Yigitcan Özer, and Eckehard Steinbach. 2016. Deep learning for surface material classification using haptic and visual information. IEEE Transactions on Multimedia 18, 12 (2016), 2407–2416.
Index Terms
- Predicting Perceptual Haptic Attributes of Textured Surface from Tactile Data Based on Deep CNN-LSTM Network
Recommendations
Adaptation to Force in the Haptic Rendering of Virtual Environments
EuroHaptics '08: Proceedings of the 6th international conference on Haptics: Perception, Devices and ScenariosRealistic haptic rendering is one of the most challenging issues in the field of virtual reality. The intent is for the user to experience the same kinesthetic sensations in the virtual realm as they would in the real world. Therefore, we need to know ...
Tactile brush: drawing on skin with a tactile grid display
CHI '11: Proceedings of the SIGCHI Conference on Human Factors in Computing SystemsTactile Brush is an algorithm that produces smooth, two-dimensional tactile moving strokes with varying frequency, intensity, velocity and direction of motion. The design of the algorithm is derived from the results of psychophysical investigations of ...
The Effect of Texture on Perceiving Haptic Stiffness
ICBAKE '11: Proceedings of the 2011 International Conference on Biometrics and Kansei EngineeringThe sense of touch is essential to perform discriminating tasks in virtual reality applications or virtual training systems. To discriminate between different virtual objects, a human operator needs to clearly perceive the stiffness of each object. ...
Comments
Please enable JavaScript to view thecomments powered by Disqus.Information & Contributors
Information
Published In
October 2023
542 pages
ISBN:9798400703287
DOI:10.1145/3611659
Copyright © 2023 ACM.
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].
Sponsors
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 09 October 2023
Check for updates
Author Tags
Qualifiers
- Research-article
- Research
- Refereed limited
Funding Sources
- Ministry of Science and ICT Korea under the ITRC support program
Conference
VRST 2023: 29th ACM Symposium on Virtual Reality Software and Technology
October 9 - 11, 2023
Christchurch, New Zealand
Acceptance Rates
Overall Acceptance Rate 66 of 254 submissions, 26%
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 322Total Downloads
- Downloads (Last 12 months)207
- Downloads (Last 6 weeks)14
Reflects downloads up to 13 Dec 2024
Other Metrics
Citations
View Options
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign inFull Access
View options
View or Download as a PDF file.
PDFeReader
View online with eReader.
eReaderHTML Format
View this article in HTML Format.
HTML Format