abstract

Interactive Visualization of Deep Learning Models in an Immersive Environment

Authors:

Hikaru Nagasaka,

Motoya IzuharaAuthors Info & Claims

VRST '21: Proceedings of the 27th ACM Symposium on Virtual Reality Software and Technology

Article No.: 82, Pages 1 - 3

https://doi.org/10.1145/3489849.3489956

Published: 08 December 2021 Publication History

Get Access

Abstract

The development of deep learning (DL) models has been prevalent among software engineers. However, it is difficult for non-experts to analyze and understand their behavior. Hence, we propose an interactive visualization system of DL models in an immersive environment. Because an immersive environment offers unlimited displays and visualization of high-dimensional data, it enables a comprehensive analysis on data propagations through the layers, and compares the multiple performance metrics. In this research, we implemented a prototype system, demonstrated it to machine learning engineers, and discussed the future benefits of visualizing DL models in an immersive environment. Accordingly, our concept received positive feedback; however, we inferred that most of the engineers consider the visualization technology as a unique introduction to the immersive environment.

References

[1]

Adam W Harley. 2015. An interactive node-link visualization of convolutional neural networks. In International Symposium on Visual Computing. Springer, Cham, Switzerland, AG, 867–877. https://doi.org/10.1007/978-3-319-27857-5_77

Crossref

Google Scholar

[2]

Andrew G Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetto, and Hartwig Adam. 2017. MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications. CoRR abs/1704.04861(2017). arxiv:1704.04861http://arxiv.org/abs/1704.04861

Google Scholar

[3]

Alex Krizhevsky 2009. Learning Multiple Layers of Features from Tiny Images. (2009). Retrieved October 8, 2021 from https://www.cs.toronto.edu/~kriz/cifar.html

Google Scholar

[4]

D Mané 2015. TensorBoard: TensorFlow’s visualization toolkit. Retrieved October 8, 2021 from https://tensorboard.dev/

Google Scholar

[5]

Nadine Meissler, Annika Wohlan, Nico Hochgeschwender, and Andreas Schreiber. 2019. Using Visualization of Convolutional Neural Networks in Virtual Reality for Machine Learning Newcomers. In 2019 IEEE International Conference on Artificial Intelligence and Virtual Reality (AIVR). IEEE, 152–1526. https://doi.org/10.1109/AIVR46125.2019.00031

Crossref

Google Scholar

[6]

Ramprasaath R. Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. 2017. Grad-CAM: Visual Explanations From Deep Networks via Gradient-Based Localization. In 2017 IEEE International Conference on Computer Vision (ICCV). IEEE. https://doi.org/10.1109/ICCV.2017.74

Crossref

Google Scholar

[7]

Ronell Sicat, Jiabao Li, JunYoung Choi, Maxime Cordeil, Won-Ki Jeong, Benjamin Bach, and Hanspeter Pfister. 2018. DXR: A Toolkit for Building Immersive Data Visualizations. IEEE transactions on visualization and computer graphics 25, 1(2018), 715–725. https://doi.org/10.1109/TVCG.2018.2865152

Digital Library

Google Scholar

[8]

Karen Simonyan and Andrew Zisserman. 2015. Very Deep Convolutional Networks for Large-Scale Image Recognition. In 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings, Yoshua Bengio and Yann LeCun (Eds.). http://arxiv.org/abs/1409.1556

Google Scholar

[9]

Laurens van der Maaten and Geoffrey Hinton. 2008. Visualizing Data using t-SNE. Journal of Machine Learning Research 9, 86 (2008), 2579–2605. http://jmlr.org/papers/v9/vandermaaten08a.html

Google Scholar

[10]

Kevin C. VanHorn, Meyer Zinn, and Murat Can Cobanoglu. 2019. Deep Learning Development Environment in Virtual Reality. CoRR abs/1906.05925(2019). arxiv:1906.05925http://arxiv.org/abs/1906.05925

Google Scholar

[11]

Xufan Zhang, Ziyue Yin, Yang Feng, Qingkai Shi, Jia Liu, and Zhenyu Chen. 2019. NeuralVis: Visualizing and Interpreting Deep Learning Models. In 2019 34th IEEE/ACM International Conference on Automated Software Engineering (ASE). IEEE, 1106–1109. https://doi.org/10.1109/ASE.2019.00113

Digital Library

Google Scholar

Cited By

View all

Friedl‐Knirsch JPointecker FPfistermüller SStach CAnthes CRoth D(2024)A Systematic Literature Review of User Evaluation in Immersive AnalyticsComputer Graphics Forum10.1111/cgf.1511143:3Online publication date: 10-Jun-2024
https://doi.org/10.1111/cgf.15111
Kashiyama MHirokawa MMatsuno RSakuma KItoh T(2024)Interactive Visualization of Ensemble Decision Trees Based on the Relations Among Weak Learners2024 28th International Conference Information Visualisation (IV)10.1109/IV64223.2024.00028(1-6)Online publication date: 22-Jul-2024
https://doi.org/10.1109/IV64223.2024.00028
Bhan SVarma LManiar KLobo RShah SPatil N(2023)Interactive Gaming Experience in VR Integrated with Machine LearningAdvances and Applications of Artificial Intelligence & Machine Learning10.1007/978-981-99-5974-7_56(703-714)Online publication date: 15-Nov-2023
https://doi.org/10.1007/978-981-99-5974-7_56

Recommendations

TangibleData: Interactive Data Visualization with Mid-Air Haptics
VRST '21: Proceedings of the 27th ACM Symposium on Virtual Reality Software and Technology

In this paper, we investigate the effects of mid-air haptics in interactive 3D data visualization. We build an interactive 3D data visualization tool that adapts hand gestures and mid-air haptics to provide tangible interaction in VR using ultrasound ...
Designing Spatial Visualization and Interactions of Immersive Sankey Diagram in Virtual Reality
MM '24: Proceedings of the 32nd ACM International Conference on Multimedia

Virtual reality (VR) is a revolutionary method of presenting data visualizations, which brings potential possibilities for enhancing analytical activities. However, applying this method to visualize complex data flows remains largely underexplored, ...
Building a Prototype for Easy to Use Collaborative Immersive Analytics
Computational Science – ICCS 2021
Abstract
The increase in the size and complexity of today’s datasets creates a need to develop and experiment with novel data visualization methods. One of these innovations is immersive analytics, in which extended reality technologies such as virtual ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

VRST '21: Proceedings of the 27th ACM Symposium on Virtual Reality Software and Technology

December 2021

563 pages

ISBN:9781450390927

DOI:10.1145/3489849

Editors:
Yuichi Itoh
Aoyama Gakuin University, Japan
,
Kazuki Takashima
Tohoku University, Japan
,
Parinya Punpongsanon
Osaka University, Japan
,
Misha Sra
University of California, Santa Barbara, USA
,
Kazuyuki Fujita
Tohoku University, Japan
,
Shigeo Yoshida
The University of Tokyo, Japan
,
Tham Piumsomboon
University of Canterbury, New Zealand

Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for third-party components of this work must be honored. For all other uses, contact the Owner/Author.

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 08 December 2021

Check for updates

Author Tags

Qualifiers

Abstract
Research
Refereed limited

Conference

VRST '21

Sponsor:

VRST '21: 27th ACM Symposium on Virtual Reality Software and Technology

December 8 - 10, 2021

Osaka, Japan

Acceptance Rates

Overall Acceptance Rate 66 of 254 submissions, 26%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

2
Total Citations
View Citations
260
Total Downloads

Downloads (Last 12 months)66
Downloads (Last 6 weeks)1

Reflects downloads up to 17 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

View all

Friedl‐Knirsch JPointecker FPfistermüller SStach CAnthes CRoth D(2024)A Systematic Literature Review of User Evaluation in Immersive AnalyticsComputer Graphics Forum10.1111/cgf.1511143:3Online publication date: 10-Jun-2024
https://doi.org/10.1111/cgf.15111
Kashiyama MHirokawa MMatsuno RSakuma KItoh T(2024)Interactive Visualization of Ensemble Decision Trees Based on the Relations Among Weak Learners2024 28th International Conference Information Visualisation (IV)10.1109/IV64223.2024.00028(1-6)Online publication date: 22-Jul-2024
https://doi.org/10.1109/IV64223.2024.00028
Bhan SVarma LManiar KLobo RShah SPatil N(2023)Interactive Gaming Experience in VR Integrated with Machine LearningAdvances and Applications of Artificial Intelligence & Machine Learning10.1007/978-981-99-5974-7_56(703-714)Online publication date: 15-Nov-2023
https://doi.org/10.1007/978-981-99-5974-7_56

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

HTML Format

View this article in HTML Format.

HTML Format

Abstract

References

Cited By

Recommendations

TangibleData: Interactive Data Visualization with Mid-Air Haptics

Designing Spatial Visualization and Interactions of Immersive Sankey Diagram in Virtual Reality

Building a Prototype for Easy to Use Collaborative Immersive Analytics

Comments

Information

Published In

Sponsors

Publisher

Publication History

Check for updates

Author Tags

Qualifiers

Conference

Acceptance Rates

Contributors

Other Metrics

Bibliometrics

Article Metrics

Other Metrics

Citations

Cited By

Login options

Full Access

View options

PDF

eReader

HTML Format

Figures

Other

Share

Share this Publication link

Share on social media

Affiliations