demonstration

Facial Expression Mapping inside Head Mounted Display by Embedded Optical Sensors

Authors:

Maki SugimotoAuthors Info & Claims

UIST '16 Adjunct: Adjunct Proceedings of the 29th Annual ACM Symposium on User Interface Software and Technology

Pages 91 - 92

https://doi.org/10.1145/2984751.2985714

Published: 16 October 2016 Publication History

Get Access

Abstract

Head Mounted Display (HMD) provides an immersive ex-perience in virtual environments for various purposes such as for games and communication. However, it is difficult to capture facial expression in a HMD-based virtual environ-ment because the upper half of user's face is covered up by the HMD. In this paper, we propose a facial expression mapping technology between user and a virtual avatar using embedded optical sensors and machine learning. The distance between each sensor and surface of the face is measured by the optical sensors that are attached inside the HMD. Our system learns the sensor values of each facial expression by neural network and creates a classifier to estimate the current facial expression.

Supplementary Material

suppl.mov (uistde0137-file3.mp4)

Supplemental video

Download
109.51 MB

References

[1]

Li, H., Trutoiu, L., Olszewski, K., Wei, L., Trutna, T., Hsieh, P.-L., Nicholls, A., and Ma, C. Facial performance sensing head-mounted display. ACM Transactions on Graphics (Proceedings SIGGRAPH 2015) 34, 4 (July 2015).

Digital Library

Google Scholar

[2]

Masai, K., Sugiura, Y., Ogata, M., Kunze, K., Inami, M., and Sugimoto, M. Facial expression recognition in daily life by embedded photo reflective sensors on smart eyewear. In Proceedings of the 21st International Conference on Intelligent User Interfaces, IUI'16, ACM (New York, NY, USA, 2016), 317--326.

Digital Library

Google Scholar

[3]

Scheirer, J., Fernandez, R., and Picard, R. W. Expression glasses: A wearable device for facial expression recognition, 1999.

Digital Library

Google Scholar

Cited By

View all

Kim DHarrison C(2023)Pantœnna: Mouth pose estimation for ar/vr headsets using low-profile antenna and impedance characteristic sensingProceedings of the 36th Annual ACM Symposium on User Interface Software and Technology10.1145/3586183.3606805(1-12)Online publication date: 29-Oct-2023
https://dl.acm.org/doi/10.1145/3586183.3606805
Wen LZhou JHuang WChen F(2022)A Survey of Facial Capture for Virtual RealityIEEE Access10.1109/ACCESS.2021.313820010(6042-6052)Online publication date: 2022
https://doi.org/10.1109/ACCESS.2021.3138200
Baloup MPietrzak THachet MCasiez G(2021)Non-isomorphic Interaction Techniques for Controlling Avatar Facial Expressions in VRProceedings of the 27th ACM Symposium on Virtual Reality Software and Technology10.1145/3489849.3489867(1-10)Online publication date: 8-Dec-2021
https://dl.acm.org/doi/10.1145/3489849.3489867
Show More Cited By

Index Terms

Facial Expression Mapping inside Head Mounted Display by Embedded Optical Sensors
1. Human-centered computing

Recommendations

Facial performance sensing head-mounted display

There are currently no solutions for enabling direct face-to-face interaction between virtual reality (VR) users wearing head-mounted displays (HMDs). The main challenge is that the headset obstructs a significant portion of a user's face, preventing ...
Expression-invariant face recognition by facial expression transformations

In this paper, we present a method of expression-invariant face recognition that transforms input face image with an arbitrary expression into its corresponding neutral facial expression image. When a new face image with an arbitrary expression is ...
Wearable head-mounted 3D tactile display application scenarios
MobileHCI '16: Proceedings of the 18th International Conference on Human-Computer Interaction with Mobile Devices and Services Adjunct

Current generation virtual reality (VR) and augmented reality (AR) head-mounted displays (HMDs) usually include no or only a single vibration motor for haptic feedback and do not use it for guidance. In a previous work, we presented HapticHead, a ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

UIST '16 Adjunct: Adjunct Proceedings of the 29th Annual ACM Symposium on User Interface Software and Technology

October 2016

244 pages

ISBN:9781450345316

DOI:10.1145/2984751

General Chairs:
Jun Rekimoto
The University of Tokyo
,
Takeo Igarashi
The University of Tokyo
,
Program Chairs:
Jacob O. Wobbrock
University of Washington
,
Daniel Avrahami
FXPAL

Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for third-party components of this work must be honored. For all other uses, contact the Owner/Author.

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 16 October 2016

Check for updates

Author Tags

Qualifiers

Demonstration

Funding Sources

Japan Society for the Promotion of Science

Conference

UIST '16

Sponsor:

UIST '16: The 29th Annual ACM Symposium on User Interface Software and Technology

October 16 - 19, 2016

Tokyo, Japan

Acceptance Rates

UIST '16 Adjunct Paper Acceptance Rate 79 of 384 submissions, 21%;

Overall Acceptance Rate 355 of 1,733 submissions, 20%

Upcoming Conference

UIST '25

Sponsor:
sigchi
sigchi

The 38th Annual ACM Symposium on User Interface Software and Technology

September 28 - October 1, 2025

Busan , Republic of Korea

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

6
Total Citations
View Citations
303
Total Downloads

Downloads (Last 12 months)12
Downloads (Last 6 weeks)2

Reflects downloads up to 11 Dec 2024

Other Metrics

View Author Metrics

Citations

Cited By

View all

Kim DHarrison C(2023)Pantœnna: Mouth pose estimation for ar/vr headsets using low-profile antenna and impedance characteristic sensingProceedings of the 36th Annual ACM Symposium on User Interface Software and Technology10.1145/3586183.3606805(1-12)Online publication date: 29-Oct-2023
https://dl.acm.org/doi/10.1145/3586183.3606805
Wen LZhou JHuang WChen F(2022)A Survey of Facial Capture for Virtual RealityIEEE Access10.1109/ACCESS.2021.313820010(6042-6052)Online publication date: 2022
https://doi.org/10.1109/ACCESS.2021.3138200
Baloup MPietrzak THachet MCasiez G(2021)Non-isomorphic Interaction Techniques for Controlling Avatar Facial Expressions in VRProceedings of the 27th ACM Symposium on Virtual Reality Software and Technology10.1145/3489849.3489867(1-10)Online publication date: 8-Dec-2021
https://dl.acm.org/doi/10.1145/3489849.3489867
Iravantchi YZhang YBernitsas EGoel MHarrison CBrewster SFitzpatrick GCox AKostakos V(2019)InterferiProceedings of the 2019 CHI Conference on Human Factors in Computing Systems10.1145/3290605.3300506(1-13)Online publication date: 2-May-2019
https://dl.acm.org/doi/10.1145/3290605.3300506
Hickson SDufour NSud AKwatra VEssa I(2019)Eyemotion: Classifying Facial Expressions in VR Using Eye-Tracking Cameras2019 IEEE Winter Conference on Applications of Computer Vision (WACV)10.1109/WACV.2019.00178(1626-1635)Online publication date: Jan-2019
https://doi.org/10.1109/WACV.2019.00178
Ando TKubo YShizuki BTakahashi SGajos KMankoff JHarrison C(2017)CanalSenseProceedings of the 30th Annual ACM Symposium on User Interface Software and Technology10.1145/3126594.3126649(679-689)Online publication date: 20-Oct-2017
https://dl.acm.org/doi/10.1145/3126594.3126649

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Cited By

Index Terms

Recommendations

Facial performance sensing head-mounted display

Expression-invariant face recognition by facial expression transformations

Wearable head-mounted 3D tactile display application scenarios