Siyou Pei, Alexander Chen, Jaewook Lee, Yang Zhang 🏆 CHI '22 Honorable Mention
📘 paper 🎬 preview 🎙️ full presentation :desktop_computer: lab website
In the digital reality, there are many objects to retrieve and interact with. This rich set of objects means a lot to the user experience. To play with so many objects, the prevailing method is users hold hand controllers. But holding something all the time can be cumbersome. We want to bring up a new interaction technique that is more readily available.
In the rock-paper-scissors game, people use hands in different shapes to imitate rock, paper and scissors. It is intuitive and self-revealing.
So we wonder, can we generalize this idea to many other objects and interactions in AR and VR? Then, we propose Hand Interfaces!
We propose Hand Interfaces - a new free-hand interaction technique that allows users to embody objects through imitating them with hands. In short, hands now BECOME those virtual objects.
Further, Hand Interfaces not only supports object retrieval but also interactive control.
If you have a Meta Oculus Quest headset, we have a minimal app for you to install easily. You only need the Scissors_Demo.apk in this repo, nothing else. The app contains a pair of scissors. You can perform a peace gesture to retrieve the scissors in the air, and close your index and middle fingers to snip virtual objects in front of you (e.g. a cake, a bread, or even a phone!).
To install it, please
- Download the Scissors_Demo.apk in this repo.
- Make sure your Quest account has enabled developer mode. Official tutorial
- Sideload the apk onto your Quest headset through Oculus Developer Hub, or SideQuest or ADB commands.
- Find the apk file in your Oculus library - unknown sources. Have fun!
The 11 objects used in user studies are included in this github project. The two tasks - object retrieval and interactive control - are split into separate scenes in user studies.
To build and run the full project, we need to set up our Quest device for Unity development. This is a tutorial I made when I was a TA of a VR course.
handinterfaces/Assets/Scenes/prototypes stores early versions of idea prototypes.
handinterfaces/Assets/Scenes/user study contains what we showed to participants during user studies. The unity scenes starting with "Retr" are for "object retrieval", while those starting with "Inter" refer to "interactive control".
As indicated in the paper, DM, VG, HI are abbreviations for "Drop-down Menu", "VirtualGrasp" and "Hand Interfaces", FG, VM, HI are abbreviations for "Fist Gesture", "Virtual Manipulation" and "Hand Interfaces".
handinterfaces/Assets/Example Applications/ includes three applications mentioned in the paper, they are "education", "entertainment", and "IoT". Demo videos can be found here.
Feel free to create a new issue in the 'Issues' tab!
https://github.com/DavidArayan/ezy-slice
https://github.com/dilmerv/VRDraw
https://github.com/pharan/Unity-MeshSaver
In addition, many thanks to Quentin Valembois, Dilmer Valecillos and Antony Vitillo for their contribution to the AR/VR content creation community.
Special thanks to the authors of VirtualGrasp, Yukang Yan, et al. for inspiring this thread of work in designing AR/VR interfaces around hands’ expressivity.
@inproceedings{pei2022hand,
author = {Pei, Siyou and Chen, Alexander and Lee, Jaewook and Zhang, Yang},
title = {Hand Interfaces: Using Hands to Imitate Objects in AR/VR for Expressive Interactions},
year = {2022},
isbn = {9781450391573},
publisher = {Association for Computing Machinery},
address = {New York, NY, USA},
url = {https://doi.org/10.1145/3491102.3501898},
doi = {10.1145/3491102.3501898},
booktitle = {Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems},
articleno = {429},
numpages = {16},
keywords = {Interaction design, AR/VR, Embodiment, Free-hand interactions, Imitation, On-body interactions},
location = {New Orleans, LA, USA},
series = {CHI '22}
}
Thank you for reading! Isn't it exciting to create the next-gen user experience for XR :D
You may also find me @SiyouPei