More Web Proxy on the site http://driver.im/

poster

Recognizing Gestures on Projected Button Widgets with an RGB-D Camera Using a CNN

Authors:

Hideto OdaAuthors Info & Claims

ISS '18: Proceedings of the 2018 ACM International Conference on Interactive Surfaces and Spaces

Pages 369 - 374

https://doi.org/10.1145/3279778.3279907

Published: 19 November 2018 Publication History

Abstract

Projector-camera systems can turn any surface such as tabletops and walls into an interactive display. A basic problem is to recognize the gesture actions on the projected UI widgets. Previous approaches using finger template matching or occlusion patterns have issues with environmental lighting conditions, artifacts and noise in the video images of a projection, and inaccuracies of depth cameras. In this work, we propose a new recognizer that employs a deep neural net with an RGB-D camera; specifically, we use a CNN (Convolutional Neural Network) with optical flow computed from the color and depth channels. We evaluated our method on a new dataset of RGB-D videos of 12 users interacting with buttons projected on a tabletop surface.

Supplementary Material

ZIP File (isspp13.zip)

PDF of posterboard.

Download
341.19 KB

References

[1]

Borkowski, S., Crowley, J.L., Letessier, J., Berard, F. User-centric design of a vision system for interactive applications. Proc. ICVS '06.

Digital Library

[2]

Farneback, G. Two-frame motion estimation based on polynomial expansion. Proc. SCIA '03, pp. 363--370.

Digital Library

[3]

Harrison, C., Benko, H., Wilson, A. OmniTouch: Wearable multitouch interaction everywhere. Proc. UIST '11, pp. 441--450.

Digital Library

[4]

Intel RealSense. https://software.intel.com/en-us/realsense/d400

[5]

Kinect. http://www.xbox.com/en-US/KINECT

[6]

Kjeldsen, R., C., Pingali, G., Hartman, J., Levas, T., Podlaseck, M. Interacting with steerable projected displays. Proc. FGR '02, pp. 402--407.

Digital Library

[7]

LeCun, Y., Bengio, Y., Hinton, G. Deep learning. Nature, vol. 521: 436--444 (2015).

[8]

Microsoft Cognitive Toolkit (CNTK). https://www.microsoft.com/en-us/cognitive-toolkit

[9]

OpenCV. http://opencv.willowgarage.com/wiki

[10]

Pinhanez, C., Kjeldsen, R., Tang, L., Levas, A., Podlaseck, M., Sukaviriya, N. and Pingali, G. Creating touch-screens anywhere with interactive projected displays. Proc. ACM Multimedia '03 (Demo), pp. 460--461.

Digital Library

[11]

Simonyan, K., Zisserman, A. Two-stream convolutional networks for action recognition in videos. Proc. NIPS '14.

Digital Library

[12]

Tang, H., Chiu, P., Liu, Q. Gesture Viewport: Interacting with media content using finger gestures on any surface. ICME '14 demo.

[13]

Wang, J., Liu, Z., Wu, Y., Yuan, J. Mining actionlet ensemble for action recognition with depth cameras. Proc. CVPR '12, pp. 1290--1297.

Digital Library

[14]

Wellner, P. The DigitalDesk calculator: tangible manipulation on a desk top display. Proc. UIST '91, pp. 27--33.

Digital Library

[15]

Xiao, R., Harrison, C., Hudson, S. WorldKit: Rapid and easy creation of ad-hoc interactive applications on everyday surfaces. Proc. CHI '13, pp. 879--888.

Digital Library

[16]

Yuan, S., Ye, Q., Stenger, B., Jain, S., Kim, T.-K. BigHand2.2M Benchmark: Hand pose dataset and state of the art analysis. Proc. CVPR '17, pp. 2605--2613.

Cited By

Cheng XHuang B(2022)CSI-Based Human Continuous Activity Recognition Using GMM–HMMIEEE Sensors Journal10.1109/JSEN.2022.319824822:19(18709-18717)Online publication date: 1-Oct-2022
https://doi.org/10.1109/JSEN.2022.3198248
Mthunzi EGetschmann CEchtler F(2021)Fast 3D point-cloud segmentation for interactive surfacesCompanion Proceedings of the 2021 Conference on Interactive Surfaces and Spaces10.1145/3447932.3491141(33-37)Online publication date: 14-Nov-2021
https://dl.acm.org/doi/10.1145/3447932.3491141
Hu YBejarano SHoffman G(2020)ShadowSenseProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/34322024:4(1-24)Online publication date: 18-Dec-2020
https://dl.acm.org/doi/10.1145/3432202
Show More Cited By

Index Terms

Recognizing Gestures on Projected Button Widgets with an RGB-D Camera Using a CNN
1. Human-centered computing
  1. Human computer interaction (HCI)
    1. Interaction techniques
      1. Gestural input

Recommendations

Contact area interaction with sliding widgets
UIST '09: Proceedings of the 22nd annual ACM symposium on User interface software and technology

We show how to design touchscreen widgets that respond to a finger's contact area. In standard touchscreen systems a finger often appears to touch several screen objects, but the system responds as though only a single pixel is touched. In contact area ...
Recognizing Touch Gestures for Social Human-Robot Interaction
ICMI '15: Proceedings of the 2015 ACM on International Conference on Multimodal Interaction

In this study, we performed touch gesture recognition on two sets of data provided by "Recognition of Social Touch Gestures Challenge 2015". For the first dataset, dubbed Corpus of Social Touch (CoST), touch is performed on a mannequin arm, whereas for ...
Controlling widgets with one power-up button
UIST '13: Proceedings of the 26th annual ACM symposium on User interface software and technology

The Power-up Button is a physical button that combines pressure and proximity sensing to enable gestural interaction with one thumb. Combined with a gesture recognizer that takes the hand's anatomy into account, the Power-up Button can recognize six ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences

ISS '18: Proceedings of the 2018 ACM International Conference on Interactive Surfaces and Spaces

November 2018

499 pages

ISBN:9781450356947

DOI:10.1145/3279778

General Chairs:
Hideki Koike
Tokyo Institute of Technology, Japan
,
Carlo Ratti
Massachusetts Institute of Technology, USA
,
Yuichiro Takeuchi
Sony Computer Science Laboratories Inc., Japan
,
Program Chairs:
Kentaro Fukuchi
Meiji University, Japan
,
Stacey Scott
University of Guelph, Canada
,
Diego Martinez Plasencia
University of Sussex, United Kingdom

Copyright © 2018 Owner/Author.

Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for third-party components of this work must be honored. For all other uses, contact the Owner/Author.

Sponsors

SIGCHI: ACM Special Interest Group on Computer-Human Interaction

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 19 November 2018

Check for updates

Author Tags

Qualifiers

Poster

Conference

ISS '18

Sponsor:

SIGCHI

ISS '18: 2018 ACM International Conference on Interactive Surfaces and Spaces

November 25 - 28, 2018

Tokyo, Japan

Acceptance Rates

ISS '18 Paper Acceptance Rate 28 of 105 submissions, 27%;

Overall Acceptance Rate 147 of 533 submissions, 28%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

4
Total Citations
View Citations
116
Total Downloads

Downloads (Last 12 months)4
Downloads (Last 6 weeks)0

Reflects downloads up to 15 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Cheng XHuang B(2022)CSI-Based Human Continuous Activity Recognition Using GMM–HMMIEEE Sensors Journal10.1109/JSEN.2022.319824822:19(18709-18717)Online publication date: 1-Oct-2022
https://doi.org/10.1109/JSEN.2022.3198248
Mthunzi EGetschmann CEchtler F(2021)Fast 3D point-cloud segmentation for interactive surfacesCompanion Proceedings of the 2021 Conference on Interactive Surfaces and Spaces10.1145/3447932.3491141(33-37)Online publication date: 14-Nov-2021
https://dl.acm.org/doi/10.1145/3447932.3491141
Hu YBejarano SHoffman G(2020)ShadowSenseProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/34322024:4(1-24)Online publication date: 18-Dec-2020
https://dl.acm.org/doi/10.1145/3432202
Bu QYang GMing XZhang TFeng JZhang J(2020)Deep transfer learning for gesture recognition with WiFi signalsPersonal and Ubiquitous Computing10.1007/s00779-019-01360-826:3(543-554)Online publication date: 21-Jan-2020
https://doi.org/10.1007/s00779-019-01360-8

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents