More Web Proxy on the site http://driver.im/

short-paper

Meyendtris: a hands-free, multimodal tetris clone using eye tracking and passive BCI for intuitive neuroadaptive gaming

Authors:

Laurens R. Krol,

Sarah-Christin Freytag,

Thorsten O. ZanderAuthors Info & Claims

ICMI '17: Proceedings of the 19th ACM International Conference on Multimodal Interaction

Pages 433 - 437

https://doi.org/10.1145/3136755.3136805

Published: 03 November 2017 Publication History

Abstract

This paper introduces a completely hands-free version of Tetris that uses eye tracking and passive brain-computer interfacing (a real-time measurement and interpretation of brain activity) to replace existing game elements, as well as introduce novel ones. In Meyendtris, dwell time-based eye tracking replaces the game's direct control elements, i.e. the movement of the tetromino. In addition to that, two mental states of the player influence the game in real time by means of passive brain-computer interfacing. First, a measure of the player's relaxation is used to modulate the speed of the game (and the corresponding music). Second, when upon landing of a tetromino a state of error perception is detected in the player's brain, this last landed tetromino is destroyed. Together, this results in a multimodal, hands-free version of the classic Tetris game that is no longer hindered by manual input bottlenecks, while engaging novel mental abilities of the player.

References

[1]

Kai Keng Ang, Zheng Yang Chin, Haihong Zhang, and Cuntai Guan. 2008. Filter Bank Common Spatial Pattern (FBCSP) in Brain-Computer Interface. In IEEE International Joint Conference on Neural Networks (IJCNN), 2008. 2390–2397.

[2]

Benjamin Blankertz, Steven Lemm, Matthias S Treder, Stefan Haufe, and Klaus-Robert Müller. 2011. Single-trial analysis and classification of ERP components – A tutorial. NeuroImage 56, 2 (2011), 814–825.

[3]

D.H. Cymek, A.C. Venjakob, S. Ruff, O.H.M Lutz, S. Hofman, and M. Roetting. 2014. Entering PIN codes by smooth pursuit eye movements. Journal of Eye Movement Research 7(4) (2014), 1.

[4]

Arnaud Delorme, Tim Mullen, Christian Kothe, Zeynep Akalin Acar, Nima Bigdely-Shamlo, Andrey Vankov, and Scott Makeig. 2011. EEGLAB, SIFT, NFT, BCILAB, and ERICA: New Tools for Advanced EEG Processing. Computational Intelligence and Neuroscience 2011, Article 10 (2011), 1 pages.

Digital Library

[5]

Kate C Ewing, Stephen H Fairclough, and Kiel Gilleade. 2016. Evaluation of an Adaptive Game that Uses EEG Measures Validated during the Design Process as Inputs to a Biocybernetic Loop. Frontiers in Human Neuroscience 10 (2016), 223.

[6]

L. A. Farwell and E. Donchin. 1988. Talking off the top of your head: toward a mental prosthesis utilizing event-related brain potentials. Electroencephalography and Clinical Neurophysiology 70, 6 (1988), 510–523.

[7]

William J Gehring, Yanni Liu, Joseph M Orr, and Joshua Carp. 2012. The errorrelated negativity (ERN/Ne). In Oxford handbook of event-related potential components, Steven J. Luck and Emily S. Kappenman (Eds.). Oxford University Press, New York, NY, 231–291. 0120

[8]

Kiel M Gilleade and Alan Dix. 2004. Using Frustration in the Design of Adaptive Videogames. In Proceedings of the 2004 ACM SIGCHI International Conference on Advances in Computer Entertainment Technology (ACE ’04). ACM, New York, NY, USA, 228–232.

Digital Library

[9]

Mike Goslin and Mark R Mine. 2004. The Panda3D graphics engine. Computer 37, 10 (2004), 112–114.

Digital Library

[10]

K. Holmqvist. 2011. Eye Tracking. A comprehensive guide to methods and measures. Oxford University Press.

[11]

R.J.K. Jakob. 1995. Virtual environments and advanced interface design. In Eye Tracking and Advanced Interface Design, W. Barfield and T.A. III. Furness (Eds.). Oxford University Press, New York, NY, 258–288.

Digital Library

[12]

O.H.M. Lutz, A.C. Venjakob, and S. Ruff. 2015. SMOOVS: Towards calibration-free text entry by gaze using smooth pursuit movements. Journal of Eye Movement Research 8(1):2 (2015), 1–11.

[13]

David Marshall, Damien Coyle, Shane Wilson, and Michael Callaghan. 2013. Games, Gameplay, and BCI: The State of the Art. IEEE Transactions on Computational Intelligence and AI in Games 5, 2 (2013), 82–99. TCIAIG.2013.2263555

[14]

Tim R Mullen, Christian A E Kothe, Yu M Chi, Alejandro Ojeda, Trevor Kerth, Scott Makeig, Tzyy-Ping Jung, and Gert Cauwenberghs. 2015. Real-time neuroimaging and cognitive monitoring using wearable dry EEG. IEEE Transactions on Biomedical Engineering 62, 11 (2015), 2553–2567. TBME.2015.2481482

[15]

Janna Protzak, Klas Ihme, and Thorsten O Zander. 2013. A Passive Brain-Computer Interface for Supporting Gaze-Based Human-Machine Interaction. In Universal Access in Human-Computer Interaction. Design Methods, Tools, and Interaction Techniques for eInclusion, Constantine Stephanidis and Margherita Antona (Eds.). Lecture Notes in Computer Science, Vol. 8009. Springer, Berlin Heidelberg, Germany, 662–671.

Digital Library

[16]

Sergei L. Shishkin, Yuri O. Nuzhdin, Evgeny P. Svirin, Alexander G. Trofimov, Anastasia A. Fedorova, Bogdan L. Kozyrskiy, and Boris M. Velichkovsky. 2016.

[17]

EEG Negativity in Fixations Used for Gaze-Based Control: Toward Converting Intentions into Actions with an Eye-Brain-Computer Interface. Frontiers in Neuroscience 10 (2016), 528.

[18]

B. van de Laar, H. Gürkök, D. Plass-Oude Bos, M. Poel, and A. Nijholt. 2013. Experiencing BCI Control in a Popular Computer Game. IEEE Transactions on Computational Intelligence and AI in Games 5, 2 (2013), 176–184.

[19]

Jacques J. Vidal. 1973. Toward direct brain-computer communication. Annual Review of Biophysics and Bioengineering 2, 1 (1973), 157–180.

[20]

Jonathan R. Wolpaw and Dennis J. McFarland. 2008. Brain-Computer Interface Operation of Robotic and Prosthetic Devices. Computer 41 (2008), 52–56.

Digital Library

[21]

Jonathan R Wolpaw and Elizabeth W Wolpaw. 2012. Brain-computer interfaces: something new under the sun. In Brain-Computer Interfaces: Principles and practice, Jonathan R Wolpaw and Elizabeth W Wolpaw (Eds.). Oxford University Press, Oxford, UK, 3–12.

[22]

Thorsten O. Zander, Lena M. Andreessen, Angela Berg, Maurice Bleuel, Juliane Pawlitzki, Lars Zawallich, Laurens R. Krol, and Klaus Gramann. 2017. Evaluation of a Dry EEG System for Application of Passive Brain-Computer Interfaces in Autonomous Driving. Frontiers in Human Neuroscience 11 (2017), 78.

[23]

Thorsten O. Zander, Jonas Brönstrup, Romy Lorenz, and Laurens R. Krol. 2014. Towards BCI-based Implicit Control in Human-Computer Interaction. In Advances in Physiological Computing, Stephen H. Fairclough and Kiel Gilleade (Eds.). Springer, Berlin, Germany, 67–90.

[24]

Thorsten O. Zander, Matti Gärtner, Christian Andreas Kothe, and Roman Vilimek. 2010. Combining Eye Gaze Input With a Brain-Computer Interface for Touchless Human-Computer Interaction. International Journal of Human-Computer Interaction 27, 1 (2010), 38–51.

[25]

Thorsten O. Zander and Sabine Jatzev. 2012. Context-aware brain-computer interfaces: exploring the information space of user, technical system and environment. Journal of Neural Engineering 9, 1 (2012), 016003.

[26]

Thorsten O. Zander and Christian Andreas Kothe. 2011. Towards passive braincomputer interfaces: applying brain-computer interface technology to humanmachine systems in general. Journal of Neural Engineering 8, 2 (2011), 025005.

Cited By

Reddy GProulx MHirshfield LRies A(2024)Towards an Eye-Brain-Computer Interface: Combining Gaze with the Stimulus-Preceding Negativity for Target Selections in XRProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3641925(1-17)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613904.3641925
Vigoroso LRoccato MCavallo ECaffaro F(2024)Intention to Adopt Digital Games for Safety Training in Young Farm Operators: The Role of Ease of Use, Perceived Usefulness, and Game Design CharacteristicsInternational Journal of Human–Computer Interaction10.1080/10447318.2024.2310353(1-10)Online publication date: 6-Feb-2024
https://doi.org/10.1080/10447318.2024.2310353
Dallant JIacono J(2024)How fast can we play Tetris greedily with rectangular pieces?Theoretical Computer Science10.1016/j.tcs.2024.114405992:COnline publication date: 21-Apr-2024
https://dl.acm.org/doi/10.1016/j.tcs.2024.114405
Show More Cited By

Index Terms

Meyendtris: a hands-free, multimodal tetris clone using eye tracking and passive BCI for intuitive neuroadaptive gaming
1. Human-centered computing
  1. Human computer interaction (HCI)
    1. Interaction paradigms
    2. Interaction techniques

Recommendations

Evaluating User Experience of Actual and Imagined Movement in BCI Gaming

Most research on Brain-Computer Interfaces BCI focuses on developing ways of expression for disabled people who are not able to communicate through other means. Recently it has been shown that BCI can also be used in games to give users a richer ...
Playstyles in Tetris: Beyond Player Skill, Score, and Competition
Entertainment Computing – ICEC 2022
Abstract
In this study, we looked at playstyles in Tetris in order to explore how players construct meaning in abstract video games. To do this, we looked at playstyles that go beyond metrics related to player skill, score and competition. Indeed, non-...
The Berlin Brain-Computer Interface (BBCI) --- towards a new communication channel for online control in gaming applications

The investigation of innovative Human-Computer Interfaces (HCI) provides a challenge for future multimedia research and development. Brain-Computer Interfaces (BCI) exploit the ability of human communication and control bypassing the classical ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences

ICMI '17: Proceedings of the 19th ACM International Conference on Multimodal Interaction

November 2017

676 pages

ISBN:9781450355438

DOI:10.1145/3136755

General Chairs:
Edward Lank
University of Waterloo, Canada
,
Alessandro Vinciarelli
University of Glasgow, UK
,
Program Chairs:
Eve Hoggan
Aarhus University, Denmark
,
Sriram Subramanian
University of Sussex, UK
,
Stephen A. Brewster
University of Glasgow, UK

Copyright © 2017 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Sponsors

SIGCHI: ACM Special Interest Group on Computer-Human Interaction

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 03 November 2017

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Short-paper

Conference

ICMI '17

Sponsor:

SIGCHI

ICMI '17: INTERNATIONAL CONFERENCE ON MULTIMODAL INTERACTION

November 13 - 17, 2017

Glasgow, UK

Acceptance Rates

ICMI '17 Paper Acceptance Rate 65 of 149 submissions, 44%;

Overall Acceptance Rate 453 of 1,080 submissions, 42%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

11
Total Citations
View Citations
293
Total Downloads

Downloads (Last 12 months)35
Downloads (Last 6 weeks)4

Reflects downloads up to 01 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Reddy GProulx MHirshfield LRies A(2024)Towards an Eye-Brain-Computer Interface: Combining Gaze with the Stimulus-Preceding Negativity for Target Selections in XRProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3641925(1-17)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613904.3641925
Vigoroso LRoccato MCavallo ECaffaro F(2024)Intention to Adopt Digital Games for Safety Training in Young Farm Operators: The Role of Ease of Use, Perceived Usefulness, and Game Design CharacteristicsInternational Journal of Human–Computer Interaction10.1080/10447318.2024.2310353(1-10)Online publication date: 6-Feb-2024
https://doi.org/10.1080/10447318.2024.2310353
Dallant JIacono J(2024)How fast can we play Tetris greedily with rectangular pieces?Theoretical Computer Science10.1016/j.tcs.2024.114405992:COnline publication date: 21-Apr-2024
https://dl.acm.org/doi/10.1016/j.tcs.2024.114405
Kaklauskas AAbraham AUbarte IKliukas RLuksaite VBinkyte-Veliene AVetloviene IKaklauskiene L(2022)A Review of AI Cloud and Edge Sensors, Methods, and Applications for the Recognition of Emotional, Affective and Physiological StatesSensors10.3390/s2220782422:20(7824)Online publication date: 14-Oct-2022
https://doi.org/10.3390/s22207824
Putze FPutze SSagehorn MMicek CSolovey E(2022)Understanding HCI Practices and Challenges of Experiment Reporting with Brain Signals: Towards Reproducibility and ReuseACM Transactions on Computer-Human Interaction10.1145/349055429:4(1-43)Online publication date: 31-Mar-2022
https://dl.acm.org/doi/10.1145/3490554
Mancini MCherubino PCartocci GMartinez ABorghini GGuastamacchia Edi Flumeri GRossi DModica EMenicocci SLupo VTrettel ABabiloni F(2021)Forefront Users’ Experience Evaluation by Employing Together Virtual Reality and Electroencephalography: A Case Study on Cognitive Effects of ScentsBrain Sciences10.3390/brainsci1102025611:2(256)Online publication date: 18-Feb-2021
https://doi.org/10.3390/brainsci11020256
Andreu-Perez AKiani MAndreu-Perez JReddy PAndreu-Abela JPinto MIzzetoglu K(2021)Single-Trial Recognition of Video Gamer’s Expertise from Brain Haemodynamic and Facial Emotion ResponsesBrain Sciences10.3390/brainsci1101010611:1(106)Online publication date: 14-Jan-2021
https://doi.org/10.3390/brainsci11010106
Woźniak MSikorski PWróbel-Lachowska MBartłomiejczyk NDominiak JGrudzień KRomanowski A(2021)Enhancing In-game Immersion Using BCI-controlled MechanicsProceedings of the 27th ACM Symposium on Virtual Reality Software and Technology10.1145/3489849.3489862(1-6)Online publication date: 8-Dec-2021
https://dl.acm.org/doi/10.1145/3489849.3489862
Ivaylov ILazarova MManolova A(2021)Multimodal Motor Imagery BCI Based on EEG and NIRS2021 56th International Scientific Conference on Information, Communication and Energy Systems and Technologies (ICEST)10.1109/ICEST52640.2021.9483551(73-76)Online publication date: 16-Jun-2021
https://doi.org/10.1109/ICEST52640.2021.9483551
Kosuru RLingelbach KBui MVukelić M(2019)MindTrainProceedings of Mensch und Computer 201910.1145/3340764.3344884(643-647)Online publication date: 8-Sep-2019
https://dl.acm.org/doi/10.1145/3340764.3344884
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents