More Web Proxy on the site http://driver.im/

research-article

Enhanced Videogame Livestreaming by Reconstructing an Interactive 3D Game View for Spectators

Authors:

Jeremy Hartmann,

Daniel VogelAuthors Info & Claims

CHI '22: Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems

Article No.: 499, Pages 1 - 14

https://doi.org/10.1145/3491102.3517521

Published: 29 April 2022 Publication History

Abstract

Many videogame players livestream their gameplay so remote spectators can watch for enjoyment, fandom, and to learn strategies and techniques. Current approaches capture the player’s rendered RGB view of the game, and then encode and stream it as a 2D live video feed. We extend this basic concept by also capturing the depth buffer, camera pose, and projection matrix from the rendering pipeline of the videogame and package them all within a MPEG-4 media container. Combining these additional data streams with the RGB view, our system builds a real-time, cumulative 3D representation of the live game environment for spectators. This enables each spectator to individually control a personal game view in 3D. This means they can watch the game from multiple perspectives, enabling a new kind of videogame spectatorship experience.

Supplementary Material

MP4 File (3491102.3517521-video-figure.mp4)

Video Figure

Download
144.89 MB

MP4 File (3491102.3517521-video-preview.mp4)

Video Preview

Download
16.72 MB

References

[1]

Activision. 2022. Call of Duty®: Black Ops - Cold War | Popular FPS Game. https://www.callofduty.com/ca/en/blackopscoldwar. (Accessed on 01/06/2022).

[2]

Helmut Alt. 2009. The Computational Geometry of Comparing Shapes. Springer Berlin Heidelberg, Berlin, Heidelberg, 235–248. https://doi.org/10.1007/978-3-642-03456-5_16

Digital Library

[3]

AltspaceVR. 2021. AltspaceVR | Be there, together.https://altvr.com/. (Accessed on 05/06/2021).

[4]

Apple. 2021. HTTP Live Streaming (HLS) - Apple Developer. https://developer.apple.com/streaming/. (Accessed on 09/01/2021).

[5]

Hugh Bailey. 2021. Open Broadcaster Software | OBS. https://obsproject.com/. (Accessed on 09/03/2021).

[6]

Louis Bavoil and Miguel Sainz. 2008. Screen space ambient occlusion. NVIDIA developer information: http://developers. nvidia. com 6 (2008).

[7]

Bigscreen. 2021. Bigscreen. https://www.bigscreenvr.com/. (Accessed on 05/02/2021).

[8]

Michael Broxton, John Flynn, Ryan Overbeck, Daniel Erickson, Peter Hedman, Matthew Duvall, Jason Dourgarian, Jay Busch, Matt Whalen, and Paul Debevec. 2020. Immersive light field video with a layered mesh representation. ACM Transactions on Graphics 39, 4 (jul 2020), 15. https://doi.org/10.1145/3386569.3392485

Digital Library

[9]

Carlos Campos, Richard Elvira, Juan J.Gomez Rodriguez, Jose M.M. Montiel, and Juan D. Tardos. 2021. ORB-SLAM3: An Accurate Open-Source Library for Visual, Visual-Inertial, and Multimap SLAM. IEEE Transactions on Robotics 37, 6 (2021), 1874–1890. https://doi.org/10.1109/TRO.2021.3075644 arxiv:2007.11898

[10]

Gifford Cheung and Jeff Huang. 2011. Starcraft from the Stands: Understanding the Game Spectator. In Proceedings of the 2011 annual conference on Human factors in computing systems - CHI ’11. ACM Press, New York, New York, USA, 763. https://doi.org/10.1145/1978942.1979053

Digital Library

[11]

Elizabeth F Churchill, David N Snowdon, and Alan J Munro. 2012. Collaborative virtual environments: digital places and spaces for interaction. Springer Science & Business Media.

[12]

Andon M. Coleman. 2021. The Complete Guide to SK | Special K - The Official Wiki. https://wiki.special-k.info/. (Accessed on 05/11/2021).

[13]

Yann Collet. 2021. LZ4 - Extremely fast compression. https://lz4.github.io/lz4/. (Accessed on 09/04/2021).

[14]

John Downs, Frank Vetere, Steve Howard, Steve Loughnan, and Wally Smith. 2014. Audience Experience in Social Videogaming: Effects of Turn Expectation and Game Physicality. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. ACM, New York, NY, USA, 3473–3482. https://doi.org/10.1145/2556288.2556965

Digital Library

[15]

Lisa A Elkin, Matthew Kay, James J Higgins, and Jacob O Wobbrock. 2021. An aligned rank transform procedure for multifactor contrast tests. In Proceedings of the ACM Symposium on User Interface Software and Technology (UIST ’21). https://doi.org/10.5281/zenodo.594511

Digital Library

[16]

Katharina Emmerich, Andrey Krekhov, Sebastian Cmentowski, and Jens Krueger. 2021. Streaming VR Games to the Broad Audience: A Comparison of the First-Person and Third-Person Perspectives. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. 1–14. https://doi.org/10.1145/3411764.3445515 arxiv:2101.04449

Digital Library

[17]

FFmpeg. 2021. FFmpeg. https://www.ffmpeg.org/. (Accessed on 08/06/2021).

[18]

Epic Games. 2021. Fortnite | Free-to-Play Cross-Platform Game - Fortnite. https://www.epicgames.com/fortnite/en-US/home. (Accessed on 08/04/2021).

[19]

Resolution Games. 2019. Acron: Attack of the Squirrels!https://www.resolutiongames.com/acron. (Accessed on 09/06/2021).

[20]

Riot Games. 2022. League of Legends. https://www.leagueoflegends.com/en-us/. (Accessed on 01/06/2022).

[21]

D. A. Green. 2011. A colour scheme for the display of astronomical intensity images. Bulletin of the Astronomical Society of India 39, 2 (aug 2011), 289–295. arxiv:1108.5083http://arxiv.org/abs/1108.5083

[22]

Jan Gugenheimer, Evgeny Stemasov, Julian Frommel, and Enrico Rukzio. 2017. ShareVR: Enabling Co-Located Experiences for Virtual Reality between HMD and Non-HMD Users. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems. ACM, New York, NY, USA, 4021–4033. https://doi.org/10.1145/3025453.3025683

Digital Library

[23]

Jan Gugenheimer, Evgeny Stemasov, Harpreet Sareen, and Enrico Rukzio. 2018. FaceDisplay: Towards Asymmetric Multi-User Interaction for Nomadic Virtual Reality. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. ACM, New York, NY, USA, 1–13. https://doi.org/10.1145/3173574.3173628

Digital Library

[24]

William A. Hamilton, Oliver Garretson, and Andruid Kerne. 2014. Streaming on twitch: Fostering participatory communities of play within live mixed media. In Proceedings of the 32nd annual ACM conference on Human factors in computing systems - CHI ’14. ACM Press, New York, New York, USA, 1315–1324. https://doi.org/10.1145/2556288.2557048

Digital Library

[25]

Jeremy Hartmann, Christian Holz, Eyal Ofek, and Andrew D. Wilson. 2019. RealityCheck: Blending Virtual Environments with Situated Physical Reality. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems - CHI ’19. ACM Press, New York, New York, USA, 1–12. https://doi.org/10.1145/3290605.3300577

Digital Library

[26]

Jeremy Hartmann, Yen-ting Yeh, and Daniel Vogel. 2020. AAR: Augmenting a Wearable Augmented Reality Display with an Actuated Head-Mounted Projector. In Proceedings of the 33rd Annual ACM Symposium on User Interface Software and Technology. ACM, New York, NY, USA, 445–458. https://doi.org/10.1145/3379337.3415849

Digital Library

[27]

Linjia He, Hongsong Li, Tong Xue, Deyuan Sun, Shoulun Zhu, and Gangyi Ding. 2018. Am I in the theater? Usability Study of Live Performance Based Virtual Reality. In Proceedings of the 24th ACM Symposium on Virtual Reality Software and Technology. ACM, New York, NY, USA, 1–11. https://doi.org/10.1145/3281505.3281508

Digital Library

[28]

Sebastian Herscher, Connor DeFanti, Nicholas Gregory Vitovitch, Corinne Brenner, Haijun Xia, Kris Layng, and Ken Perlin. 2019. CAVRN: An exploration and evaluation of a collective audience virtual reality nexus experience. In UIST 2019 - Proceedings of the 32nd Annual ACM Symposium on User Interface Software and Technology. ACM, New York, NY, USA, 1137–1150. https://doi.org/10.1145/3332165.3347929

Digital Library

[29]

Galen Hunt and Doug Brubacher. 1999. Detours: Binary interception of Win32 functions. 3rd USENIX Windows NT Symposium(1999). http://research.microsoft.com/sn/detours

[30]

LIV Inc. 2021. LIV | Your VR capture toolbox. https://www.liv.tv/. (Accessed on 05/02/2021).

[31]

Youtube Inc.2021. YouTube Gaming. https://www.youtube.com/gaming. (Accessed on 05/04/2021).

[32]

Twitch Interactive. 2021. Twitch. https://www.twitch.tv/. (Accessed on 05/04/2021).

[33]

ISO/IEC 14496-11 2015. Information technology — Coding of audio-visual objects — Part 11: Scene description and application engine. Standard. International Organization for Standardization, Geneva, CH.

[34]

Pascal Jansen, Fabian Fischbach, Jan Gugenheimer, Evgeny Stemasov, Julian Frommel, and Enrico Rukzio. 2020. ShARe: Enabling Co-Located Asymmetric Multi-User Interaction for Augmented Reality Head-Mounted Displays. In Proceedings of the 33rd Annual ACM Symposium on User Interface Software and Technology. ACM, New York, NY, USA, 459–471. https://doi.org/10.1145/3379337.3415843

Digital Library

[35]

Tatsuyoshi Kaneko, Hiroyuki Tarumi, Keiya Kataoka, Yuki Kubochi, Daiki Yamashita, Tomoki Nakai, and Ryota Yamaguchi. 2019. Supporting the sense of unity between remote audiences in VR-based remote live music support system KSA2. In Proceedings - 2018 IEEE International Conference on Artificial Intelligence and Virtual Reality, AIVR 2018. 124–127. https://doi.org/10.1109/AIVR.2018.00025

[36]

Dennis L. Kappen, Pejman Mirza-Babaei, Jens Johannsmeier, Daniel Buckstein, James Robb, and Lennart E. Nacke. 2014. Engaged By Boos and Cheers: The Effect of Co-Located Game Audiences on Social Player Experience. In Proceedings of the first ACM SIGCHI annual symposium on Computer-human interaction in play. ACM, New York, NY, USA, 151–160. https://doi.org/10.1145/2658537.2658687

Digital Library

[37]

Shunichi Kasahara and Jun Rekimoto. 2014. JackIn: Integrating first-person view with out-of-body vision generation for human-human augmentation. In ACM International Conference Proceeding Series. Association for Computing Machinery. https://doi.org/10.1145/2582051.2582097

Digital Library

[38]

Mehdi Kaytoue, Arlei Silva, Loïc Cerf, Wagner Meira, and Chedy Raïssi. 2012. Watch me Playing, I am a Professional: a First Study on Video Game Live Streaming. In Proceedings of the 21st international conference companion on World Wide Web - WWW ’12 Companion. ACM Press, New York, New York, USA, 1181. https://doi.org/10.1145/2187980.2188259

Digital Library

[39]

Andrey Krekhov, Daniel Preuß, Sebastian Cmentowski, and Jens Krüger. 2020. Silhouette Games: An Interactive One-Way Mirror Approach to Watching Players in VR. In Proceedings of the Annual Symposium on Computer-Human Interaction in Play. New York, NY, USA. https://doi.org/10.1145/3410404.3414247

Digital Library

[40]

Jie Li, Yiping Kong, Thomas Röggla, Francesca De Simone, Swamy Ananthanarayan, Huib de Ridder, Abdallah El Ali, and Pablo Cesar. 2019. Measuring and Understanding Photo Sharing Experiences in Social Virtual Reality. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems. ACM, New York, NY, USA, 1–14. https://doi.org/10.1145/3290605.3300897

Digital Library

[41]

Yunpeng Liu, Stephan Beck, Renfang Wang, Jin Li, Huixia Xu, Shijie Yao, Xiaopeng Tong, and Bernd Froehlich. 2015. Hybrid Lossless-Lossy Compression for Real-Time Depth-Sensor Streams in 3D Telepresence Applications. In Advances in Multimedia Information Processing – PCM 2015, Vol. 9314. Springer International Publishing, Cham, 442–452. https://doi.org/10.1007/978-3-319-24075-6_43

[42]

Bernhard Maurer, Ilhan Aslan, Martin Wuchse, Katja Neureiter, and Manfred Tscheligi. 2015. Gaze-Based Onlooker Integration: Exploring the In-Between of Active Player and Passive Spectator in Co-Located Gaming. In Proceedings of the 2015 Annual Symposium on Computer-Human Interaction in Play. ACM, New York, NY, USA, 163–173. https://doi.org/10.1145/2793107.2793126

Digital Library

[43]

Patrick Mours. 2021. Reshade. https://reshade.me/. (Accessed on 09/01/2021).

[44]

Moving Picture Experts Group (MPEG). 2021. MPEG – The Moving Picture Experts Group. https://www.mpegstandards.org/. (Accessed on 05/13/2021).

[45]

Fabrizio Pece, Jan Kautz, and Tim Weyrich. 2011. Adapting standard video codecs for depth streaming. In Joint Virtual Reality Conference of EGVE 2011 - The 17th Eurographics Symposium on Virtual Environments, EuroVR 2011 - The 8th EuroVR (INTUITION) Conference. 59–66. https://doi.org/10.2312/EGVE/JVRC11/059-066

[46]

WebM Project. 2021. The WebM Project | Welcome to the WebM Project. https://www.webmproject.org/. (Accessed on 09/01/2021).

[47]

Jana Rambusch, Anna - Sofia Alklind Taylor, and Tarja Susi. 2017. A pre-study on spectatorship in eSports. In Spectating Play. 13TH ANNUAL GAME RESEARCH LAB SPRING SEMINAR. 24–25.

[48]

Jean-Paul Sartre. 1943. Being and Nothingness. Éditions Gallimard. 628 pages.

[49]

Max Sjöblom and Juho Hamari. 2017. Why do people watch others play video games? An empirical study on the motivations of Twitch users. Computers in Human Behavior 75 (oct 2017), 985–996. https://doi.org/10.1016/j.chb.2016.10.019

Digital Library

[50]

Bijan Stephen. 2021. CodeMiko will see you now - The Verge. https://www.theverge.com/22370260/codemiko-twitch-interview-stream-technician. (Accessed on 05/02/2021).

[51]

Burak S. Tekin and Stuart Reeves. 2017. Ways of Spectating: Unravelling Spectator Participation in Kinect Play. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems, Vol. 2017-May. ACM, New York, NY, USA, 1558–1570. https://doi.org/10.1145/3025453.3025813

Digital Library

[52]

Emmanuel Thomas. [n.d.]. The ’MP4’ Registration Authority. https://mp4ra.org/. (Accessed on 05/18/2021).

[53]

Balasaravanan Thoravi Kumaravel, Cuong Nguyen, Stephen DiVerdi, and Bjoern Hartmann. 2020. TransceiVR: Bridging Asymmetrical Communication Between VR Users and External Collaborators. In Proceedings of the 33rd Annual ACM Symposium on User Interface Software and Technology. ACM, New York, NY, USA, 182–195. https://doi.org/10.1145/3379337.3415827

Digital Library

[54]

Wolfgang Tschauko. 2021. VR Giants on Steam. https://store.steampowered.com/app/1124160/VR_Giants/. (Accessed on 09/06/2021).

[55]

Viswanath Venkatesh. 2000. Determinants of perceived ease of use: Integrating control, intrinsic motivation, and emotion into the technology acceptance model. Information systems research 11, 4 (2000), 342–365.

Digital Library

[56]

Chiu-Hsuan Wang, Seraphina Yong, Hsin-Yu Chen, Yuan-Syun Ye, and Liwei Chan. 2020. HMD Light: Sharing In-VR Experience via Head-Mounted Projector for Asymmetric Interaction. In Proceedings of the 33rd Annual ACM Symposium on User Interface Software and Technology. ACM, New York, NY, USA, 472–486. https://doi.org/10.1145/3379337.3415847

Digital Library

[57]

Thomas Wiegand, G.J. Sullivan, G. Bjontegaard, and Ajay Luthra. 2003. Overview of the H.264/AVC Video Coding Standard. IEEE Transactions on Circuits and Systems for Video Technology 13, 7 (jul 2003), 560–576. https://doi.org/10.1109/TCSVT.2003.815165

Digital Library

[58]

Jacob O Wobbrock, Leah Findlater, Darren Gergle, and James J Higgins. 2011. The aligned rank transform for nonparametric factorial analyses using only anova procedures. In Proceedings of the 2011 annual conference on Human factors in computing systems - CHI ’11. ACM Press, New York, New York, USA, 143. https://doi.org/10.1145/1978942.1978963

Digital Library

[59]

Wenqi Xian, Jia-Bin Huang, Johannes Kopf, and Changil Kim. 2021. Space-time Neural Irradiance Fields for Free-Viewpoint Video. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). 9421–9431.

[60]

Hiromu Yakura and Masataka Goto. 2020. Enhancing Participation Experience in VR Live Concerts by Improving Motions of Virtual Audience Avatars. In 2020 IEEE International Symposium on Mixed and Augmented Reality (ISMAR). IEEE, 555–565. https://doi.org/10.1109/ISMAR50242.2020.00083

Cited By

Aghahoseini PDavid MBunt A(2024)Investigating the Role of Real-Time Chat Summaries in Supporting Live StreamersProceedings of the 50th Graphics Interface Conference10.1145/3670947.3670980(1-12)Online publication date: 3-Jun-2024
https://dl.acm.org/doi/10.1145/3670947.3670980
Do NLe KLy DFjeld MTran M(2024)XRPublicSpectator: Towards Public Mixed Reality Viewing in Collocated Asymmetric GroupsExtended Abstracts of the CHI Conference on Human Factors in Computing Systems10.1145/3613905.3650796(1-7)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613905.3650796
Wu LLiu QZhao JLank E(2023)Interactions across Displays and Space: A Study of Virtual Reality Streaming Practices on TwitchProceedings of the ACM on Human-Computer Interaction10.1145/36264737:ISS(242-265)Online publication date: 1-Nov-2023
https://dl.acm.org/doi/10.1145/3626473

Index Terms

Enhanced Videogame Livestreaming by Reconstructing an Interactive 3D Game View for Spectators

Recommendations

An interactive snowboard game in virtual environment
Edutainment'07: Proceedings of the 2nd international conference on Technologies for e-learning and digital entertainment

In this paper we implement the 3D virtual interactive snowboard game. It is not an on-line game but a bodily sensible game. We develop the terrain rendering engine which is designed for our interactive snowboard simulation game. It is developed by using ...
Tabletop Game Design for Videogame Designers
Engaging in Videogame Play: An Activity-Centric Analysis of the Player Experience
OzCHI '15: Proceedings of the Annual Meeting of the Australian Special Interest Group for Computer Human Interaction

This paper focuses on examining play activities in people's favourite videogame experience. Through interviews with 30 videogame players we discovered which types of play activities are most appealing. Our research identifies the level of appeal of a ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences

CHI '22: Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems

April 2022

10459 pages

ISBN:9781450391573

DOI:10.1145/3491102

Editors:
Simone Barbosa
PUC-Rio, Brazil
,
Cliff Lampe
University of Michigan, USA
,
Caroline Appert
Université Paris-Saclay, France
,
David A. Shamma
Toyota Research Institute, USA
,
Steven Drucker
Microsoft Research, USA
,
Julie Williamson
University of Glasgow, UK
,
Koji Yatani
University of Tokyo, Japan

Copyright © 2022 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Sponsors

SIGCHI: ACM Special Interest Group on Computer-Human Interaction

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 29 April 2022

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed limited

Funding Sources

Ontario Early Researcher Award
Natural Sciences and Engineering Research Council of Canada
Canada Foundation for Innovation

Conference

CHI '22

Sponsor:

SIGCHI

CHI '22: CHI Conference on Human Factors in Computing Systems

April 29 - May 5, 2022

LA, New Orleans, USA

Acceptance Rates

Overall Acceptance Rate 6,199 of 26,314 submissions, 24%

Upcoming Conference

CHI 2025

Sponsor:
sigchi

ACM CHI Conference on Human Factors in Computing Systems

April 26 - May 1, 2025

Yokohama , Japan

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

3
Total Citations
View Citations
500
Total Downloads

Downloads (Last 12 months)72
Downloads (Last 6 weeks)7

Reflects downloads up to 13 Dec 2024

Other Metrics

View Author Metrics

Citations

Cited By

Aghahoseini PDavid MBunt A(2024)Investigating the Role of Real-Time Chat Summaries in Supporting Live StreamersProceedings of the 50th Graphics Interface Conference10.1145/3670947.3670980(1-12)Online publication date: 3-Jun-2024
https://dl.acm.org/doi/10.1145/3670947.3670980
Do NLe KLy DFjeld MTran M(2024)XRPublicSpectator: Towards Public Mixed Reality Viewing in Collocated Asymmetric GroupsExtended Abstracts of the CHI Conference on Human Factors in Computing Systems10.1145/3613905.3650796(1-7)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613905.3650796
Wu LLiu QZhao JLank E(2023)Interactions across Displays and Space: A Study of Virtual Reality Streaming Practices on TwitchProceedings of the ACM on Human-Computer Interaction10.1145/36264737:ISS(242-265)Online publication date: 1-Nov-2023
https://dl.acm.org/doi/10.1145/3626473

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

HTML Format

View this article in HTML Format.

Media

Figures

Other

Tables

View Table of Contents