More Web Proxy on the site http://driver.im/

research-article

Open access

Low-latency cloud-based volumetric video streaming using head motion prediction

Authors:

Dimitri Podborski,

Thomas Buchholz,

Thomas Schierl,

Cornelius HellgeAuthors Info & Claims

NOSSDAV '20: Proceedings of the 30th ACM Workshop on Network and Operating Systems Support for Digital Audio and Video

Pages 27 - 33

https://doi.org/10.1145/3386290.3396933

Published: 08 June 2020 Publication History

Abstract

Volumetric video is an emerging key technology for immersive representation of 3D spaces and objects. Rendering volumetric video requires lots of computational power which is challenging especially for mobile devices. To mitigate this, we developed a streaming system that renders a 2D view from the volumetric video at a cloud server and streams a 2D video stream to the client. However, such network-based processing increases the motion-to-photon (M2P) latency due to the additional network and processing delays. In order to compensate the added latency, prediction of the future user pose is necessary. We developed a head motion prediction model and investigated its potential to reduce the M2P latency for different look-ahead times. Our results show that the presented model reduces the rendering errors caused by the M2P latency compared to a baseline system in which no prediction is performed.

References

[1]

Bernard D. Adelstein, Thomas G. Lee, and Stephen R. Ellis. 2003. Head Tracking Latency in Virtual Environments: Psychophysics and a Model. Proceedings of the Human Factors and Ergonomics Society Annual Meeting 47, 20 (Oct. 2003), 2083--2087.

[2]

Htrotugu Akaike. 1973. Maximum likelihood identification of Gaussian autoregressive moving average models. Biometrika 60, 2 (1973), 255--265.

[3]

R.S. Allison, L.R. Harris, M. Jenkin, U. Jasiobedzka, and J.E. Zacher. 2001. Tolerance of temporal delay in virtual environments. In Proceedings IEEE Virtual Reality 2001. IEEE Comput. Soc, 247--254.

[4]

Tamay Aykut, Mojtaba Karimi, Christoph Burgmair, Andreas Finkenzeller, Christoph Bachhuber, and Eckehard Steinbach. 2018. Delay compensation for a telepresence System with 3D 360 degree vision based on deep head motion prediction and dynamic FoV adaptation. IEEE Robotics and Automation Letters 3, 4 (2018), 4343--4350.

[5]

Ou Bai, Varun Rathi, Peter Lin, Dandan Huang, Harsha Battapady, Ding-Yu Fei, Logan Schneider, Elise Houdayer, Xuedong Chen, and Mark Hallett. 2011. Prediction of human voluntary movement before it occurs. Clinical Neurophysiology 122, 2 (2011), 364--372.

[6]

Yanan Bao, Huasen Wu, Tianxiao Zhang, Albara Ah Ramli, and Xin Liu. 2016. Shooting a moving target: Motion-prediction-based transmission for 360-degree videos. In 2016 IEEE International Conference on Big Data (Big Data). IEEE, 1161--1170.

[7]

Yanan Bao, Tianxiao Zhang, Amit Pande, Huasen Wu, and Xin Liu. 2017. Motion-Prediction-Based Multicast for 360-Degree Video Transmissions. In 2017 14th Annual IEEE International Conference on Sensing, Communication, and Networking (SECON). IEEE, 1--9.

[8]

Yair Barniv, Mario Aguilar, and Erion Hasanbelliu. 2005. Using EMG to anticipate head motion for virtual-environment applications. IEEE Transactions on Biomedical Engineering 52, 6 (2005), 1078--1093.

[9]

FFmpeg. 2019. H.264 Video Encoding Guide. https://trac.ffmpeg.org/wiki/Encode/H.264. Online; accessed: 2020-03-26.

[10]

C. Holmberg, S. Hakansson, and G. Eriksson. 2015. Web real-time communication use cases and requirements. RFC 7478.

[11]

Rob J Hyndman and George Athanasopoulos. 2018. Forecasting: principles and practice. OTexts.

[12]

Kishor Koirala, Meera Dasog, Pu Liu, and Edward A Clancy. 2015. Using the electromyogram to anticipate torques about the elbow. IEEE Transactions on Neural Systems and Rehabilitation Engineering 23, 3 (2015), 396--402.

[13]

Steve LaValle and Peter Giokaris. 2015. Perception based predictive tracking for head mounted displays. US Patent No. 9348410B2, Filed May 22, 2014, Issued Jun. 6., 2015.

[14]

Peter Lincoln, Alex Blate, Montek Singh, Turner Whitted, Andrei State, Anselmo Lastra, and Henry Fuchs. 2016. From motion to photons in 80 microseconds: Towards minimal latency for virtual and augmented reality. IEEE transactions on visualization and computer graphics 22, 4 (2016), 1367--1376.

Digital Library

[15]

Simone Mangiante, Guenter Klas, Amit Navon, Zhuang GuanHua, Ju Ran, and Marco Dias Silva. 2017. VR is on the edge: How to deliver 360 videos in mobile networks. In Proceedings of the Workshop on Virtual Reality and Augmented Reality Network. ACM, 30--35.

Digital Library

[16]

NVIDIA. 2019. NVIDIA CloudXR Delivers Low-Latency AR/VR Streaming Over 5G Networks to Any Device. https://blogs.nvidia.com/blog/2019/10/22/nvidia-cloudxr. Online; accessed: 2020-03-26.

[17]

Stefano Petrangeli, Gwendal Simon, Haoliang Wang, and Vishy Swaminathan. 2019. Dynamic Adaptive Streaming for Augmented Reality Applications. In 2019 IEEE International Symposium on Multimedia (ISM). IEEE, 56--567.

[18]

Feng Qian, Bo Han, Jarrell Pair, and Vijay Gopalakrishnan. 2019. Toward practical volumetric video streaming on commodity smartphones. In Proceedings of the 20th International Workshop on Mobile Computing Systems and Applications. ACM, 135--140.

Digital Library

[19]

James Robinson and Cameron McCormack. 2015. Timing control for script-based animations. W3C Working Draft. https://www.w3.org/TR/2015/NOTE-animation-timing-20150922

[20]

Yago Sanchez, Gurdeep Singh Bhullar, Robert Skupin, Cornelius Hellge, and Thomas Schierl. 2019. Delay impact on MPEG OMAFâĂ&Zacute;s tile-based viewport-dependent 360Âř video streaming. IEEE Journal on Emerging and Selected Topics in Circuits and Systems (2019).

[21]

O Schreer, I Feldmann, P Kauff, P Eisert, D Tatzelt, C Hellge, K Müller, T Ebner, and S Bliedung. 2019. Lessons learnt during one year of commercial volumetric video production. In 2019 IBC conference. IBC.

[22]

Sebastian Schwarz, Marius Preda, Vittorio Baroncini, Madhukar Budagavi, Pablo Cesar, Philip A Chou, Robert A Cohen, Maja Krivokuća, Sébastien Lasserre, Zhu Li, et al. 2018. Emerging MPEG standards for point cloud compression. IEEE Journal on Emerging and Selected Topics in Circuits and Systems 9, 1 (2018), 133--148.

[23]

Skipper Seabold and Josef Perktold. 2010. statsmodels: Econometric and statistical modeling with python. In 9th Python in Science Conference. inproceedings.

[24]

Andrew Segall, Vittorio Baroncini, Jill Boyce, Jianle Chen, and Teruhiko Suzuki. 2017. Joint call for proposals on video compression with capability beyond HEVC. In JVET-H1002.

[25]

Shu Shi, Varun Gupta, Michael Hwang, and Rittwik Jana. 2019. Mobile VR on edge cloud: a latency-driven design. In Proceedings of the 10th ACM Multimedia Systems Conference. ACM, 222--231.

Digital Library

[26]

Shu Shi and Cheng-Hsin Hsu. 2015. A survey of interactive remote rendering systems. Comput. Surveys 47, 4 (May 2015), 1--29.

Digital Library

[27]

Ken Shoemake. 1985. Animating rotation with quaternion curves. In ACM SIGGRAPH computer graphics, Vol. 19. ACM, 245--254.

Digital Library

[28]

Twitch. 2018. Using Netflix machine learning to analyze Twitch stream picture quality. https://streamquality.report/docs/report.html. Online; accessed: 2020-03-26.

[29]

Jeroen van der Hooft, Tim Wauters, Filip De Turck, Christian Timmerer, and Hermann Hellwagner. 2019. Towards 6DoF HTTP adaptive streaming through point cloud compression. In Proceedings of the 27th ACM International Conference on Multimedia. 2405--2413.

Digital Library

[30]

Daniel Wagner. 2018. Motion-to-photon latency in mobile AR and VR. https://daqri.com/blog/motion-to-photon-latency. Online; accessed: 2020-03-26.

Cited By

Choi HKomuro NKim W(2024)Microservices-Based Resource Provisioning for Multi-User Cloud VR in Edge NetworksElectronics10.3390/electronics1315307713:15(3077)Online publication date: 3-Aug-2024
https://doi.org/10.3390/electronics13153077
Qian PWang NHeng FUdora CTafazolli R(2024)Enabling User Intent-based Network Path Adaptation for Live Volumetric Streaming2024 IFIP Networking Conference (IFIP Networking)10.23919/IFIPNetworking62109.2024.10619068(395-403)Online publication date: 3-Jun-2024
https://doi.org/10.23919/IFIPNetworking62109.2024.10619068
Shi JZhang MShen LLiu JZhang YPu LXu JRizk AVega M(2024)Towards Full-scene Volumetric Video Streaming via Spatially Layered Representation and NeRF GenerationProceedings of the 34th edition of the Workshop on Network and Operating System Support for Digital Audio and Video10.1145/3651863.3651879(22-28)Online publication date: 15-Apr-2024
https://dl.acm.org/doi/10.1145/3651863.3651879
Show More Cited By

Index Terms

Low-latency cloud-based volumetric video streaming using head motion prediction

Recommendations

Kalman Filter-based Head Motion Prediction for Cloud-based Mixed Reality
MM '20: Proceedings of the 28th ACM International Conference on Multimedia

Volumetric video allows viewers to experience highly-realistic 3D content with six degrees of freedom in mixed reality (MR) environments. Rendering complex volumetric videos can require a prohibitively high amount of computational power for mobile ...
Cloud rendering-based volumetric video streaming system for mixed reality services
MMSys '20: Proceedings of the 11th ACM Multimedia Systems Conference

Volumetric video is an emerging technology for immersive representation of 3D spaces that captures objects from all directions using multiple cameras and creates a dynamic 3D model of the scene. However, processing volumetric content requires high ...
Adaptive Streaming of Visual Volumetric Video-based Coding Media
MMVE '23: Proceedings of the 15th International Workshop on Immersive Mixed and Virtual Environment Systems

High-quality 3D point clouds have recently emerged as an advanced representation of immersive media, enabling new forms of interaction in virtual environments and augmented reality applications. A point cloud consists of a set of points represented in ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences

NOSSDAV '20: Proceedings of the 30th ACM Workshop on Network and Operating Systems Support for Digital Audio and Video

June 2020

73 pages

ISBN:9781450379458

DOI:10.1145/3386290

General Chair:
M. Reha Civanlar
Ozyegin University
,
Program Chairs:
Lucile Sassatelli
Universite Cote d'Azur
,
Jong-Seok Lee
Yonsei University

Copyright © 2020 Owner/Author.

Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for third-party components of this work must be honored. For all other uses, contact the Owner/Author.

Sponsors

SIGMM: ACM Special Interest Group on Multimedia

In-Cooperation

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 08 June 2020

Check for updates

Author Tags

Qualifiers

Research-article

Conference

MMSys '20

Sponsor:

SIGMM

MMSys '20: 11th ACM Multimedia Systems Conference

June 10 - 11, 2020

Istanbul, Turkey

Acceptance Rates

NOSSDAV '20 Paper Acceptance Rate 10 of 22 submissions, 45%;

Overall Acceptance Rate 118 of 363 submissions, 33%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

32
Total Citations
View Citations
1,790
Total Downloads

Downloads (Last 12 months)353
Downloads (Last 6 weeks)24

Reflects downloads up to 14 Dec 2024

Other Metrics

View Author Metrics

Citations

Cited By

Choi HKomuro NKim W(2024)Microservices-Based Resource Provisioning for Multi-User Cloud VR in Edge NetworksElectronics10.3390/electronics1315307713:15(3077)Online publication date: 3-Aug-2024
https://doi.org/10.3390/electronics13153077
Qian PWang NHeng FUdora CTafazolli R(2024)Enabling User Intent-based Network Path Adaptation for Live Volumetric Streaming2024 IFIP Networking Conference (IFIP Networking)10.23919/IFIPNetworking62109.2024.10619068(395-403)Online publication date: 3-Jun-2024
https://doi.org/10.23919/IFIPNetworking62109.2024.10619068
Shi JZhang MShen LLiu JZhang YPu LXu JRizk AVega M(2024)Towards Full-scene Volumetric Video Streaming via Spatially Layered Representation and NeRF GenerationProceedings of the 34th edition of the Workshop on Network and Operating System Support for Digital Audio and Video10.1145/3651863.3651879(22-28)Online publication date: 15-Apr-2024
https://dl.acm.org/doi/10.1145/3651863.3651879
Fang TNiu CSun YLv CJiang XXue BWu FChen GGanesan DLane NShi W(2024)An End-to-End, Low-Cost, and High-Fidelity 3D Video Pipeline for Mobile DevicesProceedings of the 30th Annual International Conference on Mobile Computing and Networking10.1145/3636534.3690685(1162-1176)Online publication date: 4-Dec-2024
https://dl.acm.org/doi/10.1145/3636534.3690685
Yu DChen RLi XXiao MZhang GLiu Y(2024)A GPU-Enabled Real-Time Framework for Compressing and Rendering Volumetric VideosIEEE Transactions on Computers10.1109/TC.2023.334310473:3(789-800)Online publication date: 1-Mar-2024
https://dl.acm.org/doi/10.1109/TC.2023.3343104
Wu YChen CLi THsieh H(2024)Towards Optimal Multiview Transcoding for Edge-Assisted Wireless Volumetric StreamingICC 2024 - IEEE International Conference on Communications10.1109/ICC51166.2024.10622856(4096-4101)Online publication date: 9-Jun-2024
https://doi.org/10.1109/ICC51166.2024.10622856
Yeregui IMejías DPacho GViola RAstorga JMontagud M(2024)Edge Rendering Architecture for multiuser XR Experiences and E2E Performance Assessment2024 IEEE International Symposium on Broadband Multimedia Systems and Broadcasting (BMSB)10.1109/BMSB62888.2024.10608249(1-7)Online publication date: 19-Jun-2024
https://doi.org/10.1109/BMSB62888.2024.10608249
Kumar TSharma PTanwar JAlsghier HBhushan SAlhumyani HSharma VAlutaibi A(2024)Cloud‐based video streaming servicesCAAI Transactions on Intelligence Technology10.1049/cit2.122999:2(265-285)Online publication date: 14-Mar-2024
https://dl.acm.org/doi/10.1049/cit2.12299
Enenche PKim DYou D(2024)On the road to the metaverse: Point cloud video streaming: Perspectives and enablersICT Express10.1016/j.icte.2024.11.001Online publication date: Nov-2024
https://doi.org/10.1016/j.icte.2024.11.001
Mukawa H(2024) 43‐1: Invited Paper: Review and Perspective of XR Technologies for Immersive Experience SID Symposium Digest of Technical Papers10.1002/sdtp.1758455:1(559-562)Online publication date: 30-Jul-2024
https://doi.org/10.1002/sdtp.17584
Show More Cited By

View Options

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Media

Figures

Other

Tables

View Table of Contents