More Web Proxy on the site http://driver.im/

research-article

Subjective quality evaluation of tile-based streaming for omnidirectional videos

Authors:

Ashutosh Singla,

Alexander Raake,

Britta Meixner,

Thomas BuchholzAuthors Info & Claims

MMSys '19: Proceedings of the 10th ACM Multimedia Systems Conference

Pages 232 - 242

https://doi.org/10.1145/3304109.3306218

Published: 18 June 2019 Publication History

Abstract

In viewport-adaptive streaming of omnidirectional video, only the field of view is streamed in high quality. While this has significant benefits over streaming the entire 360 sphere, no standard test method for perceived quality and simulator sickness is available to evaluate the quality of experience (QoE) of such streaming approaches. QoE testing is important as tile-based viewport-adaptive streaming technologies are replacing classical approaches because of significant bandwidth savings and increase in viewing quality. In this work, we propose a testbed, a test method, as well as test metrics for QoE tests of viewport-adaptive streaming approaches. The proposed method is validated in two different test setups, using a specific tile-based streaming technology available in the market. The chosen input variables (videos sequences, resolution, bandwidth, and network round-trip delay) are tested for their statistical significance. We found that our test method is suitable for QoE testing of viewport-adaptive streaming technologies. We also found that simulator sickness scores increase with test duration, but that breaks between tests reduce this effect. With our systematic test approach, it is possible to compare metrics among different test setups. On the tested technology, we found that a typical network delay (47 ms) only has a minimal effect on the quality ratings. Furthermore, the magnitude of the network delay does not influence simulator sickness for the system we have tested.

References

[1]

P. R. Alface, J. Macq, and N. Verzijp. 2012. Interactive omnidirectional video delivery: A bandwidth-effective approach. Bell Labs Technical Journal 16, 4 (March 2012), 135--147.

Digital Library

[2]

Stacy A Balk, Mary Anne Bertola, and Vaughan W Inman. 2013. Simulator sickness questionnaire: Twenty years later. In Proceedings of the Seventh International Driving Symposium on Human Factors in Driver Assessment, Training, and Vehicle Design (2013), 257--263.

[3]

Y. Bao, H. Wu, T. Zhang, A. A. Ramli, and X. Liu. 2016. Shooting a moving target: Motion-prediction-based transmission for 360-degree videos. (Dec 2016), 1161--1170.

[4]

John G. Beerends and Frank E. De Caluwe. 1999. The Influence of Video Quality on Perceived Audio Quality and Vice Versa. J. Audio Eng. Soc 47, 5 (1999), 355--362. http://www.aes.org/e-lib/browse.cfm?elib=12105

[5]

X. Corbillon, G. Simon, A. Devlic, and J. Chakareski. 2017. Viewport-adaptive navigable 360-degree video delivery. In 2017 IEEE International Conference on Communications (ICC) (May 2017), 1--7.

[6]

Igor D.D. Curcio, Henri Toukomaa, and Deepa Naik. 2017. Bandwidth Reduction of Omnidirectional Viewport-Dependent Video Streaming via Subjective Quality Assessment. In Proceedings of the 2Nd International Workshop on Multimedia Alternate Realities (AltMM '17). ACM, New York, NY, USA, 9--14.

Digital Library

[7]

V. R. Gaddam, M. Riegler, R. Eg, C. Griwodz, and P. Halvorsen. 2016. Tiling in Interactive Panoramic Video: Approaches and Evaluation. IEEE Transactions on Multimedia 18, 9 (Sept 2016), 1819--1831.

Digital Library

[8]

Arnob Ghosh, Vaneet Aggarwal, and Feng Qian. 2017. A Rate Adaptation Algorithm for Tile-based 360-degree Video Streaming. CoRR abs/1704.08215 (2017). arXiv:1704.08215 http://arxiv.org/abs/1704.08215

[9]

Mario Graf, Christian Timmerer, and Christopher Mueller. 2017. Towards Bandwidth Efficient Adaptive Streaming of Omnidirectional Video over HTTP: Design, Implementation, and Evaluation. In Proceedings of the 8th ACM on Multimedia Systems Conference (2017), 261--271.

Digital Library

[10]

Mohammad Hosseini and Viswanathan Swaminathan. 2016. Adaptive 360 (VR) Video Streaming: Divide and Conquer! CoRR abs/1609.08729 (2016). arXiv:1609.08729 http://arxiv.org/abs/1609.08729

[11]

S. Ishihara. 1997. The Series of Plates Designed as a test for Colour Deficiency. http://www.dfis.ubi.pt/~hgil/P.V.2/Ishihara/Ishihara.24.Plate.TEST.Book.pdf (2018/05/16). (1997).

[12]

ITU-R. January 2012. Recommendation ITU-R BT.500-13: Methodology for the subjective assessment of the quality of television pictures. (January 2012).

[13]

ITU-T. April 2008. Recommendation ITU-TP.910: Subjective video quality assessment methods for multimedia applications. (April 2008).

[14]

ITU-T. Mar 2016. Subjective assessment methods for 3D video, P.915. (Mar 2016).

[15]

Robert S. Kennedy, Norman E. Lane, Kevin S. Berbaum, and Michael G. Lilienthal. 1993. Simulator Sickness Questionnaire: An Enhanced Method for Quantifying Simulator Sickness. The International Journal of Aviation Psychology 3, 3 (1993), 203--220.

[16]

K. Misra, A. Segall, M. Horowitz, S. Xu, A. Fuldseth, and M. Zhou. 2013. An Overview of Tiles in HEVC. IEEE Journal of Selected Topics in Signal Processing 7, 6 (Dec 2013), 969--977.

[17]

Raoul Monnier, R. van Brandenburg, and R. Koenen. 2017. Streaming UHD-Quality VR at Realistic Bitrates: Mission Impossible? 2017 NAB Broadcast Engineering and Information Technology Conference (BEITC) (2017), 1--8.

[18]

Stefano Petrangeli, Viswanathan Swaminathan, Mohammad Hosseini, and Filip De Turck. 2017. An HTTP/2-Based Adaptive Streaming Framework for 360-degree Virtual Reality Videos. In Proceedings of the 2017 ACM on Multimedia Conference (MM '17). ACM, New York, NY, USA, 306--314.

Digital Library

[19]

Bappaditya Ray, Joel Jung, and Mohamed-Chaker Larabi. 2018. A Low-Complexity Video Encoder for Equirectangular Projected 360 Video Content. In 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 1723--1727.

[20]

R. Schatz, A. Sackl, C. Timmerer, and B. Gardlo. 2017. Towards subjective quality of experience assessment for omnidirectional video streaming. In 2017 Ninth International Conference on Quality of Multimedia Experience (QoMEX) (May 2017), 1--6.

[21]

Ashutosh Singla, Stephan Fremerey, Werner Robitza, Pierre Lebreton, and Alexander Raake. 2017. Comparison of Subjective Quality Evaluation for HEVC Encoded Omnidirectional Videos at Different Bit-rates for UHD and FHD Resolution. In Proceedings of the on Thematic Workshops of ACM Multimedia 2017 (Thematic Workshops '17). ACM, New York, NY, USA, 511--519.

Digital Library

[22]

A. Singla, S. Fremerey, W. Robitza, and A. Raake. 2017. Measuring and comparing QoE and simulator sickness of omnidirectional videos in different head mounted displays. In 2017 Ninth International Conference on Quality of Multimedia Experience (QoMEX) (May 2017), 1--6.

[23]

Ashutosh Singla, Werner Robitza, and Alexander Raake. 2018. Comparison of subjective quality evaluation methods for omnidirectional videos with DSIS and Modified ACR. Electronic Imaging 2018 (2018), 1--6.

[24]

G. J. Sullivan, J. Ohm, W. Han, and T. Wiegand. 2012. Overview of the High Efficiency Video Coding (HEVC) Standard. IEEE Transactions on Circuits and Systems for Video Technology 22, 12 (Dec 2012), 1649--1668.

Digital Library

[25]

A. Tirumala, M. Gates, F. Qin, J. Dugan, and J. Ferguson. 2005. Iperf: The TCP/UDP Bandwidth Measurement Tool. https://iperf.fr/ (2018/06/20). (2005).

[26]

T. Tominaga, T. Hayashi, J. Okamoto, and A. Takahashi. 2010. Performance comparisons of subjective quality assessment methods for mobile video. In 2010 Second International Workshop on Quality of Multimedia Experience (QoMEX) (June 2010), 82--87.

[27]

Huyen T. T. Tran, Nam Pham Ngoc, Cuong T. Pham, Yong Ju Jung, and Truong Cong Thang. 2017. A subjective study on QoE of 360 video for VR communication. 2017 IEEE 19th International Workshop on Multimedia Signal Processing (MMSP) (2017), 1--6.

[28]

E. Upenik, M. Rerabek, and T. Ebrahimi. 2016. Testbed for subjective evaluation of omnidirectional visual content. In 2016 Picture Coding Symposium (PCS). 1--5.

[29]

R. van Brandenburg, R. Koenen, and D. Sztykman. 2017. CDN optimisation for VR streaming. https://www.ibc.org/tech-advances/cdn-optimisation-for-vr-streaming-/2457.article (2018/06/12). (Oct. 2017).

[30]

Alireza Zare, Alireza Aminlou, Miska M. Hannuksela, and Moncef Gabbouj. 2016. HEVC-compliant Tile-based Streaming of Panoramic Video for Virtual Reality Applications. In Proceedings of the 2016 ACM on Multimedia Conference (MM '16). ACM, New York, NY, USA, 601--605.

Digital Library

Cited By

Liu JMa YZhao QYu YChang QFang Y(2023)Research on Compressed sensing and Transmission Methods of Video Resources Based on Virtual Reality2023 4th International Conference on Computer Engineering and Intelligent Control (ICCEIC)10.1109/ICCEIC60201.2023.10426627(171-175)Online publication date: 20-Oct-2023
https://doi.org/10.1109/ICCEIC60201.2023.10426627
van der Hooft JAmirpour HVega MSanchez YSchatz RSchierl TTimmerer C(2023)A Tutorial on Immersive Video Delivery: From Omnidirectional Video to HolographyIEEE Communications Surveys & Tutorials10.1109/COMST.2023.326325225:2(1336-1375)Online publication date: Oct-2024
https://doi.org/10.1109/COMST.2023.3263252
Göring SRao RRaake A(2023)Quality assessment of higher resolution images and videos with remote testingQuality and User Experience10.1007/s41233-023-00055-68:1Online publication date: 13-Apr-2023
https://doi.org/10.1007/s41233-023-00055-6
Show More Cited By

Index Terms

Subjective quality evaluation of tile-based streaming for omnidirectional videos

Recommendations

Comparison of Subjective Quality Evaluation for HEVC Encoded Omnidirectional Videos at Different Bit-rates for UHD and FHD Resolution
Thematic Workshops '17: Proceedings of the on Thematic Workshops of ACM Multimedia 2017

In this paper, we perform subjective quality evaluation studies for HEVC/H.265-encoded omnidirectional videos at different bit-rates for two different resolutions (FHD and UHD) on an Oculus Rift. Results of these tests provide insight into appropriate ...
Bandwidth Reduction of Omnidirectional Viewport-Dependent Video Streaming via Subjective Quality Assessment
AltMM '17: Proceedings of the 2nd International Workshop on Multimedia Alternate Realities

Omnidirectional video is more and more widespread in consumer electronics and professional capture devices, as well over Internet via novel streaming services. Omnidirectional video requires a large streaming bandwidth. To date there is little knowledge ...
Optimized Viewport Dependent Streaming of Stereoscopic Omnidirectional Video
PV '18: Proceedings of the 23rd Packet Video Workshop

Streaming the whole omnidirectional video at high quality is less efficient in terms of bandwidth requirements and decoding complexity, since only a small part of the 360-degree horizontal field of view can be seen by a user at a given point in time. In ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences

MMSys '19: Proceedings of the 10th ACM Multimedia Systems Conference

June 2019

374 pages

ISBN:9781450362979

DOI:10.1145/3304109

General Chair:
Michael Zink
University of Massachusetts Amherst
,
Program Chairs:
Laura Toni
University College London
,
Ali C. Begen
Özyeğin University/Networked Media

Copyright © 2019 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGMM: ACM Special Interest Group on Multimedia

In-Cooperation

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 18 June 2019

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

Deutsche Telekom AG

Conference

MMSys '19

Sponsor:

SIGMM

MMSys '19: 10th ACM Multimedia Systems Conference

June 18 - 21, 2019

Massachusetts, Amherst

Acceptance Rates

MMSys '19 Paper Acceptance Rate 40 of 82 submissions, 49%;

Overall Acceptance Rate 176 of 530 submissions, 33%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

35
Total Citations
View Citations
460
Total Downloads

Downloads (Last 12 months)23
Downloads (Last 6 weeks)0

Reflects downloads up to 09 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Liu JMa YZhao QYu YChang QFang Y(2023)Research on Compressed sensing and Transmission Methods of Video Resources Based on Virtual Reality2023 4th International Conference on Computer Engineering and Intelligent Control (ICCEIC)10.1109/ICCEIC60201.2023.10426627(171-175)Online publication date: 20-Oct-2023
https://doi.org/10.1109/ICCEIC60201.2023.10426627
van der Hooft JAmirpour HVega MSanchez YSchatz RSchierl TTimmerer C(2023)A Tutorial on Immersive Video Delivery: From Omnidirectional Video to HolographyIEEE Communications Surveys & Tutorials10.1109/COMST.2023.326325225:2(1336-1375)Online publication date: Oct-2024
https://doi.org/10.1109/COMST.2023.3263252
Göring SRao RRaake A(2023)Quality assessment of higher resolution images and videos with remote testingQuality and User Experience10.1007/s41233-023-00055-68:1Online publication date: 13-Apr-2023
https://doi.org/10.1007/s41233-023-00055-6
Singla ASingla A(2023)Simulator Sickness Assessment for $$360^\circ $$ VideosAssessment of Visual Quality and Simulator Sickness for Omnidirectional Videos10.1007/978-3-031-49988-3_6(85-96)Online publication date: 11-Nov-2023
https://doi.org/10.1007/978-3-031-49988-3_6
Singla ASingla A(2023)Visual Quality Assessment for $$360^\circ $$ VideosAssessment of Visual Quality and Simulator Sickness for Omnidirectional Videos10.1007/978-3-031-49988-3_5(65-84)Online publication date: 11-Nov-2023
https://doi.org/10.1007/978-3-031-49988-3_5
Singla ASingla A(2023)Technical SetupAssessment of Visual Quality and Simulator Sickness for Omnidirectional Videos10.1007/978-3-031-49988-3_3(29-49)Online publication date: 11-Nov-2023
https://doi.org/10.1007/978-3-031-49988-3_3
Wong EWahab NSaeed FAlharbi N(2022)360-Degree Video Bandwidth Reduction: Technique and Approaches Comprehensive ReviewApplied Sciences10.3390/app1215758112:15(7581)Online publication date: 28-Jul-2022
https://doi.org/10.3390/app12157581
KOIKE MURATA YYAMAGISHI K(2022)Bitstream-Quality-Estimation Model for Tile-Based VR Video Streaming ServicesIEICE Transactions on Communications10.1587/transcom.2021EBP3163E105.B:8(1002-1013)Online publication date: 1-Aug-2022
https://doi.org/10.1587/transcom.2021EBP3163
Liu DWang YZhao CLu YHu Y(2022)An Investigation of User Experiences on VR Videos in a Home EnvironmentProceedings of the 2022 7th International Conference on Multimedia and Image Processing10.1145/3517077.3517085(49-53)Online publication date: 14-Jan-2022
https://dl.acm.org/doi/10.1145/3517077.3517085
Fan CHung THsu C(2022)Modeling the User Experience of Watching 360° Videos with Head-Mounted DisplaysACM Transactions on Multimedia Computing, Communications, and Applications10.1145/346382518:1(1-23)Online publication date: 27-Jan-2022
https://dl.acm.org/doi/10.1145/3463825
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents