More Web Proxy on the site http://driver.im/

research-article

Streaming Video QoE Modeling and Prediction: A Long Short-Term Memory Approach

Authors:

Nagabhushan Eswara,

Anand Panchbhai,

Soumen Chakraborty,

Hemanth P. Sethuram,

Sumohana S. ChannappayyaAuthors Info & Claims

IEEE Transactions on Circuits and Systems for Video Technology, Volume 30, Issue 3

Pages 661 - 673

https://doi.org/10.1109/TCSVT.2019.2895223

Published: 01 March 2020 Publication History

Abstract

Due to the rate adaptation in hypertext transfer protocol adaptive streaming, the video quality delivered to the client keeps varying with time depending on the end-to-end network conditions. Moreover, the varying network conditions could also lead to the video client running out of the playback content resulting in rebuffering events. These factors affect the user satisfaction and cause degradation of the user quality of experience (QoE). Hence, it is important to quantify the perceptual QoE of the streaming video users and to monitor the same in a continuous manner so that the QoE degradation can be minimized. However, the continuous evaluation of QoE is challenging as it is determined by complex dynamic interactions among the QoE influencing factors. Toward this end, we present long short-term memory (LSTM)-QoE, a recurrent neural network-based QoE prediction model using an LSTM network. The LSTM-QoE is a network of cascaded LSTM blocks to capture the nonlinearities and the complex temporal dependencies involved in the time-varying QoE. Based on an evaluation over several publicly available continuous QoE datasets, we demonstrate that the LSTM-QoE has the capability to model the QoE dynamics effectively. We compare the proposed model with the state-of-the-art QoE prediction models and show that it provides an excellent performance across these datasets. Furthermore, we discuss the state space perspective for the LSTM-QoE and show the efficacy of the state space modeling approaches for the QoE prediction.

References

[1]

“Cisco visual networking index: Global mobile data traffic forecast update, 2016–2021,” Cisco, San Jose, CA, USA, White Paper 1454457600805266, 2017.

[2]

I. Sodagar, “The MPEG-DASH standard for multimedia streaming over the Internet,” IEEE Multimedia, vol. 18, no. 4, pp. 62–67, Apr. 2011.

Digital Library

[3]

V. Joseph and G. de Veciana, “NOVA: QoE-driven optimization of DASH-based video delivery in networks,” in Proc. IEEE Conf. Comput. Commun. (INFOCOM), Apr./May 2014, pp. 82–90.

[4]

N. Eswaraet al., “A continuous QoE evaluation framework for video streaming over HTTP,” IEEE Trans. Circuits Syst. Video Technol., vol. 28, no. 11, pp. 3236–3250, Nov. 2018.

Digital Library

[5]

C. G. Bampis, Z. Li, and A. C. Bovik, “Continuous prediction of streaming video QoE using dynamic networks,” IEEE Signal Process. Lett., vol. 24, no. 7, pp. 1083–1087, Jul. 2017.

[6]

D. Ghadiyaram, J. Pan, and A. C. Bovik, “Learning a continuous-time streaming video QoE model,” IEEE Trans. Image Process., vol. 27, no. 5, pp. 2257–2271, May 2018.

[7]

C. G. Bampis, Z. Li, A. K. Moorthy, I. Katsavounidis, A. Aaron, and A. C. Bovik, “Study of temporal effects on subjective video quality of experience,” IEEE Trans. Image Process., vol. 26, no. 11, pp. 5217–5231, Nov. 2017.

Digital Library

[8]

P. Reichl, S. Egger, R. Schatz, and A. D’Alconzo, “The logarithmic nature of QoE and the role of the weber-fechner law in QoE assessment,” in Proc. IEEE Int. Conf. Commun. (ICC), May 2010, pp. 1–5.

[9]

M. Fiedler, T. Hoßfeld, and P. Tran-Gia, “A generic quantitative relationship between quality of experience and quality of service,” IEEE Netw., vol. 24, no. 2, pp. 36–41, Mar./Apr. 2010.

Digital Library

[10]

R. K. P. Mok, E. W. W. Chan, and R. K. C. Chang, “Measuring the quality of experience of HTTP video streaming,” in Proc. 12th IFIP/IEEE Int. Symp. Integr. Netw. Manage. (IM) Workshops, May 2011, pp. 485–492.

[11]

O. Oyman and S. Singh, “Quality of experience for HTTP adaptive streaming services,” IEEE Commun. Mag., vol. 50, no. 4, pp. 20–27, Apr. 2012.

[12]

A. Balachandran, V. Sekar, A. Akella, S. Seshan, I. Stoica, and H. Zhang, “Developing a predictive model of quality of experience for Internet video,” ACM SIGCOMM Comput. Commun. Rev., vol. 43, no. 4, pp. 339–350, Aug. 2013.

Digital Library

[13]

T. De Pessemier, K. De Moor, W. Joseph, L. De Marez, and L. Martens, “Quantifying the influence of rebuffering interruptions on the user’s quality of experience during mobile video watching,” IEEE Trans. Broadcast., vol. 59, no. 1, pp. 47–61, Mar. 2013.

[14]

T. Hoßfeld, R. Schatz, E. Biersack, and L. Plissonneau, Data Traffic Monitoring and Analysis: From Measurement, Classification, and Anomaly Detection to Quality of Experience. Berlin, Germany: Springer, 2013, pp. 264–301.

[15]

D. Ghadiyaram, J. Pan, and A. C. Bovik, “A time-varying subjective quality model for mobile streaming videos with stalling events,” Proc. SPIE, vol. 9599, p. 959911, Sep. 2015.

[16]

N. Eswara, S. Channappayya, A. Kumar, and K. Kuchi, “eTVSQ based video rate adaptation in cellular networks with $\alpha$ -fair resource allocation,” in Proc. IEEE Wireless Commun. Netw. Conf. (WCNC), Apr. 2016, pp. 1–6.

[17]

K. Zeng, H. Yeganeh, and Z. Wang, “Quality-of-experience of streaming video: Interactions between presentation quality and playback stalling,” in Proc. IEEE Int. Conf. Image Process. (ICIP), Sep. 2016, pp. 2405–2409.

[18]

C. Chen, L. K. Choi, G. de Veciana, C. Caramanis, R. W. Heath, Jr., and A. C. Bovik, “Modeling the time—Varying subjective quality of HTTP video streams with rate adaptations,” IEEE Trans. Image Process., vol. 23, no. 5, pp. 2206–2221, May 2014.

[19]

K. Seshadrinathan and A. C. Bovik, “Motion tuned spatio-temporal quality assessment of natural videos,” IEEE Trans. Image Process., vol. 19, no. 2, pp. 335–350, Feb. 2010.

Digital Library

[20]

P. V. Vu, C. T. Tu, and D. M. Chandler, “A spatiotemporal most-apparent-distortion model for video quality assessment,” in Proc. 18th IEEE Int. Conf. Image Process. (ICIP), Sep. 2011, pp. 2505–2508.

[21]

R. Soundararajan and A. C. Bovik, “Video quality assessment by reduced reference spatio-temporal entropic differencing,” IEEE Trans. Circuits Syst. Video Technol., vol. 23, no. 4, pp. 684–694, Apr. 2013.

Digital Library

[22]

M. A. Saad, A. C. Bovik, and C. Charrier, “Blind prediction of natural video quality,” IEEE Trans. Image Process., vol. 23, no. 3, pp. 1352–1365, Mar. 2014.

Digital Library

[23]

A. Mittal, M. A. Saad, and A. C. Bovik, “A completely blind video integrity oracle,” IEEE Trans. Image Process., vol. 25, no. 1, pp. 289–300, Jan. 2016.

Digital Library

[24]

K. Manasa and S. S. Channappayya, “An optical flow-based full reference video quality assessment algorithm,” IEEE Trans. Image Process., vol. 25, no. 6, pp. 2480–2492, Jun. 2016.

Digital Library

[25]

K. Manasa and S. S. Channappayya, “An optical flow-based no-reference video quality assessment algorithm,” in Proc. IEEE Int. Conf. Image Process. (ICIP), Sep. 2016, pp. 2400–2404.

[26]

S. Winkler and P. Mohandas, “The evolution of video quality measurement: From PSNR to hybrid metrics,” IEEE Trans. Broadcast., vol. 54, no. 3, pp. 660–668, Sep. 2008.

[27]

S. Chikkerur, V. Sundaram, M. Reisslein, and L. J. Karam, “Objective video quality assessment methods: A classification, review, and performance comparison,” IEEE Trans. Broadcast., vol. 57, no. 2, pp. 165–182, Jun. 2011.

[28]

Z. Wang, E. P. Simoncelli, and A. C. Bovik, “Multiscale structural similarity for image quality assessment,” in Proc. 27th Asilomar Conf. Signals, Syst. Comput., vol. 2, Nov. 2003, pp. 1398–1402.

[29]

A. Balachandran, V. Sekar, A. Akella, S. Seshan, I. Stoica, and H. Zhang, “A quest for an Internet video quality-of-experience metric,” in Proc. 11th ACM Workshop Hot Topics Netw., Oct. 2012, pp. 97–102.

[30]

D. Ghadiyaram, A. C. Bovik, H. Yeganeh, R. Kordasiewicz, and M. Gallant, “Study of the effects of stalling events on the quality of experience of mobile streaming videos,” in Proc. IEEE Global Conf. Signal Inf. Process. (GlobalSIP), Dec. 2014, pp. 989–993.

[31]

N. Eswara, H. P. Sethuram, S. Chakraborty, K. Kuchi, A. Kumar, and S. Channappayya, “Modeling continuous video QoE evolution: A state space approach,” in Proc. IEEE Int. Conf. Multimedia Expo (ICME), Jul. 2018, pp. 1–6.

[32]

C. G. Bampis, Z. Li, I. Katsavounidis, and A. C. Bovik, “Recurrent and dynamic models for predicting streaming video quality of experience,” IEEE Trans. Image Process., vol. 27, no. 7, pp. 3316–3331, Jul. 2018.

[33]

D. E. Pearson, “Viewer response to time-varying video quality,” Proc. SPIE, vol. 3299, pp. 16–25, Jul. 1998.

[34]

Quality of Experience Requirements for IPTV Services, document ITU-T G.1080, Dec. 2008.

[35]

D. G. Albrecht and W. S. Geisler, “Motion selectivity and the contrast-response function of simple cells in the visual cortex,” Vis. Neurosci., vol. 7, no. 6, pp. 531–546, Dec. 1991.

[36]

K. Seshadrinathan, R. Soundararajan, A. C. Bovik, and L. K. Cormack, “Study of subjective and objective quality assessment of video,” IEEE Trans. Image Process., vol. 19, no. 6, pp. 1427–1441, Jun. 2010.

Digital Library

[37]

K. Seshadrinathan and A. C. Bovik, “Temporal hysteresis model of time varying subjective video quality,” in Proc. IEEE Int. Conf. Acoust., Speech Signal Process. (ICASSP), May 2011, pp. 1153–1156.

[38]

S. Hochreiter and J. Schmidhuber, “Long short-term memory,” Neural Comput., vol. 9, no. 8, pp. 1735–1780, Nov. 1997.

Digital Library

[39]

F. A. Gers, J. Schmidhuber, and F. Cummins, “Learning to forget: Continual prediction with LSTM,” Neural Comput., vol. 12, no. 10, pp. 2451–2471, Oct. 2000.

Digital Library

[40]

A. Graves, Supervised Sequence Labelling with Recurrent Neural Networks vol. 385. Berlin, Germany: Springer-Verlag, 2012. [Online]. Available: https://www.spiedigitallibrary.org/conference-proceedings-of-spie/9599/959911/A-time-varying-subjective-quality-model-for-mobile-streaming-videos/10.1117/12.2188882.short?SSO=1

[41]

J. Donahueet al., “Long-term recurrent convolutional networks for visual recognition and description,” in Proc. IEEE Conf. Comput. Vis. Pattern Recognit. (CVPR), Jun. 2015, pp. 2625–2634.

[42]

K. Xuet al., “Show, attend and tell: Neural image caption generation with visual attention,” in Proc. 32nd Int. Conf. Mach. Learn. (ICML), 2015, pp. 2048–2057.

[43]

T. Mikolov, M. Karafiát, L. Burget, J. Černocký, and S. Khudanpur, “Recurrent neural network based language model,” in Proc. 11th Annu. Conf. Int. Speech Commun. Assoc., vol. 2, 2010, pp. 1045–1048.

[44]

S. Gu, Z. Ghahramani, and R. E. Turner, “Neural adaptive sequential Monte Carlo,” in Proc. Adv. Neural Inf. Process. Syst. (NIPS), 2015, pp. 2629–2637.

[45]

I. Sutskever, O. Vinyals, and Q. V. Le, “Sequence to sequence learning with neural networks,” in Proc. Adv. Neural Inf. Process. Syst. (NIPS), 2014, pp. 3104–3112.

[46]

X. Zheng, M. Zaheer, A. Ahmed, Y. Wang, E. P. Xing, and A. J. Smola. (2017). “State space LSTM models with particle MCMC inference.” [Online]. Available: https://arxiv.org/abs/1711.11179

[47]

D. Ghadiyaram, J. Pan, and A. C. Bovik, “A subjective and objective study of stalling events in mobile streaming videos,” IEEE Trans. Circuits Syst. Video Technol., vol. 29, no. 1, pp. 183–197, Jan. 2019.

Digital Library

[48]

N. Eswaraet al., “A linear regression framework for assessing time-varying subjective quality in HTTP streaming,” in Proc. IEEE Global Conf. Signal Inf. Process. (GlobalSIP), Nov. 2017, pp. 31–35.

[49]

A. Mittal, R. Soundararajan, and A. C. Bovik, “Making a ‘completely blind’ image quality analyzer,” IEEE Signal Process. Lett., vol. 20, no. 3, pp. 209–212, Mar. 2013.

[50]

F. Cholletet al. (2015). Keras. [Online]. Available: https://keras.io

Cited By

Van Damme Svan der Hooft JDe Turck FTorres Vega M(2024)Analyzing the applicability of psychometric QoE modeling for projection-based point cloud video quality assessmentJournal on Image and Video Processing10.1186/s13640-024-00655-y2024:1Online publication date: 19-Nov-2024
https://dl.acm.org/doi/10.1186/s13640-024-00655-y
Jia ZMin XSun WZhai G(2024)Continuous and Overall Quality of Experience Evaluation for Streaming Video Based on Rich Features Exploration and Dual-Stage AttentionIEEE Transactions on Circuits and Systems for Video Technology10.1109/TCSVT.2024.341894134:11_Part_2(11709-11723)Online publication date: 25-Jun-2024
https://dl.acm.org/doi/10.1109/TCSVT.2024.3418941
Zhang YCheng SGuo ZZhang X(2024)Inferring Video Streaming Quality of Real-Time Communication Inside NetworkIEEE Transactions on Circuits and Systems for Video Technology10.1109/TCSVT.2024.337560434:8(7756-7770)Online publication date: 1-Aug-2024
https://dl.acm.org/doi/10.1109/TCSVT.2024.3375604
Show More Cited By

Index Terms

Streaming Video QoE Modeling and Prediction: A Long Short-Term Memory Approach

Index terms have been assigned to the content through auto-classification.

Recommendations

A Short History of Rome
Adaptation of ANN Based Video Stream QoE Prediction Model
Proceedings of the 15th Pacific-Rim Conference on Advances in Multimedia Information Processing --- PCM 2014 - Volume 8879

Pseudo-Subjective Quality Assessment PSQA is an effective way to prediction the Quality of experience QoE of video stream. The ANN-based PSQA model gives a decent QoE prediction accuracy when it is tested under the same condition as training. However, ...
A Short View of the Whole Scripture History

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image IEEE Transactions on Circuits and Systems for Video Technology

IEEE Transactions on Circuits and Systems for Video Technology Volume 30, Issue 3

March 2020

300 pages

ISSN:1051-8215

Issue’s Table of Contents

1051-8215 © 2019 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See https://www.ieee.org/publications/rights/index.html for more information.

Publisher

IEEE Press

Publication History

Published: 01 March 2020

Qualifiers

Research-article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

20
Total Citations
View Citations
0
Total Downloads

Downloads (Last 12 months)0
Downloads (Last 6 weeks)0

Reflects downloads up to 05 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Van Damme Svan der Hooft JDe Turck FTorres Vega M(2024)Analyzing the applicability of psychometric QoE modeling for projection-based point cloud video quality assessmentJournal on Image and Video Processing10.1186/s13640-024-00655-y2024:1Online publication date: 19-Nov-2024
https://dl.acm.org/doi/10.1186/s13640-024-00655-y
Jia ZMin XSun WZhai G(2024)Continuous and Overall Quality of Experience Evaluation for Streaming Video Based on Rich Features Exploration and Dual-Stage AttentionIEEE Transactions on Circuits and Systems for Video Technology10.1109/TCSVT.2024.341894134:11_Part_2(11709-11723)Online publication date: 25-Jun-2024
https://dl.acm.org/doi/10.1109/TCSVT.2024.3418941
Zhang YCheng SGuo ZZhang X(2024)Inferring Video Streaming Quality of Real-Time Communication Inside NetworkIEEE Transactions on Circuits and Systems for Video Technology10.1109/TCSVT.2024.337560434:8(7756-7770)Online publication date: 1-Aug-2024
https://dl.acm.org/doi/10.1109/TCSVT.2024.3375604
Darwich MBayoumi M(2024)Video quality adaptation using CNN and RNN models for cost-effective and scalable video streaming ServicesCluster Computing10.1007/s10586-024-04315-827:5(6355-6375)Online publication date: 1-Aug-2024
https://dl.acm.org/doi/10.1007/s10586-024-04315-8
Li YDeng GBai CYang JWang GZhang HBai JYuan HXu MWang S(2023)Demystifying the QoS and QoE of Edge-hosted Video Streaming Applications in the Wild with SNESetProceedings of the ACM on Management of Data10.1145/36267231:4(1-29)Online publication date: 12-Dec-2023
https://dl.acm.org/doi/10.1145/3626723
Lee KFang JSun YHsu CGutiérrez JFarias MCortés C(2023)Modeling Gamer Quality-of-Experience Using a Real Cloud VR Gaming TestbedProceedings of the 15th International Workshop on Immersive Mixed and Virtual Environment Systems10.1145/3592834.3592877(12-17)Online publication date: 7-Jun-2023
https://dl.acm.org/doi/10.1145/3592834.3592877
Choi WYoon JCosta XAl Hassanieh HAsadi ACox LPerino DWidmer JGiustiniano D(2023)UBR: User-Centric QoE-Based Rate Adaptation for Dynamic Network ConditionsProceedings of the 29th Annual International Conference on Mobile Computing and Networking10.1145/3570361.3615756(1-3)Online publication date: 2-Oct-2023
https://dl.acm.org/doi/10.1145/3570361.3615756
Duanmu ZLiu WChen DLi ZWang ZWang YGao W(2023)A Bayesian Quality-of-Experience Model for Adaptive Streaming VideosACM Transactions on Multimedia Computing, Communications, and Applications10.1145/349143218:3s(1-24)Online publication date: 11-Feb-2023
https://dl.acm.org/doi/10.1145/3491432
Carvalho MMacedo D(2023)Container Scheduling in Co-Located Environments Using QoE AwarenessIEEE Transactions on Network and Service Management10.1109/TNSM.2023.324409020:3(3247-3260)Online publication date: 1-Sep-2023
https://dl.acm.org/doi/10.1109/TNSM.2023.3244090
Feng YZhao G(2022)Implementation of Short Video Click-Through Rate Estimation Model Based on Cross-Media Collaborative Filtering Neural NetworkComputational Intelligence and Neuroscience10.1155/2022/49519122022Online publication date: 1-Jan-2022
https://dl.acm.org/doi/10.1155/2022/4951912
Show More Cited By

View Options

View options

Media

Figures

Other

Tables

View Issue’s Table of Contents