More Web Proxy on the site http://driver.im/

research-article

Optimal Volumetric Video Streaming With Hybrid Saliency Based Tiling

Authors:

Han HuAuthors Info & Claims

IEEE Transactions on Multimedia, Volume 25

Pages 2939 - 2953

https://doi.org/10.1109/TMM.2022.3153208

Published: 01 January 2023 Publication History

Abstract

Volumetric video enables a six-degree-of-freedom (6DoF) immersive viewing experience and has a wide range of applications in entertainment and education, among others. Most existing approaches to volumetric video streaming are extensions of VR video streaming solutions that do not take into account user behavior and the properties of the video during the tiling process, and the complexity of decoding is high. To this end, we study volumetric video streaming in this paper and address the research questions mentioned above. In particular, we first propose a hybrid visual saliency and hierarchical clustering empowered 3D tiling scheme that better matches the user’s field of view (FoV). Then, we build a quality of experience (QoE) model considering the volumetric video features as the optimization objective. In addition to the usual encoded version, we introduce the reconstructed version (i.e., decoded version, which allows the user to skip the decoding process and thus reduces the decoding overhead) and propose a joint computational and communication resource allocation scheme to achieve a trade-off between communication and computational resources to maximize the QoE. We perform exhaustive simulations and build a prototype system to verify the performance of the proposed tiling and transmission scheme. The results show that the proposed tiling and transmission scheme performs significantly better than the comparison schemes.

References

[1]

A. Clemm, M. T. Vega, H. K. Ravuri, T. Wauters, and F. D. Turck, “Toward truly immersive holographic-type communication: Challenges and solutions,” IEEE Commun. Mag., vol. 58, no. 1, pp. 93–99, Jan. 2020.

Digital Library

[2]

Z. Liu et al., “Point cloud video streaming: Challenges and solutions,” IEEE Netw., vol. 35, no. 5, pp. 202–209, Sep./Oct. 2021.

Digital Library

[3]

B. Han, Y. Liu, and F. Qian, “ViVo: Visibility-aware mobile volumetric video streaming,” in Proc. 26th Annu. Int. Conf. Mobile Comput. Netw., 2020, pp. 1–13.

[4]

K. Lee, J. Yi, Y. Lee, S. Choi, and Y. M. Kim, “GROOT: A real-time streaming system of high-fidelity volumetric videos,” in Proc. 26th Annu. Int. Conf. Mobile Comput. Netw., 2020, pp. 1–14.

[5]

A. Javaheri, C. Brites, F. M. B. Pereira, and J. M. Ascenso, “Point cloud rendering after coding: Impacts on subjective and objective quality,” IEEE Trans. Multimedia, vol. 23, pp. 4049–4064, 2021.

[6]

Z. Liu et al., “Fuzzy logic-based adaptive point cloud video streaming,” IEEE Open J. Comput. Soc., vol. 1, pp. 121–130, 2020.

[7]

Y. Guo et al., “Deep learning for 3D point clouds: A survey,” IEEE Trans. Pattern Anal. Mach. Intell., vol. 43, no. 12, pp. 4338–4364, Dec. 2021.

Digital Library

[8]

K. Long, Y. Cui, C. Ye, and Z. Liu, “Optimal wireless streaming of multi-quality 360 VR video by exploiting natural, relative smoothness-enabled, and transcoding-enabled multicast opportunities,” IEEE Trans. Multimedia, vol. 23, pp. 3670–3683, 2021.

Digital Library

[9]

S. Schwarz et al., “Emerging MPEG standards for point cloud compression,” IEEE Trans. Emerg. Sel. Topics Circuits Syst., vol. 9, no. 1, pp. 133–148, Mar. 2019.

[10]

E. d’Eon, B. Harrison, T. Myers, and P. A. Chou, “8i voxelized full bodies-a voxelized point cloud dataset,” in Proc. ISO/IEC JTC1/SC29 Joint WG11/WG1 (MPEG/JPEG) Input Document WG11M40059/WG1M74006, Geneva, Jan. 2017.

[11]

Google, “Draco: 3D data compression,” 2018, Accessed: Feb. 26, 2022. [Online]. Available: http://github.com/google/draco

[12]

L. Wang, C. Li, W. Dai, J. Zou, and H. Xiong, “QoE-driven and tile-based adaptive streaming for point clouds,” in Proc. ICASSP IEEE Int. Conf. Acoust., Speech, Signal Process., 2021, pp. 1930–1934.

[13]

J. Li, C. Zhang, Z. Liu, W. Sun, and Q. Li, “Joint communication and computational resource allocation for QoE-driven point cloud video streaming,” in Proc. ICC IEEE Int. Conf. Commun., 2020, pp. 1–6, ISSN: 1938–1883.

[14]

H. Zhang et al., “DeepQoE: A multimodal learning framework for video quality of experience (QoE) prediction,” IEEE Trans. Multimedia, vol. 22, no. 12, pp. 3210–3223, 2020.

Digital Library

[15]

G. Meynet, J. Digne, and G. Lavoué, “PC-MSDM: A quality metric for 3D point clouds,” in Proc. 11th Int. Conf. Qual. Multimedia Experience, 2019, pp. 1–3.

[16]

J. van der Hooft et al., “Objective and subjective QoE evaluation for adaptive point cloud streaming,” in Proc. 12th Int. Conf. Qual. Multimedia Experience, 2020, pp. 1–6.

[17]

V. F. Figueiredo, G. L. Sandri, R. L. de Queiroz, and P. A. Chou, “Saliency maps for point clouds,” in Proc. IEEE 22nd Int. Workshop Multimedia Signal Process., 2020, pp. 1–5.

[18]

M. Abid, M. P. D. Silva, and P. L. Callet, “Towards visual saliency computation on 3D graphical contents for interactive visualization,” in Proc. IEEE Int. Conf. Image Process., 2020, pp. 3448–3452.

[19]

X. Ding, W. Lin, Z. Chen, and X. Zhang, “Point cloud saliency detection by local and global feature fusion,” IEEE Trans. Image Process., vol. 28, no. 11, pp. 5379–5393, Nov. 2019.

Digital Library

[20]

J. Yun and J. Sim, “Supervoxel-based saliency detection for large-scale colored 3D point clouds,” in Proc. IEEE Int. Conf. Image Process., 2016, pp. 4062–4066.

[21]

F. P. Tasse, J. Kosinka, and N. Dodgson, “Cluster-based point set saliency,” in Proc. IEEE Int. Conf. Comput. Vis., 2015, pp. 163–171.

[22]

B. K. Horn and B. G. Schunck, “Determining optical flow,” Artif. Intell., vol. 17, no. 1-3, pp. 185–203, 1981.

[23]

A. Dewan, T. Caselitz, G. D. Tipaldi, and W. Burgard, “Rigid scene flow for 3D lidar scans,” in Proc. IEEE/RSJ Int. Conf. Intell. Robots Syst., 2016, pp. 1765–1770.

[24]

X. Liu, C. R. Qi, and L. J. Guibas, “FlowNet3D: Learning scene flow in 3D point clouds,” in Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit., 2019, pp. 529–537.

[25]

A. Behl, D. Paschalidou, S. Donné, and A. Geiger, “PointflowNet: Learning representations for rigid motion estimation from point clouds,” in Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit., 2019, pp. 7954–7963.

[26]

C. Dorea, E. M. Hung, and R. L. de Queiroz, “Local texture and geometry descriptors for fast block-based motion estimation of dynamic voxelized point clouds,” in Proc. IEEE Int. Conf. Image Process., 2019, pp. 3721–3725.

[27]

A. L. Souto, R. L. de Queiroz, and C. Dorea, “A 3D motion vector database for dynamic point clouds,” Aug. 2020, arXiv:2008.08438.

[28]

M. Hosseini and C. Timmerer, “Dynamic adaptive point cloud streaming,” in Proc. 23rd Packet Video Workshop, 2018, pp. 25–30.

Digital Library

[29]

J. van der Hooft, T. Wauters, F. De Turck, C. Timmerer, and H. Hellwagner, “Towards 6DoF HTTP adaptive streaming through point cloud compression,” in Proc. 27th ACM Int. Conf. Multimedia, 2019, pp. 2405–2413.

[30]

J. Park, P. A. Chou, and J.-N. Hwang, “Rate-utility optimized streaming of volumetric media for augmented reality,” IEEE Trans. Emerg. Sel. Topics Circuits Syst., vol. 9, no. 1, pp. 149–162, Mar. 2019.

[31]

F. Qian, B. Han, J. Pair, and V. Gopalakrishnan, “Toward practical volumetric video streaming on commodity smartphones,” in Proc. 20th Int. Workshop Mobile Comput. Syst. Appl., 2019, pp. 135–140.

Digital Library

[32]

T. Fujihashi, T. Koike-Akino, T. Watanabe, and P. V. Orlik, “Holocast : Hybrid digital-analog transmission for graceful point cloud delivery with graph fourier transform,” IEEE Trans. Multimedia, to be published.

[33]

S. Gül, D. Podborski, T. Buchholz, T. Schierl, and C. Hellge, “Low-latency cloud-based volumetric video streaming using head motion prediction,” in Proc. 30th ACM Workshop Netw. Operating Syst. Support Digit. Audio Video, 2020, pp. 27–33.

Digital Library

[34]

S. Subramanyam, I. Viola, A. Hanjalic, and P. Cesar, “User centered adaptive streaming of dynamic point clouds with low complexity tiling,” in Proc. 28th ACM Int. Conf. Multimedia, 2020, pp. 3669–3677.

Digital Library

[35]

E. Shtrom, G. Leifman, and A. Tal, “Saliency detection in large point sets,” in Proc. IEEE Int. Conf. Comput. Vis., 2013, pp. 3591–3598.

[36]

S. Salti, F. Tombari, and L. D. Stefano, “Shot: Unique signatures of histograms for surface and texture description,” Comput. Vis. Image Understanding, vol. 125, pp. 251–264, 2014.

[37]

A. Maximo, R. Patro, A. Varshney, and R. Farias, “A robust and rotationally invariant local surface descriptor with applications to non-local mesh processing,” Graphical Models, vol. 73, no. 5, pp. 231–242, 2011.

Digital Library

[38]

R. B. Rusu, N. Blodow, and M. Beetz, “Fast point feature histograms (FPFH) for 3D registration,” in Proc. IEEE Int. Conf. Robot. Automat., 2009, pp. 3212–3217.

[39]

A. F. R. Guarda, N. M. M. Rodrigues, and F. Pereira, “Constant size point cloud clustering: A compact, non-overlapping solution,” IEEE Trans. Multimedia, vol. 23, pp. 77–91, 2021.

[40]

J. L. Bentley, “Multidimensional binary search trees used for associative searching,” Commun. ACM, vol. 18, no. 9, pp. 509–517, Sep. 1975.

Digital Library

[41]

X. Yin, A. Jindal, V. Sekar, and B. Sinopoli, “A control-theoretic approach for dynamic adaptive video streaming over http,” SIGCOMM Comput. Commun. Rev., vol. 45, no. 4, pp. 325–338, Aug. 2015.

Digital Library

[42]

R. M. Kolpakov and M. A. Posypkin, “Upper and lower bounds for the complexity of the branch and bound method for the knapsack problem,” Discrete Math. Appl., vol. 20, no. 1, pp. 95–112, 2010.

[43]

J. Zheng, T. Ji, M. Li, Q. Wu, and P. Wu, “Constrained optimization applying decomposed unlimited point method based on KKT condition,” in Proc. 5th Comput. Sci. Electron. Eng. Conf., 2013, pp. 87–91.

[44]

D. Raca, D. Leahy, C.J. Sreenan, and J. J. Quinlan, “Beyond throughput, the next generation: A 5G dataset with channel and context metrics,” in Proc. 11th ACM Multimedia Syst. Conf., 2020, pp. 303–308.

[45]

J. Li, X. Wang, Z. Liu, and Q. Li, “A QoE model in point cloud video streaming,” 2022, arXiv:2111.02985.

Cited By

Zhong JZhang HJia QWu JWang PWang HLiu LZhang XGuo Z(2024)Low-bitrate Volumetric Video Streaming with Depth ImageProceedings of the 2024 SIGCOMM Workshop on Emerging Multimedia Systems10.1145/3672196.3673397(39-44)Online publication date: 4-Aug-2024
https://dl.acm.org/doi/10.1145/3672196.3673397
Gurel ZZengin ABegen AAhsan SKondrad LKammachi-Sreedhar KGül SIllahi GCurcio I(2024)V2RAProceedings of the 16th International Workshop on Immersive Mixed and Virtual Environment Systems10.1145/3652212.3652226(50-56)Online publication date: 15-Apr-2024
https://dl.acm.org/doi/10.1145/3652212.3652226
Shi YClement BOoi W(2024)QV4Proceedings of the 15th ACM Multimedia Systems Conference10.1145/3625468.3647619(144-154)Online publication date: 15-Apr-2024
https://dl.acm.org/doi/10.1145/3625468.3647619
Show More Cited By

Recommendations

Design of Finite-Length Precoded EWF Codes for Scalable Video Streaming

Expanding window fountain (EWF) codes, which can provide unequal erasure protection property, are used as an efficient application-layer forward error correction solution for scalable multimedia data transmission over packet networks. Similar to Raptor ...
Adaptive Streaming of Visual Volumetric Video-based Coding Media
MMVE '23: Proceedings of the 15th International Workshop on Immersive Mixed and Virtual Environment Systems

High-quality 3D point clouds have recently emerged as an advanced representation of immersive media, enabling new forms of interaction in virtual environments and augmented reality applications. A point cloud consists of a set of points represented in ...
Volumetric Video Use Cases for XR Immersive Streaming
ICEMT '24: Proceedings of the 2024 8th International Conference on Education and Multimedia Technology

This paper proposes a summarized analyses for volumetric video streaming use cases and its applicability for immersive extended reality applications. It also closely evaluates and illustrates the usability of standardized technologies for one of the use ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image IEEE Transactions on Multimedia

IEEE Transactions on Multimedia Volume 25, Issue

2023

8932 pages

ISSN:1520-9210

Issue’s Table of Contents

1520-9210 © 2022 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See https://www.ieee.org/publications/rights/index.html for more information.

Publisher

IEEE Press

Publication History

Published: 01 January 2023

Qualifiers

Research-article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

12
Total Citations
View Citations
0
Total Downloads

Downloads (Last 12 months)0
Downloads (Last 6 weeks)0

Reflects downloads up to 11 Dec 2024

Other Metrics

View Author Metrics

Citations

Cited By

Zhong JZhang HJia QWu JWang PWang HLiu LZhang XGuo Z(2024)Low-bitrate Volumetric Video Streaming with Depth ImageProceedings of the 2024 SIGCOMM Workshop on Emerging Multimedia Systems10.1145/3672196.3673397(39-44)Online publication date: 4-Aug-2024
https://dl.acm.org/doi/10.1145/3672196.3673397
Gurel ZZengin ABegen AAhsan SKondrad LKammachi-Sreedhar KGül SIllahi GCurcio I(2024)V2RAProceedings of the 16th International Workshop on Immersive Mixed and Virtual Environment Systems10.1145/3652212.3652226(50-56)Online publication date: 15-Apr-2024
https://dl.acm.org/doi/10.1145/3652212.3652226
Shi YClement BOoi W(2024)QV4Proceedings of the 15th ACM Multimedia Systems Conference10.1145/3625468.3647619(144-154)Online publication date: 15-Apr-2024
https://dl.acm.org/doi/10.1145/3625468.3647619
Liang ZLiu JDasari MWang F(2024)Fumos: Neural Compression and Progressive Refinement for Continuous Point Cloud Video StreamingIEEE Transactions on Visualization and Computer Graphics10.1109/TVCG.2024.337209630:5(2849-2859)Online publication date: 4-Mar-2024
https://dl.acm.org/doi/10.1109/TVCG.2024.3372096
Huang YBai BZhu YQiao XSu XYang LZhang P(2024)ISCom: Interest-Aware Semantic Communication Scheme for Point Cloud Video Streaming on Metaverse XR DevicesIEEE Journal on Selected Areas in Communications10.1109/JSAC.2023.334543042:4(1003-1021)Online publication date: 1-Apr-2024
https://dl.acm.org/doi/10.1109/JSAC.2023.3345430
Zou JWang KZhang KKassim M(2024)Perspective of virtual machine consolidation in cloud computing: a systematic surveyTelecommunications Systems10.1007/s11235-024-01184-987:2(257-285)Online publication date: 1-Oct-2024
https://dl.acm.org/doi/10.1007/s11235-024-01184-9
Ravuri HStruye Jvan der Hooft JWauters TDe Turck FFamaey JTorres Vega M(2024)Context-Aware and Reliable Transport Layer Framework for Interactive Immersive Media Delivery Over Millimeter WaveJournal of Network and Systems Management10.1007/s10922-024-09845-532:4Online publication date: 15-Aug-2024
https://dl.acm.org/doi/10.1007/s10922-024-09845-5
Liu JWang YWang YWang YCui SWang FDasari MJiang JGorlatova M(2023)Mobile Volumetric Video Streaming System through Implicit Neural RepresentationProceedings of the 2023 Workshop on Emerging Multimedia Systems10.1145/3609395.3610593(1-7)Online publication date: 10-Sep-2023
https://dl.acm.org/doi/10.1145/3609395.3610593
Hu KYang HJin YLiu JChen YZhang MWang FEl Saddik AMei TCucchiara RBertini MTobon Vallejo DAtrey PHossain M(2023)Understanding User Behavior in Volumetric Video Watching: Dataset, Analysis and PredictionProceedings of the 31st ACM International Conference on Multimedia10.1145/3581783.3613810(1108-1116)Online publication date: 26-Oct-2023
https://dl.acm.org/doi/10.1145/3581783.3613810
Ran YZhang THuang WXia SLuo JCosta XHassanieh HAsadi ACox LPerino DWidmer JGiustiniano D(2023)iSAW: Intelligent Super-Resolution-Assisted Adaptive WebRTC Video StreamingProceedings of the 29th Annual International Conference on Mobile Computing and Networking10.1145/3570361.3614072(1-3)Online publication date: 2-Oct-2023
https://dl.acm.org/doi/10.1145/3570361.3614072
Show More Cited By

View Options

View options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Media

Figures

Other

Tables

View Issue’s Table of Contents