Human-Aware Dynamic Hierarchical Network Control for Distributed Metaverse Services
Pages 629 - 642
Abstract
Metaverse has emerged as a revolutionary technique for transforming the way people interact with digital content, which relies on a distributed computing and communication infrastructure, encompassing terminal users, edge servers, and cloud servers. However, the rapid evolution of the Metaverse presents challenges that surpass the capabilities of existing communication and network infrastructures, particularly on network bandwidth and latency. Additionally, human experience becomes a critical factor in this domain. Therefore, we introduce a human-aware hierarchical software defined network (SDN) architecture consisting of a Metaverse cloud layer, a mobile edge computing (MEC) server empowered edge layer, and a distributed terminal layer. Each MEC server dynamically controls a multi-antenna base station (BS) and several reconfigurable intelligent surfaces (RISs) according to the terminal immersive experience requirements in real-time. To overcome the bandwidth limitation, we propose a novel smart reconfigurable spatial reuse new radio in unlicensed spectrum (NR-U) framework, which can realize customizable communications through flexibly and coordinately reconfiguring beams among the coordination between BSs and RISs. The objective function is formulated as a Lyapunov optimization based decentralized partially-observable Markov decision process (Dec-POMDP) problem to maximize the spectral efficiency while guaranteeing the latency and reliability requirements in Metaverse, via a joint user selection, phase-shift control, and beam coordination strategy. To solve the above non-convex, strongly coupled, and mixed integer nonlinear programming (MINLP), we propose a novel multi-agent hierarchical deep reinforcement learning (MAHDRL) algorithm that integrates deep Q-network (DQN) to solve discrete problems, deep deterministic policy gradient (DDPG) to solve continuous problems, and mixing network to capture complex interactions between multiple agents. Numerical results demonstrate the effectiveness of the proposed algorithm and verify the performance improvements compared to traditional multi-agent deep reinforcement learning (MADRL) algorithms.
References
[1]
Y. Wanget al., “A survey on metaverse: Fundamentals, security, and privacy,” IEEE Commun. Surveys Tuts., vol. 25, no. 1, pp. 319–352, 1st Quart., 2023.
[2]
Y. Fu, C. Li, F. R. Yu, T. H. Luan, P. Zhao, and S. Liu, “A survey of blockchain and intelligent networking for the metaverse,” IEEE Internet Things J., vol. 10, no. 4, pp. 3587–3610, Feb. 2023.
[3]
H. Du, B. Ma, D. Niyato, J. Kang, Z. Xiong, and Z. Yang, “Rethinking quality of experience for metaverse services: A consumer-based economics perspective,” IEEE Netw., early access, Feb. 8, 2023. 10.1109/MNET.131.2200503.
[4]
W. Y. B. Limet al., “Realizing the metaverse with edge intelligence: A match made in heaven,” IEEE Wireless Commun., vol. 30, no. 4, pp. 64–71, Aug. 2023.
[5]
Y. Jianget al., “Reliable distributed computing for metaverse: A hierarchical game-theoretic approach,” IEEE Trans. Veh. Technol., vol. 72, no. 1, pp. 1084–1100, Jan. 2023.
[6]
Y. Hanet al., “A dynamic hierarchical framework for IoT-assisted digital twin synchronization in the metaverse,” IEEE Internet Things J., vol. 10, no. 1, pp. 268–284, Jan. 2023.
[7]
D. Van Huynh, S. R. Khosravirad, A. Masaracchia, O. A. Dobre, and T. Q. Duong, “Edge intelligence-based ultra-reliable and low-latency communications for digital twin-enabled metaverse,” IEEE Wireless Commun. Lett., vol. 11, no. 8, pp. 1733–1737, Aug. 2022.
[8]
H. Duet al., “Exploring attention-aware network resource allocation for customized metaverse services,” IEEE Netw., early access, Dec. 26, 2022. 10.1109/MNET.128.2200338.
[9]
R. Liu, Q. Chen, G. Yu, G. Y. Li, and Z. Ding, “Resource management in LTE-U systems: Past, present, and future,” IEEE Open J. Veh. Technol., vol. 1, pp. 1–17, 2020.
[10]
S. Lagenet al., “New radio beam-based access to unlicensed spectrum: Design challenges and solutions,” IEEE Commun. Surveys Tuts., vol. 22, no. 1, pp. 8–37, 1st Quart., 2020.
[11]
C. Chen, R. Ratasuk, and A. Ghosh, “Downlink performance analysis of LTE and WiFi coexistence in unlicensed bands with a simple listen-before-talk scheme,” in Proc. IEEE 81st Veh. Technol. Conf. (VTC Spring), May 2015, pp. 1–5.
[12]
G. Geraci, A. Garcia-Rodriguez, D. López-Pérez, A. Bonfante, L. Galati Giordano, and H. Claussen, “Operating massive MIMO in unlicensed bands for enhanced coexistence and spatial reuse,” IEEE J. Sel. Areas Commun., vol. 35, no. 6, pp. 1282–1293, Jun. 2017.
[13]
Q. Chen, K. Yang, H. Jiang, and M. Qiu, “Joint beamforming coordination and user selection for CoMP-enabled NR-U networks,” IEEE Internet Things J., vol. 9, no. 16, pp. 14530–14541, Aug. 2022.
[14]
X. Xu, Q. Chen, H. Jiang, and J. Huang, “Millimeter-wave NR-U and WiGig coexistence: Joint user grouping, beam coordination, and power control,” IEEE Trans. Wireless Commun., vol. 21, no. 4, pp. 2352–2367, Apr. 2022.
[15]
E. Basar, M. Di Renzo, J. De Rosny, M. Debbah, M.-S. Alouini, and R. Zhang, “Wireless communications through reconfigurable intelligent surfaces,” IEEE Access, vol. 7, pp. 116753–116773, 2019.
[16]
Q. Wu and R. Zhang, “Intelligent reflecting surface enhanced wireless network via joint active and passive beamforming,” IEEE Trans. Wireless Commun., vol. 18, no. 11, pp. 5394–5409, Nov. 2019.
[17]
M. Hua, Q. Wu, D. W. K. Ng, J. Zhao, and L. Yang, “Intelligent reflecting surface-aided joint processing coordinated multipoint transmission,” IEEE Trans. Commun., vol. 69, no. 3, pp. 1650–1665, Mar. 2021.
[18]
H. Guo, Y.-C. Liang, J. Chen, and E. G. Larsson, “Weighted sum-rate maximization for reconfigurable intelligent surface aided wireless networks,” IEEE Trans. Wireless Commun., vol. 19, no. 5, pp. 3064–3076, May 2020.
[19]
X. Xu, Q. Chen, X. Mu, Y. Liu, and H. Jiang, “Graph-embedded multi-agent learning for smart reconfigurable THz MIMO-NOMA networks,” IEEE J. Sel. Areas Commun., vol. 40, no. 1, pp. 259–275, Jan. 2022.
[20]
P. Wang, J. Fang, X. Yuan, Z. Chen, and H. Li, “Intelligent reflecting surface-assisted millimeter wave communications: Joint active and passive precoding design,” IEEE Trans. Veh. Technol., vol. 69, no. 12, pp. 14960–14973, Dec. 2020.
[21]
M. D. Renzoet al., “Smart radio environments empowered by reconfigurable AI meta-surfaces: An idea whose time has come,” EURASIP J. Wireless Commun. Netw., vol. 2019, no. 1, pp. 1–20, Dec. 2019.
[22]
T. Hou, Y. Liu, Z. Song, X. Sun, and Y. Chen, “MIMO-NOMA networks relying on reconfigurable intelligent surface: A signal cancellation-based design,” IEEE Trans. Commun., vol. 68, no. 11, pp. 6932–6944, Nov. 2020.
[23]
W. Shi, X. Zhou, L. Jia, Y. Wu, F. Shu, and J. Wang, “Enhanced secure wireless information and power transfer via intelligent reflecting surface,” IEEE Commun. Lett., vol. 25, no. 4, pp. 1084–1088, Apr. 2021.
[24]
Y. Zhang, M. Qiu, C.-W. Tsai, M. M. Hassan, and A. Alamri, “Health-CPS: Healthcare cyber-physical system assisted by cloud and big data,” IEEE Syst. J., vol. 11, no. 1, pp. 88–95, Mar. 2017.
[25]
N. S. Perovic, L.-N. Tran, M. Di Renzo, and M. F. Flanagan, “Achievable rate optimization for MIMO systems with reconfigurable intelligent surfaces,” IEEE Trans. Wireless Commun., vol. 20, no. 6, pp. 3865–3882, Jun. 2021.
[26]
H. Duet al., “Attention-aware resource allocation and QoE analysis for metaverse xURLLC services,” IEEE J. Sel. Areas Commun., vol. 41, no. 7, pp. 2158–2175, Jul. 2023.
[27]
S. Bouchard, J. St-Jacques, G. Robillard, and P. Renaud, “Anxiety increases the feeling of presence in virtual reality,” Presence, vol. 17, no. 4, pp. 376–391, Aug. 2008.
[28]
Z. Long, H. Dong, and A. E. Saddik, “Human-centric resource allocation for the metaverse with multi-access edge computing,” IEEE Internet Things J., early access.
[29]
C. Amato, G. Chowdhary, A. Geramifard, N. K Üre, and M. J. Kochenderfer, “Decentralized control of partially observable Markov decision processes,” in Proc. 52nd IEEE Conf. Decis. Control, Dec. 2013, pp. 2398–2405.
[30]
M. J. Neely, “Stochastic network optimization with application to communication and queueing systems,” Synth. Lectures Commun. Netw., vol. 3, no. 1, pp. 1–211, Jan. 2010.
[31]
V. Mnihet al., “Playing Atari with deep reinforcement learning,” 2013, arXiv:1312.5602.
[32]
V. Mnihet al., “Human-level control through deep reinforcement learning,” Nature, vol. 518, no. 7540, pp. 529–533, 2015.
[33]
D. Silver, G. Lever, N. Heess, T. Degris, D. Wierstra, and M. Riedmiller, “Deterministic policy gradient algorithms,” in Proc. Conf. Mach. Learn., 2014, pp. 387–395.
[34]
T. P. Lillicrapet al., “Continuous control with deep reinforcement learning,” 2015, arXiv:1509.02971.
[35]
J. Xionget al., “Parametrized deep Q-networks learning: Reinforcement learning with discrete-continuous hybrid action space,” 2018, arXiv:1810.06394.
[36]
L. Busoniu, R. Babuska, and B. De Schutter, “A comprehensive survey of multiagent reinforcement learning,” IEEE Trans. Syst., Man, Cybern., C, vol. 38, no. 2, pp. 156–172, Mar. 2008.
[37]
Y. Wu, T. Zhao, H. Yan, M. Liu, and N. Liu, “Hierarchical hybrid multi-agent deep reinforcement learning for peer-to-peer energy trading among multiple heterogeneous microgrids,” IEEE Trans. Smart Grid, vol. 14, no. 6, pp. 4649–4665, Nov. 2023.
[38]
J. Xu, X. Kang, R. Zhang, Y.-C. Liang, and S. Sun, “Optimization for master-UAV-powered auxiliary-aerial-IRS-assisted IoT networks: An option-based multi-agent hierarchical deep reinforcement learning approach,” IEEE Internet Things J., vol. 9, no. 22, pp. 22887–22902, Nov. 2022.
[39]
R. Lowe, Y. I. Wu, A. Tamar, J. Harb, O. P. Abbeel, and I. Mordatch, “Multi-agent actor-critic for mixed cooperative-competitive environments,” in Proc. Adv. Neural Inf. Process. Syst. (NeurIPS), vol. 30, 2017, pp. 1–12.
[40]
T. Rashid, M. Samvelyan, C. Schroeder, G. Farquhar, J. Foerster, and S. Whiteson, “QMIX: Monotonic value function factorisation for deep multi-agent reinforcement learning,” in Proc. Conf. Mach. Learn., 2018, pp. 4295–4304.
[41]
Part 11: Wireless Lan Medium Access Control (MAC) and Physical Layer (PHY) Specifications, IEEE Standard, 2012.
[42]
M. Qiu and E. Sha, “Cost minimization while satisfying hard/soft timing constraints for heterogeneous embedded systems,” ACM Trans. Des. Autom. Electr. Syst., vol. 14, no. 25, pp. 1–30, Apr. 2009.
Index Terms
- Human-Aware Dynamic Hierarchical Network Control for Distributed Metaverse Services
Index terms have been assigned to the content through auto-classification.
Recommendations
Automated control for SLA-aware elastic clouds
FeBiD '10: Proceedings of the Fifth International Workshop on Feedback Control Implementation and Design in Computing Systems and NetworksAlthough Cloud Computing provides a means to support remote, on-demand access top a set of computing resources, its ad-hoc management for quality-of-service and SLA poses significant challenges to the performance, availability and economical costs of ...
Dynamic Migration of Cloud Services
NCCA '14: Proceedings of the 2014 IEEE 3rd Symposium on Network Cloud Computing and Applications (ncca 2014)In the last couple of years, cloud computing records a rapidly increasing popularity due to scalable, elastic infrastructures that are charged according to their usage and do not have to be maintained. However, the offerings of cloud providers sometimes ...
QoS-Aware Task Offloading in Distributed Cloudlets with Virtual Network Function Services
MSWiM '17: Proceedings of the 20th ACM International Conference on Modelling, Analysis and Simulation of Wireless and Mobile SystemsPushing the cloud frontier to the network edge has attracted tremendous interest not only from cloud operators of the IT service/software industry but also from network service operators that provide various network services for mobile users. In ...
Comments
Please enable JavaScript to view thecomments powered by Disqus.Information & Contributors
Information
Published In
0733-8716 © 2024 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See https://www.ieee.org/publications/rights/index.html for more information.
Publisher
IEEE Press
Publication History
Published: 05 February 2024
Qualifiers
- Research-article
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 0Total Downloads
- Downloads (Last 12 months)0
- Downloads (Last 6 weeks)0
Reflects downloads up to 09 Jan 2025