[go: up one dir, main page]
More Web Proxy on the site http://driver.im/ Skip to main content
Springer Nature Link
Log in

Variable Hybrid Action Space Deep Q-Networks for Optimal Power Allocation and User Association in Heterogeneous Networks

  • Research
  • Published:
Wireless Personal Communications Aims and scope Submit manuscript

Abstract

Heterogeneous networks (HetNets) are essential in contemporary wireless communication networks as they help operators address challenges related to coverage, capacity, and quality. HetNets also enable the efficient use of resources and preparedness for future wireless technologies. The efficient allocation of limited resources to user equipments (UEs) is a critical problem that can be tackled using joint resource allocation and user association (JRAUA) techniques. However, this problem is challenging due to its non-convex and combinatorial nature, which is further complicated by the hybrid action space that involves both continuous actions and discrete actions. Consequently, determining the most efficient approach for JRAUA is a challenging endeavor. The objective of this study is to address the above challenge and improve the energy efficiency of networks by examining the JRAUA problem in downlink HetNets using orthogonal frequency division multiple access (OFDMA). This work presents a new strategy, called the Variable Hybrid Action Space—Deep Q-Network (VHAS-DQN), to optimize the learning policy and improve the average cumulative reward by incorporating constraints like wireless backhaul capacity of each UE and quality-of-service (QoS) into the learning process. To demonstrate the advantages of the JRAUA model in terms of energy efficiency while guaranteeing that the backhaul capacity constraints and quality of service (QoS) criteria are met, the simulation results from the VHAS-DQN were compared with other experimental case studies.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
£29.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price includes VAT (United Kingdom)

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Algorithm 1
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8

Similar content being viewed by others

Availability of Data and Materials

No associated data.

References

  1. Saad, W., Bennis, M., & Chen, M. (2020). A vision of 6G wireless systems: applications, trends, technologies, and open research problems. IEEE Network, 34(3), 134–142. https://doi.org/10.1109/MNET.001.1900287

    Article  Google Scholar 

  2. Dryjanski, M., & Kliks, A. (2020). A hierarchical and modular radio resource management architecture for 5G and beyond. IEEE Communications Magazine, 58(7), 28–34. https://doi.org/10.1109/MCOM.001.1900796

    Article  Google Scholar 

  3. Wu, Q., Li, G. Y., Chen, W., Ng, D. W. K., & Schober, R. (2017). An overview of sustainable green 5G networks. IEEE Wireless Communications, 24(4), 72–80. https://doi.org/10.1109/MWC.2017.1600343

    Article  Google Scholar 

  4. Mili, M. R., Hamdi, K. A., Marvasti, F., & Bennis, M. (2016). Joint optimization for optimal power allocation in OFDMA femtocell networks. IEEE Communications Letters, 20(1), 133–136. https://doi.org/10.1109/LCOMM.2015.2497697

    Article  Google Scholar 

  5. Amiri, R., Almasi, M. A., Andrews, J. G., & Mehrpouyan, H. (2019). Reinforcement learning for self-organization and power control of two-tier heterogeneous networks. IEEE Transactions on Wireless Communications, 18(8), 3933–3947. https://doi.org/10.1109/TWC.2019.2919611

    Article  Google Scholar 

  6. Teng, Y., Liu, M., Yu, F. R., Leung, V. C. M., Song, M., & Zhang, Y. (2019). Resource allocation for ultra-dense networks: A survey, some research issues and challenges. IEEE Communications Surveys and Tutorials, 21(3), 2134–2168. https://doi.org/10.1109/COMST.2018.2867268

    Article  Google Scholar 

  7. Zhou, T., Liu, Z., Zhao, J., Li, C., & Yang, L. (2018). Joint user association and power control for load balancing in downlink heterogeneous cellular networks. IEEE Transactions on Vehicular Technology, 67(3), 2582–2593. https://doi.org/10.1109/TVT.2017.2768574

    Article  Google Scholar 

  8. Wen, Z., Zhu, G., Ni, M., & Lin, S. (2017). User association-based interference management in ultra-dense networks. In 2017 IEEE international symposium on antennas and propagation & USNC/URSI national radio science meeting (pp. 1903–1904). IEEE. https://doi.org/10.1109/APUSNCURSINRSM.2017.8072994

  9. Deng, D., Li, X., Zhao, M., Rabie, K. M., & Kharel, R. (2020). Deep learning-based secure mimo communications with imperfect CSI for heterogeneous networks. Sensors (Switzerland), 20(6), 1–15. https://doi.org/10.3390/s20061730

    Article  Google Scholar 

  10. Liu, S., He, J., & Wu, J. (2021). Dynamic cooperative spectrum sensing based on deep multi-user reinforcement learning. Applied Sciences (Switzerland), 11(4), 1–16. https://doi.org/10.3390/app11041884

    Article  Google Scholar 

  11. Munaye, Y. Y., Juang, R. T., Lin, H. P., Tarekegn, G. B., & Lin, D. B. (2021). Deep reinforcement learning based resource management in UAV-assisted iot networks. Applied Sciences (Switzerland), 11(5), 1–20. https://doi.org/10.3390/app11052163

    Article  Google Scholar 

  12. Meng, F., Chen, P., Wu, L., & Cheng, J. (2020). Power allocation in multi-user cellular networks: Deep reinforcement learning approaches. IEEE Transactions on Wireless Communications, 19(10), 6255–6267. https://doi.org/10.1109/TWC.2020.3001736

    Article  Google Scholar 

  13. Ma, H., Zhang, H., Wang, X., & Cheng, J. (2017). Backhaul-aware user association and resource allocation for massive MIMO-enabled HetNets. IEEE Communications Letters, 21(12), 2710–2713. https://doi.org/10.1109/LCOMM.2017.2755021

    Article  Google Scholar 

  14. Hasan, M. K., Shahjalal, M., Islam, M. M., Alam, M. M., Ahmed, M. F., & Jang, Y. M. (2020). The role of deep learning in NOMA for 5G and beyond communications. 2020 international conference on artificial intelligence in information and communication, ICAIIC 2020 (pp. 303–307). https://doi.org/10.1109/ICAIIC48513.2020.9065219

  15. Nasir, Y. S., & Guo, D. (2019). Multi-agent deep reinforcement learning for dynamic power allocation in wireless networks. IEEE Journal on Selected Areas in Communications, 37(10), 2239–2250. https://doi.org/10.1109/JSAC.2019.2933973

    Article  Google Scholar 

  16. He, C., Hu, Y., Chen, Y., & Zeng, B. (2019). Joint power allocation and channel assignment for NOMA with deep reinforcement learning. IEEE Journal on Selected Areas in Communications, 37(10), 2200–2210. https://doi.org/10.1109/JSAC.2019.2933762

    Article  Google Scholar 

  17. Chen, Y., Li, J., Chen, W., Lin, Z., & Vucetic, B. (2016). Joint user association and resource allocation in the downlink of heterogeneous networks. IEEE Transactions on Vehicular Technology, 65(7), 5701–5706. https://doi.org/10.1109/TVT.2015.2452953

    Article  Google Scholar 

  18. Han, Q., Yang, B., Miao, G., Chen, C., Wang, X., & Guan, X. (2017). Backhaul-aware user association and resource allocation for energy-constrained HetNets. IEEE Transactions on Vehicular Technology, 66(1), 580–593. https://doi.org/10.1109/TVT.2016.2533559

    Article  Google Scholar 

  19. Amiri, R., Mehrpouyan, H., Fridman, L., Mallik, R. K., Nallanathan, A., & Matolak, D. (2018). A machine learning approach for power allocation in HetNets considering QoS. In IEEE international conference on communications (Vol. 2018-May, pp. 1–7). IEEE. https://doi.org/10.1109/ICC.2018.8422864

  20. Ahmed, K. I., & Hossain, E. (2019). A deep Q-learning method for downlink power allocation in multi-cell networks. Retrieved from http://arxiv.org/abs/1904.13032

  21. Xu, Z., Wang, Y., Tang, J., Wang, J., & Gursoy, M. C. (2017). A deep reinforcement learning based framework for power-efficient resource allocation in cloud RANs. In 2017 IEEE International Conference on Communications (ICC) (pp. 1–6). IEEE. https://doi.org/10.1109/ICC.2017.7997286

  22. Wei, Y., Yu, F. R., Song, M., & Han, Z. (2018). User scheduling and resource allocation in HetNets with hybrid energy supply: An actor-critic reinforcement learning approach. IEEE Transactions on Wireless Communications, 17(1), 680–692. https://doi.org/10.1109/TWC.2017.2769644

    Article  Google Scholar 

  23. Li, D., Zhang, H., Long, K., Huangfu, W., Dong, J., & Nallanathan, A. (2019). User association and power allocation based on Q-learning in ultra dense heterogeneous networks. In 2019 IEEE global communications conference, GLOBECOM 2019—proceedings. https://doi.org/10.1109/GLOBECOM38437.2019.9013455

  24. Zhang, Y., Kang, C., Ma, T., Teng, Y., & Guo, D. (2018). Power allocation in multi-cell networks using deep reinforcement learning. In IEEE vehicular technology conference, 2018-Augus (pp. 1–6). https://doi.org/10.1109/VTCFall.2018.8690757

  25. Huang, L., Bi, S., & Zhang, Y. J. A. (2020). Deep reinforcement learning for online computation offloading in wireless powered mobile-edge computing networks. IEEE Transactions on Mobile Computing, 19(11), 2581–2593. https://doi.org/10.1109/TMC.2019.2928811

    Article  Google Scholar 

  26. Yau, K. L. A., Komisarczuk, P., & Teal, P. D. (2012). Reinforcement learning for context awareness and intelligence in wireless networks: Review, new features and open issues. Journal of Network and Computer Applications, 35(1), 253–267. https://doi.org/10.1016/j.jnca.2011.08.007

    Article  Google Scholar 

  27. Asheralieva, A., & Miyanaga, Y. (2016). An autonomous learning-based algorithm for joint channel and power level selection by D2D pairs in heterogeneous cellular networks. IEEE Transactions on Communications, 64(9), 3996–4012. https://doi.org/10.1109/TCOMM.2016.2593468

    Article  Google Scholar 

  28. Ghadimi, E., Davide Calabrese, F., Peters, G., & Soldati, P. (2017). A reinforcement learning approach to power control and rate adaptation in cellular networks. IEEE International Conference on Communications. https://doi.org/10.1109/ICC.2017.7997440

    Article  Google Scholar 

  29. Zhang, T., Zhu, K., & Wang, J. (2021). Energy-efficient mode selection and resource allocation for D2D-enabled heterogeneous networks: A deep reinforcement learning approach. IEEE Transactions on Wireless Communications, 20(2), 1175–1187. https://doi.org/10.1109/TWC.2020.3031436

    Article  Google Scholar 

  30. Li, Z., & Guo, C. (2020). Multi-agent deep reinforcement learning based spectrum allocation for D2D underlay communications. IEEE Transactions on Vehicular Technology, 69(2), 1828–1840. https://doi.org/10.1109/TVT.2019.2961405

    Article  Google Scholar 

  31. Sun, M., Jin, Y., Wang, S., & Mei, E. (2022). Joint deep reinforcement learning and unsupervised learning for channel selection and power control in D2D networks. Entropy, 24(12), 1363–1378. https://doi.org/10.3390/e24121722

    Article  MathSciNet  Google Scholar 

  32. Park, H., & Lim, Y. (2020). Reinforcement learning for energy optimization with 5G communications in vehicular social networks. Sensors, 20(8), 2361. https://doi.org/10.3390/s20082361

    Article  Google Scholar 

  33. Xu, Z., Tang, J., Yin, C., Wang, Y., Xue, G., Wang, J., & Gursoy, M. C. (2022). ReCARL: Resource allocation in cloud RANs with deep reinforcement learning. IEEE Transactions on Mobile Computing, 21(7), 2533–2545. https://doi.org/10.1109/TMC.2020.3044282

    Article  Google Scholar 

  34. Lu, Y., Lu, H., Cao, L., Wu, F., & Zhu, D. (2018). Learning deterministic policy with target for power control in wireless networks. In 2018 IEEE global communications conference, GLOBECOM 2018—Proceedings, (pp. 1–7). https://doi.org/10.1109/GLOCOM.2018.8648056

  35. Khalili, A., Akhlaghi, S., Tabassum, H., & Ng, D. W. K. (2020). Joint user association and resource allocation in the uplink of heterogeneous networks. IEEE Wireless Communications Letters, 9(6), 804–808. https://doi.org/10.1109/LWC.2020.2970696

    Article  Google Scholar 

  36. Jabeen, S., & Ho, P. H. (2019). A benchmark for joint channel allocation and user scheduling in flexible heterogeneous networks. IEEE Transactions on Vehicular Technology, 68(9), 9233–9244. https://doi.org/10.1109/TVT.2019.2930884

    Article  Google Scholar 

  37. Zhang, H., Huang, S., Jiang, C., Long, K., Leung, V. C. M., & Poor, H. V. (2017). Energy efficient user association and power allocation in millimeter-wave-based ultra dense networks with energy harvesting base stations. IEEE Journal on Selected Areas in Communications, 35(9), 1936–1947. https://doi.org/10.1109/JSAC.2017.2720898

    Article  Google Scholar 

  38. Ye, Q., Rong, B., Chen, Y., Al-Shalash, M., Caramanis, C., & Andrews, J. G. (2013). User association for load balancing in heterogeneous cellular networks. IEEE Transactions on Wireless Communications, 12(6), 2706–2716. https://doi.org/10.1109/TWC.2013.040413.120676

    Article  Google Scholar 

  39. Kim, T., & Chang, J. M. (2018). QoS-aware energy-efficient association and resource scheduling for HetNets. IEEE Transactions on Vehicular Technology, 67(1), 650–664. https://doi.org/10.1109/TVT.2017.2737629

    Article  Google Scholar 

  40. Ye, Q., Zhuang, W., Zhang, S., Jin, A. L., Shen, X., & Li, X. (2018). Dynamic radio resource slicing for a two-tier heterogeneous wireless network. IEEE Transactions on Vehicular Technology, 67(10), 9896–9910. https://doi.org/10.1109/TVT.2018.2859740

    Article  Google Scholar 

  41. Luo, X. (2017). Delay-oriented QoS-aware user association and resource allocation in heterogeneous cellular networks. IEEE Transactions on Wireless Communications, 16(3), 1809–1822. https://doi.org/10.1109/TWC.2017.2654458

    Article  Google Scholar 

  42. Asheralieva, A., & Miyanaga, Y. (2017). Optimal contract design for joint user association and intercell interference mitigation in heterogeneous LTE-A Networks with asymmetric information. IEEE Transactions on Vehicular Technology, 66(6), 5284–5300. https://doi.org/10.1109/TVT.2016.2615849

    Article  Google Scholar 

  43. Liu, J., Tao, X., & Lu, J. (2019). Mobility-aware centralized reinforcement learning for dynamic resource allocation in HetNets. In 2019 IEEE Global Communications Conference, GLOBECOM 2019—Proceedings (pp. 1–6). https://doi.org/10.1109/GLOBECOM38437.2019.9013191

  44. De Domenico, A., &Ktenas, D. (2018). Reinforcement learning for interference-aware cell DTX in heterogeneous networks. In IEEE Wireless Communications and Networking Conference, WCNC (pp. 1–6). https://doi.org/10.1109/WCNC.2018.8376993

  45. Yang, H., Alphones, A., Zhong, W. D., Chen, C., & Xie, X. (2020). Learning-Based energy-efficient resource management by heterogeneous RF/VLC for ultra-reliable low-latency industrial IoT networks. IEEE Transactions on Industrial Informatics, 16(8), 5565–5576. https://doi.org/10.1109/TII.2019.2933867

    Article  Google Scholar 

  46. Hsieh, C. K., Chan, K. L., & Chien, F. T. (2021). Energy-efficient power allocation and user association in heterogeneous networks with deep reinforcement learning. Applied Sciences (Switzerland), 11(9), 4135. https://doi.org/10.3390/app11094135

    Article  Google Scholar 

  47. Mohajer, A., Daliri, M. S., Mirzaei, A., Ziaeddini, A., Nabipour, M., & Bavaghar, M. (2022). Heterogeneous computational resource allocation for NOMA: Toward green mobile edge-computing systems. IEEE Transactions on Services Computing, 16(2), 1225–1238.

    Article  Google Scholar 

  48. Dong, S., Zhan, J., Hu, W., Mohajer, A., Bavaghar, M., & Mirzaei, A. (2023). Energy-efficient hierarchical resource allocation in uplink-downlink decoupled NOMA HetNets. IEEE Transactions on Network and Service Management, 20(3), 3380–3395.

    Google Scholar 

  49. Mohajer, A., Sorouri, F., Mirzaei, A., Ziaeddini, A., Rad, K. J., & Bavaghar, M. (2022). Energy-aware hierarchical resource management and backhaul traffic optimization in heterogeneous cellular networks. IEEE Systems Journal, 16(4), 5188–5199.

    Article  Google Scholar 

  50. Hausknecht, M., Stone, P., & Mc, O. P. (2016, July). On-policy vs. off-policy updates for deep reinforcement learning. In Deep reinforcement learning: Frontiers and challenges, IJCAI 2016 Workshop. New York, NY, USA: AAAI Press.

  51. Karandikar, R. L., & Vidyasagar, M. (2021). Convergence of batch asynchronous stochastic approximation with applications to reinforcement learning, 1, 1–28. Retrieved from http://arxiv.org/abs/2109.03445

  52. Zhang, S., & Sutton, R. S. (2017). A deeper look at experience replay. Retrieved from http://arxiv.org/abs/1712.01275

  53. Schaul, T., Quan, J., Antonoglou, I., & Silver, D. (2016). Prioritized experience replay. In 4th International conference on learning representations, ICLR 2016—conference track proceedings (pp. 1–21).

Download references

Funding

No funding.

Author information

Authors and Affiliations

  1. Vasavi College of Engineering, Hyderabad, Telangana, India

    Aruna Valasa

  2. National Institute of Technology, Warangal, Telangana, India

    Anjaneyulu Lokam & Chayan Bhar

Authors
  1. Aruna Valasa
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Anjaneyulu Lokam
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Chayan Bhar
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

Aruna Valasa wrote the manuscript text , Lokam Anjaneyulu and Chayan Bhar reviewed the manuscript.

Corresponding author

Correspondence to Aruna Valasa.

Ethics declarations

Conflict of interest

The authors have no conflict of interest.

Ethical Approval

Ethical approval was not necessary for this study.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Valasa, A., Lokam, A. & Bhar, C. Variable Hybrid Action Space Deep Q-Networks for Optimal Power Allocation and User Association in Heterogeneous Networks. Wireless Pers Commun 136, 233–259 (2024). https://doi.org/10.1007/s11277-024-11255-4

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11277-024-11255-4

Keywords

Search

Navigation