Journal Article2023Cited by 17

Multiagent Best Routing in High-Mobility Digital-Twin-Driven Internet of Vehicles

Md Zahangir Alam, Komal S Khan, Abbas Jamalipour

IEEE IoT Journal

Institute of Electrical and Electronics Engineers (IEEE), Vol. 11, Issue 8, pp. 13708 - 13721, ISBN: 2327-4662

  • Industry, Innovation and Infrastructure — Relevance: 64%
DOI: 10.1109/jiot.2023.3338020URL

Abstract

Low-delay high-gain optimal multi-hop routing path is crucial to guarantee both the latency and reliability requirements for infotainment services in the high mobility internet of vehicles (IoVs) subject to queue stability. The high mobility in multi-hop IoVs reduces reliability and energy efficiency, and becomes bottleneck for the optimal route solution using classical optimization methods. To a great extent, deep reinforcement learning (DRL)-based method is not applicable in IoVs environment because of the continuously changing topology and space complexity, which grows exponentially with the number of state variables as well as the relaying hops. Usually, in multi-hop scenario, network reliability and latency are affected by mobility as well as average hop count, which limit the vehicle-to-vehicle (V2V) link connectivity. To cope with this problem, in this paper, we formulate a minimum hop count delay-sensitive buffer-aided optimization problem in a dynamic complex multi-hop vehicular topology using a digital twin-enabled dynamic coordination graph (DCG). Particularly, for the first time, a DCG-based multi-agent deep deterministic policy gradient (DCG-MADDPG) decentralized algorithm is proposed that combines the advantage of DCG and MADDPG to model continuously changing topology and find the optimal routing solutions by cooperative learning in the aforementioned communications. The proposed DCG-MADDPG coordinated learning trains each agent towards highly reliable and low latency optimal decision-making path solutions while maintaining queue stability and convergence on the way to a desired state. Experimental results reveal that the proposed coordinated learning algorithm outperforms the existing learning in terms of energy consumption and latency at less computational complexity.

Keywords

  • digital twin
  • queue stability
  • multi-hop routing
  • MADDPG
  • infotainment services
  • delay-sensitive routing
  • cooperative multi-agent learning
  • energy efficiency
  • high mobility
  • dynamic topology

References

  1. 1.Peter Grossman. (2009). Introduction to algorithms. , 1–14[10.1007/978-0-230-37405-8_1]
  2. 2.Wiering, Marco, Otterlo, Martijn. (2012). Reinforcement Learning. Adaptation, learning, and optimization[10.1007/978-3-642-27645-3]
  3. 3.Suzhi Bi, Liang Huang, Hui Wang, Ying-Jun Angela Zhang. (2021). Lyapunov-Guided Deep Reinforcement Learning for Stable Online Computation Offloading in Mobile-Edge Computing Networks. IEEE Transactions on Wireless Communications, 20(11), 7519–7537[10.1109/twc.2021.3085319]
  4. 4.Ke Zhang, Jiayu Cao, Yan Zhang. (2021). Adaptive Digital Twin and Multiagent Deep Reinforcement Learning for Vehicular Edge Computing and Networks. IEEE Transactions on Industrial Informatics, 18(2), 1405–1413[10.1109/tii.2021.3088407]
  5. 5.Lei Liu, Ming Zhao, Miao Yu, Mian Ahmad Jan, Dapeng Lan, Amirhosein Taherkordi. (2022). Mobility-Aware Multi-Hop Task Offloading for Autonomous Driving in Vehicular Edge Computing and Networks. IEEE Transactions on Intelligent Transportation Systems, 1–14[10.1109/tits.2022.3142566]
  6. 6.Yueyue Dai, Ke Zhang, Sabita Maharjan, Yan Zhang. (2020). Deep Reinforcement Learning for Stochastic Computation Offloading in Digital Twin Networks. IEEE Transactions on Industrial Informatics, 17(7), 4968–4977[10.1109/tii.2020.3016320]
  7. 7.Salman Raza, Shangguang Wang, Manzoor Ahmed, Muhammad Rizwan Anwar, Muhammad Ayzed Mirza, Wali Ullah Khan. (2021). Task Offloading and Resource Allocation for IoV Using 5G NR-V2X Communication. IEEE Internet of Things Journal, 9(13), 10397–10410[10.1109/jiot.2021.3121796]
  8. 8.Chao Yu, Xin Wang, Xin Xu, Minjie Zhang, Hongwei Ge, Jiankang Ren, Liang Sun, Bingcai Chen, Guozhen Tan. (2019). Distributed Multiagent Coordinated Learning for Autonomous Driving in Highways Based on Dynamic Coordination Graphs. IEEE Transactions on Intelligent Transportation Systems, 21(2), 735–748[10.1109/tits.2019.2893683]
  9. 9.Md. Zahangir Alam, Abbas Jamalipour. (2022). Multi-Agent DRL-Based Hungarian Algorithm (MADRLHA) for Task Offloading in Multi-Access Edge Computing Internet of Vehicles (IoVs). IEEE Transactions on Wireless Communications, 21(9), 7641–7652[10.1109/twc.2022.3160099]
  10. 10.Mohammad Parvini, Mohammad Reza Javan, Nader Mokari, Bijan Abbasi, Eduard A. Jorswieck. (2023). AoI-Aware Resource Allocation for Platoon-Based C-V2X Networks via Multi-Agent Multi-Task Reinforcement Learning. IEEE Transactions on Vehicular Technology, 72(8), 9880–9896[10.1109/tvt.2023.3259688]
  11. 11.Jie Tian, Qianqian Liu, Haixia Zhang, Dalei Wu. (2021). Multiagent Deep-Reinforcement-Learning-Based Resource Allocation for Heterogeneous QoS Guarantees for Vehicular Networks. IEEE Internet of Things Journal, 9(3), 1683–1695[10.1109/jiot.2021.3089823]
  12. 12.Yu Gong, Yifei Wei, Zhiyong Feng, F. Richard Yu, Yan Zhang. (2022). Resource Allocation for Integrated Sensing and Communication in Digital Twin Enabled Internet of Vehicles. IEEE Transactions on Vehicular Technology, 72(4), 4510–4524[10.1109/tvt.2022.3228583]
  13. 13.Zipeng Li, Lin Xiang, Xiaohu Ge, Guoqiang Mao, Han-Chieh Chao. (2020). Latency and Reliability of mmWave Multi-Hop V2V Communications Under Relay Selections. IEEE Transactions on Vehicular Technology, 69(9), 9807–9821[10.1109/tvt.2020.3002903]
  14. 14.Liwei Geng, Hongbo Zhao, Jiayue Wang, Aryan Kaushik, Shuai Yuan, Wenquan Feng. (2023). Deep-Reinforcement-Learning-Based Distributed Computation Offloading in Vehicular Edge Computing Networks. IEEE Internet of Things Journal, 10(14), 12416–12433[10.1109/jiot.2023.3247013]
  15. 15.Anitha Saravana Kumar, Lian Zhao, Xavier Fernando. (2023). Task Offloading and Resource Allocation in Vehicular Networks: A Lyapunov-Based Deep Reinforcement Learning Approach. IEEE Transactions on Vehicular Technology, 72(10), 13360–13373[10.1109/tvt.2023.3271613]
  16. 16.Zhenchuan Sun, Yijun Mo, Chen Yu. (2021). Graph-Reinforcement-Learning-Based Task Offloading for Multiaccess Edge Computing. IEEE Internet of Things Journal, 10(4), 3138–3150[10.1109/jiot.2021.3123822]
  17. 17.Guifu Ma, Xiaowei Wang, Manjiang Hu, Wenjie Ouyang, Xiaolong Chen, Yang Li. (2022). DRL-Based Computation Offloading With Queue Stability for Vehicular-Cloud-Assisted Mobile Edge Computing Systems. IEEE Transactions on Intelligent Vehicles, 8(4), 2797–2809[10.1109/tiv.2022.3225147]
  18. 18.Xiao Du, Ting Wang, Qiang Feng, Chenhui Ye, Tao Tao, Lu Wang, Yuanming Shi, Mingsong Chen. (2022). Multi-Agent Reinforcement Learning for Dynamic Resource Management in 6G in-X Subnetworks. IEEE Transactions on Wireless Communications, 22(3), 1900–1914[10.1109/twc.2022.3207918]
  19. 19.Muhammad Awais Javed, Tu N. Nguyen, Jawad Mirza, Junaid Ahmed, Bakhtiar Ali. (2022). Reliable Communications for Cybertwin-Driven 6G IoVs Using Intelligent Reflecting Surfaces. IEEE Transactions on Industrial Informatics, 18(11), 7454–7462[10.1109/tii.2022.3151773]
  20. 20.Yejun He, Mengna Yang, Zhou He, Mohsen Guizani. (2022). Resource Allocation Based on Digital Twin-Enabled Federated Learning Framework in Heterogeneous Cellular Network. IEEE Transactions on Vehicular Technology, 72(1), 1149–1158[10.1109/tvt.2022.3205778]
  21. 21.Homa Maleki, Mehmet Başaran, Lütfiye Durak-Ata. (2023). Handover-Enabled Dynamic Computation Offloading for Vehicular Edge Computing Networks. IEEE Transactions on Vehicular Technology, 72(7), 9394–9405[10.1109/tvt.2023.3247889]
  22. 22.Bo Fan, Zixun Su, Yanyan Chen, Yuan Wu, Chengzhong Xu, Tony Q. S. Quek. (2022). Ubiquitous Control Over Heterogeneous Vehicles: A Digital Twin Empowered Edge AI Approach. IEEE Wireless Communications, 30(1), 166–173[10.1109/mwc.012.2100587]
  23. 23.Ying Ju, Haoyu Wang, Yuchao Chen, Tong-Xing Zheng, Qingqi Pei, Jinhong Yuan, Naofal Al-Dhahir. (2023). Deep Reinforcement Learning Based Joint Beam Allocation and Relay Selection in mmWave Vehicular Networks. IEEE Transactions on Communications, 71(4), 1997–2012[10.1109/tcomm.2023.3240754]
  24. 24.Yunlong Lu, Bo Ai, Zhangdui Zhong, Yan Zhang. (2022). Energy-Efficient Task Transfer in Wireless Computing Power Networks. IEEE Internet of Things Journal, 10(11), 9353–9365[10.1109/jiot.2022.3223690]
  25. 25.Yilong Hui, Gaosheng Zhao, Changle Li, Nan Cheng, Zhisheng Yin, Tom H. Luan, Xiao Xiao. (2022). Digital Twins Enabled On-Demand Matching for Multi-Task Federated Learning in HetVNets. IEEE Transactions on Vehicular Technology, 72(2), 2352–2364[10.1109/tvt.2022.3211005]
  26. 26.Zhen Gao, Lei Yang, Yu Dai. (2023). Fast Adaptive Task Offloading and Resource Allocation in Large-Scale MEC Systems via Multiagent Graph Reinforcement Learning. IEEE Internet of Things Journal, 11(1), 758–776[10.1109/jiot.2023.3285950]
  27. 27.Armeline D. Mafuta, Bodhaswar T. J. Maharaj, Attahiru S. Alfa. (2022). Decentralized Resource Allocation-Based Multiagent Deep Learning in Vehicular Network. IEEE Systems Journal, 17(1), 87–98[10.1109/jsyst.2022.3163235]
  28. 28.Farooque Hassan Kumbhar, Soo Young Shin. (2021). Novel Vehicular Compatibility-Based <i>Ad Hoc</i> Message Routing Scheme in the Internet of Vehicles Using Machine Learning. IEEE Internet of Things Journal, 9(4), 2817–2828[10.1109/jiot.2021.3093545]
  29. 29.Allan M. de Souza, Horacio F. Oliveira, Zhongliang Zhao, Torsten Braun, Antonio A.F. Loureiro, Leandro A. Villas. (2020). Enhancing Sensing and Decision-Making of Automated Driving Systems With Multi-Access Edge Computing and Machine Learning. IEEE Intelligent Transportation Systems Magazine, 14(1), 44–56[10.1109/mits.2019.2953513]
  30. 30.Liyan Sui, Xin Guan, Chen Cui, Haiyang Jiang, Heng Pan, Tomoaki Ohtsuki. (2022). Graph Learning Empowered Situation Awareness in Internet of Energy With Graph Digital Twin. IEEE Transactions on Industrial Informatics, 19(5), 7268–7277[10.1109/tii.2022.3227641]
  31. 31.Nan Geng, Qinbo Bai, Chenyi Liu, Tian Lan, Vaneet Aggarwal, Yuan Yang, Mingwei Xu. (2023). A Reinforcement Learning Framework for Vehicular Network Routing Under Peak and Average Constraints. IEEE Transactions on Vehicular Technology, 72(5), 6753–6764[10.1109/tvt.2023.3235946]
  32. 32.Chang-Sik Choi, François Baccelli. (2023). LOS Coverage Area in Vehicular Networks with Cox-Distributed Roadside Units and Relays. IEEE Transactions on Vehicular Technology, 72(6), 7772–7782[10.1109/tvt.2023.3238730]
  33. 33.Farooque Hassan Kumbhar, Soo Young Shin. (2022). Innovating Multi-Objective Optimal Message Routing for Unified High Mobility Networks. IEEE Transactions on Vehicular Technology, 72(5), 6571–6583[10.1109/tvt.2022.3232567]
  34. 34.Md Zahangir Alam, Forough Shirin Abkenar, Iwan Adhicandra, Sarumathi Murali, Abbas Jamalipour. (2020). Low-Delay Path Selection for Cluster-Based Buffer-Aided Vehicular Communications. IEEE Transactions on Vehicular Technology, 69(9), 9356–9363[10.1109/tvt.2020.2976926]
  35. 35.Xinchen Lyu, Chenyu Zhang, Chenshan Ren, Yanzhao Hou. (2022). Distributed Graph-Based Optimization of Multicast Data Dissemination for Internet of Vehicles. IEEE Transactions on Intelligent Transportation Systems, 24(3), 3117–3128[10.1109/tits.2022.3226198]
  36. 36.Zhijuan Li, Yanbin Wang, Jing Zhao. (2022). Reliability Evaluation of IEEE 802.11p Broadcast Ad Hoc Networks on the Highway. IEEE Transactions on Vehicular Technology, 71(7), 7428–7444[10.1109/tvt.2022.3167095]
  37. 37.Fenghui Zhang, Michael Mao Wang, Ruilong Deng, Xinsheng Zhao. (2022). High-Reliability and Low-Energy Sensor Sharing in Vehicle Platoon Based on Multihop Millimeter-Wave Communication. IEEE Internet of Things Journal, 9(19), 18514–18526[10.1109/jiot.2022.3161088]
  38. 38.Mulei Ma, Chenyu Gong, Liantao Wu, Yang Yang. (2022). FLIRRAS: Fast Learning With Integrated Reward and Reduced Action Space for Online Multitask Offloading. IEEE Internet of Things Journal, 10(6), 5406–5417[10.1109/jiot.2022.3222295]
  39. 39.Gaosheng Zhao, Yilong Hui, Changle Li, Nan Cheng, Zhisheng Yin, Xiao Xiao, Tom H. Luan. (2023). When Autonomous Vehicles Meet Accidents: A DT-Enabled Post-Accident Maintenance Scheme. IEEE Internet of Things Journal, 10(24), 22233–22245[10.1109/jiot.2023.3303490]

Cited By

  1. 1.Xingchi Chen, Xiaoyang Lu, Qing Li, Dazhou Li, Fa Zhu. (2024). Integration of LLM and Human–AI Coordination for Power Dispatching With Connected Electric Vehicles Under SAGVNs. IEEE Transactions on Vehicular Technology, 74(2), 1992–2002[10.1109/tvt.2024.3434969]
  2. 2.Xiaohui Gu, Wei Duan, Guoan Zhang, Jia Hou, Limei Peng, Miaowen Wen, Feifei Gao, Min Chen, Pin-Han Ho. (2025). Digital Twin Technology for Intelligent Vehicles and Transportation Systems: A Survey on Applications, Challenges and Future Directions. IEEE Communications Surveys & Tutorials, 28, 3235–3271[10.1109/comst.2025.3581152]
  3. 3.Reza Poorzare, Dimitris N. Kanellopoulos, Varun Kumar Sharma, Poulami Dalapati, Oliver P. Waldhorst. (2025). Network Digital Twin Toward Networking, Telecommunications, and Traffic Engineering: A Survey. IEEE Access, 13, 16489–16538[10.1109/access.2025.3531947]
  4. 4.Yixin He, Fanghui Huang, Dawei Wang, Bin Chen, Tao Li, Ruonan Zhang. (2025). Performance Analysis and Optimization Design of AAV-Assisted Vehicle Platooning in NOMA-Enhanced Internet of Vehicles. IEEE Transactions on Intelligent Transportation Systems, 26(6), 8810–8819[10.1109/tits.2025.3542402]
  5. 5.Ping Lang, Daxin Tian, Xu Han, Peiyu Zhang, Xuting Duan, Jianshan Zhou, Victor C.M. Leung. (2024). Towards 6G vehicular networks: Vision, technologies, and open challenges. Computer Networks, 257, 110916[10.1016/j.comnet.2024.110916]
  6. 6.Md Zahangir Alam, Suryaia Rahman, Md Asif Bin Khaled, Ashraful Islam, Abbas Jamalipour. (2025). A Graph-Assisted Digital-Twin-Driven Multiagent Shared Offloading for Internet of Vehicles. IEEE Internet of Things Journal, 12(11), 17349–17363[10.1109/jiot.2025.3538657]
  7. 7.Selvi M, Mayur K. L, Santhosh Kumar SVN, Thangaramya K. (2025). A comprehensive review on authentication, threats and privacy preserving challenges for securing smart transportation infrastructure. Peer-to-Peer Networking and Applications, 18(6)[10.1007/s12083-025-02125-2]
  8. 8.Xubin Wu, Guoqiang Wu, Xuewei Zhang, Qiliang Yang, Liqiang Xie. (2026). Hierarchical Decoupling Digital Twin Modeling Method for Topological Systems: A Case Study of Water Purification Systems. Technologies, 14(1), 42[10.3390/technologies14010042]
  9. 9.Wenjun Xu, Hao Wei, Fengyu Wang, Wanli Ni, Ping Zhang. (2025). SemAudio: Semantic Communication for Audio Streaming Transmission. IEEE Transactions on Vehicular Technology, 75(2), 2411–2425[10.1109/tvt.2025.3598973]
  10. 10.Mitra Valizadeh, Miadreza Shafie-khah, Iman Pishkar, Reza Kia, Pegah Mahoor. (2026). Digital twins for energy management: bridging virtual intelligence and physical systems. Applied Energy, 417, 128050[10.1016/j.apenergy.2026.128050]
  11. 11.Alex Edwin Guillen Bonilla, Kerly Ángela Álvarez Cadena, Denisse Paola Guillen Martínez, Jaime David Sevilla Carrasco. (2025). Tendencias del Aprendizaje Cooperativo en la enseñanza de Ciencias Experimentales: Un Enfoque Bibliométrico. Estudios y Perspectivas Revista Científica y Académica, 4(4), 1940–1959[10.61384/r.c.a..v4i4.771]
  12. 12.Amir Alavi, Hamidreza Alavi. (2026). Digital Twin-Enabled Vehicle Routing in Logistics and Transportation. Sensors, 26(19), 5986[10.3390/s26195986]
  13. 13.Yan Geng. (2026). Construction of Autonomous Collaborative Resource Control System for Electric Vehicles Based on Intelligent Sensing Internet of Vehicles. International Journal of Intelligent Transportation Systems Research[10.1007/s13177-026-00684-z]
  14. 14.Wei Feng, Wenyang Gao, Jianping Yao, Longyu Zhou, Chenggang Yan, Tony Q. S. Quek. (2026). Prediction-Assisted Multi-UAV Online Service Migration and Trajectory Control for MEC-Empowered Vehicular Networks. IEEE Transactions on Mobile Computing, 25(10), 18299–18312[10.1109/tmc.2026.3700894]
  15. 15.Hanzhen Chen, Xing Tang, Chuhao Fu, Jing Wang, Siwei Wei, Jiaojiao Yu, Guangli Xiang. (2026). Intersection-based routing with spectrum awareness for cognitive Internet of Vehicles. Vehicular Communications, 60, 101048[10.1016/j.vehcom.2026.101048]
  16. 16.Daqian Liu, Xiaoyong Shi, Yuntao Shi, Zhenwu Lei. (2025). HNGC: Hierarchical Network Graph Clustering Scheme to Improve V2V Multihop Routing in Urban mmWave IoV. IEEE Internet of Things Journal, 13(4), 7272–7287[10.1109/jiot.2025.3640081]
  17. 17.Yitao Zhao, Jiahao Li, Xin Shen, Chuanxu Chen. (2025). Distributed bottleneck game-based routing in application-driven network architecture. Journal of Computational Methods in Sciences and Engineering[10.1177/14727978251366508]
← All publications