Cited By
View all- Qin BPan HDai YSi XHuang XYuen CZhang Y(2024)Machine and Deep Learning for Digital Twin Networks: A SurveyIEEE Internet of Things Journal10.1109/JIOT.2024.341673311:21(34694-34716)Online publication date: 1-Nov-2024
Human-Autonomy Teaming on Autonomous Vehicles with Large Language Model-Enabled Human Digital Twins
Authors: 
Can Cui, Yunsheng Ma, Xu Cao, Wenqian Ye, Ziran WangAuthors Info & Claims
Pages 319 - 324
Abstract
The development of autonomous vehicles is dramatically reshaping the transportation landscape, bringing new challenges and opportunities in human-machine interaction. As autonomous vehicles evolve, understanding and responding to human intent becomes significant, and therefore require new ways of human-autonomy teaming. A human digital twin (HDT) is a virtual representation of an individual driver, capturing their preferences, behaviors, and physiological states, enabling machines to better understand and predict human needs and responses. In this paper, we explore how large language models (LLMs), like GPT-4 and LLaMA, together with HDTs are changing the way humans team up with autonomous vehicles. These LLMs help make our conversations with vehicles more natural and intuitive. By pairing them in HDTs, we can get real-time feedback and smarter responses. This combination offers not just easier control but also safer driving experiences. We will break down how this works, why it matters, and what we might expect in the future.
References
[1]
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances in neural information processing systems 33 (2020), 1877--1901.
[2]
Arquimedes Canedo. 2016. Industrial IoT lifecycle via digital twins. In Proceedings of the Eleventh IEEE/ACM/IFIP International Conference on Hardware/Software Codesign and System Synthesis. ACM, 29.
[3]
Can Cui, Yunsheng Ma, Xu Cao, Wenqian Ye, and Ziran Wang. 2023. Drive as You Speak: Enabling Human-Like Interaction with Large Language Models in Autonomous Vehicles. arXiv:2309.10228 [cs.HC]
[4]
Can Cui, Yunsheng Ma, Xu Cao, Wenqian Ye, and Ziran Wang. 2023. Receive, Reason, and React: Drive as You Say with Large Language Models in Autonomous Vehicles. arXiv:2310.08034 [cs.HC]
[5]
Can Cui, Yunsheng Ma, Juanwu Lu, and Ziran Wang. 2023. Radar Enlighten the Dark: Enhancing Low-Visibility Perception for Automated Vehicles with Camera-Radar Fusion. IEEE International Conference on Intelligent Transportation Systems (ITSC) (May 2023). arXiv:2305.17318 [cs.CV].
[6]
Nanyi Fei, Zhiwu Lu, Yizhao Gao, Guoxing Yang, Yuqi Huo, Jingyuan Wen, Haoyu Lu, Ruihua Song, Xin Gao, Tao Xiang, et al. 2022. Towards artificial general intelligence via a multimodal foundation model. Nature Communications 13, 1 (2022), 3094.
[7]
Edward Glaessgen and David Stargel. 2012. The digital twin paradigm for future NASA and US Air Force vehicles. In 53rd AIAA/ASME/ASCE/AHS/ASC Structures, Structural Dynamics and Materials Conference 20th AIAA/ASME/AHS Adaptive Structures Conference 14th AIAA. 1818.
[8]
Wenlong Huang, Chen Wang, Ruohan Zhang, Yunzhu Li, Jiajun Wu, and Li Fei-Fei. 2023. VoxPoser: Composable 3D Value Maps for Robotic Manipulation with Language Models. arXiv preprint arXiv:2307.05973 (2023).
[9]
Jacob Devlin Ming-Wei Chang Kenton and Lee Kristina Toutanova. 2019. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In Proceedings of NAACL-HLT. 4171--4186.
[10]
Jinkyu Kim, Teruhisa Misu, Yi-Ting Chen, Ashish Tawari, and John Canny. 2019. Grounding human-to-vehicle advice for self-driving vehicles. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 10591--10599.
[11]
Jinkyu Kim, Anna Rohrbach, Trevor Darrell, John Canny, and Zeynep Akata. 2018. Textual explanations for self-driving vehicles. In Proceedings of the European conference on computer vision (ECCV). 563--578.
[12]
Xishun Liao, Xuanpeng Zhao, Ziran Wang, Zhouqiao Zhao, Kyungtae Han, Rohit Gupta, Matthew J. Barth, and Guoyuan Wu. 2023. Driver Digital Twin for Online Prediction of Personalized Lane Change Behavior. IEEE Internet of Things Journal (2023).
[13]
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. 2023. Visual instruction tuning. arXiv preprint arXiv:2304.08485 (2023).
[14]
Yunsheng Ma, Liangqi Yuan, Amr Abdelraouf, Kyungtae Han, Rohit Gupta, Zihao Li, and Ziran Wang. 2023. M2DAR: Multi-view multi-scale driver action recognition with vision transformer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 5286--5293.
[15]
OpenAI. 2023. GPT-4 Technical Report.
[16]
Roland Rosen, Georg Von Wichert, George Lo, and Kurt D Bettenhausen. 2015. About the importance of autonomy and digital twins for the future of manufacturing. IFAC-PapersOnLine 48, 3 (2015), 567--572.
[17]
Cynthia Rudin. 2019. Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead. Nature machine intelligence 1, 5 (2019), 206--215.
[18]
Timo Schick, Jane Dwivedi-Yu, Roberto Dessì, Roberta Raileanu, Maria Lomeli, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. 2023. Toolformer: Language Models Can Teach Themselves to Use Tools.
[19]
Patrick Schramowski, Cigdem Turan, Nico Andersen, Constantin A Rothkopf, and Kristian Kersting. 2022. Large pre-trained language models contain human-like biases of what is right and wrong to do. Nature Machine Intelligence 4, 3 (2022), 258--268.
[20]
Chris Schwarz and Ziran Wang. 2022. The Role of Digital Twins in Connected and Automated Vehicles. IEEE Intelligent Transportation Systems Magazine (2022), 2--11.
[21]
NN Sriram, Tirth Maniar, Jayaganesh Kalyanasundaram, Vineet Gandhi, Brojeshwar Bhowmick, and K Madhava Krishna. 2019. Talk to the vehicle: Language conditioned autonomous navigation of self driving cars. In 2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). IEEE, 5284--5290.
[22]
Fei Tao, He Zhang, Ang Liu, and Andrew YC Nee. 2018. Digital twin in industry: State-of-the-art. IEEE Transactions on industrial informatics 15, 4 (2018), 2405--2415.
[23]
Ziran Wang, Rohit Gupta, Kyungtae Han, Haoxin Wang, Akila Ganlath, Nejib Ammar, and Prashant Tiwari. 2022. Mobility Digital Twin: Concept, Architecture, Case Study, and Future Challenges. IEEE Internet of Things Journal 9, 18 (Sept. 2022), 17452--17467.
[24]
Ziran Wang, Xishun Liao, Xuanpeng Zhao, Kyungtae Han, Prashant Tiwari, Matthew J Barth, and Guoyuan Wu. 2020. A Digital Twin Paradigm: Vehicle-to-Cloud Based Advanced Driver Assistance Systems. In 2020 IEEE 91st Vehicular Technology Conference. 1--6.
Index Terms
- Human-Autonomy Teaming on Autonomous Vehicles with Large Language Model-Enabled Human Digital Twins
Recommendations
Numerical Analysis of Tractor Accidents using Driving Simulator for Autonomous Driving Tractor
ICMRE'19: Proceedings of the 5th International Conference on Mechatronics and Robotics EngineeringAutonomous driving of automobiles is a hot research topic in recent years. The autonomous driving tractor also has been studied in the agricultural field as well as an autonomous driving automobile. On the other hand, tractor accidents frequently occur ...
Effects of various in-vehicle human–machine interfaces on drivers’ takeover performance and gaze pattern in conditionally automated vehicles
Highlights- A simulated driving experiment was conducted to explore the effects of various HMIs on drivers’ takeover performance and gaze patterns in automated vehicles.
- Peripheral HMIs can improve takeover performance and facilitate gaze patterns ...
AbstractWith the era of automated driving approaching, designing an effective and suitable human–machine interface (HMI) to present takeover requests (TORs) is critical to ensure driving safety. The present study conducted a simulated driving experiment ...
Autonomous Driving: Investigating the Feasibility of Bimodal Take-Over Requests
Autonomous vehicles will need de-escalation strategies to compensate when reaching system limitations. Car-driver handovers can be considered one possible method to deal with system boundaries. The authors suggest a bimodal auditory and visual handover ...
Comments
Please enable JavaScript to view thecomments powered by Disqus.Information & Contributors
Information
Published In
December 2023
405 pages
This work is licensed under a Creative Commons Attribution International 4.0 License.
Sponsors
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 07 August 2024
Check for updates
Author Tags
Qualifiers
- Research-article
Conference
SEC '23
Sponsor:
Acceptance Rates
Overall Acceptance Rate 40 of 100 submissions, 40%
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- View Citations1Total Citations
- 91Total Downloads
- Downloads (Last 12 months)91
- Downloads (Last 6 weeks)22
Reflects downloads up to 20 Dec 2024
Other Metrics
Citations
Cited By
View all- Qin BPan HDai YSi XHuang XYuen CZhang Y(2024)Machine and Deep Learning for Digital Twin Networks: A SurveyIEEE Internet of Things Journal10.1109/JIOT.2024.341673311:21(34694-34716)Online publication date: 1-Nov-2024
View Options
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign in