How Does Simulation-Based Testing for Self-Driving Cars Match Human Perception?
Authors: 
Christian Birchler, 
Tanzil Kombarabettu Mohammed, Pooja Rani, Teodora Nechita, Timo Kehrer, 
Sebastiano PanichellaAuthors Info & Claims
Article No.: 42, Pages 929 - 950
Abstract
Software metrics such as coverage or mutation scores have been investigated for the automated quality assessment of test suites. While traditional tools rely on software metrics, the field of self-driving cars (SDCs) has primarily focused on simulation-based test case generation using quality metrics such as the out-of-bound (OOB) parameter to determine if a test case fails or passes. However, it remains unclear to what extent this quality metric aligns with the human perception of the safety and realism of SDCs. To address this (reality) gap, we conducted an empirical study involving 50 participants to investigate the factors that determine how humans perceive SDC test cases as safe, unsafe, realistic, or unrealistic. To this aim, we developed a framework leveraging virtual reality (VR) technologies, called SDC-Alabaster, to immerse the study participants into the virtual environment of SDC simulators. Our findings indicate that the human assessment of safety and realism of failing/passing test cases can vary based on different factors, such as the test's complexity and the possibility of interacting with the SDC. Especially for the assessment of realism, the participants' age leads to a different perception. This study highlights the need for more research on simulation testing quality metrics and the importance of human perception in evaluating SDCs.
References
[1]
Raja Ben Abdessalem, Shiva Nejati, Lionel C. Briand, and Thomas Stifter. 2018. Testing vision-based control systems using learnable evolutionary algorithms. In International Conference on Software Engineering. 1016–1026. https://doi.org/10.1145/3180155.3180160
[2]
Raja Ben Abdessalem, Annibale Panichella, Shiva Nejati, Lionel C. Briand, and Thomas Stifter. 2020. Automated repair of feature interaction failures in automated driving systems. In International Symposium on Software Testing and Analysis. ACM, 88–100. https://doi.org/10.1145/3395363.3397386
[3]
Afsoon Afzal, Deborah S. Katz, Claire Le Goues, and Christopher Steven Timperley. 2020. A Study on the Challenges of Using Robotics Simulators for Testing. arXiv:2004.07368. arxiv:2004.07368
[4]
Afsoon Afzal, Deborah S. Katz, Claire Le Goues, and Christopher Steven Timperley. 2021. Simulation for Robotics Test Automation: Developer Perspectives. In Conference on Software Testing, Verification and Validation. IEEE, 263–274. https://doi.org/10.1109/ICST49551.2021.00036
[5]
Miguel Alcon, Hamid Tabani, Jaume Abella, and Francisco J. Cazorla. 2021. Enabling Unit Testing of Already-Integrated AI Software Systems: The Case of Apollo for Autonomous Driving. In Conference on Digital System Design. IEEE, 426–433. https://doi.org/10.1109/DSD53832.2021.00071
[6]
Aitor Arrieta, Maialen Otaegi, Liping Han, Goiuria Sagardui, Shaukat Ali, and Maite Arratibel. 2022. Automating Test Oracle Generation in DevOps for Industrial Elevators. In International Conference on Software Analysis, Evolution and Reengineering. IEEE, 284–288. https://doi.org/10.1109/SANER53432.2022.00044
[7]
Aitor Arrieta, Shuai Wang, Goiuria Sagardui, and Leire Etxeberria. 2019. Search-Based test case prioritization for simulation-Based testing of cyber-Physical system product lines. J. Syst. Softw., 149 (2019), 1–34. https://doi.org/10.1016/j.jss.2018.09.055
[8]
BBC. 2023. Robots to do 39% of domestic chores by 2033, say experts. https://www.bbc.com/news/technology-64718842 Accessed: 2023-01-04
[9]
BeamNG.tech. [n. d.]. BeamNG.research. https://documentation.beamng.com/beamng_tech/ Accessed: 2022-07-31
[10]
Christian Birchler, Sajad Khatiri, Bill Bosshard, Alessio Gambi, and Sebastiano Panichella. 2023. Machine learning-based test selection for simulation-based testing of self-driving cars software. Empir. Softw. Eng., 28, 3 (2023), 71. https://doi.org/10.1007/s10664-023-10286-y
[11]
Christian Birchler, Sajad Khatiri, Pouria Derakhshanfar, Sebastiano Panichella, and Annibale Panichella. 2023. Single and Multi-objective Test Cases Prioritization for Self-driving Cars in Virtual Environments. ACM Trans. Softw. Eng. Methodol., 32, 2 (2023), 28:1–28:30. https://doi.org/10.1145/3533818
[12]
Christian Birchler, Tanzil Kombarabettu Mohammed, Pooja Rani, Teodora Nechita, Timo Kehrer, and Sebastiano Panichella. 2024. Replication Package - "How does Simulation-based Testing for Self-driving Cars match Human Perception?". https://doi.org/10.5281/zenodo.10570961
[13]
Christian Birchler, Tanzil Kombarabettu Mohammed, Pooja Rani, Teodora Nechita, Timo Kehrer, and Sebastiano Panichella. 2024. Replication Package - "How does Simulation-based Testing for Self-driving Cars match Human Perception?". https://doi.org/10.5281/zenodo.10570960
[14]
Christian Birchler, Cyrill Rohrbach, Hyeongkyun Kim, Alessio Gambi, Tianhai Liu, Jens Horneber, Timo Kehrer, and Sebastiano Panichella. 2023. TEASER: Simulation-Based CAN Bus Regression Testing for Self-Driving Cars Software. In International Conference on Automated Software Engineering. 2058–2061. https://doi.org/10.1109/ASE56229.2023.00154
[15]
Tim Bohne, Gurunatraj Parthasarathy, and Benjamin Kisliuk. 2023. A systematic approach to the development of long-term autonomous robotic systems for agriculture. In 43. GIL-Jahrestagung, Resiliente Agri-Food-Systeme (LNI, Vol. P-330). Gesellschaft für Informatik e.V., 285–290. https://dl.gi.de/20.500.12116/40260
[16]
Ezequiel Castellano, Ahmet Cetinkaya, Cédric Ho Thanh, Stefan Klikovits, Xiaoyi Zhang, and Paolo Arcaini. 2021. Frenetic at the SBST 2021 Tool Competition. In International Workshop on Search-Based Software Testing. IEEE, 36–37. https://doi.org/10.1109/SBST52555.2021.00016
[17]
Yevgen Chebotar, Ankur Handa, Viktor Makoviychuk, Miles Macklin, Jan Issac, Nathan D. Ratliff, and Dieter Fox. 2019. Closing the Sim-to-Real Loop: Adapting Simulation Randomization with Real World Experience. In International Conference on Robotics and Automation. IEEE, 8973–8979. https://doi.org/10.1109/ICRA.2019.8793789
[18]
Shafiul Azam Chowdhury, Sohil Lal Shrestha, Taylor T. Johnson, and Christoph Csallner. 2020. SLEMI: equivalence modulo input (EMI) based mutation of CPS models for finding compiler bugs in Simulink. In International Conference on Software Engineering. 335–346. https://doi.org/10.1145/3377811.3380381
[19]
Jack Collins, Ross Brown, Jurgen Leitner, and David Howard. 2020. Traversing the reality gap via simulator tuning. arXiv preprint arXiv:2003.01369.
[20]
Hugo Leonardo da Silva Araujo, Mohammad Reza Mousavi, and Mahsa Varshosaz. 2023. Testing, Validation, and Verification of Robotic and Autonomous Systems: A Systematic Review. ACM Trans. Softw. Eng. Methodol., 32, 2 (2023), 51:1–51:61. https://doi.org/10.1145/3542945
[21]
Jyotirmoy V. Deshmukh, Marko Horvat, Xiaoqing Jin, Rupak Majumdar, and Vinayak S. Prabhu. 2017. Testing Cyber-Physical Systems through Bayesian Optimization. ACM Trans. Embed. Comput. Syst., 16, 5s (2017), 170:1–170:18. https://doi.org/10.1145/3126521
[22]
Alexey Dosovitskiy, Germán Ros, Felipe Codevilla, Antonio M. López, and Vladlen Koltun. 2017. CARLA: An Open Urban Driving Simulator. In Annual Conference on Robot Learning (Proceedings of Machine Learning Research, Vol. 78). PMLR, 1–16. http://proceedings.mlr.press/v78/dosovitskiy17a.html
[23]
Alessio Gambi, Tri Huynh, and Gordon Fraser. 2019. Automatically reconstructing car crashes from police reports for testing self-driving cars. In International Conference on Software Engineering: Companion Proceedings. IEEE / ACM, 290–291. https://doi.org/10.1109/ICSE-Companion.2019.00119
[24]
Alessio Gambi, Tri Huynh, and Gordon Fraser. 2019. Generating effective test cases for self-driving cars from police reports. In Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering. ACM, 257–267. https://doi.org/10.1145/3338906.3338942
[25]
Alessio Gambi, Gunel Jahangirova, Vincenzo Riccio, and Fiorella Zampetti. 2022. SBST Tool Competition 2022. In International Workshop on Search-Based Software Testing. IEEE, 25–32. https://doi.org/10.1145/3526072.3527538
[26]
Joshua Garcia, Yang Feng, Junjie Shen, Sumaya Almanee, Yuan Xia, and Qi Alfred Chen. 2020. A comprehensive study of autonomous vehicle bugs. In International Conference on Software Engineering. ACM, 385–396. https://doi.org/10.1145/3377811.3380397
[27]
BeamNG GmbH. 2023. BeamNG.tech. https://beamng.tech/
[28]
BeamNG GmbH. 2023. Publications based on BeamNG.tech. https://beamng.tech/research/
[29]
The Guardian. 2018. Self-driving Uber kills Arizona woman in first fatal crash involving pedestrian. https://www.theguardian.com/technology/2018/mar/19/uber- self-driving-car-kills-woman-arizona-tempe
[30]
Rodrigo Gutiérrez-Moreno, Rafael Barea, Elena López Guillén, Javier Araluce, and Luis Miguel Bergasa. 2022. Reinforcement Learning-Based Autonomous Driving at Intersections in CARLA Simulator. Sensors, 22, 21 (2022), 8373. https://doi.org/10.3390/s22218373
[31]
Carl Hildebrandt and Sebastian G. Elbaum. 2021. World-in-the-Loop Simulation for Autonomous Systems Validation. In International Conference on Robotics and Automation. IEEE, 10912–10919. https://doi.org/10.1109/ICRA48506.2021.9561240
[32]
Adrian Hoff, Michael Nieke, and Christoph Seidl. 2021. Towards immersive software archaeology: regaining legacy systems’ design knowledge via interactive exploration in virtual reality. In Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering. ACM, 1455–1458. https://doi.org/10.1145/3468264.3473128
[33]
Jiawei Huang and Alexander Klippel. 2020. The Effects of Visual Realism on Spatial Memory and Exploration Patterns in Virtual Reality. In Symposium on Virtual Reality Software and Technology. ACM, 18:1–18:11. https://doi.org/10.1145/3385956.3418945
[34]
Jiawei Huang, Melissa S. Lucash, Mark B. Simpson, Casey Helgeson, and Alexander Klippel. 2019. Visualizing Natural Environments from Data in Virtual Reality: Combining Realism and Uncertainty. In Conference on Virtual Reality and 3D User Interfaces. IEEE, 1485–1488. https://doi.org/10.1109/VR.2019.8797996
[35]
Gunel Jahangirova, David Clark, Mark Harman, and Paolo Tonella. 2016. Test oracle assessment and improvement. In International Symposium on Software Testing and Analysis. ACM, 247–258. https://doi.org/10.1145/2931037.2931062
[36]
Gunel Jahangirova, Andrea Stocco, and Paolo Tonella. 2021. Quality Metrics and Oracles for Autonomous Vehicles Testing. In Conference on Software Testing, Verification and Validation. IEEE, 194–204. https://doi.org/10.1109/ICST49551.2021.00030
[37]
Sajad Khatiri, Sebastiano Panichella, and Paolo Tonella. 2023. Simulation-based Test Case Generation for Unmanned Aerial Vehicles in the Neighborhood of Real Flights. In International Conference on Software Testing, Verification and Validation. IEEE, 281–292. https://doi.org/10.1109/ICST57152.2023.00034
[38]
Sajad Khatiri, Sebastiano Panichella, and Paolo Tonella. 2024. Simulation-based Testing of Unmanned Aerial Vehicles with Aerialist. In International Conference on Software Engineering (ICSE).
[39]
Sajad Khatiri, Prasun Saurabh, Timothy Zimmermann, Charith Munasinghe, Christian Birchler, and Sebastiano Panichella. 2024. SBFT Tool Competition 2024 - CPS-UAV Test Case Generation Track. In IEEE/ACM International Workshop on Search-Based and Fuzz Testing, SBFT@ICSE 2024.
[40]
Sylvain Koos, Jean-Baptiste Mouret, and Stéphane Doncieux. 2013. The Transferability Approach: Crossing the Reality Gap in Evolutionary Robotics. IEEE Trans. Evol. Comput., 17, 1 (2013), 122–145. https://doi.org/10.1109/TEVC.2012.2185849
[41]
Timothy E Lee, Jonathan Tremblay, Thang To, Jia Cheng, Terry Mosier, Oliver Kroemer, Dieter Fox, and Stan Birchfield. 2020. Camera-to-robot pose estimation from a single image. In International Conference on Robotics and Automation. 9426–9432.
[42]
Rohit Mehra, Vibhu Saujanya Sharma, Vikrant Kaulgud, Sanjay Podder, and Adam P. Burden. 2020. Immersive IDE: Towards Leveraging Virtual Reality for creating an Immersive Software Development Environment. In International Conference on Software Engineering, Workshops. ACM, 177–180. https://doi.org/10.1145/3387940.3392234
[43]
Rohit Mehra, Vibhu Saujanya Sharma, Vikrant Kaulgud, Sanjay Podder, and Adam P. Burden. 2020. Towards Immersive Comprehension of Software Systems Using Augmented Reality - An Empirical Evaluation. In International Conference on Automated Software Engineering. IEEE, 1267–1269. https://doi.org/10.1145/3324884.3418907
[44]
Claudio Menghi, Shiva Nejati, Khouloud Gaaloul, and Lionel C. Briand. 2019. Generating automated and online test oracles for Simulink models with continuous and uncertain behaviors. In Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering. 27–38. https://doi.org/10.1145/3338906.3338920
[45]
Leonel Merino, Mircea Lungu, and Christoph Seidl. 2020. Unleashing the Potentials of Immersive Augmented Reality for Software Engineering. In International Conference on Software Analysis, Evolution and Reengineering. IEEE, 517–521. https://doi.org/10.1109/SANER48275.2020.9054812
[46]
Elena Molina, Alejandro Ríos Jerez, and Núria Pelechano Gómez. 2020. Avatars rendering and its effect on perceived realism in Virtual Reality. In International Conference on Artificial Intelligence and Virtual Reality. IEEE, 222–225. https://doi.org/10.1109/AIVR50618.2020.00046
[47]
Saasha Nair, Sina Shafaei, Daniel Auge, and Alois C. Knoll. 2021. An Evaluation of "Crash Prediction Networks" (CPN) for Autonomous Driving Scenarios in CARLA Simulator. In Workshop on Artificial Intelligence Safety (CEUR Workshop Proceedings, Vol. 2808). CEUR-WS.org. http://ceur-ws.org/Vol-2808/Paper_10.pdf
[48]
Anthony Ngo, Max Paul Bauer, and Michael Resch. 2021. A Multi-Layered Approach for Measuring the Simulation-to-Reality Gap of Radar Perception for Autonomous Driving. In International Intelligent Transportation Systems Conference. IEEE, 4008–4014. https://doi.org/10.1109/ITSC48978.2021.9564521
[49]
Vuong Nguyen, Stefan Huber, and Alessio Gambi. 2021. SALVO: Automated Generation of Diversified Tests for Self-driving Cars from Existing Maps. In International Conference on Artificial Intelligence Testing. IEEE, 128–135. https://doi.org/10.1109/AITEST52744.2021.00033
[50]
2020. NVIDIA DRIVE Constellation. https://developer.nvidia.com/drive/drive-constellation
[51]
Sebastiano Panichella, Alessio Gambi, Fiorella Zampetti, and Vincenzo Riccio. 2021. SBST Tool Competition 2021. In International Workshop on Search-Based Software Testing. IEEE, 20–27. https://doi.org/10.1109/SBST52555.2021.00011
[52]
Mingyu Park, Hoon Jang, Taejoon Byun, and Yunja Choi. 2020. Property-based testing for LG home appliances using accelerated software-in-the-loop simulation. In International Conference on Software Engineering. ACM, 120–129. https://doi.org/10.1145/3377813.3381346
[53]
Yi-Hao Peng, Carolyn Yu, Shi-Hong Liu, Chung-Wei Wang, Paul Taele, Neng-Hao Yu, and Mike Y. Chen. 2020. WalkingVibe: Reducing Virtual Reality Sickness and Improving Realism while Walking in VR using Unobtrusive Head-mounted Vibrotactile Feedback. In Conference on Human Factors in Computing Systems. ACM, 1–12. https://doi.org/10.1145/3313831.3376847
[54]
Andrea Piazzoni, Jim Cherian, Mohamed Azhar, Jing Yew Yap, James Lee Wei Shung, and Roshan Vijay. 2021. ViSTA: a Framework for Virtual Scenario-based Testing of Autonomous Vehicles. In International Conference on Artificial Intelligence Testing. IEEE, 143–150. https://doi.org/10.1109/AITEST52744.2021.00035
[55]
Fabio Reway, Abdul Hoffmann, Diogo Wachtel, Werner Huber, Alois C. Knoll, and Eduardo Parente Ribeiro. 2020. Test Method for Measuring the Simulation-to-Reality Gap of Camera-based Object Detection Algorithms for Autonomous Driving. In Intelligent Vehicles Symposium. IEEE, 1249–1256. https://doi.org/10.1109/IV47402.2020.9304567
[56]
Guodong Rong, Byung Hyun Shin, Hadi Tabatabaee, Qiang Lu, Steve Lemke, Martins Mozeiko, Eric Boise, Geehoon Uhm, Mark Gerow, Shalin Mehta, Eugene Agafonov, Tae Hyung Kim, Eric Sterner, Keunhae Ushiroda, Michael Reyes, Dmitry Zelenkovsky, and Seonman Kim. 2020. LGSVL Simulator: A High Fidelity Simulator for Autonomous Driving. 1–6. https://doi.org/10.1109/ITSC45102.2020.9294422
[57]
Erica Salvato, Gianfranco Fenu, Eric Medvet, and Felice Andrea Pellegrino. 2021. Crossing the Reality Gap: A Survey on Sim-to-Real Transferability of Robot Controllers in Reinforcement Learning. IEEE Access, 9 (2021), 153171–153187. https://doi.org/10.1109/ACCESS.2021.3126658
[58]
Vibhu Saujanya Sharma, Rohit Mehra, Vikrant Kaulgud, and Sanjay Podder. 2018. An immersive future for software engineering: avenues and approaches. In International Conference on Software Engineering: New Ideas and Emerging Results. ACM, 105–108. https://doi.org/10.1145/3183399.3183414
[59]
Vibhu Saujanya Sharma, Rohit Mehra, Vikrant Kaulgud, and Sanjay Podder. 2019. An extended reality approach for creating immersive software project workspaces. In International Workshop on Cooperative and Human Aspects of Software Engineering. IEEE / ACM, 27–30. https://doi.org/10.1109/CHASE.2019.00013
[60]
Gustavo Silvera, Abhijat Biswas, and Henny Admoni. 2022. DReyeVR: Democratizing Virtual Reality Driving Simulation for Behavioural & Interaction Research. In International Conference on Human-Robot Interaction, Daisuke Sakamoto, Astrid Weiss, Laura M. Hiatt, and Masahiro Shiomi (Eds.). IEEE / ACM, 639–643. https://doi.org/10.1109/HRI53351.2022.9889526
[61]
Andrea Di Sorbo, Fiorella Zampetti, Aaron Visaggio, Massimiliano Di Penta, and Sebastiano Panichella. 2023. Automated Identification and Qualitative Characterization of Safety Concerns Reported in UAV Software Platforms. ACM Trans. Softw. Eng. Methodol., 32, 3 (2023), 67:1–67:37. https://doi.org/10.1145/3564821
[62]
Donna Spencer. 2009. Card sorting: Designing usable categories. Rosenfeld Media.
[63]
Jack Stilgoe. 2021. How can we know a self-driving car is safe? Ethics Inf. Technol., 23, 4 (2021), 635–647. https://doi.org/10.1007/s10676-021-09602-1
[64]
Andrea Stocco, Brian Pulfer, and Paolo Tonella. 2023. Mind the Gap! A Study on the Transferability of Virtual Versus Physical-World Testing of Autonomous Driving Systems. IEEE Trans. Software Eng., 49, 4 (2023), 1928–1940. https://doi.org/10.1109/TSE.2022.3202311
[65]
Andrea Stocco, Michael Weiss, Marco Calzana, and Paolo Tonella. 2020. Misbehaviour prediction for autonomous driving systems. In International Conference on Software Engineering. ACM, 359–371. https://doi.org/10.1145/3377811.3380353
[66]
Shuncheng Tang, Zhenya Zhang, Yi Zhang, Jixiang Zhou, Yan Guo, Shuang Liu, Shengjian Guo, Yan-Fu Li, Lei Ma, Yinxing Xue, and Yang Liu. 2023. A Survey on Automated Driving System Testing: Landscapes and Trends. ACM Trans. Softw. Eng. Methodol., 32, 5 (2023), 124:1–124:62. https://doi.org/10.1145/3579642
[67]
Valerio Terragni, Gunel Jahangirova, Mauro Pezzè, and Paolo Tonella. 2021. Improving assertion oracles with evolutionary computation. In Genetic and Evolutionary Computation Conference, Companion Volume. ACM, 45–46. https://doi.org/10.1145/3449726.3462722
[68]
Valerio Terragni, Gunel Jahangirova, Paolo Tonella, and Mauro Pezzè. 2020. Evolutionary improvement of assertion oracles. In Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering. ACM, 1178–1189. https://doi.org/10.1145/3368089.3409758
[69]
Valerio Terragni, Gunel Jahangirova, Paolo Tonella, and Mauro Pezzè. 2021. GAssert: A Fully Automated Tool to Improve Assertion Oracles. In International Conference on Software Engineering: Companion Proceedings. IEEE, 85–88. https://doi.org/10.1109/ICSE-Companion52605.2021.00042
[70]
Christopher Steven Timperley, Afsoon Afzal, Deborah S Katz, Jam Marcos Hernandez, and Claire Le Goues. 2018. Crashing simulated planes is cheap: Can simulation detect robotics bugs early? In International Conference on Software Testing, Verification and Validation. 331–342.
[71]
Dinghua Wang, Shuqing Li, Guanping Xiao, Yepang Liu, and Yulei Sui. 2021. An exploratory study of autopilot software bugs in unmanned aerial vehicles. In Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering. ACM, 20–31. https://doi.org/10.1145/3468264.3468559
[72]
Lingfeng Wang and K.C. Tan. 2005. Software testing for safety critical applications. IEEE Instrumentation & Measurement Magazine, 8, 2 (2005), 38–47. https://doi.org/10.1109/MIM.2005.1438843
[73]
Franz Wotawa. 2021. On the Use of Available Testing Methods for Verification & Validation of AI-based Software and Systems. In Workshop on Artificial Intelligence Safety (CEUR Workshop Proceedings, Vol. 2808). CEUR-WS.org. http://ceur-ws.org/Vol-2808/Paper_29.pdf
[74]
Qinghua Xu, Shaukat Ali, and Tao Yue. 2021. Digital Twin-based Anomaly Detection in Cyber-physical Systems. In Conference on Software Testing, Verification and Validation. IEEE, 205–216. https://doi.org/10.1109/ICST49551.2021.00031
[75]
Fiorella Zampetti, Ritu Kapur, Massimiliano Di Penta, and Sebastiano Panichella. 2022. An empirical characterization of software bugs in open-source Cyber–Physical Systems. Journal of Systems and Software, 192 (2022), 111425. issn:0164-1212 https://doi.org/10.1016/j.jss.2022.111425
[76]
Eleni Zapridou, Ezio Bartocci, and Panagiotis Katsaros. 2020. Runtime Verification of Autonomous Driving Systems in CARLA. In Runtime Verification - International Conference (Lecture Notes in Computer Science, Vol. 12399). Springer, 172–183. https://doi.org/10.1007/978-3-030-60508-7_9
[77]
Fangyi Zhang, Jürgen Leitner, Zongyuan Ge, Michael Milford, and Peter Corke. 2019. Adversarial discriminative sim-to-real transfer of visuo-motor policies. Int. J. Robotics Res., 38, 10-11 (2019), https://doi.org/10.1177/0278364919870227
[78]
Wei Zhang, Siyu Fu, Zixu Cao, Zhiyuan Jiang, Shunqing Zhang, and Shugong Xu. 2020. An SDR-in-the-Loop Carla Simulator for C-V2X-Based Autonomous Driving. In Conference on Computer Communications. IEEE, 1270–1271. https://doi.org/10.1109/INFOCOMWKSHPS50562.2020.9162743
[79]
Husheng Zhou, Wei Li, Zelun Kong, Junfeng Guo, Yuqun Zhang, Bei Yu, Lingming Zhang, and Cong Liu. 2020. DeepBillboard: systematic physical-world testing of autonomous driving systems. In International Conference on Software Engineering. 347–358. https://doi.org/10.1145/3377811.3380422
Index Terms
- How Does Simulation-Based Testing for Self-Driving Cars Match Human Perception?
Recommendations
Automatically testing self-driving cars with search-based procedural content generation
ISSTA 2019: Proceedings of the 28th ACM SIGSOFT International Symposium on Software Testing and AnalysisSelf-driving cars rely on software which needs to be thoroughly tested. Testing self-driving car software in real traffic is not only expensive but also dangerous, and has already caused fatalities. Virtual tests, in which self-driving car software is ...
Machine learning-based test selection for simulation-based testing of self-driving cars software
AbstractSimulation platforms facilitate the development of emerging Cyber-Physical Systems (CPS) like self-driving cars (SDC) because they are more efficient and less dangerous than field operational test cases. Despite this, thoroughly testing SDCs in ...
Cost-effective simulation-based test selection in self-driving cars software▪
AbstractSimulation environments are essential for the continuous development of complex cyber-physical systems such as self-driving cars (SDCs). Previous results on simulation-based testing for SDCs have shown that many automatically generated tests do ...
Highlights- A structural refactoring and extension of the SDC-Scissor framework to provide an extendable open API
- An extension of original datasets that include new configurations of the test subject
- Integration of additional ML models for ...
Comments
Please enable JavaScript to view thecomments powered by Disqus.Information & Contributors
Information
Published In
July 2024
2770 pages
Copyright © 2024 Owner/Author.
This work is licensed under a Creative Commons Attribution International 4.0 License.
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 12 July 2024
Published in PACMSE Volume 1, Issue FSE
Badges
Author Tags
Qualifiers
- Research-article
Funding Sources
- Horizon 2020 Framework Programme
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 351Total Downloads
- Downloads (Last 12 months)351
- Downloads (Last 6 weeks)96
Reflects downloads up to 13 Dec 2024
Other Metrics
Citations
Cited By
View allView Options
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign in