[go: up one dir, main page]
More Web Proxy on the site http://driver.im/ skip to main content
10.1145/3510003.3510163acmconferencesArticle/Chapter ViewAbstractPublication PagesicseConference Proceedingsconference-collections
research-article

Towards training reproducible deep learning models

Published: 05 July 2022 Publication History

Abstract

Reproducibility is an increasing concern in Artificial Intelligence (AI), particularly in the area of Deep Learning (DL). Being able to reproduce DL models is crucial for AI-based systems, as it is closely tied to various tasks like training, testing, debugging, and auditing. However, DL models are challenging to be reproduced due to issues like randomness in the software (e.g., DL algorithms) and non-determinism in the hardware (e.g., GPU). There are various practices to mitigate some of the aforementioned issues. However, many of them are either too intrusive or can only work for a specific usage context. In this paper, we propose a systematic approach to training reproducible DL models. Our approach includes three main parts: (1) a set of general criteria to thoroughly evaluate the reproducibility of DL models for two different domains, (2) a unified framework which leverages a record-and-replay technique to mitigate software-related randomness and a profile-and-patch technique to control hardware-related non-determinism, and (3) a reproducibility guideline which explains the rationales and the mitigation strategies on conducting a reproducible training process for DL models. Case study results show our approach can successfully reproduce six open source and one commercial DL models.

References

[1]
2021 (accessed August, 2021). An open source platform for the machine learning lifecycle. https://mlflow.org/
[2]
2021 (accessed August, 2021). Assessment List for Trustworthy Artificial Intelligence (ALTAI) for self-assessment. https://digital-strategy.ec.europa.eu/en/library/assessment-list-trustworthy-artificial-intelligence-altai-self-assessment
[3]
2021 (accessed August, 2021). The CIFAR-10 and CIFAR-100 datasets. https://www.cs.toronto.edu/~kriz/cifar.html
[4]
2021 (accessed August, 2021). CUDA Toolkit. https://developer.nvidia.com/cuda-toolkit
[5]
2021 (accessed August, 2021). Determined AI Reproducibility. https://docs.determined.ai/latest/topic-guides/training/reproducibility.html
[6]
2021 (accessed August, 2021). Determinism in Deep Learning (S9911). https://developer.download.nvidia.com/video/gputechconf/gtc/2019/presentation/s9911-determinism-in-deep-learning.pdf
[7]
2021 (accessed August, 2021). Inconsistent results when using two styles of import statements - Issue 14672. https://github.com/keras-team/keras/issues/14672
[8]
2021 (accessed August, 2021). Migrate your TensorFlow 1 code to TensorFlow 2. https://www.tensorflow.org/guide/migrate
[9]
2021 (accessed August, 2021). The Mnist Database of handwritten digits. http://yann.lecun.com/exdb/mnist/
[10]
2021 (accessed August, 2021). Most common random seeds. https://www.kaggle.com/residentmario/kernel16e284dcb7
[11]
2021 (accessed August, 2021). Notes from the AI Frontier Insights from Hundreds of Use Cases. https://www.mckinsey.com/featured-insights/artificial-intelligence/notes-from-the-ai-frontier-applications-and-value-of-deep-learning
[12]
2021 (accessed August, 2021). Open-source Version Control System for Machine Learning Projects. https://dvc.org/
[13]
2021 (accessed August, 2021). Proposal for a REGULATION OF THE EUROPEAN PARLIAMENT AND OF THE COUNCIL LAYING DOWN HARMONISED RULES ON ARTIFICIAL INTELLIGENCE (ARTIFICIAL INTELLIGENCE ACT) AND AMENDING CERTAIN UNION LEGISLATIVE ACTS. https://eur-lex.europa.eu/legal-content/EN/TXT/?uri=CELEX%3A52021PC0206
[14]
2021 (accessed August, 2021). Reproducibility in Pytorch. https://pytorch.org/docs/stable/notes/randomness.html
[15]
2021 (accessed August, 2021). Tensorflow Determinism. https://github.com/NVIDIA/framework-determinism
[16]
2021 (accessed August, 2021). TensorFlow GPU Support. https://www.tensorflow.org/install/source#gpu
[17]
2021 (accessed August, 2021). Tensorflow RFC for determinism. https://github.com/tensorflow/community/blob/master/rfcs/20210119-determinism.md
[18]
2021 (accessed August, 2021). Testing for Deploying Machine Learning Models. https://developers.google.com/machine-learning/testing-debugging/pipeline/deploying
[19]
2021 (accessed August, 2021). The Machine Learning Reproducibility Checklist. https://www.cs.mcgill.ca/~jpineau/ReproducibilityChecklist.pdf
[20]
2021 (accessed August, 2021). Unreproducible results when directly import keras in CPU environment - Issue 14671. https://github.com/keras-team/keras/issues/14671
[21]
2022 (accessed Feb, 2022). The replication package. https://github.com/nemo9cby/ICSE2022Rep
[22]
Martín Abadi, Paul Barham, Jianmin Chen, Zhifeng Chen, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Geoffrey Irving, Michael Isard, Manjunath Kudlur, Josh Levenberg, Rajat Monga, Sherry Moore, Derek Gordon Murray, Benoit Steiner, Paul A. Tucker, Vijay Vasudevan, Pete Warden, Martin Wicke, Yuan Yu, and Xiaoqiang Zheng. 2016. TensorFlow: A System for Large-Scale Machine Learning. In 12th USENIX Symposium on Operating Systems Design and Implementation, OSDI 2016, Savannah, GA, USA, November 2--4, 2016. USENIX Association, 265--283.
[23]
Saleema Amershi, Andrew Begel, Christian Bird, Robert DeLine, Harald C. Gall, Ece Kamar, Nachiappan Nagappan, Besmira Nushi, and Thomas Zimmermann. 2019. Software engineering for machine learning: a case study. In Proceedings of the 41st International Conference on Software Engineering: Software Engineering in Practice, ICSE (SEIP) 2019, Montreal, QC, Canada, May 25--31, 2019, Helen Sharp and Mike Whalen (Eds.). IEEE / ACM, 291--300.
[24]
Dario Amodei, Sundaram Ananthanarayanan, Rishita Anubhai, Jingliang Bai, Eric Battenberg, Carl Case, Jared Casper, Bryan Catanzaro, Jingdong Chen, Mike Chrzanowski, Adam Coates, Greg Diamos, Erich Elsen, Jesse H. Engel, Linxi Fan, Christopher Fougner, Awni Y. Hannun, Billy Jun, Tony Han, Patrick LeGresley, Xi-angang Li, Libby Lin, Sharan Narang, Andrew Y. Ng, Sherjil Ozair, Ryan Prenger, Sheng Qian, Jonathan Raiman, Sanjeev Satheesh, David Seetapun, Shubho Sengupta, Chong Wang, Yi Wang, Zhiqian Wang, Bo Xiao, Yan Xie, Dani Yogatama, Jun Zhan, and Zhenyao Zhu. 2016. Deep Speech 2 : End-to-End Speech Recognition in English and Mandarin. In Proceedings of the 33nd International Conference on Machine Learning, ICML 2016, New York City, NY, USA, June 19--24, 2016 (JMLR Workshop and Conference Proceedings).
[25]
Amine Barrak, Ellis E. Eghan, and Bram Adams. 2021. On the Co-evolution of ML Pipelines and Source Code - Empirical Study of DVC Projects. In 28th IEEE International Conference on Software Analysis, Evolution and Reengineering, SANER 2021, Honolulu, HI, USA, March 9--12, 2021. IEEE, 422--433.
[26]
Miles Brundage, Shahar Avin, Jasmine Wang, Haydn Belfield, Gretchen Krueger, Gillian K. Hadfield, Heidy Khlaaf, Jingying Yang, Helen Toner, Ruth Fong, Tegan Maharaj, Pang Wei Koh, Sara Hooker, Jade Leung, Andrew Trask, Emma Bluemke, Jonathan Lebensbold, Cullen O'Keefe, Mark Koren, Theo Ryffel, J. B. Rubinovitz, Tamay Besiroglu, Federica Carugati, Jack Clark, Peter Eckersley, Sarah de Haas, Maritza Johnson, Ben Laurie, Alex Ingerman, Igor Krawczuk, Amanda Askell, Rosario Cammarota, Andrew Lohn, David Krueger, Charlotte Stix, Peter Henderson, Logan Graham, Carina Prunkl, Bianca Martin, Elizabeth Seger, Noa Zilberman, Seán Ó hÉigeartaigh, Frens Kroeger, Girish Sastry, Rebecca Kagan, Adrian Weller, Brian Tse, Elizabeth Barnes, Allan Dafoe, Paul Scharre, Ariel HerbertVoss, Martijn Rasser, Shagun Sodhani, Carrick Flynn, Thomas Krendl Gilbert, Lisa Dyer, Saif Khan, Yoshua Bengio, and Markus Anderljung. 2020. Toward Trustworthy AI Development: Mechanisms for Supporting Verifiable Claims. CoRR abs/2004.07213 (2020). arXiv:2004.07213 https://arxiv.org/abs/2004.07213
[27]
Sharan Chetlur, Cliff Woolley, Philippe Vandermersch, Jonathan Cohen, John Tran, Bryan Catanzaro, and Evan Shelhamer. 2014. cuDNN: Efficient Primitives for Deep Learning. CoRR abs/1410.0759 (2014). arXiv:1410.0759 http://arxiv.org/abs/1410.0759
[28]
Cédric Colas, Olivier Sigaud, and Pierre-Yves Oudeyer. 2018. How Many Random Seeds? Statistical Power Analysis in Deep Reinforcement Learning Experiments. CoRR abs/1806.08295 (2018). arXiv:1806.08295 http://arxiv.org/abs/1806.08295
[29]
Andre Esteva, Alexandre Robicquet, Bharath Ramsundar, Volodymyr Kuleshov, Mark DePristo, Katherine Chou, Claire Cui, Greg Corrado, Sebastian Thrun, and Jeff Dean. 2019. A guide to deep learning in healthcare. Nature medicine 25, 1 (2019), 24--29.
[30]
Timnit Gebru, Jamie Morgenstern, Briana Vecchione, Jennifer Wortman Vaughan, Hanna M. Wallach, Hal Daumé III, and Kate Crawford. 2018. Datasheets for Datasets. CoRR abs/1803.09010 (2018). arXiv:1803.09010 http://arxiv.org/abs/1803.09010
[31]
Simos Gerasimou, Hasan Ferit Eniser, Alper Sen, and Alper Cakan. 2020. Importance-driven deep learning system testing. In ICSE '20: 42nd International Conference on Software Engineering, Seoul, South Korea, 27 June - 19 July, 2020, Gregg Rothermel and Doo-Hwan Bae (Eds.). ACM, 702--713.
[32]
Sindhu Ghanta, Lior Khermosh, Sriram Subramanian, Vinay Sridhar, Swaminathan Sundararaman, Dulcardo Arteaga, Qianmei Luo, Drew Roselli, Dhananjoy Das, and Nisha Talagala. 2018. A systems perspective to reproducibility in production machine learning domain. (2018).
[33]
David Goldberg. 1991. What Every Computer Scientist Should Know About Floating-Point Arithmetic. ACM Comput. Surv. 23, 1 (1991), 5--48.
[34]
Sorin Mihai Grigorescu, Bogdan Trasnea, Tiberiu T. Cocias, and Gigel Macesanu. 2020. A survey of deep learning techniques for autonomous driving. J. Field Robotics 37, 3 (2020), 362--386.
[35]
Jiazhen Gu, Huanlin Xu, Haochuan Lu, Yangfan Zhou, and Xin Wang. 2021. Detecting Deep Neural Network Defects with Data Flow Analysis. In 51st Annual IEEE/IFIP International Conference on Dependable Systems and Networks Workshops, DSN Workshops 2021, Taipei, Taiwan, June 21--24, 2021.
[36]
Xiaodong Gu, Hongyu Zhang, and Sunghun Kim. 2018. Deep code search. In Proceedings of the 40th International Conference on Software Engineering, ICSE 2018, Gothenburg, Sweden, May 27 - June 03, 2018, Michel Chaudron, Ivica Crnkovic, Marsha Chechik, and Mark Harman (Eds.). ACM, 933--944.
[37]
Odd Erik Gundersen and Sigbjørn Kjensmo. [n.d.]. State of the Art: Reproducibility in Artificial Intelligence. In Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence, (AAAI-18).
[38]
Qianyu Guo, Sen Chen, Xiaofei Xie, Lei Ma, Qiang Hu, Hongtao Liu, Yang Liu, Jianjun Zhao, and Xiaohong Li. 2019. An Empirical Study Towards Characterizing Deep Learning Development and Deployment Across Different Frameworks and Platforms. In 34th IEEE/ACM International Conference on Automated Software Engineering, ASE 2019, San Diego, CA, USA, November 11--15, 2019.
[39]
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep Residual Learning for Image Recognition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2016, Las Vegas, NV, USA, June 27--30, 2016. IEEE Computer Society, 770--778.
[40]
Peter Henderson, Riashat Islam, Philip Bachman, Joelle Pineau, Doina Precup, and David Meger. 2018. Deep Reinforcement Learning That Matters. In Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence, (AAAI-18). AAAI Press.
[41]
Matthew Hutson. 2018. Artificial intelligence faces reproducibility crisis. Science (New York, N.Y.) 359 (02 2018), 725--726.
[42]
Frank Hutter, Lars Kotthoff, and Joaquin Vanschoren (Eds.). 2019. Automated Machine Learning - Methods, Systems, Challenges. Springer.
[43]
Samuel Idowu, Daniel Strüber, and Thorsten Berger. 2021. Asset Management in Machine Learning: A Survey. In 43rd IEEE/ACM International Conference on Software Engineering: Software Engineering in Practice, ICSE (SEIP) 2021, Madrid, Spain, May 25--28, 2021. IEEE, 51--60.
[44]
Richard Isdahl and Odd Erik Gundersen. 2019. Out-of-the-box reproducibility: A survey of machine learning platforms. In 2019 15th international conference on eScience (eScience). IEEE, 86--95.
[45]
Hadi Jooybar, Wilson W. L. Fung, Mike O'Connor, Joseph Devietti, and Tor M. Aamodt. 2013. GPUDet: a deterministic GPU architecture. In Architectural Support for Programming Languages and Operating Systems, ASPLOS 2013, Houston, TX, USA, March 16--20, 2013. ACM, 1--12.
[46]
Yann LeCun, Yoshua Bengio, and Geoffrey E. Hinton. 2015. Deep learning. Nat. (2015).
[47]
Y. Lecun, L. Bottou, Y. Bengio, and P. Haffner. 1998. Gradient-based learning applied to document recognition. Proc. IEEE (1998).
[48]
Brian Lee, Andrew Jackson, Tom Madams, Seth Troisi, and Derek Jones. 2019. Minigo: A Case Study in Reproducing Reinforcement Learning Research. In Reproducibility in Machine Learning, ICLR 2019 Workshop, New Orleans, Louisiana, United States, May 6, 2019. OpenReview.net.
[49]
Wei Li. 2017. cifar-10-cnn: Play deep learning with CIFAR datasets. https://github.com/BIGBALLON/cifar-10-cnn.
[50]
Chao Liu, Cuiyun Gao, Xin Xia, David Lo, John C. Grundy, and Xiaohu Yang. 2020. On the Replicability and Reproducibility of Deep Learning in Software Engineering. CoRR abs/2006.14244 (2020). arXiv:2006.14244 https://arxiv.org/abs/2006.14244
[51]
Lei Ma, Felix Juefei-Xu, Fuyuan Zhang, Jiyuan Sun, Minhui Xue, Bo Li, Chunyang Chen, Ting Su, Li Li, Yang Liu, Jianjun Zhao, and Yadong Wang. 2018. DeepGauge: multi-granularity testing criteria for deep learning systems. In Proceedings of the 33rd ACM/IEEE International Conference on Automated Software Engineering, ASE 2018, Montpellier, France, September 3--7, 2018, Marianne Huchard, Christian Kästner, and Gordon Fraser (Eds.). ACM, 120--131.
[52]
Shiqing Ma, Yingqi Liu, Wen-Chuan Lee, Xiangyu Zhang, and Ananth Grama. 2018. MODE: automated neural network model debugging via state differential analysis and input selection. In Proceedings of the 2018 ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering, ESEC/SIGSOFT FSE 2018, Lake Buena Vista, FL, USA, November 04--09, 2018, Gary T. Leavens, Alessandro Garcia, and Corina S. Pasareanu (Eds.).
[53]
Margaret Mitchell, Simone Wu, Andrew Zaldivar, Parker Barnes, Lucy Vasserman, Ben Hutchinson, Elena Spitzer, Inioluwa Deborah Raji, and Timnit Gebru. 2019. Model Cards for Model Reporting. In Proceedings of the Conference on Fairness, Accountability, and Transparency, FAT* 2019, Atlanta, GA, USA, January 29--31, 2019, danah boyd and Jamie H. Morgenstern (Eds.). ACM, 220--229.
[54]
David Lorge Parnas. 2017. The real risks of artificial intelligence. Commun. ACM 60, 10 (2017), 27--31.
[55]
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Köpf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. 2019. PyTorch: An Imperative Style, High-Performance Deep Learning Library. In Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8--14, 2019, Vancouver, BC, Canada, Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d'Alché-Buc, Emily B. Fox, and Roman Garnett (Eds.). 8024--8035.
[56]
Hung Viet Pham, Shangshu Qian, Jiannan Wang, Thibaud Lutellier, Jonathan Rosenthal, Lin Tan, Yaoliang Yu, and Nachiappan Nagappan. [n.d.]. Problems and Opportunities in Training Deep Learning Software Systems: An Analysis of Variance. In 35th IEEE/ACM International Conference on Automated Software Engineering, ASE 2020, Melbourne, Australia, September 21--25, 2020.
[57]
Joelle Pineau, Philippe Vincent-Lamarre, Koustuv Sinha, Vincent Larivière, Alina Beygelzimer, Florence d'Alché Buc, Emily Fox, and Hugo Larochelle. 2020. Improving Reproducibility in Machine Learning Research (A Report from the NeurIPS 2019 Reproducibility Program). arXiv:2003.12206 [cs.LG]
[58]
Edward Raff. 2019. A Step Toward Quantifying Independently Reproducible Machine Learning Research. In Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8--14, 2019, Vancouver, BC, Canada, Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d'Alché-Buc, Emily B. Fox, and Roman Garnett (Eds.).
[59]
Jeanine Romano, Jeffrey D Kromrey, Jesse Coraggio, and Jeff Skowronek. 2006. Appropriate statistics for ordinal level data: Should we really be using t-test and Cohen'sd for evaluating group differences on the NSSE and other surveys. In annual meeting of the Florida Association of Institutional Research, Vol. 13.
[60]
John K. Salmon, Mark A. Moraes, Ron O. Dror, and David E. Shaw. 2011. Parallel random numbers: as easy as 1, 2, 3. In Conference on High Performance Computing Networking, Storage and Analysis, SC 2011, Seattle, WA, USA, November 12--18, 2011. ACM, 16:1--16:12.
[61]
Simone Scardapane and Dianhui Wang. 2017. Randomness in neural networks: an overview. Wiley Interdiscip. Rev. Data Min. Knowl. Discov. 7, 2 (2017).
[62]
Joel Scheuner, Jürgen Cito, Philipp Leitner, and Harald C. Gall. 2015. Cloud WorkBench: Benchmarking IaaS Providers based on Infrastructure-as-Code. In Proceedings of the 24th International Conference on World Wide Web Companion, WWW 2015, Florence, Italy, May 18--22, 2015 - Companion Volume. ACM, 239--242.
[63]
Peter Sugimura and Florian Hartl. 2018. Building a reproducible machine learning pipeline. arXiv preprint arXiv:1810.04570 (2018).
[64]
Peter Sugimura and Florian Hartl. 2018. Building a Reproducible Machine Learning Pipeline. CoRR abs/1810.04570 (2018). arXiv:1810.04570 http://arxiv.org/abs/1810.04570
[65]
Rachael Tatman, Jake VanderPlas, and Sohier Dane. 2018. A practical taxonomy of reproducibility for machine learning research. (2018).
[66]
Ruben Vicente-Saez and Clara Martinez-Fuentes. 2018. Open Science now: A systematic literature review for an integrated definition. Journal of business research 88 (2018), 428--436.
[67]
Michael Woelfle, Piero Olliaro, and Matthew H Todd. 2011. Open science is a research accelerator. Nature chemistry 3, 10 (2011), 745--748.
[68]
Curtis Yanko. 2021 (accessed August, 2021). Using a Software Bill of Materials (SBOM) is Going Mainstream. https://blog.sonatype.com/software-bill-of-materials-going-mainstream
[69]
Sergey Zagoruyko and Nikos Komodakis. 2016. Wide Residual Networks. In Proceedings of the British Machine Vision Conference 2016, BMVC 2016, York, UK, September 19--22, 2016. BMVA Press. http://www.bmva.org/bmvc/2016/papers/paper087/index.html

Cited By

View all
  • (2024)A Survey on Stability of Learning with Limited Labelled Data and its Sensitivity to the Effects of RandomnessACM Computing Surveys10.1145/369133957:1(1-40)Online publication date: 7-Oct-2024
  • (2024)MLXP: A framework for conducting replicable experiments in PythonProceedings of the 2nd ACM Conference on Reproducibility and Replicability10.1145/3641525.3663648(134-144)Online publication date: 18-Jun-2024
  • (2024)Investigating Reproducibility in Deep Learning-Based Software Fault Prediction2024 IEEE 24th International Conference on Software Quality, Reliability and Security (QRS)10.1109/QRS62785.2024.00038(306-317)Online publication date: 1-Jul-2024
  • Show More Cited By

Recommendations

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences
ICSE '22: Proceedings of the 44th International Conference on Software Engineering
May 2022
2508 pages
ISBN:9781450392211
DOI:10.1145/3510003
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

In-Cooperation

  • IEEE CS

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 05 July 2022

Permissions

Request permissions for this article.

Check for updates

Author Tags

  1. artificial intelligence
  2. deep learning
  3. reproducibility
  4. software engineering

Qualifiers

  • Research-article

Conference

ICSE '22
Sponsor:

Acceptance Rates

Overall Acceptance Rate 276 of 1,856 submissions, 15%

Upcoming Conference

ICSE 2025

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • Downloads (Last 12 months)198
  • Downloads (Last 6 weeks)15
Reflects downloads up to 22 Dec 2024

Other Metrics

Citations

Cited By

View all
  • (2024)A Survey on Stability of Learning with Limited Labelled Data and its Sensitivity to the Effects of RandomnessACM Computing Surveys10.1145/369133957:1(1-40)Online publication date: 7-Oct-2024
  • (2024)MLXP: A framework for conducting replicable experiments in PythonProceedings of the 2nd ACM Conference on Reproducibility and Replicability10.1145/3641525.3663648(134-144)Online publication date: 18-Jun-2024
  • (2024)Investigating Reproducibility in Deep Learning-Based Software Fault Prediction2024 IEEE 24th International Conference on Software Quality, Reliability and Security (QRS)10.1109/QRS62785.2024.00038(306-317)Online publication date: 1-Jul-2024
  • (2024)Confronting the Reproducibility Crisis: A Case Study of Challenges in Cybersecurity AI2024 Cyber Awareness and Research Symposium (CARS)10.1109/CARS61786.2024.10778911(1-6)Online publication date: 28-Oct-2024
  • (2024)A systematic review of deep learning-based denoising for low-dose computed tomography from a perceptual quality perspectiveBiomedical Engineering Letters10.1007/s13534-024-00419-714:6(1153-1173)Online publication date: 30-Aug-2024
  • (2024)Towards enhancing the reproducibility of deep learning bugs: an empirical studyEmpirical Software Engineering10.1007/s10664-024-10579-w30:1Online publication date: 9-Nov-2024
  • (2024)Challenges and practices of deep learning model reengineering: A case study on computer visionEmpirical Software Engineering10.1007/s10664-024-10521-029:6Online publication date: 20-Aug-2024
  • (2023)Variance of ML-based software fault predictors: are we really improving fault prediction?2023 49th Euromicro Conference on Software Engineering and Advanced Applications (SEAA)10.1109/SEAA60479.2023.00021(78-85)Online publication date: 6-Sep-2023
  • (2023)Reusing Deep Learning Models: Challenges and Directions in Software Engineering2023 IEEE John Vincent Atanasoff International Symposium on Modern Computing (JVA)10.1109/JVA60410.2023.00015(17-30)Online publication date: 5-Jul-2023
  • (2023)FedSlice: Protecting Federated Learning Models from Malicious Participants with Model SlicingProceedings of the 45th International Conference on Software Engineering10.1109/ICSE48619.2023.00049(460-472)Online publication date: 14-May-2023
  • Show More Cited By

View Options

Login options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Media

Figures

Other

Tables

Share

Share

Share this Publication link

Share on social media