More Web Proxy on the site http://driver.im/

research-article

Analysis of test log information through interactive visualizations

Authors:

Marcelo SchotsAuthors Info & Claims

ICPC '18: Proceedings of the 26th Conference on Program Comprehension

Pages 156 - 166

https://doi.org/10.1145/3196321.3196345

Published: 28 May 2018 Publication History

Abstract

A fundamental activity to achieve software quality is software testing, whose results are typically stored in log files. These files contain the richest and more detailed source of information for developers trying to understand failures and identify their potential causes. Analyzing and understanding the information presented in log files, however, can be a complex task, depending on the amount of errors and the variety of information. Some previously proposed tools try to visualize test information, but they have limited interaction and present a single perspective of such data. This paper presents ASTRO, an infrastructure that extracts information from a number of log files and presents it in multi-perspective interactive visualizations that aim at easing and improving the developers' analysis process. A study carried out with practitioners from 3 software test factories indicated that ASTRO helps to analyze information of interest, with less accuracy in tasks that involved assimilation of information from different perspectives. Based on their difficulties, participants also provided feedback for improving the tool.

References

[1]

Rui Abreu, Peter Zoeteweij, and Arjan JC Van Gemund. 2007. On the accuracy of spectrum-based fault localization. In Testing: Academic and Industrial Conference Practice and Research Techniques-MUTATION, 2007. TAICPART-MUTATION 2007. IEEE, 89--98.

Digital Library

[2]

Hudson Borges, Andre Hora, and Marco Tulio Valente. 2016. Understanding the factors that impact the popularity of GitHub repositories. In 32th International Conference on Software Maintenance and Evolution (ICSME, 2016). Raleigh, North Carolina, USA, 334--344.

[3]

Michael Bostock, Vadim Ogievetsky, and Jeffrey Heer. 2011. D3: Data-Driven Documents. IEEE Transactions on Visualization and Computer Graphics 17, 12 (2011), 2301--2309.

Digital Library

[4]

Mattias Bybro and S Arnborg. 2003. A mutation testing tool for Java programs. Master's thesis, Stockholm University, Stockholm, Sweden (2003).

[5]

Diego Cardoso Borda Castro. 2018. Análise e Compreensão de Arquivos de Log Gerados por Ferramentas de Teste de Software. Undergraduate final project. Rio de Janeiro State University, Rio de Janeiro, Brazil.

[6]

James S Collofello and Scott N Woodfield. 1989. Evaluating the effectiveness of reliability-assurance techniques. Journal of Systems and Software 9, 3 (1989), 191--195.

Digital Library

[7]

Bas Cornelissen, Andy Zaidman, Danny Holten, Leon Moonen, Arie van Deursen, and Jarke J. van Wijk. 2008. Execution trace analysis through massive sequence and circular bundle views. Journal of Systems and Software 81, 12 (2008), 2252--2268.

Digital Library

[8]

Stephan Diehl. 2007. Software visualization: visualizing the structure, behaviour, and evolution of software. Springer Science & Business Media.

Digital Library

[9]

George Din, Justyna Zander, and Stephan Pietsch. 2004. Test Execution Logging and Visualisation Techniques. In 17th International Conference Software and Systems Engineering and their Applications, Paris, France.

[10]

githut. 2014. githut.info. http://githut.info/. (2014).

[11]

Carlos Gouveia, José Campos, and Rui Abreu. 2013. Using HTML5 visualizations in software fault localization. In 1st IEEE Working Conference on Software Visualization (VISSOFT 2013). 1--10.

[12]

IEEE. 2008. Draft IEEE Standard for Software and System Test Documentation (Revision of IEEE 829-1998). IEEE Unapproved Draft Std P829/D11, Feb 2008 (2008).

[13]

IEEE. 2016. ieee.org. http://spectrum.ieee.org/computing/software/the-2016-top-programming-languages. (July 2016).

[14]

ISO/IEC/IEEE. 2013. ISO/IEC/IEEE International Standard for Software and systems engineering - Software testing, Part 1: Concepts and definitions. ISO/IEC/IEEE 29119-1:2013(E) (Sept 2013), 1--64.

[15]

ISO/IEC/IEEE. 2013. ISO/IEC/IEEE International Standard for Software and systems engineering - Software testing, Part 3: Test documentation. ISO/IEC/IEEE 29119-3:2013(E) (Sept 2013), 1--138.

[16]

Riitta Jääskeläinen. 2010. Think-aloud protocol. Handbook of translation studies 1 (2010), 371--374.

[17]

James A Jones, Mary Jean Harrold, and John Stasko. 2002. Visualization of test information to assist fault localization. In 24th International Conference on Software Engineering (ICSE 2002). Orlando, FL, USA, 467--477.

Digital Library

[18]

Tomek Kaczanowski. 2013. Practical Unit Testing with JUnit and Mockito. Tomasz Kaczanowski.

Digital Library

[19]

Jens Knodel, Dirk Muthig, and Matthias Naab. 2006. Understanding Software Architectures by Visualization-An Experiment with Graphical Elements. In 13th Working Conference on Reverse Engineering (WCRE 2006). Benevento, Italy, 39--50.

Digital Library

[20]

Qingzhou Luo, Farah Hariri, Lamyaa Eloussi, and Darko Marinov. 2014. An empirical analysis of flaky tests. In 22nd ACM SIGSOFT International Symposium on Foundations of Software Engineering (FSE 2014). Hong Kong, China, 643--653.

Digital Library

[21]

Glenford J Myers, Corey Sandler, and Tom Badgett. 2011. The art of software testing. John Wiley & Sons.

Digital Library

[22]

Kshirasagar Naik and Priyadarshi Tripathy. 2011. Software testing and quality assurance: theory and practice. John Wiley & Sons.

Digital Library

[23]

Stanislaw Osinski and Dawid Weiss. 2005. A concept-driven algorithm for clustering search results. IEEE Intelligent Systems 20, 3 (2005), 48--54.

Digital Library

[24]

pitest. 2017. pitest. http://pitest.org/. (2017).

[25]

Marcelo Schots and Cláudia Werner. 2015. On Mapping Goals and Visualizations: Towards Identifying and Addressing Information Needs. In 3rd Workshop on Software Visualization, Evolution, and Maintenance (VEM 2015). 25--32.

[26]

Gregory Tassey. 2002. The economic impacts of inadequate infrastructure for software testing. National Institute of Standards and Technology, RTI Project 7007, 011 (2002).

[27]

Jan Valdman. 2001. Log file analysis. Tech. Rep. DCSE/TR-2001-04 (2001).

[28]

Iris Vessey. 1985. Expertise in debugging computer programs: A process analysis. International Journal of Man-Machine Studies 23, 5 (1985), 459--494.

Cited By

Cury OAvelino GSantos Neto PBritto RTúlio Valente M(2022)Identifying Source Code File ExpertsProceedings of the 16th ACM / IEEE International Symposium on Empirical Software Engineering and Measurement10.1145/3544902.3546243(125-136)Online publication date: 19-Sep-2022
https://dl.acm.org/doi/10.1145/3544902.3546243
Zheng WLiu GZhang MChen XZhao W(2021)Research Progress of Flaky Tests2021 IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER)10.1109/SANER50967.2021.00081(639-646)Online publication date: Mar-2021
https://doi.org/10.1109/SANER50967.2021.00081
Kim JSavchenko VShin KSorokin KJeon HPankratenko GMarkov SKim CRothermel GBae D(2020)Automatic abnormal log detection by analyzing log history for providing debugging insightProceedings of the ACM/IEEE 42nd International Conference on Software Engineering: Software Engineering in Practice10.1145/3377813.3381371(71-80)Online publication date: 27-Jun-2020
https://dl.acm.org/doi/10.1145/3377813.3381371
Show More Cited By

Index Terms

Analysis of test log information through interactive visualizations
1. Software and its engineering
  1. Software creation and management
    1. Software verification and validation
      1. Operational analysis
      2. Software defect analysis
        Software testing and debugging

Recommendations

Constructing Interactive Visualizations with iVoLVER
CHI EA '16: Proceedings of the 2016 CHI Conference Extended Abstracts on Human Factors in Computing Systems

iVoLVER, the Interactive Visual Language for Visualization Extraction and Reconstruction, is a web-based pen and touch system that graphically supports the construction of interactive visualizations and allows the extraction of data from different types ...
Information Visualization for Interactive Information Retrieval
CHIIR '18: Proceedings of the 2018 Conference on Human Information Interaction & Retrieval

As search tasks move beyond targeted search and into the domain of complex search, a substantial cognitive burden is placed on the searcher to craft and refine their queries, evaluate and explore among the search results, and ultimately make use of what ...
Information visualizations for knowledge acquisition: The impact of dimensionality and color coding

Information visualizations - interactive graphical representations of large amounts of abstract data which do not have a natural visual representation - have mainly been used to support information retrieval. This article investigates whether ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences

ICPC '18: Proceedings of the 26th Conference on Program Comprehension

May 2018

423 pages

ISBN:9781450357142

DOI:10.1145/3196321

General Chair:
Foutse Khomh
École Polytechnique de Montréal, Canada
,
Program Chairs:
Chanchal K. Roy
University of Saskatchewan, Canada
,
Janet Siegmund
University of Passau, Germany

Copyright © 2018 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGSOFT: ACM Special Interest Group on Software Engineering
IEEE-CS: Computer Society

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 28 May 2018

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Conference

ICSE '18

Sponsor:

SIGSOFT
IEEE-CS

ICSE '18: 40th International Conference on Software Engineering

May 28 - 29, 2018

Gothenburg, Sweden

Upcoming Conference

ICSE 2025

2025 IEEE/ACM 46th International Conference on Software Engineering

April 26 - May 3, 2025

Ottawa , ON , Canada

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

5
Total Citations
View Citations
193
Total Downloads

Downloads (Last 12 months)13
Downloads (Last 6 weeks)1

Reflects downloads up to 12 Dec 2024

Other Metrics

View Author Metrics

Citations

Cited By

Cury OAvelino GSantos Neto PBritto RTúlio Valente M(2022)Identifying Source Code File ExpertsProceedings of the 16th ACM / IEEE International Symposium on Empirical Software Engineering and Measurement10.1145/3544902.3546243(125-136)Online publication date: 19-Sep-2022
https://dl.acm.org/doi/10.1145/3544902.3546243
Zheng WLiu GZhang MChen XZhao W(2021)Research Progress of Flaky Tests2021 IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER)10.1109/SANER50967.2021.00081(639-646)Online publication date: Mar-2021
https://doi.org/10.1109/SANER50967.2021.00081
Kim JSavchenko VShin KSorokin KJeon HPankratenko GMarkov SKim CRothermel GBae D(2020)Automatic abnormal log detection by analyzing log history for providing debugging insightProceedings of the ACM/IEEE 42nd International Conference on Software Engineering: Software Engineering in Practice10.1145/3377813.3381371(71-80)Online publication date: 27-Jun-2020
https://dl.acm.org/doi/10.1145/3377813.3381371
Campusano MBergel A(2019)VizRob: Effective Visualizations to Debug Robotic Behaviors2019 Third IEEE International Conference on Robotic Computing (IRC)10.1109/IRC.2019.00021(86-93)Online publication date: Feb-2019
https://doi.org/10.1109/IRC.2019.00021
Artho CPande MTang QZimmermann TLawall JMarinov D(2019)Visual analytics for concurrent Java executionsProceedings of the 34th IEEE/ACM International Conference on Automated Software Engineering10.1109/ASE.2019.00112(1102-1105)Online publication date: 10-Nov-2019
https://dl.acm.org/doi/10.1109/ASE.2019.00112

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents