More Web Proxy on the site http://driver.im/

research-article

Developer reading behavior while summarizing Java methods: size and context matters

Authors:

Natalia Dragan,

Hend Alrasheed,

Jonathan I. MaleticAuthors Info & Claims

ICSE '19: Proceedings of the 41st International Conference on Software Engineering

Pages 384 - 395

https://doi.org/10.1109/ICSE.2019.00052

Published: 25 May 2019 Publication History

Abstract

An eye-tracking study of 18 developers reading and summarizing Java methods is presented. The developers provide a written summary for methods assigned to them. In total, 63 methods are used from five different systems. Previous studies on this topic use only short methods presented in isolation usually as images. In contrast, this work presents the study in the Eclipse IDE allowing access to all the source code in the system. The developer can navigate via scrolling and switching files while writing the summary. New eye-tracking infrastructure allows for this improvement in the study environment. Data collected includes eye gazes on source code, written summaries, and time to complete each summary. Unlike prior work that concluded developers focus on the signature the most, these results indicate that they tend to focus on the method body more than the signature. Moreover, both experts and novices tend to revisit control flow terms rather than reading them for a long period. They also spend a significant amount of gaze time and have higher gaze visits when they read call terms. Experts tend to revisit the body of the method significantly more frequently than its signature as the size of the method increases. Moreover, experts tend to write their summaries from source code lines that they read the most.

References

[1]

A. J. Ko, B. A. Myers, M. J. Coblenz, and H. H. Aung, "An exploratory study of how developers seek, relate, and collect relevant information during software maintenance tasks," IEEE Transactions on Software Engineering, vol. 32, no. 12, pp. 971--987, 2006.

Digital Library

[2]

T. D. LaToza, G. Venolia, and R. DeLine, "Maintaining mental models: A study of developer work habits," in Proceedings of the 28th International Conference on Software Engineering, ser. ICSE '06, 2006, pp. 492--501.

Digital Library

[3]

B. Fluri, M. Wursch, and H. C. Gall, "Do code and comments co-evolve? on the relation between source code and comment changes," in 14th Working Conference on Reverse Engineering (WCRE 2007), 2007, pp. 70--79.

Digital Library

[4]

M. E. Crosby and J. Stelovsky, "How do we read algorithms? a case study," Computer, vol. 23, no. 1, pp. 25--35, 1990.

Digital Library

[5]

S. C. B. de Souza, N. Anquetil, and K. M. de Oliveira, "A study of the documentation essential to software maintenance," in Proceedings of the 23rd Annual International Conference on Design of Communication: Documenting & Designing for Pervasive Information, ser. SIGDOC '05, 2005, pp. 68--75.

Digital Library

[6]

G. Sridhara, E. Hill, D. Muppaneni, L. Pollock, and K. Vijay-Shanker, "Towards automatically generating summary comments for java methods," in Proceedings of the IEEE/ACM International Conference on Automated Software Engineering, ser. ASE '10, 2010, pp. 43--52.

Digital Library

[7]

P. W. McBurney and C. McMillan, "Automatic documentation generation via source code summarization of method context," in Proceedings of the 22Nd International Conference on Program Comprehension, ser. ICPC 2014, 2014, pp. 279--290.

Digital Library

[8]

S. Haiduc, J. Aponte, L. Moreno, and A. Marcus, "On the use of automated text summarization techniques for summarizing source code," in 2010 17th Working Conference on Reverse Engineering, 2010, pp. 35--44.

Digital Library

[9]

B. P. Eddy, J. A. Robinson, N. A. Kraft, and J. C. Carver, "Evaluating source code summarization techniques: Replication and expansion," in 2013 21st International Conference on Program Comprehension (ICPC), 2013, pp. 13--22.

[10]

L. Moreno, J. Aponte, G. Sridhara, A. Marcus, L. Pollock, and K. Vijay-Shanker, "Automatic generation of natural language summaries for java classes," in 2013 21st International Conference on Program Comprehension (ICPC), 2013, pp. 23--32.

[11]

N. J. Abid, N. Dragan, M. L. Collard, and J. I. Maletic, "Using stereotypes in the automatic generation of natural language summaries for c++ methods," in 2015 IEEE International Conference on Software Maintenance and Evolution (ICSME), 2015, pp. 561--565.

Digital Library

[12]

G. Sridhara, L. Pollock, and K. Vijay-Shanker, "Automatically detecting and describing high level actions within methods," in Proceedings of the 33rd International Conference on Software Engineering, ser. ICSE '11, 2011, pp. 101--110.

Digital Library

[13]

G. Sridhara, "Generating parameter comments and integrating with method summaries," in 2011 IEEE 19th International Conference on Program Comprehension, 2011, pp. 71--80.

Digital Library

[14]

S. Badihi and A. Heydarnoori, "Crowdsummarizer: Automated generation of code summaries for java programs through crowdsourcing," IEEE Software, vol. 34, no. 2, pp. 71--80, 2017.

Digital Library

[15]

Y. Wan, Z. Zhao, M. Yang, G. Xu, H. Ying, J. Wu, and S. Y. Philip, "Improving automatic source code summarization via deep reinforcement learning," in The 33rd IEEE/ACM International Conference on Automated Software Engineering, 2018.

Digital Library

[16]

L. Moreno and A. Marcus, "Automatic software summarization: the state of the art," in 2017 IEEE/ACM 39th International Conference on Software Engineering Companion (ICSE-C), 2017, pp. 511--512.

Digital Library

[17]

P. Rodeghero, C. McMillan, P. W. McBurney, N. Bosch, and S. D'Mello, "Improving automated source code summarization via an eye-tracking study of programmers," in Proceedings of the 36th International Conference on Software Engineering, ser. ICSE 2014, 2014, pp. 390--401.

Digital Library

[18]

T. R. Shaffer, J. L. Wise, B. M. Walters, S. C. Müller, M. Falcone, and B. Sharif, "itrace: Enabling eye tracking on software artifacts within the ide to support software engineering tasks," in Proceedings of the 2015 10th Joint Meeting on Foundations of Software Engineering, ser. ESEC/FSE 2015, 2015, pp. 954--957.

Digital Library

[19]

D. T. Guarnera, C. A. Bryant, A. Mishra, J. I. Maletic, and B. Sharif, "itrace: Eye tracking infrastructure for development environments," in Proceedings of the 2018 ACM Symposium on Eye Tracking Research & Applications, ser. ETRA '18, 2018, pp. 105:1--105:3.

Digital Library

[20]

Z. Sharafi, Z. Soh, and Y. Guéhéneuc, "A systematic literature review on the usage of eye-tracking in software engineering," Information and Software Technology, vol. 67, pp. 79 -- 107, 2015.

Digital Library

[21]

B. Sharif, J. Meinken, T. Shaffer, and H. Kagdi, "Eye movements in software traceability link recovery," Empirical Software Engineering, vol. 22, no. 3, pp. 1063--1102, Jun 2017.

Digital Library

[22]

Z. Sharafi, T. Shaffer, B. Sharif, and Y. Guéhéneuc, "Eye-tracking metrics in software engineering," in 2015 Asia-Pacific Software Engineering Conference (APSEC), 2015, pp. 96--103.

[23]

U. Obaidellah, M. Al Haek, and P. C.-H. Cheng, "A survey on the usage of eye-tracking in computer programming," ACM Comput. Surv., vol. 51, no. 1, pp. 5:1--5:58, Jan. 2018.

Digital Library

[24]

T. Barik, J. Smith, K. Lubick, E. Holmes, J. Feng, E. Murphy-Hill, and C. Parnin, "Do developers read compiler error messages?" in Proceedings of the 39th International Conference on Software Engineering, ser. ICSE '17, 2017, pp. 575--585.

Digital Library

[25]

R. Bednarik and M. Tukiainen, "An eye-tracking methodology for characterizing program comprehension processes," in Proceedings of the 2006 Symposium on Eye Tracking Research & Applications, ser. ETRA '06, 2006, pp. 125--132.

Digital Library

[26]

R. Bednarik, "Temporal eye-tracking data: Evolution of debugging strategies with multiple representations," in Proceedings of the 2008 Symposium on Eye Tracking Research & Applications, ser. ETRA '08, 2008, pp. 99--102.

Digital Library

[27]

T. Busjahn, R. Bednarik, A. Begel, M. Crosby, J. H. Paterson, C. Schulte, B. Sharif, and S. Tamm, "Eye movements in code reading: Relaxing the linear order," in 2015 IEEE 23rd International Conference on Program Comprehension, 2015, pp. 255--265.

Digital Library

[28]

P. Rodeghero and C. McMillan, "An empirical study on the patterns of eye movement during summarization tasks," in 2015 ACM/IEEE International Symposium on Empirical Software Engineering and Measurement (ESEM), vol. 00, 2015, pp. 1--10.

[29]

A. Jbara and D. G. Feitelson, "How programmers read regular code: a controlled experiment using eye tracking," Empirical Software Engineering, vol. 22, no. 3, pp. 1440--1477, Jun 2017.

Digital Library

[30]

S. Lee, D. Hooshyar, H. Ji, K. Nam, and H. Lim, "Mining biometric data to predict programmer expertise and task difficulty," Cluster Computing, Jan 2017.

[31]

H. Uwano, M. Nakamura, A. Monden, and K.-i. Matsumoto, "Analyzing individual performance of source code review using reviewers eye movement," in Proceedings of the 2006 Symposium on Eye Tracking Research & Applications, ser. ETRA '06, 2006, pp. 133--140.

Digital Library

[32]

B. Sharif, M. Falcone, and J. I. Maletic, "An eye-tracking study on the role of scan time in finding source code defects," in Proceedings of the Symposium on Eye Tracking Research and Applications, ser. ETRA '12, 2012, pp. 381--384.

Digital Library

[33]

A. Begel and H. Vrzakova, "Eye movements in code review," in Proceedings of the Workshop on Eye Movements in Programming, ser. EMIP '18, 2018, pp. 5:1--5:5.

Digital Library

[34]

P. Rodeghero, C. Liu, P. W. McBurney, and C. McMillan, "An eye-tracking study of java programmers and application to source code summarization," IEEE Transactions on Software Engineering, vol. 41, no. 11, pp. 1038--1054, 2015.

Digital Library

[35]

K. Kevic, B. M. Walters, T. R. Shaffer, B. Sharif, D. C. Shepherd, and T. Fritz, "Tracing software developers' eyes and interactions for change tasks," in Proceedings of the 2015 10th Joint Meeting on Foundations of Software Engineering, ser. ESEC/FSE 2015, 2015, pp. 202--213.

Digital Library

[36]

K. Kevic, B. Walters, T. Shaffer, B. Sharif, D. C. Shepherd, and T. Fritz, "Eye gaze and interaction contexts for change tasks - observations and potential," Journal of Systems and Software, vol. 128, pp. 252--266, 2017.

Digital Library

[37]

X. Wang, L. Pollock, and K. Vijay-Shanker, "Developing a model of loop actions by mining loop characteristics from a large code corpus," in 2015 IEEE International Conference on Software Maintenance and Evolution (ICSME), 2015, pp. 51--60.

Digital Library

[38]

X. Wang, "Automatically generating natural language descriptions for object-related statement sequences," in 2017 IEEE 24th International Conference on Software Analysis, Evolution and Reengineering (SANER), 2017, pp. 205--216.

[39]

N. Dragan, M. L. Collard, and J. I. Maletic, "Reverse engineering method stereotypes," in 2006 22nd IEEE International Conference on Software Maintenance, 2006, pp. 24--34.

Digital Library

[40]

N. Dragan, "Automatic identification of class stereotypes," in 2010 IEEE International Conference on Software Maintenance, 2010, pp. 1--10.

Digital Library

[41]

N. Abid, N. Dragan, M. L. Collard, and J. I. Maletic, "The evaluation of an approach for automatic generated documentation," in 2017 IEEE International Conference on Software Maintenance and Evolution (ICSME), 2017, pp. 307--317.

[42]

P. W. Mcburney and C. Mcmillan, "An empirical study of the textual similarity between source code and source code summaries," Empirical Softw. Engg., vol. 21, no. 1, pp. 17--42, Feb. 2016.

Digital Library

[43]

K. Rayner, "Eye movements in reading and information processing: 20 years of research," vol. 124, no. 3, pp. 372--422, 00 1998.

[44]

P. Olsson, "Real-time and offline filters for eye tracking," p. 42, 2007.

[45]

M. L. Collard, M. J. Decker, and J. I. Maletic, "Lightweight transformation and fact extraction with the srcml toolkit," in 2011 IEEE 11th International Working Conference on Source Code Analysis and Manipulation, 2011, pp. 173--184.

Digital Library

[46]

B. Liblit, A. Begel, and E. Sweetser, "Cognitive perspectives on the role of naming in computer programs," in Proceedings of the 18th Annual Psychology of Programming Workshop, ser. PPIG '06, Brighton, United Kingdom, 2006.

[47]

S. Siegel, Nonparametric statistics for the behavioral sciences, ser. McGraw-Hill series in psychology.

[48]

H. Abdi. (2007) The kendall rank correlation coefficient. {Online}. Available: http://www.utdallas.edu/~herve/Abdi-KendallCorrelation2007-pretty.pdf

Cited By

Wong ABucci PBeschastnikh IFedorova A(2024)Making Sense of Multi-threaded Application Performance at Scale with NonSequiturProceedings of the ACM on Programming Languages10.1145/36897938:OOPSLA2(2325-2354)Online publication date: 8-Oct-2024
https://dl.acm.org/doi/10.1145/3689793
Karas ZBansal AZhang YLi TMcMillan CHuang Y(2024)A Tale of Two Comprehensions? Analyzing Student Programmer Attention during Code SummarizationACM Transactions on Software Engineering and Methodology10.1145/366480833:7(1-37)Online publication date: 26-Aug-2024
https://dl.acm.org/doi/10.1145/3664808
Grabinger LHauser FWolff CMottok J(2024)On Eye Tracking in Software EngineeringSN Computer Science10.1007/s42979-024-03045-35:6Online publication date: 26-Jul-2024
https://dl.acm.org/doi/10.1007/s42979-024-03045-3
Show More Cited By

Recommendations

Java 1.2 Developer's Handbook
A Deeper Analysis of AOI Coverage in Code Reading
ETRA '21 Short Papers: ACM Symposium on Eye Tracking Research and Applications

The proportion of areas of interest that are covered with gaze is employed as metric to compare natural-language text and source code reading, as well as novice and expert programmers’ code reading behavior. Two levels of abstraction are considered for ...
Beyond gaze: preliminary analysis of pupil dilation and blink rates in an fMRI study of program comprehension
EMIP '18: Proceedings of the Workshop on Eye Movements in Programming

Researchers have been employing psycho-physiological measures to better understand program comprehension, for example simultaneous fMRI and eye tracking to validate top-down comprehension models. In this paper, we argue that there is additional value in ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences

ICSE '19: Proceedings of the 41st International Conference on Software Engineering

May 2019

1318 pages

General Chair:
Joanne M. Atlee
University of Waterloo, Canada
,
Program Chairs:
Tevfik Bultan
University of California, Santa Barbara
,
Jon Whittle
Monash University, Australia

Sponsors

SIGSOFT: ACM Special Interest Group on Software Engineering
IEEE-CS: Computer Society

Publisher

IEEE Press

Publication History

Published: 25 May 2019

Check for updates

Badges

Author Tags

Qualifiers

Research-article

Conference

ICSE '19

Sponsor:

SIGSOFT
IEEE-CS

ICSE '19: 41st International Conference on Software Engineering

May 25 - 31, 2019

Quebec, Montreal, Canada

Acceptance Rates

Overall Acceptance Rate 276 of 1,856 submissions, 15%

Upcoming Conference

ICSE 2025

2025 IEEE/ACM 46th International Conference on Software Engineering

April 26 - May 3, 2025

Ottawa , ON , Canada

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

17
Total Citations
View Citations
231
Total Downloads

Downloads (Last 12 months)18
Downloads (Last 6 weeks)3

Reflects downloads up to 13 Dec 2024

Other Metrics

View Author Metrics

Citations

Cited By

Wong ABucci PBeschastnikh IFedorova A(2024)Making Sense of Multi-threaded Application Performance at Scale with NonSequiturProceedings of the ACM on Programming Languages10.1145/36897938:OOPSLA2(2325-2354)Online publication date: 8-Oct-2024
https://dl.acm.org/doi/10.1145/3689793
Karas ZBansal AZhang YLi TMcMillan CHuang Y(2024)A Tale of Two Comprehensions? Analyzing Student Programmer Attention during Code SummarizationACM Transactions on Software Engineering and Methodology10.1145/366480833:7(1-37)Online publication date: 26-Aug-2024
https://dl.acm.org/doi/10.1145/3664808
Grabinger LHauser FWolff CMottok J(2024)On Eye Tracking in Software EngineeringSN Computer Science10.1007/s42979-024-03045-35:6Online publication date: 26-Jul-2024
https://dl.acm.org/doi/10.1007/s42979-024-03045-3
Mansoor NPeterson CDodd MSharif B(2023)Assessing the Effect of Programming Language and Task Type on Eye Movements of Computer Science StudentsACM Transactions on Computing Education10.1145/363253024:1(1-38)Online publication date: 14-Nov-2023
https://dl.acm.org/doi/10.1145/3632530
Wyrich MBogner JWagner S(2023)40 Years of Designing Code Comprehension Experiments: A Systematic Mapping StudyACM Computing Surveys10.1145/362652256:4(1-42)Online publication date: 9-Nov-2023
https://dl.acm.org/doi/10.1145/3626522
Bansal ASharif BMcMillan C(2023)Towards Modeling Human Attention from Eye Movements for Neural Source Code SummarizationProceedings of the ACM on Human-Computer Interaction10.1145/35911367:ETRA(1-19)Online publication date: 18-May-2023
https://dl.acm.org/doi/10.1145/3591136
Aljehane SSharif BMaletic J(2023)Studying Developer Eye Movements to Measure Cognitive Workload and Visual Effort for Expertise AssessmentProceedings of the ACM on Human-Computer Interaction10.1145/35911357:ETRA(1-18)Online publication date: 18-May-2023
https://dl.acm.org/doi/10.1145/3591135
Park KWeill-Tessier PBrown NSharif BJensen NKölling M(2023)An eye tracking study assessing the impact of background styling in code editors on novice programmers’ code understandingProceedings of the 2023 ACM Conference on International Computing Education Research - Volume 110.1145/3568813.3600133(444-463)Online publication date: 7-Aug-2023
https://dl.acm.org/doi/10.1145/3568813.3600133
Wu JClause J(2023)Automated Identification of Uniqueness in JUnit TestsACM Transactions on Software Engineering and Methodology10.1145/353331332:1(1-32)Online publication date: 13-Feb-2023
https://dl.acm.org/doi/10.1145/3533313
Kuang PSöderberg ENiehorster DHöst MBavota GHao D(2023)Toward Gaze-Assisted Developer ToolsProceedings of the 45th International Conference on Software Engineering: New Ideas and Emerging Results10.1109/ICSE-NIER58687.2023.00015(49-54)Online publication date: 17-May-2023
https://dl.acm.org/doi/10.1109/ICSE-NIER58687.2023.00015
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents