Assessing the Effect of Programming Language and Task Type on Eye Movements of Computer Science Students

Learning programming involves reading and comprehension, which in turn means that findings of programming comprehension studies can help computer science educators with shaping their course content and updating their teaching methods to enhance learning. There is a challenge, however, in relating the findings from empirical studies to teaching methods and learning. Izu et al. provide some examples of teaching methods and materials related to program comprehension [27]. In their critical review, Schulte et al. [59] analyze and compare the different programming comprehension models and provide some insights on how these models can be applied to teaching methods and provide students with better and more effective learning tasks. They conclude that the role of domain knowledge for program comprehension should be highlighted more in education, the instruments used in empirical studies might be useful to test learning outcomes, and the differences in expert and novice understanding of programs should also be discussed and investigated in programming education. They state that experts have a flexible and navigational mental representation (i.e., their representations are more than the sum of the elements from reading) of programs, which is in line with findings of Busjahn et al. [14] who found that novices have a more linear reading method when working on programs compared to experts. Additionally, There have been studies on challenges and barriers in learning how to program, and how the programming language affects learning. Stefik et al. [71] conducted a study on how novices learn syntax, and how learning varies across different programming languages. Their results showed the importance of syntax for novice programming, how variations in syntax affect the accuracy rate, and that some syntactic designs in languages were easier to comprehend for novices compared to others.

Due to the importance of learning programming, computer education researchers are interested in how students read and trace code, which is directly related to code comprehension. There are several works that have investigated the relationship between reading, tracing, and writing skills in programming students who have recently started to learn how to code [18, 36, 37, 78]. They all found direct relationships between tracing and reading code and code comprehension, and that students who write better code are better at tracing and explaining it as well. In this article, we use eye tracking as a method to track student gazes on the code, which can give us insight into how they trace and understand code to solve specific problems.

There are several other approaches for analyzing the patterns of learning in students. Allevato et al. [4] analyzed the sequence of submitted assignments from students and allowed them to change their code so that it could pass the grading criteria and test cases. They realized that students who did better on assignments made more increasing changes and worked incrementally, compared to students who did poorly on assignments who made decreasing changes. Mansoor et al. [39] studied how students comprehend and learn the Alloy language, a specification language based on first-order logic. They created detailed tutorials for all participants, taught the language in some classes, and recruited some students from those classes. Additionally, they recruited non-novices who already knew the language, to compare the work patterns of novices and non-novices. They found a similar pattern of incremental changes when looking at the Alloy analyzer interaction logs, and that novice participants who made more edits and executed the models more often, had higher accuracy scores. Piech et al. [54] used another method to model how students learn, studying how they get to their final solution by capturing snapshots from compilations to analyze the changes between each compile. They present how their modeling can inform about the similarity and differences of learning patterns, and be a predictive model about each student’s progress over the course of an assignment. We believe that using eye tracking while studying a participant’s problem solving patterns, gives us more insight into how they make changes and why they make those changes on code, and learning these patterns can be very beneficial for educational purposes. If, as an instructor, you are able to see how your student is reading the code in real time, then you can instruct them to correct their focus so they can get to the bug quicker.

Previous studies have also investigated the different approaches programmers employ to achieve program comprehension. When tasked with a fill-in-the-blank line in a program, programmers employed several different strategies to understand the program before filling in the line with the correct code [19]. The authors found that most programmers began trying to identify the subgoals of a program such as looping through an array or a maximum algorithm. If a failure to understand a subgoal occurs, then additional strategies are employed to resolve the failure. Margulieux and Morrison et al. have also studied subgoal labels in Python and Java [41, 43].

In recent years, eye-tracking studies have been performed to investigate program comprehension in novice and expert developers. In this section, we present some studies that are closely related. For further information on the state of eye-tracking studies done on program comprehension, we direct the reader to prior systematic literature reviews [46, 62].

Given the importance of reading in code comprehension tasks, it logically follows that important insights can be gained from tracking eye movements while participants work on code. Busjahn et al. [15] present eye tracking as a tool to complement the methods used in computer science education research. Eye movements are an objective resource when it comes to studying a programmer’s mental model and reading patterns [1, 50]. Eye movements are a proxy for attention, which provide insight into what information people are considering and in what order they do specific tasks. Given these properties and what can be learned from eye-tracking data, it adds a lot of value to studies that want to explore comprehension through analyzing reading patterns.

Busjahn et al. conducted a study to look into the differences in how individuals read code versus how they read words, with an additional focus on programmer expertise [14]. Fourteen novices and nine professional software developers had their eye movements tracked while they read Java code. They found that novices looked at code in the same linear fashion that is observed when individuals read text (approximately 80% of the time). Experts, however, read code in a much less linear fashion. Since this study’s focus is on comparing reading patterns between experts and novices, they did not ask the participants to work on various types of tasks and read code in different languages. Our article tries to compare comprehension patterns in different types of tasks and languages instead.

We summarize a few relevant studies done using eye tracking in the program comprehension field. Peterson et al. examine lines developers familiar with open source systems view during summarization and try to correlate line length with the total duration of time spent on the line [51]. One of their findings is that smaller methods tend to have shorter overall fixation durations but have significantly longer durations per line. In another study the authors also investigated the information seeking behavior via eye movements of developers on Stack Overflow, which showed the importance of code snippets in the questions and answers, and showed that participants did not look at the title of a post, tags, or votes compared to the rest of the text [53]. Saddler et al. examine developer reading behavior on Stack Overflow while they search for information related to fixing bugs and building new features [57]. However, here their focus was more on the reading patterns on Stack Overflow instead of the code itself. Kevic et al. conducted one of the first eye-tracking studies on bug fixing in open source software in the Eclipse IDE using an early prototype of the iTrace framework [33]. Their study investigated how developers navigate change tasks, and they found that developers focus on a few methods while working on the tasks, and read small parts of the code within those methods to complete the tasks. Jbara et al. conducted an eye-tracking study to measure the time and effort spent reading and understanding regular code [28]. They define regular code as code that includes repetitions of the same basic pattern and is considered to be significantly longer than a non-regular version. They point out that initial code segments are read more than the later ones in regular code and also that code reading was far from being linear, as is also pointed out by Busjahn et al. [14]. Obaidellah et al. [47] look at novice programmer gaze patterns on pseudocode using eye tracking on 51 undergraduate CS students showing that as difficulty increases, the regressions between areas of interest also tend to increase. Hu et al. demonstrate that high-performing students had long fixation durations for analytical problems (more structured) and the problem-solving stage, whereas shorter fixations at the problem exploration stage of interactive problems (less structured) [26]. This study also uses images and short code snippets. Titus et al. showed via an eye-tracking study that CS students found reading error messages equally hard compared to source code [7]. Abid et al. conducted an eye-tracking study analysis of the use of top-down versus bottom-up models used during code summarization tasks [1]. They found that, on average, experts and novices read methods using the bottom-up (more focused) mental model than using top-down (bouncing around), and on average, novices spent longer gaze time during the bottom-up process than experts. Aschwanden and Crosby show that beacons are usually present in code when the longest fixation duration is over a thousand milliseconds [6]. This study shows that beacons can be based on the code content and domain of study.

Turner et al. [75] conducted an eye-tracking study comparing the accuracy and speed of both bug fixing tasks and overview tasks written in Python and C++. They found that there were no significant differences in accuracy or timing between the tasks based on the language they were written in, but they did find that there was a significant difference in the fixation rate on buggy lines of code between Python and C++. This is the only previous study we are aware of that compares two programming languages for bug localization and program overview tasks. Our article, in addition to using C++ and Python, also looks at different types of tasks that possibly require different behaviors to perform them correctly, as by nature, a programmer will approach a bug fix very differently from a feature addition. In addition, it is a more realistic study that covers realistic tasks that are more than just a few lines long. Our study is fundamentally different in data collection and instrumentation as well. Moreover, a more comprehensive visualization of fixation transitions between the lines of code is presented via scarfplots.

Recently, Kather et al. [30] studied code composition and planning while programming and they investigated the effects of composition strategies and familiarity with code on program comprehension in an eye-tracking study with students. Using eye-tracking data and retrospective interviews, students’ reading patterns were analyzed, and their mental models were studied. They found that familiarity with the template of the program makes it easier to create schemata. This study also uses images for the stimuli and excessively large areas of interest to analyze the data, which might miss some intricate details of how students navigate between chunks of code. It also does not allow the students to interact realistically with the code snippets.

To the best of our knowledge, we are not aware of any studies that compare eye movements on C++ and Python with respect to different types of software tasks in realistic scenarios, such as using an IDE. We bridge this gap in the literature and add to the empirical evidence by discussing the differences and similarities of comprehension behaviors of student programmers who have worked on these different tasks.

In this section, we review various models and theoretical frameworks in the field of program comprehension. Program comprehension is a sub-field of software engineering that deals with building a mental representation (albeit subjective) of the code while solving a task. Storey et al. provide a consolidated review of all the theories, methods, and tools developed in the software engineering space for program comprehension [72].

Schulte et al. compare and contrast different program comprehension models (from an educational perspective) and discuss how a block model [58] for program comprehension is mapped to various other prior models [59]. Several theories were proposed in the early 1980s. Brooks introduced the concept of top-down comprehension [12], driven mainly by a hypothesis and beacons [80] in the code. Soloway and Ehrlich used a similar model using programming plans or rules of discourse that are used to form a mental representation [70]. Schneiderman et al. present a bottom-up comprehension model where programmers start with individual code items to get to higher level abstractions of what the code does [66]. Pennington et al. discuss a framework where two models, program/control flow and data flow, evolve simultaneously [50]. Letovsky provides a more opportunistic model approach where programmers use and switch between top-down and bottom-up models as needed [35]. Von Mayrhauser et al. build on previous models to introduce an integrated metamodel that consists of a top-down model, a program model, and a situation model [79] where programmers switch between these and build them simultaneously.

With respect to determining program complexity from a cognitive perspective, Duran et al. use Cognitive Load Theory and the Model of Hierarchical Complexity that extends Soloway’s plan-based analysis of programs to a finer granularity [21]. Ajami et al. also look at code complexity and how syntax, predicates, and idioms could have an effect on it [3]. They found for loops to be significantly harder than ifs and that counting down is harder than counting up. However, they point out that there could be other factors besides the use of known idioms and syntactic structures that could affect code complexity, and more empirical evidence is required. Katzmarski and Koschke provide a programmer centric view of complexity and show that this does not coincide with complexity metrics rankings [31]. They point out that data-flow metrics align better with programmer viewpoints than control-flow metrics but even that is loosely correlated. Yu et al. provide a survey on software complexity metrics that could be used to determine task variability in program comprehension studies [83].

Izu et al. identify learning activities that address key components of program comprehension and provide a theoretical learning trajectory to guide teachers in selecting further activities in CS courses [27]. Tshukudu and Cutts propose a model describing how student novices are affected while learning different programming languages [74]. They studied students transitioning from Python to Java and vice versa and proposed ways to ease the transition process. Teague et al. use a neo-Piagetian framework that describes cognitive development stages that students go through to study simple programming concepts [73]. They show that students make many mistakes and focus on superficial aspects of the task until they reach the operational stage, at which decentration of focus occurs. That is the point where the cognitive ability to reason about abstractions and adapt skills to tasks that are closely related is formed. Clear et al. have published a report on the BRACElet project that has contributed key findings on how novices learn to program [16]. They also provide guidelines for programming problems for novices. Cunningham et al. provide support for sketching program traces on paper for code reading that correlates with greater success on code reading problems that involve loops, arrays, and conditionals [18]. They used this concept of sketching on new task types such as code writing, code ordering, and code fixing and found that different types of sketching were used for these tasks, not always with increased performance.

Not related directly to program comprehension, but generally to CS education, Nelson and Ko discuss that although theory can be helpful in interpreting designs and results, sometimes it can also inhibit progress [45]. We need to pay special attention to this observation, especially with respect to eye-tracking studies, because we have just scratched the surface when it comes to using eye tracking as a means to learn how students and experts work. There are not many studies that are conducted using eye tracking where one can do a meta analysis to come up with a theory on how students work. We may develop a working theory on how eye movements occur in different tasks, but we still need more empirical evidence to validate such behaviors.

The participants were mainly students from a large Midwestern university in the United States. Fourteen volunteers participated in the study.

Table 1 shows a summary of the demographic information collected from the participants. Eleven participants were male and three of them were female. Eight participants were between 18 and 24 years old, two participants were between 25 and 34 years old, one was between 34 and 44 years, and three were over 45 years old. There were six undergraduate students, four graduate students, and four non-students (who had just graduated) among the participants. Nine participants did not have any industry employment and experience, and five participants indicated that they had industry experience. Netbeans was the most used IDE among the participants, with six participants choosing it as one of the IDEs they use for programming. Eclipse and Visual Studio were the next popular choices, appearing in the participants’ answers five and four times, respectively.

We asked the participants to self-report their programming skills and experience levels. Siegmund et al. [68] state that self estimation is a reliable measurement of programming skills and experience. Eight participants rated their design skills as average, five rated them as above average/good, and one rated them as excellent. Six participants rated their programming skills as average, six rated them as above average/good, and two rated their skills as excellent. As for programming language specific questions, five participants ranked their C++ skills as beginner level, five ranked their skills as intermediate, one ranked their skills as average, and finally, three participants ranked their skills as advanced. Five participants had between 1 and 2 years of experience in C++ programming, three participants had between 3 and 5 years of experience, three participants had between 6 and 10 years of experience, two participants had more than 10 years of experience, and finally, one participant had no experience in C++ programming. Subsequent questions were about the participants’ skills in Python. Five participants said that they did not know Python. One ranked their skills as beginner level, five ranked their skills as intermediate, and three ranked their skills as advanced. Five participants had no experience in Python programming, five participants had between 1 and 2 years of experience, three participants had between 3 and 5 years of experience, and one participant had between 6 and 10 years of experience. Finally, the participants were asked to list the languages they could program in. Java was mentioned in the answers nine times, with C++, C and Python coming as the next most mentioned answers, respectively.

The students were recruited from a class that taught Python as an introductory programming language. All of the students were also knowledgeable in C++. There were no incentives for their participation. They were all in the CS program. None of them were students of the authors. The authors did not know any of the students personally. All participation was voluntary and done via an announcement. The study took place in a quiet eye-tracking lab where only the moderator and the participant were present without any outside distractions. The moderator was there to ensure the participant was seated at the correct distance from the eye tracker and to perform the calibration. They did not interact with the participants during the experiment. The University’s Institutional Review Board approved the study prior to its implementation.

The four different combinations of programming language and task types used in this experiment are listed in Table 2. A high level description of the programs and the programming constructs that are present in the program are also listed. Two tasks were presented in Python, and two were presented in C++. From each language category, one task was a bug fixing task, and the other was a feature addition task.

For the bug fixing tasks, we asked the participants to find the bug located in the program, write the line number they thought contained the bug, and attempt to fix the bug. They were also given the expected input and output of the program. For the feature addition tasks, we gave the participants a description of the program’s current capabilities and a description of an additional feature that they had to implement. Figure 1 shows Stimulus 1 and Stimulus 4. A complete replication package with all the tasks, programs, and eye movement data is available at Reference [40].

Participants were given all four tasks in randomly generated order. They had access to the source code in Geany,² the console output of the program, and the requirements of the task. Requirements included the expected input and output for the bug fixing tasks and the additional feature that needed to be implemented for the feature addition tasks. Figure 2 shows an image of the screen setup and these three areas. There was a trial task given to familiarize participants with the IDE setup so they could ask questions. We did not collect eye-tracking data for the trial task.

We now provide some rationale for why we chose these two types of tasks (new feature and bug fix). As developers, we perform a variety of tasks on a daily basis, such as bug fixing, feature addition, refactoring, code review, testing, reading requirements, reading to comprehend code, summarizing code, and many more. Almost all of the eye-tracking studies in program comprehension are on tasks that involve participants summarizing Java code, and very few are on fixing bugs. There are none on adding new features. Moreover, all studies (except for a few) are on short code snippets and all on Java. Besides Turner et al. [75] there are no published studies looking into eye movements on Python that we are aware of. It has been shown by Abid et al. that results derived from short code snippets are not always consistent with when you use realistic programs within an IDE to test developers [2]. To bridge this gap, we chose two of the activities we believe developers spend a lot of time on, i.e., fixing bugs, and adding new features. In the future, we will add more task categories as provided by Murphy et al. [44].

The goal of this article was to see how participants fare on different types of tasks. The tasks themselves are different categories and should not be considered comparable. The goal was to see how the same individual’s eye movements differed between the different types of tasks. The two tasks chosen are representative of what software developers typically do, i.e., fix bugs and implement new features, as also evidenced by many issue tracker systems in open source projects.

Our underlying assumption (based on theoretical frameworks such as Reference [18] that looked at different tasks albeit without eye tracking) is that bug fixing and new feature tasks would require different levels of comprehension and problem solving skills. For bug fixes, developers generally start with the bug report and/or expected input/output and try to figure out which line the bug is on by tracing backward to find the line via stack traces or some other tracing method. With new feature tasks, developers do not do as much tracing, since they are implementing forward based on the requirements they read and what the expected feature should do. Because of these reasons, we believe that solving these tasks would generate different user behaviors.

For the bug fixing tasks, the requirements of the task were somewhat comparable. One task reverses a word or phrase (C++) and the other creates a palindrome (Python). The new feature tasks, however, were slightly different, albeit they used similar constructs listed in Table 2. Since the study design is within subjects, giving very similar programs across languages would cause learning effects that we wanted to avoid. Since we recruited our participants from a Python class, we also wanted to make sure we chose stimuli with concepts already taught in the class. We asked the instructor for their syllabus and weekly schedule to ensure we used programs and concepts that was known to the students. We did not use verbatim any code from the class itself. We were not instructors for the course.

Note that the goal of this article was not to do a side by side comparison of the same task in C++ versus Python. Instead, it was to see how each participant understood C++ versus Python in two task categories. To account for the difference in lines of code in the tasks, we make sure we normalize our fixations per character, because otherwise, longer programs will always have more fixations as there is more to read (see Section 4.6 for more details on normalization). For future work, we plan to evaluate different program complexities [3, 21] within each task category. However, task complexity was not the scope of this article.

After each task, we asked the participants to rate the difficulty of the tasks, with the options: “Easy,” “Average,” and “Difficult.” For the statistical analysis, we assigned the numbers 1, 2, and 3 to these choices, respectively. We also asked the participants to rate their confidence level about each task, with the options “Not Confident,” “Somewhat Not Confident,” “Somewhat Confident,” and “Very Confident.” For the statistical analysis, we mapped these choices to the numbers 1, 2, 3, and 4, respectively.

We provide definitions for basic terminology we use throughout the article to help provide the reader with context for our study.

Program comprehension is a sub-field of software engineering/computer education that deals with a user building a mental representation (albeit subjective) of the code while solving a task.

Task Type refers to the various possible types of tasks a developer (in this case, a student) may engage in. Possibilities could be bug fixing, new feature addition, refactoring, code review, testing, and so on. In this article, we only evaluate two task types (bug fix and new feature addition).

Task refers to the actual set of artifacts that falls into the specific task category. For a bug fix task, this would be the code in the IDE, the program requirements, and expected input and output of the program. For the new feature task it would be the starter code in the IDE, current description of the program, and a description of the additional feature to be implemented. In both cases, the console output was also available to the participant. The participant is expected to engage with these artifacts to produce a result. In the case of the bug fixing task, the result would be the line that had the bug and a fix for the bug. For the new feature addition tasks, the result would be the newly written code that implements the new feature.

Bug localization (in our study) refers to the time when the participants read the line containing the bug, prior to any edits made, but do not fix the bug.

Stimuli is eye-tracking terminology and simply means anything that is tracked on the screen by the eye tracker. In our case, the Geany IDE was the main stimulus that contained within it all the artifacts that the participant saw.

Chunks refer to a line or set of contiguous lines of code with a specific logical and semantic meaning. We also refer to them as beacons [6, 80].

Areas of Interest (AOI) refer to parts of the stimulus on which eye-tracking metrics are recorded. Examples could be chunks in the code editor, the requirements area of the IDE, or the console output. The AOI is usually defined by the researcher.

Fixation is the stabilization of the eyes on an object of interest for a certain period of time. Fixations are made up of multiple raw gazes and have a duration associated with them, which we refer to as the fixation duration. Most processing happens during fixations, which is why they are a standard measure in most eye-tracking studies.

Scanpath refers to the directed path formed by saccades between fixations. It determines how the eye navigates across the stimuli.

Reading behavior refers to the percentages of fixations that appear on the various AOIs in question.

Navigation behavior refers to the scanpath on source code over time.

Editing refers to the act of modifying the code to fix a bug or implement a new feature.

We present the study procedure, including experimental setup, eye-tracking hardware, and discuss the steps for pre-processing the eye-tracking data to produce fixations on parts of the stimuli.

The measures used in this experiment are based on best practices guidelines reported in the field of program comprehension, software engineering, and eye tracking [61]. We direct the reader to Duchowski et al. [20] for a detailed theoretical description of all eye-tracking measures. Table 3 describes the metrics used to compare participants’ behavior while working on the four tasks. We specify the research questions, the metrics used to answer the questions, and the definition of the metrics. We chose metrics based on fixations, a group of metrics used in eye-tracking studies in software engineering [60, 61] that are used to measure visual effort. In prior studies, areas of interest with higher fixation count and duration are believed to have attracted more visual attention or that understanding them required more effort [61, 63, 64]. We calculated the total fixation count and duration over the given tasks, per character and specific AOIs. Furthermore, we calculated the mean fixation duration during each specific task as well. The fixation count and duration serve as a measure of visual effort when it comes to solving the different types of tasks (bug fixing and feature addition) and comparing the different programming languages (Python and C++) in our research questions.

Next, we explain why we use the fixation count per character as a metric. To compare eye-tracking data across the different programming languages, we first need to normalize the data. The programming languages C++ and Python have very different semantic structures and different lines of code. There might not be a direct equivalent construct between the languages. To account for the difference in lines of code in the tasks, we make sure we normalize our fixations, because indeed longer programs will have more fixations. We account for this in our analysis by normalizing by character. To do this, we divide eye movement duration over a token. So if the total fixation duration is 400 \(\text{ms}\) on a token of four characters, the normalized total time is 100 \(\text{ms}\). This has been done in prior work as well by Madi et al. [38] and Abid et al. [2].

Given that some of our participants stated that they did not know Python, we looked at their confidence levels in their answers and understanding of each task. This information showed us that even though these participants did not consider themselves knowledgeable in Python, they mostly had a good understanding of the tasks, and we believe that due to this fact, we can include them in the analysis for answering the research questions. The following is the description of confidence levels for each task. Since all the participants were recruited from the same Python course, they were all learning Python. We asked the students to “Rate your Python programming skills” and they had the following choices: “I don’t know Python,” “Beginner,” “Intermediate,” and “Advanced.” We also asked them to “Select years of experience in programming with Python,” and the choices were: “None,” “Between 1 and 2,” “Between 3 and 5,” “Between 6 and 10,” “More than 10.” We believe some of the students misunderstood these questions as asking about experiences prior to taking the course.

Research question 1 asks about the reading and navigation differences between C++ and Python tasks. The null and alternate hypotheses for this question are as follows.

\(LD_0\): The programming language used for the tasks does not affect the visual effort of the participants working on those tasks.

\(LD_A\): The programming language used for the tasks affects the visual effort of the participants working on those tasks.

To test our hypothesis, we calculated accuracy, time, overall fixation count and duration, and AOI fixation count and duration for the tasks. Note that we use fixation metrics as a proxy for visual effort as stated in Section 4.6. We then compared these metrics in different languages. Table 4 summarizes the metrics for this research question and shows the statistical tests for language-based differences in reading and code navigation.

We present the null and alternate hypotheses for the research question on task type differences.

\(TD_0\): The task type (bug fixing vs feature addition) does not affect the visual effort of the participants working on those tasks.

\(TD_A\): The task type (bug fixing vs feature addition) affects the visual effort of the participants working on those tasks.

Once again, we calculated accuracy, time, overall fixation count and duration, and AOI fixation count and duration for the tasks for testing our hypothesis. Table 5 summarizes the metrics for this research question and shows the statistical tests for task-based differences in reading and code navigation.

Next, we turn our analysis to the two bug fixing tasks. To address the third research question, we look at the scan patterns (scan paths) of the participants and the distribution of fixations on the buggy lines. This research question is exploratory in nature and does not have a formal hypothesis. Since we stop mapping gazes to lines when we detect edits (refer to Section 4.5.3 for more information), we report eye gaze distributions in this section until the first edit. We refer to this as the bug localization phase. Note that there were a lot fewer edits in the bug fix task, since most bugs were limited to 1 line and required minor changes.

Since new feature tasks, by definition, require the students to change the code, we investigate editing behavior to address this research question. We explain in Section 4.5.3 why eye-tracking measures are not used during editing as they are not reliably mapped to edited code and no vendor-based software supports this to date. Research prototypes such as Reference [22] have some support for editing (only in Atom), however, our study was not done with their framework. We plan on using the iTrace framework for future studies as it significantly simplifies the mapping of gaze data on tokens [23, 84]. Similar to RQ3, this research question is exploratory in nature and does not have a formal hypothesis. Table 9 shows the different metrics used in RQ4 and the results of the statistical tests comparing them.

We discuss the possible threats to validity for internal, external, construct, and conclusion and state how we tried to mitigate them.

Internal validity: While we do compare tasks based on the language and task type to determine if language and task type have an effect on eye movement patterns of programmers, it is possible for a different but comparable task to have a different difficulty level to a certain programmer, meaning that even if we present equally difficult but different tasks some programmers may find one task to be more difficult for them to complete. In addition, we tried to have bug fixing tasks and feature addition tasks have a similar level of difficulty. However, we do not claim that these are representative of all bug fix and feature addition tasks.

External validity: For external validity, the small size of the programs used may limit the generalizability of our results. In addition, our participants were mainly students. Because of this, our results may not generalize to a larger population of programmers, including professionals.

Construct validity: Addressing construct validity, we used some thresholds in our analysis. For example, in RQ3, we looked at the first five fixations before any fixation on the line containing the bug. We also chose to use ten segments in the bug localization timeline. Increasing the segment count will provide additional granularity, but the small sample count may cause gaps in the dataset to appear. We believe that ten segments balance these two goals for our purposes. In RQ4, we used the time until the first edit as a proxy for when participants were done understanding the program and began to add a feature. However, a participant can begin adding a feature and continue reading the program for comprehension after editing has begun. In RQ3, we looked at the distribution of fixation duration over the AOIs of the source code. These AOIs are mostly line-based AOIs, but some closely related lines were grouped together as a single chunk. Chunk-based analysis has been conducted in previous studies [56], but the decision over which lines to group together can influence the analysis. We mitigate this by only directly comparing the distribution of fixation durations over AOIs of the same task. These source code AOIs are also only tracked before an edit occurs. To mitigate this, we only used the source code AOIs for the bug localization part of the bug fixing tasks, which should occur in its entirety before they attempt to fix the bug.

Conclusion validity: Finally, for conclusion validity, we used the appropriate statistical tests for our inferential statistics.

With respect to RQ1, which tried to determine differences in programming languages, we did indeed see a difference based on the normalized total fixation count between C++ and Python. Students spent more time looking at the Console output when performing Python tasks. We do not believe this is due to noise in the data, because both Python and C++ had I/O operations (see Table 4.3). We believe that it might be possible that the Python programs were easier to change and the students got immediate feedback in the console. The data seems to support this assumption. This result seems to align with Tshukudu and Cutts [74], where different models were needed to transition between programming languages. Murphy et al. also point to the Console view being selected the most during their collection of interaction data from Java developers in Eclipse [44].

With respect to RQ2-RQ4, which tried to determine differences in task type, the results show significant differences in the Total Fixation Count and Total Fixation Duration overall (normalized per character), indicating that participants had more frequent and longer normalized fixations overall in the bug fixing tasks. These results align with what was reported by Cunningham et al. [18], where they found the behaviors to change when different task types are used. One possible reason why the fixation count might be higher for bug fixing is because the reading strategy when looking for bugs is very different from reading code just to understand what it is doing. When finding and fixing bugs, one zeroes into certain parts and traces and re-reads them.

With respect to RQ3 and bug fixing behavior, fixations are found closer to the line containing the bug right before they look at buggy lines. This behavior is also found in prior work by Peterson et al. [52], indicating that participants view related lines together and using chunking as a mechanism to map eye gazes is replicable in other studies and tasks as well.

Finally, with RQ4, we found that for both C++ and Python, participants could identify where to start adding a feature (noted by the time to first edit) quicker and potentially iterate over solutions to solve the task correctly. Brown et al. collected five years of programmer activity data in a Java IDE, namely, BlueJ [13]. Part of this data includes edit sequences of novices. However, they state that no study has made use of the code execution and code editing sequences as of yet. It would be interesting to see how their results relate to what we found in our study with respect to the time to first edit and task performance. This is left as a future exercise. The individual differences we see in our study are also reported in Jbara et al. [28].

This is one of the first eye-tracking studies that look at the same individual performing two task types in two programming languages. None of the prior work used the visualization plots shown in Figures 5 and 6. This was a new form of data visualization and analysis that is richer than what is presented in Uwano et al. [77] and Sharif et al. [63]. In the future, researchers looking at scan patterns can compare not just lines but chunks of lines across time. This is important, because sometimes a programming plan [55] or beacon [12, 35] is not necessarily encapsulated in just one line. We believe this method of comparison opens up new avenues of research for comparing studies with each other in a more scalable way. When analyzing eye-tracking studies, it is also important to account for individual differences that are quite common. We see this in our analysis of the scarfplots in our study but also in other studies in the literature, such as in Jbara et al. [28]. CS education researchers can also benefit from these results by building better tools that guide novices in recognizing bugs, thereby advancing the state of the art of teaching novices.

The fact that we have noticed differences in task type within the same individual tells us that the type of task is extremely important, and as education researchers, we should be studying all the different types of tasks that developers perform on a daily basis. We name a few, such as refactoring, summarization, bug fixing, new feature addition, testing, and code review. Eye tracking has only mainly studied summarization, with a few papers looking into bug localization and only one on code review [11]. In addition, most work is done on the Java programming language. It is time to start branching out to other languages, using more realistic tasks, and also multiple task types with varying complexities [3, 21]. Especially now that we have eye-tracking frameworks such as iTrace [23, 84] that make the running and mapping of gazes to tokens relatively easy and straightforward.

Kersten and Murphy provide a task context model to help with developer productivity [32]. The context is created by monitoring a programmer’s activity and extracting structural relationships between program artifacts. CS education researchers can consider doing something similar with eye-tracking data where when enabled, the eyes are tracked while the programmer is fixing a bug. Later, these scan paths can eventually be used to replay the thought process to the same or another developer via visualizations. Such future tools would help recommendation systems as well, where eye gaze history could be used to recommend areas a student should look at, based on how they have viewed it in the past to keep their mental model in sync with their prior debugging session. Given the advancement in eye-tracking infrastructure [23, 84] and the affordability of research-grade trackers, this is not a far fetched goal.

Our findings show a more substantial difference in eye movement patterns in different task types than in various programming languages. This finding indicates that the comprehension patterns differ regarding the goal of the task at hand and suggests a need for finding different teaching techniques for solving various types of programming tasks, no matter what the programming language taught to the students is. Further investigation into differences of programming languages versus differences of tasks on learning and comprehension, can offer some insight into “Which programming languages should we teach to students?” [48] and can help in determining what factors other than language are the most important in program comprehension.

Due to the comprehension pattern differences between tasks, we also suggest that instructors try to include different tasks in programming homework (e.g., bug fixing, adding features, and summarization) to improve different program comprehension skills in the students, instead of only focusing on full implementation of specific problems. Students need more practice reading code that is not written by them so they can practice their program comprehension skills and learn when to switch back and forth between different models of comprehension [12, 35, 50, 58, 79]. It will also prove to be more useful in their future careers, as software developers do incremental work on partial code similar to subgoals [43] instead of always writing code from scratch.

Finally, CS educators can better support student debugging if they know what novice students typically look at during various types of tasks. They can also actively teach students not to fear editing the code early on, because we see a correlation between time to edit and accuracy in feature addition task performance.