Task estimation for software company employees based on computer interaction logs

Digital tools and services collect a growing amount of log data. In the software development industry, such data are integral and boast valuable information on user and system behaviors with a significant potential of discovering various trends and patterns. In this study, we focus on one of those potential aspects, which is task estimation. In that regard, we perform a case study by analyzing computer recorded activities of employees from a software development company. Specifically, our purpose is to identify the task of each employee. To that end, we build a hierarchical framework with a 2-stage recognition and devise a method relying on Bayesian estimation which accounts for temporal correlation of tasks. After pre-processing, we run the proposed hierarchical scheme to initially distinguish infrequent and frequent tasks. At the second stage, infrequent tasks are discriminated between them such that the task is identified definitively. The higher performance rate of the proposed method makes it favorable against the association rule-based methods and conventional classification algorithms. Moreover, our method offers significant potential to be implemented on similar software engineering problems. Our contributions include a comprehensive evaluation of a Bayesian estimation scheme on real world data and offering reinforcements against several challenges in the data set (samples with different measurement scales, dependence characteristics, imbalance, and with insignificant pieces of information).

We provide an excerpt of data at (Yücel 2021) upon concealing certain privacy sensitive information. Since the data set analyzed during the current study contains privacy information and relates to the performance of the software company, the entire set is available from the corresponding author only on reasonable request.

The codes generated during the current study are publicly available at our repository (Yücel 2020a).

1. Nevertheless, at this stage we consider two kinds of roles as software developer and team leader to demonstrate its potential.

2. The data collection campaign is carried out with the consent of the company. The subjects are informed in a clear manner about the nature and method of the research, and agreed to participate in the experiments.

3. Here, deployment refers to active use of application window, and not running as a background process.

4. Time is recorded in YYYY-MM-DD hh-mm-ss format but is illustrated in hh-mm-ss format in Table 1 for the sake of brevity. The name of the subject is replaced with a placeholder name (Zhang) in Table 1 for privacy reasons.

5. The coder is a senior student at the department of computer science.

6. Here, documentation refers to reading, writing or editing of project documentation.

7. For brevity, we use the abbreviations Doc., Admin., Leis., and Prog. in the tables.

8. These two subjects actively work for 26 days within the experimentation period, and we arbitrarily chose one day.

9. We carry out this comparison relating the two subjects (i.e. the developer and the leader) considered in the analysis.

10. The bin size of the histograms is set to 1 and no sort of optimization is performed to enhance visualization.

11. In this table, an entry of asterisk (∗) denotes “any value” (of application or window title), whereas a dash (-) denotes “no value” (i.e. no candidate).

12. Both inconclusive and uncertain cases are considered to be not-estimated.

13. Henceforth, we carry the index n of the set of descriptors Λ to the subscript of the probability density function.

14. The number of actions with alien titles is found to be 1377 for the developer and 514 for the leader.

15. Since the matrices in Table 5 are symmetric, only the upper triangular parts are presented.

16. Using only the infrequent tasks implies an inherently low number of samples.

17. Nevertheless, even when all variables (more relevant and less relevant) are considered, the proposed method still achieves comparable rates.

18. The number of nearest neighbors is considered as K = 3.

19. In the pseudo-codes of this section, N denotes the total number of lines in the log file.

20. In our case, k and r are the number of possible tasks and number of descriptor values, respectively.

21. In Tables 30–32, we denote the case where no quantile can be computed with ’-’.

We would like to thank Mr. Ryosuke Shimizu for his help in compiling the data set and annotations. We would like to thank Mr. Christian Murphy and Dr. Samantha Stever for their help in proofreading.

This work was supported by JSPS KAKENHI Grant Numbers JP18K18168 and JP20H05706. The results of this research are funded by Okayama University Dispatch Project for Female Faculties.

1. Florian Pellegrin

   Present address: Department of Computer Science, École de Technologie Supérieure, 1100 Notre-Dame St W Montreal, Quebec, H3C 1K3, Canada

Authors and Affiliations

1. Department of Computer Science, INSA Rennes, 20 Avenue des Buttes de Coesmes, 35700, Rennes, France

   Florian Pellegrin

2. Department of Computer Science, Okayama University, 3-1-1 Tsushima-naka Kita-ku, Okayama, 700-8530, Japan

   Zeynep Yücel & Akito Monden

3. Department of Computer Engineering, Faculty of Engineering, Kasetsart University, 50 Ngamwongwan Rd. Chatuchak, Bangkok, 10900, Thailand

   Pattara Leelaprute

Corresponding author

Correspondence to Zeynep Yücel.

Conflict of interest

The authors declare that they have no conflict of interest.

Communicated by: Romain Robbes

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

This work was supported by JSPS KAKENHI Grant Numbers JP18K18168 and JP20H05706. The results of this research are funded by Okayama University Dispatch Project for Female Faculties.

Appendix A: Details of Ground Truth Annotations

Table 14 depicts details on ground truth annotations regarding the two subjects. This table indicates that both the developer and the leader have quite an imbalanced distribution of tasks. Namely, the developer carries out a Test 82% of the time, whereas the leader performs Documentation 78% of the time. In addition, Test and Administration are found to be performed by both subjects, but at different rates, whereas Programming is carried out only by the developer, and Leisure and Documentation is realized only by the leader.

Appendix B: Details of Association Rules and Key Phrases for Window Titles

Table 15 depicts the entire set of association rules defined by the expert. Please note that rule numbers simply refer to the row number in the table and do not have any significance from a semantic point of view. Similarly, also the order of candidate tasks does not have any significance. Note that in the main track we provide some sample rules after translating their window titles to English, for the sake of clarity and uniformity. Here, we provide them in their original form, i.e. including Chinese and Japanese characters.

It can be seen in Table 15 that out of the 20 rules, 13 of them are definite and 7 are indefinite. Moreover, only two rules involve both application and window title in their antecedents, while 12 rules involve only application, and 6 rules involve only window titles as antecedents. The indefinite rules with a single term in their antecedents (e.g. rule 15), can be considered to pose the most serious risk from the point of view of (un)certainty of estimations.

In addition, as mentioned in Section 5.3.1, since there is virtually an infinite number of window title possibilities, we only consider a set of 50 key phrases, which commonly appear in window titles. In this study, the set of key phrases is defined by our expert. Nevertheless, this step can be also performed automatically, for instance, by integrating a standard clustering method integrated with natural language processing tools. Specifically, the expert provided a list of 50 key phrases, some of which are redundant. Namely, among the 50 key phrases defined by the expert, 10 key phrases are utilized by the developer and 25 key phrases are used by the leader. In addition, some key phrases appear in the same window title.

To solve this issue, we revised the set of key phrases by considering those that appear together as additional variable values. In that manner, we consider the developer to utilize a total of 11 distinct window titles and the developer to utilize 31 distinct window titles (aside from the alien titles).

Appendix C: Details of Estimation Performance Upon Direct Application of Association Rules

Table 16 demonstrates the number of actions, which receive 0\(\sim \)3 candidate tasks by direct application of the rules. These results indicate that the rules are able to give a single candidate task (i.e. estimation) for only 152 actions out of 1283 (i.e. 12% of the cases) regarding the leader, and 131 actions out of 1921 (i.e. 7% of the cases) regarding the developer. This means that the remaining 88% and 93% of the actions of the leader and the developer, respectively, need to be revised such that a -single- task is assigned to them.

Appendix D: Details of Post-processing of Benchmark Method

Step-1 of post-processing:

If two subsequent actions are uncertain and have a single candidate in common, then their estimated tasks are determined as this candidate.

Algorithm 1 outlines Step-1 of post-processing as a pseudo-code and Table 17 illustrates its execution on a sample -hypothetical- piece of data^{Footnote 19}.

Step-2 of post-processing:

For each estimated action, the task associated with it is propagated to its preceding (not-estimated) action, provided that that task appears among the candidates of the preceding action.

Algorithm 2 outlines Step-2 of post-processing as a pseudo-code and Table 18 illustrates its execution on a sample -hypothetical- piece of data.

Step-3 of post-processing:

Step-2 is repeated beyond immediately preceding and succeeding actions, until it reaches an estimated action. Subsequently, the same is applied for the succeeding actions.

Algorithm 3 outlines Step-3 of post-processing as a pseudo-code and Table 19 and Table 20 illustrate its execution on a sample -hypothetical- piece of data with backward and forward propagation, respectively.

Step-4 of post-processing:

For each estimated action, the task associated with it is propagated to each preceding (not-estimated) action irrespective of their candidates, until an estimated action is found. Subsequently, the same is applied for the succeeding actions.

Algorithm 4 outlines Step-4 of post-processing as a pseudo-code and Table 21 illustrates its execution on a sample -hypothetical- piece of data (with backward propagation).

Appendix E: Details of Performance After Causal Post-processing

Table 22 demonstrates estimation accuracy of the benchmark method together with causal post-processing operations. The most frequently occurring task of the developer (i.e. Test) is achieved with a high accuracy (i.e. 0.92), raising his overall estimation accuracy (i.e. 0.86). On the contrary, for the leader, the most frequently occurring task (Documentation) is detected only by 0.58, leading his overall accuracy to be only 0.63. In addition, from Table 22, it is clear that there is a larger margin of improvement in estimation of the actions of the developer (see also Table 6).

Appendix F: Details of Benchmark Performance After Non-causal Post-processing

Table 23 presents the estimation accuracy of the benchmark method followed by non-causal post-processing. It can be observed that non-causal post-processing achieves a higher accuracy both for the developer and the leader. Namely, for the developer the accuracy increases from 0.86 to 0.92 and for the leader it increases from 0.63 to 0.85. making non-causal post-processing more beneficial for the leader.

Appendix G: Statistical Properties of Descriptor Values

Table 24 presents the minimum, maximum, mean and standard deviation values relating the four ratio scale descriptors, i.e. duration δ (in sec), and number of key strokes κ, left clicks c_L and right clicks c_R. Although the tasks of the developer and the leader are distributed in quite a different manner (see Table 14), no significant distinction is present between descriptor statistics of the two subjects. In other words, the variation between the mean value of any ratio scale descriptor is within a single standard deviation concerning either of the subjects.

Appendix H: Details of Normalized Entropy Distances

Table 25 presents normalized entropy distance values computed separately for each user. This table ascertains that there is a higher degree of dependence (i.e. lower distance) between application α and window title ω for both subjects.

Here, it is noteworthy to discuss the possible drawbacks of having a 2D variable space instead of two 1D spaces, due to the obvious dependence of α and ω depicted in Table 5 and in more detail in Tables 25 and 26.

Clearly, building a 2D space introduces a larger number of bins (i.e. cells) than two 1D spaces.

However, taking a closer look at the distribution samples in the 2D variable space, we can actually confirm that this increase in number of bins does not pose a serious issue for our particular data set. Namely, the developer uses a total of 11 applications and utilizes 10 distinct window titles (including the alien titles), which leads to a variable space with 110 bins. Among those 110 bins, only 14 are observed to have nonzero observations. Since the number of samples relating the developer is 1921 (see Table 14 of Appendix Appendix), we consider the sample size to be sufficient to populate the effectual 2D variable space.

On the other hand, regarding the leader, there are 16 applications and 31 window titles (including the alien titles), which lead to a 2D space with 540 bins and 36 of them have nonzero samples. Since the leader has 1283 samples (see Table 14 of Appendix Appendix), similar to the case of the developer, we consider these to be sufficient to populate the effectual 2D variable space. Therefore, although the contribution expected from dimensionality reduction is not regarded to be high for the particular data set of interest, we still consider this attempt to understand the dependency and our strategy based on normalized entropy distance, to be potentially beneficial for other data sets and to expand the application capabilities of the proposed method.

Appendix I: Pre-processing of Ratio Scale Variables

The pre-processing operation applied on ratio scale variables is explained in Section 5.3.1. Figure 3 illustrates an example of that pre-processing operation by presenting its details on the δ descriptor relating the developer.

An example for the pre-processing of the δ variable concerning the developer

Full size image

The quantiles Q_i of that descriptor are determined by following the procedure summarized in (6) and are illustrated on the x-axis of Fig. 3 (and listed in column 2 of Table 27). Moreover, the number of observations in each of these clusters are as given in column 3 of Table 27. As explained in Section 5.3.1, since the observations are substantially imbalanced, the number of observations in these quantiles does not approximate a uniform distribution.

Appendix J: Assessing Variables’ Relevance

Section 5.3.2 elaborated on the independence of pairs of variables, whereas this section investigates the correlation of variables and tasks. It is known that automated collection tools

may register too much and too detailed information requiring a serious amount of time for the analysis (Karahasanović and Heim 2015). Namely, certain descriptors may be highly correlated with particular tasks, while some others may not present a distinction with respect to the nature of the task. In order to assess the relevance of each descriptor in task estimation, we use Cramér’s V, which is a measure of association between two variables based on Pearson’s χ² statistics (Ramachandran and Tsokos 2014) and attains a value in the range [0, 1]. In explicit terms, Cramér’s V is computed as,

where χ² comes from Pearson’s test, n is the number of observations, and k and r are number of values that each variable can attain^{Footnote 20}.

As seen in Table 28, α and ω have by far the highest V values. Therefore, considering the significant difference between the values of V regarding these two descriptors and the remaining ones, we suggest using α and ω in task estimation and omit the others. In this respect, in this work we

prefer using V for measuring the degree of correlation and determine our set of relevant descriptors in rather an empirical manner.

In addition to assessing the relevance , we used Cramér’s V also for determining the optimum number of quantiles which yields the highest correlation between the relevant descriptors (chosen as above) and tasks. Namely, we compute V by distributing the ratio scale variables into 1 ≤ q ≤ 6 quantiles and pick the optimum value of q, which yields the highest V for each variable (see Appendix Appendix). In this respect, Table 28 presents the highest values of V for each ratio scale variable.

We compute Cramér’s V for the set of tasks and each single descriptor, which has been pre-processed as described in Section 5.3.1.

10.1 J.1 Cramér’s V for Nominal Scale Variables

As mentioned in Section 3.1, the nominal scale variables are application name α and window title ω. Computing Cramér’s V accounting for the considerations explained in Appendix J, we achieved the values presented in Table 29. We observed from this table that V relating α is higher than V relating ω for both subjects. In addition, the difference in V between α and ω is larger for the developer than the leader. For instance, at Stage-1 the developer attains 0.55 and 0.26 (with a difference of 0.29) for the two descriptors, whereas the leader attains 0.83 and 0.65 (with a difference of 0.18). All tasks are correlated more strongly with α, and the effect is more pronounced for infrequent tasks (i.e. Stage-2). Moreover, V relating ω is lower for infrequent tasks (i.e. Stage-2) for both subjects (i.e. 0.26 > 0.15 and 0.65 > 0.34).

10.2 J.2 Cramér’s V for Ratio Scale Variables

In practice, there are a number of points that we need to be careful in assessing the relevance of ratio scale variables. Specifically, unlike nominal scale variables, relevance of ratio scale variables is assessed based on their empirical cdfs rather than the -raw- values. In that respect, we first need to carefully decide on the bin size of the empirical distributions, since the choice may impact on Cramér’s V.

The essential challenge in choosing the bin size relates the concentration of descriptor values in narrow ranges as mentioned in Section 3.3. One important factor leading to this issue is that the number of key strokes κ and left/right clicks c_L, c_R, attain a value of 0 for most of the actions. Thus, in building their empirical distributions (i.e. histograms), it is plausible to consider the case of having a zero value on its own. Namely, we first distinguish zero values, and use a bin dedicated to them and group all non-zero values in one cluster. This corresponds to the Q = 0 case in Tables 30, 31 and 32 below. Besides, any number Q >= 1, indicates that the nonzero values are grouped among them (into Q + 1 clusters).

Another impediment is that we cannot distribute the descriptor values into any number of bins. Specifically, given a particular Q, it may not always be possible to have distinct edges for each cluster. Namely, some quantiles may be the same, if the distribution is too imbalanced (i.e. too many values in the first few bins and too little values towards the tail). In that case, the method explained in Fig. 3 prohibits using that particular number of quantiles Q.

The final issue relates the number of clusters that we can possibly use, in particular for integer variables. Namely, the number of clusters depends on the number of distinct observations for such variables. This can be understood easily by watching, for instance, the c_R values illustrated in Fig. 4g and h of Appendix H. The number of right clicks is either 1 or 2 for the developer, whereas it is some value between 1 and 7 for the leader, although values higher than 2 are observed quite rarely. From this figure, it is clear that we cannot categorize the values in more than 2 bins for the developer and more than 7 bins for the leader (though a large number of categories are not expected to introduce significant information)^{Footnote 21}.

Histogram and cumulative histogram of duration δ for (a) the developer and (b) the leader. Similar pairs of figures for (c, d) number of key strokes κ, (e, f) number of left clicks c_L, (g, h) number of right clicks c_R.

Full size image

When interpreting the information presented in Tables 30, 31 and 32, we should note that if a certain descriptor is correlated with a particular task for one subject, but with another task for the other subject, computing Cramér’s V for both subjects at once yields a lower value than computing it for each subject on its own. As an example, consider that, for the developer, firefox.exe is highly correlated with Test, while for the leader, it is highly correlated with Leisure. This means Cramér’s V can be -relatively- high for α computed for one subject only (i.e. only the developer or only the leader). But if it is computed for both subjects at once, it decreases, since α is associated with a larger variety of tasks.

In the light of these observations, we decided to use Cramér’s V as a guideline to determine the number of clusters of empirical distributions in addition to deciding the relevance of variables.

To determine the best number of clusters such that the correlation between the descriptors and tasks is highest, we compute Cramér’s V for the number of quantiles 0 < Q < 5 as presented in Tables 30, 31 and 32. Subsequently, we pick Q, such that it yields the highest possible V (marked in boldface in these tables).

Reprints and permissions