Automated Identification and Qualitative Characterization of Safety Concerns Reported in UAV Software Platforms

In the following, we provide relevant definitions related to safety-critical systems (SCSs) that will be the subject of our study such as the definitions of accident and hazard. At same time, we only briefly elaborate on emerging types of SCSs, since a full coverage of background on SCSs is available in previous seminal studies [113, 114, 115, 116, 117, 118, 119].

A SCS can be considered any system (composed of hardware and/or software) whose failure may cause (i) death or injury to people, or (ii) damage to equipment/property, or to the environment [152, 167]. Given this definition, almost any system could be safety-critical [115]. For this reason, the key point in the safety-critical analysis is not to ensure 100% safety, but, rather, to reduce the risk that a hazardous event occurs. In this context, emerging SCSs can be represented by robotic systems [151], real-time systems [10], autonomous systems [178], and more in general CPSs [141].

SCSs complexity poses challenges on scaling agile practices [45, 156] and performing requirements engineering [126] for SCSs. Important SCS definitions related to the safety of these systems are hazard and accident. An hazard is defined as “a set of conditions that can lead to an accident, given certain environmental conditions” [115]. For instance, a hazard in a self-driving vehicle can be due to a sensor failing to detect an obstacle. An accident, also referred to as a mishap, refers to “events, or series of events, that result in death, injury, occupational illness, damage to or loss of equipment or property, or environmental harm”. [115]. For instance, the hazardous event reported above can lead to the vehicle hitting a pedestrian (i.e., accident). In this article, we focus primarily on shed some more light on the types of hazards and accidents affecting a specific type of autonomous system such as UAV.

Figure 1 shows the development cycle of UAVs [140], which includes three software phases (i.e., Perception, Planning, and Control) as well as the hardware components that interact with the environment.

The Perception phase includes state estimation [47, 50, 134, 143] and mapping [103, 136] steps. The state estimation step enables navigation and autonomous driving of UAVs [143]. The mapping step consists of computing obstacles distance information and, complementary to the state estimation step, it generates a 3D model of the environment (with 3D range measurements [103]) and models unmapped areas at run-time [136].

The Planning phase consists of finding the right trajectory between a starting point and a destination, with state-of-the-art approaches in this phase leveraging polynomial trajectories [130, 133, 148] and trajectory optimization [49] strategies. Finally, the Control phase identifies the flight control commands to be executed by UAVs to safely navigate the environment.

To enable the usage of UAVs in different contexts, their software is equipped with functions and checks enabling different autonomous or teleoperated flight modes [25, 48, 140, 166]. These are enabled by the use of a Ground Control Station (GCS), consisting of a set of hardware and software components that enable UAV operators to communicate with/control the UAVs.

In the following, we elaborate on some of the main flight modes relevant for the conducted research:

Based on the aforementioned goal, our study aims at answering the following research questions:

Figure 2 depicts the research approach, we followed to answer our research questions. Our research approach consists of four phases. The Data Collection phase aims at building the dataset to be used for answering the given research questions and it is described in Section 3.2. During the phase of Gold Standard Definition, sentences of the dataset are manually annotated with safety and non-safety labels through a process of individual analysis, review, and open discussion. This process is detailed in Section 3.3. The labeled sentences are then used to train several ML techniques discussed in Section 3.4. Finally, in the Hazards & Accidents phase, the hazards and the related accidents reported in the safety-related sentences are identified through an approach explained in Section 3.5.

The study context consists of three open-source UAVs hosted on GitHub, from which we extracted 304 issues labeled as safety-related. To identify UAV-related projects, we queried GitHub to retrieve all projects having as topic drone and/or drones, and as main programming language C or C++, the ones predominant for CPS development [154]. From the list, we removed fork projects, and projects not being forked, and ordered them based on the number of forks, i.e., used as a proxy of the project’s popularity. As an outcome, we ended up with a list of 150 candidate UAV-related projects.

Since our goal is to identify and analyze the safety-related issues occurring in UAV platforms, for each project, we downloaded the set of labels used for tagging the user-reported issues, and filtered out all the projects that do not have safety-related labels. After the filtering, we ended up with 3 UAV-related projects (as reported in Table 1), for which we downloaded the whole set of closed issues and pull requests by relying on Perceval [46]. Finally, we filtered out all the issues and pull requests not being labeled with a safety-related label, ending up with a total of 304 closed issues tagged with at least one safety label.

Table 1 reports, for each project included in our study, the total number of closed issues and pull requests, as well as, among them how many are labeled as reporting a safety concern. Note that, two out of three projects included in our study, i.e., Ardupilot and PX4-Autopilot, have been considered as references in other studies dealing with UAVs [170]. Furthermore, by looking at data reporting in Table 1, even if all the considered projects use both issues and pull requests for supporting software development and evolution, only a small percentage of them (1.67% of issues and 0.51% of pull requests) are labeled as discussing safety properties.

For each issue and pull request having a safety-related label, we used the BEE tool [153] to structure its title and description into sentences. While other bug trackers (e.g., JIRA or Bugzilla) provide explicit fields for issue descriptions, the GitHub issue tracker does not, and the issue/pull request description is usually provided in the issue’s first comment. Therefore, similar to previous work [7, 24, 108, 109, 138], here and in the remainder of the article, we refer to the first comment of an issue/pull request as the issue description.

As a result, the 304 safety-related issues have been decomposed into 1,916 sentences.

To check whether it is possible to classify issues (not) reporting safety-related concerns, we need to properly identify, among the 1,916 sentences, the ones that refer to safety properties. To do this, all authors (henceforth referred to as coders), performed a manual classification of the sentences, considering two categories, i.e., safety-related and not.

Identifying the presence of safety concerns in a piece of text is a subjective task, e.g., interpretation of the text might be different for different individuals or might depend on their cultural background. For this reason, we conducted a pilot labeling study. Specifically, after having randomly sampled 10 safety-related issues in our dataset, accounting for 49 sentences, we relied on a plenary meeting (i.e., pilot study), where, upon discussion, each sentence was properly classified as (not) describing safety-related concerns. Once the coders reached a common understanding of the labeling procedure, we assigned the remaining 1,867 sentences to two coders each, i.e., an average of \(\simeq 747\) sentences for each coder.

At the end of the labeling procedure, an open discussion was performed by adding a third coder, to check all the sentences for which there were disagreements among the original coders (284 out of 1,867). Even if the coders agreed in 84.8% of the cases on whether a sentence discusses a safety-related property, it is still possible that they agreed by chance. For this reason, we computed the Cohen’s k [29] which resulted to be 0.68 (substantial). After disagreements’ resolution, among the 1,916 sentences in our dataset, we identified 837 (43.7%) discussing safety concerns, while the remaining 1,079 did not. In other words, among the 304 safety-related issues, 274 contain at least one sentence in the title or description describing a safety-related concern. As regards the 30 safety-labeled issues in the original dataset that are not being classified as safety-related in our study, we found cases where:

The manual inspection of the 274 safety-related issues also revealed that one of these issues [59] aimed at summarizing multiple and too generic hazardous circumstances when using a tablet ground station as a user interface. For this reason, we decided not to consider such an issue when conducting the subsequent analysis (i.e., the identification of hazards and accidents). The labeled dataset is used as input for addressing both RQ\(_1\) and RQ\(_2\). Specifically, the whole labeled dataset (304 issues) is used to experiment with ML models, while the 273 issues with at least one safety-related sentence in either the title or the description are further classified to determine a set of hazards and accidents occurring in UAV systems.

After the definition of the golden standard, we benchmark different techniques (proposed for issue report classification purposes) when used to automatically identify safety-related sentences in the titles and descriptions of UAV-reported issues.

To identify hazards and related accidents, i.e., events or conditions resulting in unexpected behavior being able to damage people, the environment, and/or the equipment, we manually analyzed the 273 issues containing at least one sentence describing a safety concern, through a card sorting strategy [155]. Specifically, the manual analysis has been performed by four coders (all of them are authors of the article), guaranteeing that each bug was classified by two different coders. Since the labeling procedure did not start with a predefined set of categories related to hazards and accidents, even in this case, we performed a pilot study, on the same 10 issues used in the previous pilot study, to determine a set of coding guidelines to follow during the subsequent independent labeling procedure.

To perform the issue labeling process, we used online spreadsheets (a separate spreadsheet for each coder), where coders could use drop-down menus to select previously identified categories or add new ones when those did not fit. Note that, during the labeling process, the categories identified by one coder were not shared with the others, finally resulting in 465 different categories of hazards and 43 categories of accidents. Going over the identified categories, we found that, in several cases, different coders used different wordings for pointing out the same hazards and accidents. Further discrepancies between some of the identified categories depended on the different granularity levels used by the coders while reporting them. For these reasons, before the resolution of disagreements, two coders went through the whole set of the originally identified categories and clustered them. In the end, they obtained 20 categories describing hazards, and eight categories related to accidents. The categories that emerged from the clustering were used to refine the original labeling results. Specifically, all four coders leveraged such categories to properly discuss, through online meetings, the cases in which a disagreement between them in terms of hazards or accidents occurred. Furthermore, to reduce the agreement by chance that could have been introduced as a consequence of the clustering procedure on the original lists, we proceeded to discuss also the cases where the original coders agreed.

To sum up, out of the 273 safety-related issues analyzed, 258 clearly mentioned the hazard, while only 115 indicated the accident occurred or which might have likely occurred. The final categorization we provide is made up of 19 hazards and 7 accidents.

As detailed in Section 3.4, we experimented with different ML classifiers, namely Naive Bayes, J48, SMO, Logistic Regression, Random Forest, and fastText, leveraging different combinations of features. In the following, we report and discuss our results. Note that, we only report statistical comparisons where appropriate, because we computed statistical comparisons among all possible comparisons of treatments. Detailed statistical results can be found in the replication package.

Table 2 summarizes the results obtained when considering different ML classifiers with different combinations of features. As shown in Table 2, Random Forest, together with fastText are the classifiers showing the best performance in terms of precision, recall, and f-measure.

Based on the statistical comparison, Random Forest with BOW outperforms all other techniques (adjusted p-values \(\lt\) 0.05) with ORs ranging from 1.3 for the comparison with BOW + SMO and 2.95 for the comparison with BOW+Logistic regression. There is no statistically significant difference when augmenting BOW with OB/EB/S2R and applying Random Forest (p-value = 0.6). Indeed, the weighted precision, recall, and f-measure only slightly increase from 0.80 to 0.81. We can also notice that the introduction of n-grams in the model slightly lowers the performance to 0.79 for all metrics, yet such a difference is not statistically significant (p-value = 0.31). This may happen because terms contributing toward the classification of a safety-related sentence are unlikely to be sequences of adjacent words.

When training the classifiers using the features extracted with LSI, Logistic Regression outperforms Random Forest: 0.79 versus 0.74 on all the evaluation metrics being considered. The difference is statistically significant (p-value \(\lt\) 0.01, with an OR = 1.5). However, there is no statistically significant difference with BOW + Random Forest (p-value \(=\) 0.28). Hence the application of LSI in this context may not result as particularly convenient, given that building a LSI space is typically more expensive than just creating a simple BOW model.

As regards the classifiers showing the worst performance, we found that Naive Bayes trained with all features shows an average f-measure of 0.45, while Logistic Regression trained with BOW features has an average F-measure of 0.63. Based on the results reported in Table 2 it is possible to conclude that, for most of the ML classifiers, the performance tends to be already very positive when considering the BOW features, with marginal improvements when adding OB/EB/S2R. Only Naive Bayes does not show an improvement in terms of F-Measure results when combining different types of features. The latter might happen because the Naive Bayes classifiers show limited improvements when increasing the number of features [138, 139].

It is important to highlight that, in general, for almost all the ML models trained with multiple features, the performance tends to be better in identifying true negatives (i.e., issues not containing safety-related concerns). This may happen because the dataset used for training the classifiers is imbalanced, i.e., \(\simeq\) 56% of the sentences do not discuss any safety-related concerns.

When comparing the results obtained by Random Forest and fastText, we must mention that, as shown in Table 2, fastText and Random Forest both achieve an average f-measure value of 0.81. It is important to highlight that fastText achieves good performance by using a standard configuration (as detailed in Section 3.4.2), while Random Forest, achieves similar results only when applying several pre-processing steps and experimenting with specific sets of features. Interestingly, fastText achieves a higher recall in the Safety class, which is important to identify safety-related issues timely during the issue management process. It is worth noticing that fastText does not outperform the other experimented techniques by a large margin (as it happens in the context of assigning labels to issues by considering the whole text reported [109]) achieving comparable results to the ones achieved by the Random Forest classifier trained with M\(_{BOW}\) + M\(_{OB-EB}\) features.

Our dataset has two levels of imbalance: (i) the imbalance of the projects (i.e., a project has many more safety issues than the others) and (ii) the imbalance of the number of samples in the Safety and Non-Safety classes. These two issues are addressed in this Section (see “Imbalance handling techniques” and “Cross-project analysis” paragraphs).

Imbalance handling techniques. To check whether the results are biased due to the imbalance of the number of samples in the Safety and Non-Safety classes, we also report the results of imbalance handling techniques applied to our dataset, considering the best performing shallow ML model in all our experiments, i.e., Random Forest trained with M\(_{BOW}\) + M\(_{OB-EB}\) features. We then compare the results of Random Forest when using the default configuration and its variant using imbalance handling techniques. Specifically, the imbalance handling approach applied was conducted by performing the following steps:

Hence, we decided to use undersampling (and not oversampling) as the imbalance handling strategy since, as reported in previous work [159], oversampling methods tend to generate false examples, causing classifiers to perform well in labs but more likely to fail in practice. We follow the recommendation of such previous work [159] that suggests avoiding oversampling methods when dealing with sensitive applications such as security, autonomous driving, aviation safety, and medical applications. Given the projects investigated in our study fall in the domain of aviation safety, we decided to experiment with the undersampling strategy. Table 3 reports the results achieved by (i) Random Forest when using its default configuration (trained with M\(_{BOW}\) + M\(_{OB-EB}\) features) and 10-fold cross-validation on the unbalanced original dataset (described in Section 3.2), and (ii) the default configuration of Random Forest (trained with M\(_{BOW}\) + M\(_{OB-EB}\) features) in a 10-fold cross-validation setting by leveraging the balanced dataset obtained when applying the undersampling strategy. As we can observe from Table 3, the performance obtained by the two variants of Random Forest are almost identical, so no major improvements are achieved by balancing the dataset. We conjecture that this happens because our dataset does not present a heavy imbalance in terms of the number of samples in the Safety and Non-Safety classes. Indeed, as reported in Section 3.3, we have a fairly balanced set of sentences: i.e., 837 Safety sentences (representing 43.7% of the data), and 1,079 Non-Safety sentences (representing 56.31% of data).

Cross-project analysis. As detailed in Section 3.4, to check whether the results are biased due to the dataset imbalance (i.e., the majority of issues and pull requests considered in our dataset belong to Ardupilot), we also report results of a cross-project analysis, in which we use the best performing configurations, i.e., Random Forest trained with M\(_{BOW}\) + M\(_{OB-EB}\) features, and fastText. As it is shown in Table 4, for Random Forest, the average precision and recall values decreased compared to a 10-fold setting. Consequently, the overall F-measure decreases of 0.16. In particular, for the Random Forest algorithm, we observe a quite significant degradation in the recall concerning the identification of the sentences labeled as Safety. This performance degradation could happen because the terminology used to describe safety issues can be slightly different in Ardupilot, compared to the other projects.

When comparing the performance of fastText in the cross-project setting and the results achieved by this model in a 10-fold setting (see Table 2), we can observe that such performance only slightly decreased, i.e., the weighted average f-measure in a cross-project setting is 0.77 while it is equal to 0.81 in a 10-fold setting. Specifically, in a cross-project setting, the default implementation of fastText achieves a recall of 0.62 for the sentences in the Safety category, suggesting that the usage of word embeddings (i.e., the features’ representation strategy leveraged by fastText) allows to partially mitigate the problem of different vocabularies used in heterogeneous projects to describe safety-related problems.

Feature Importance. To better understand the features that contribute more to the classification, we computed the Mean Decrease Gini (also called Mean Decrease in Impurity) [53, 125, 164] for Random Forest, considering BOW and OB/EB/S2R features. As shown in Figure 3, we can find intuitive (top 20) features considered important for the classification of safety-related sentences. Specifically, hardware-related features (e.g., motor and vehicle) and aspects concerning concrete states (e.g., failsafe) or actions (e.g., arm and disarm) of the UAVs tend to be considered as important features by the model. Although the keyword safety is among the top 20 relevant features, it only appears in 48 out of 837 (less than 6%) safety-related sentences in our dataset.

Hyperparameter Optimization. As detailed in Section 3.4.1, we experimented with Grid search as hyperparameter optimization approach [2, 14, 15] to investigate potential optimal combinations of parameters for the selected shallow ML models. Specifically, with Grid search, we experimented with several parameter combinations for the various ML models. In total, we experimented with around 600 combinations using a 10-fold validation setting for the selected shallow ML models (all detailed results are shared in our replication package along with the Python code used to run the Grid search experiments). As is shown in Table 5, for Random Forest, the average Precision, Recall, and F-Measure values slightly increased (by about +1%) compared to its default configuration. For most of the other models, the obtained improvements are slightly more evident in terms of Precision, Recall, and F-Measure. In particular, for the J48 algorithm, we observe a +2% of improvements for all metrics. Similarly, for Naive-Bayes we observe small improvements in Recall (+1%), Precision (+2%), and F-Measure (+1%) values. For Logistic Regression we observe more substantial improvements in Recall (+16%), Precision (+16%), and F-Measure (+16%) values. SMO is the only model that did not achieve any observable improvement in terms of F-Measure. Overall, considering mainly the model having the higher values of both Precision and Recall (i.e., values \(\gt 0.80\)), Random Forest and fastText are the best performing models, as observed before, with Random Forest, slightly better in terms of F-Measure (+1%). Interestingly, after a Grid search optimization step, all experimented ML strategies are not able to achieve more than 0.82 of F-Measure, which represents—within the considered hyperparameter values—the upper bound of our experiments.

To explore the recurrent hazards and accidents reported in UAV software platforms, and answer RQ\(_2\), we performed a manual analysis of the 273 issues and pull-requests of our dataset containing at least one sentence describing a safety-related concern (as described in Section 3.5). Specifically, based on the title and the description content, each issue is assigned to at most one hazard and/or accident category. Upon completion of the manual inspection and clustering procedure (see Section 3.5), we found 19 different categories of hazards and seven categories of accidents reported in Tables 6 and 7, respectively, along with their occurrences.

As reported in Table 6, undesired hardware behavior [75] (4.40%), onboard instrumentation issues, such as GPS, GCS, or RC connection lost during flight [57] (2.56%), communication failures, such as improper communication with motors and between motors [76] (1.47%), inappropriate handling (due to either hardware or software defects) for high vibrations [89] (1.10%), and battery-related issues [77] (0.73%) are signaled as sources of hazards in only about 10% of the cases, meaning that, in the great majority of the considered reports, the hazardous circumstances depend on software-related defects. Specifically, in about 40% of the analyzed issues, the reported hazards depend on misbehaviors occurring when the system enters failsafe mode (undesired behavior on failsafe or error condition/configuration \(+\) undesired failsafe behavior, 24.18%), or the missing inhibition of certain actions (e.g., reboot while in-flight [87], motor lockdown during takeoff [95]) when the system is in specific states (14.65%).

Concerning hazards happening in failsafe mode, we distinguish misbehaviors depending on improper failsafe settings [54] (i.e., undesired failsafe behavior) from unsafe actions occurring when a failsafe (or error condition) is triggered [60] (i.e., undesired behavior on failsafe or error condition/configuration). As regards the former, Issue # 292 [54] from ardupilot having as title “Battery Monitoring RTL Bug” states a case where there is a bug inside the functionality handling the failsafe condition due to low battery. Specifically, “a low battery warning can cause a permanent RTL that can[not] be overridden”. As regards the latter, instead, Issue # 697 [60] from ardupilot clearly mentions the case where the propellers “go to 100% Full Throttle” when a radio signal loss occurs (i.e., the failsafe condition is triggered), resulting in the vehicle flipping.

On the one hand, in \(\simeq 9\%\) of the inspected reports, the risky situations are due to insufficient checks such as Issue # 6649 [79] from ardupilot where there is a need to “check that the battery voltage is above the ARMING_MIN_VOLT and ARMING_MIN_VOLT2 parameter values” before arming the vehicle. On the other hand, there are 20 (7.33%) issues/pull requests discussing hazardous situations due to an improper parameter setting, such as PR # 1516 [81] from dronin where there is the need for changing the output calibration, i.e., “offset the minimum of the calibration range up a little bit from what we send” during disarming.

Furthermore, in \(\simeq 13\%\) of the manually analyzed pull requests and issues, misleading data [65, 83] (6.59%), or missing warnings when the system is under anomalous conditions [62, 69] (5.86%) are highlighted as risky situations by both developers and/or end-users.

In the remaining cases, dangerous events are related to the inability to perform certain actions when the system is in specific states [68] (4.03%), improper mode switches (inappropriate mode changes/handling [86] \(+\) inappropriate safety switch handling [84], 5.86%), the missing detection of failures occurred [85] (2.56%), race [90] or timing [71] conditions (1.83%), impossibility to take control [63] (1.47%), and memory size issues [70] (0.73%).

Moving the attention to the accidents generated by risky situations, as expected, crashes/collisions [64, 66] or abnormal behaviors while in-flight [74] are the most recurrent ones. Specifically, (potential) accidents in these categories have been indicated in more than 20% of the manually inspected issues (see Table 7). In a further 8% of the analyzed reports, users signaled (potential) mishaps connected with problematic landing/return-to-location operations. For instance, users experienced the automated triggering of blind [93] or hard [91] landings after the occurrence of unexpected events, confirming that landing is among the most critical and accident-prone phases of a UAV flight [44]. Flyaways [161] (i.e., the devices fly off from their users) are signaled in nearly 7% of investigated issues, while stabilization/positioning accidents [88] are disclosed in less than 4% of the cases. Finally, in less than 3% of the inspected reports, users indicated (potential) operator injuries [73] (1.83%) or takeoff issues [58] (0.73%).

To better understand the cause-effect relationships between the different hazard and accident categories, Figure 4 reports the occurrences with which specific hazard categories (detailed in Table 6) co-occur with the different accident categories (see Table 7). To avoid discussing relations occurring only once in our dataset, Figure 4 only shows the co-occurrences that took place more than once. Note that, the thickness of the lines is proportional to the number of issues reporting, at the same time, a specific hazard (on the left) and a specific accident (on the right).

Out of the overall 34 issues describing undesired behaviors when the system enters the failsafe mode or encounters an error condition (see Table 6), (i) 8 (23.53%) led to landing/rtl problems, (ii) 6 (17.65%) caused crashes or collisions, and (iii) 3 (8.82%) resulted in flyaways, e.g., PR # 8039 [82] from ardupilot stating “Prevent DCM fallback from triggering a flyaway”. As regards the former, Issue # 1164 [61] from ardupilot reports a situation where the undesired behavior of the drone when a battery failsafe is triggered results in an immediate undesirable landing of the vehicles (i.e., “give the operator the possibility to calculate and configure a battery level at which it is still safe to RTL/fly to a failsafe destination before landing”). Issue #1234 [67] from dronin, instead, discusses a scenario where the drone crashes as a consequence of “motors keep running at neutral throttle until the arming timeout has expired” once a TX failsafe is triggered — “this is unsafe, as the motors will likely burn up in a crash.”

Similarly, out of the overall 32 signaled undesired failsafe behaviors (see Table 6), (i) 4 (12.50%) led to crashes or collisions, (ii) 3 (9.38%) caused landing/rtl problems, (iii) 3 (9.38%) provoked stabilization/positioning accidents, (iv) 3 (9.38%) induced abnormal in-flight behaviors, and (v) 3 (9.38%) co-occurred with flyaways. For instance, Issue #374 [55] from ardupilot clearly discussed a scenario where the “copter lost around 30m of height” due to a bug in the handling of the failsafe operation triggered by an RC-signal loss. Differently, when talking about anomalous behavior of the UAV while in flight, PR # 16594 [96] from ardupilot reports about a situation where it is mandatory to have “the EKF failsafe [being] trigger[ed] soon after the vehicle is armed in stabilizing (or any other non-GPS mode) if it does [not] have a good position estimate.”

Crashes or collisions might also be due to (i) misleading data (11.43% of reported crashes/ collisions) such as “there is no filtration in the current sensor drivers, and any noise gets passed down as obstacles; this gives a sudden jerky response by the vehicle, which may even lead to crash” [97], (ii) inadequate checks (11.43% of reported crashes/collisions), (iii) missing inhibition of certain actions in specific states (11.43% of reported crashes/collisions) like “allowing the operator to inhibit ADS-B avoidance below the specified altitude” so that it is possible to avoid “crashing into trees or buildings” as described in PR # 7074 [80] from ardupilot, (iv) undesired hardware behaviors (8.57% of reported crashes/collisions), (v) undetected failures (5.71% of reported crashes/ collisions), or (vi) inability to perform certain actions (5.71% of reported crashes/collisions).

Besides, out of the 16 issues reported as a hazard the missing or misleading communication with the pilot, 2 (12.50%) caused flyaways, and 2 (12.50%) led to abnormal in-flight behaviors, such as “add warning to user from vehicle and ground station when user is approaching the fence [so that] it’s more obvious for the user to see the vehicle distance from the fences” [62]. Abnormal in-flight behaviors might also arise when specific actions are not inhibited (17.39% of the reported abnormal in-flight behaviors), or disabled (8.70% of the reported abnormal in-flight behaviors), and in case of inadequate checks (13.04% of the reported abnormal in-flight behaviors) or high vibrations were not properly handled (8.70% of the reported abnormal in-flight behaviors). As regards the latter, PR # 12349 [89] from ardupilot describes a case where the inappropriate detection of high vibrations and loss of altitude results in a vehicle climbing very quickly while it is not expected to climb at all.

Flyaways, instead, might also depend on missing, misleading, noisy data or measurements coming from the sensors (15.79% of reported flyaways) such as “catch fly-aways caused by bad compass heading” [56] and onboard instrumentation issues (10.53% of reported flyaways), such as “sudden bad GPS position leads to Loiter flying off (GPSGlitch)” [57].

Finally, landing/rtl problems might also be caused by inadequate checks (9.52% of reported landing/rtl problems). For instance, PR # 15092 [92] from ardupilot reports a dangerous situation where “the vehicle continuously climbs or descends at its maximum rate which at best leads to a hard landing” that is caused by a missing pre-arm check of the EKF’s altitude estimate.

The results presented above, on the one hand, provide an empirical characterization of safety issues in UAV-related software. On the other hand, such results pave the way to research approaches aimed at supporting a better analysis and testing of UAV software:

As reported in previous research [31], for popular projects the number of pending issues constantly grows [110] and, despite the use of labels having a positive impact on the issue management [22], they are rarely used by GitHub developers. For this reason, researchers proposed automated approaches supporting the issue management process, with techniques leveraging machine learning [4, 16, 30], textual analysis [174], natural language parsing [41], and summarization approaches [132, 137, 147].

Other, complementary research directions concern the automated classification and analysis of the textual content of issues [181] for detecting duplicated bugs [174] and predicting reopened issues [177]. In recent years, tools have been designed to automatically estimate the issue lifetime [111, 112].

In the context of issue textual analysis, recent work [138, 172] empirically investigates the combination of ML and textual analysis techniques to automatically predict whether issues will be not fixed, by analyzing (only) the titles and descriptions of reported issues. Similarly, Cabot et al. [22] proposed labels to classify issues in open source projects, and (ii) Guo et al. [98] presented an approach to determine the bugs that will be actually fixed. Finally, Kallis et al. [108, 109] proposed Ticket Tagger, a GitHub app analyzing the issue title and description through machine learning techniques to automatically assign labels to each issue submitted on GitHub, accordingly. Based on a default implementation of fastText, Ticket Tagger can predict three categories of default labels: bug, enhancement, and question. Herbold et al. [100] demonstrated that Ticket Tagger represents one of the most effective approaches to identifying issues of the bug type.

Recently, Wang et al. [171] proposed PLPI, a framework able to identify labels with similar meanings and predict project-specific labels to address the problem deriving from the usage of custom label schemes.

Similar to previous research [138], to build the ground truth we perform a manual inspection of UAV issues. However, we specifically focus on safety-related issues, comparing the performance achieved by different ML approaches when used to predict safety-related sentences from user-reported issues of UAV platforms. Our evaluation is useful to help developers timely identify the UAV issues that are likely to be safety-related. Besides, to ease the inspection of such issues, the experimented techniques can also help highlight the specific safety-related sentences occurring in the issue titles and descriptions.

UAVs are often mentioned as fascinating examples of self-adaptive CPSs. Software engineering for these types of systems is an increasingly explored research field. In particular, recent efforts mainly focused on bug characterization [52], testing [1, 37, 180], and verification [27] of self-adaptive CPSs.

Another emerging area of research is related to the automated generation of oracles for testing and localizing faults in CPSs. For instance, Menghi et al. [131] proposed SOCRaTes, an approach to automatically generate online test oracles in Simulink able to handle CPS Simulink models featuring continuous behaviors and involving uncertainties. The oracles are generated from requirements specified in a signal logic-based language. In a similar effort, He et al. [99] proposed a system identification-based oracle for fault localization in CPS software.

Cleland-Huang and Vierhauser proposed SafetyScrum [28], an agile software development methodology that augments the Scrum life-cycle with safety-related activities to incrementally track the safety status of UAV systems. Previous research also proposed programming languages for supporting UAV-specific aspects, such as energy awareness [124] and context adaptation [21], while Liang et al. [120] focused on how Bounding Functions (i.e., dynamic checks inserted by developers to ensure that specific variables stay within a prescribed range) are used in the Paparazzi autopilot software.

Different researchers proposed methods to perform safety analysis of UAV systems. For instance, Denney and Pai [33, 34, 35, 36] focused on different aspects of UAV safety and proposed automated approaches for creating and maintaining safety assurance cases. Recently, Vierhauser et al. [168] presented an approach for deriving human-interaction hazards (hazards related to human-system interactions). In particular, the authors provided a set of domain-level hazard trees designed for the safety analysis of UAV systems.

Prior studies that most relate to ours are the ones concerning the analysis of UAV-specific software issues reported by developers and/or end-users. In this context, Wang et al. [170] conducted an empirical study to (i) characterize UAV-specific bugs, (ii) identify their root causes, and (iii) propose repairing strategies. They further identified five challenges associated with detecting and fixing such UAV-specific bugs. As in our work, to perform the analysis, they collected issues from PX4 and Ardupilot GitHub projects. Similarly, Taylor et al. [160] analyzed the root causes, severity, and position in the firmware architecture of bugs occurring in these two open-source projects. However, while Wang et al. and Taylor et al. collected the issues with the bug label assigned, we specifically focused on safety-related issues (i.e., having safety-related labels assigned), beyond collecting issues from a further project (i.e., dRonin). Unlike Wang et al. and Taylor et al., who focused on the root causes of UAV bugs, we analyzed the safety-related UAV issues to present two sets of categories modeling the main hazards and accidents occurring in the analyzed projects, beyond assessing automated strategies to support the identification of safety-related concerns occurring in such issues. Besides, we also analyzed the frequencies with which the specific hazard categories co-occur with the various accident categories, to better understand recurrent cause-effect relationships.