Assessing Human-Computer Interaction of Operating Remotely Piloted Aircraft Systems (RPAS) in Attitude (ATTI) Mode

Unmanned aircraft systems can range from the simplest form of a single unmanned platform programmed to fly a pre-set flight path for a pre-defined duration, to a system.

Comprising of the unmanned aircraft with a full suite of complementing ground equipment that provides real-time command and control of the unmanned platform itself and the payloads it’s carrying. Many different terms have been used to try and define different elements of unmanned aircraft systems. At times, a definition may have different meanings when referred to in a different context or by different individuals.

In this study, an unmanned aircraft, other than a balloon or a kite, having a mass of not more than 20 kg without its fuel but including any articles or equipment installed in or attached to the aircraft at the commencement of its flight, is defined as a RPAS or drone.

1.1 RPAS Automation and Typical Operating Modes

The fastest growth of unmanned aircraft for non- military applications is in the class of small drone (sub 20 kg), due to its relatively low cost, yet possessing the capability and technology to meet its intended objectives. Within this class of drone, multi-rotor platforms which are capable of taking off and landing vertically has generated the greatest interest among the community owing to their inherent ability to be launched and recovered from confined locations and ability to perch and stare for extended periods of time (Prior 2013). Some of the most common non-military applications include aerial photography and videography, borders surveillance and simply recreational flying.

Although relatively cheap, drones are equipped with technologies that make flying them fairly easy. For example, they can be programmed to fly along a predefined flight path automatically via a ground- based computer software that is usually very simple and intuitive to operate. Once the flight path is programmed and uploaded to the drone, the automation can make use of the Global Positioning System (GPS) signal to navigate along the planned route accurately.

Besides flying the drone along a predefined flight path based on pre-uploaded data, pilots may also choose to navigate by means of a remote controller. A typical remote controller, a description of the control input and the corresponding motion of the SUA relative to an imaginary person sitting on board the SUA facing forward are shown in Fig. 1.

A typical drone remote controller

Full size image

When the GPS signal received by the drone is sufficiently strong, the aircraft will be able to operate in GPS Mode. This results in the drone flying accurately and smoothly, by comparing the actual position of the drone with the input from the remote controller. For example, when a pilot moves only the pitch control up in GPS Mode, the drone will move forward and any lateral deviation within its operating limits will be compensated for by the automation. This results in the aircraft moving only straight ahead relative to the drone with its vertical distance from the ground being constant. In addition, operating in GPS Mode allows the position (and vertical distance) of the drone to be locked when the inputs of the flight controls (i.e. pitch, roll and yaw) are neutralised.

While automation allows novices to easily control and operate drones in GPS Mode, sometimes without much prior training, it can create a false sense of confidence that the drones can be operated without much training, if any is considered necessary at all! Currently, in the United Kingdom (UK), it is not a requirement for drones below 20 kg or its software to be certified airworthy by the European Aviation Safety Agency (EASA) or the Civil Aviation Authority (CAA 2015b). Thus, the reliability of the automation may not be sufficiently high to ensure that the drone will always be able to operate in GPS Mode, or that there will be any fail safe when the GPS signal is not received.

Moreover, certain drone operations may be required to be conducted within built-up areas, or even indoors, and this may have an adverse effect on the reception and integrity of the GPS signal. The availability of the GPS signal may also be subjected to weather such as cloud cover and precipitation. In these abovementioned examples, the drone may still be operated in a lower level of automation known as the Attitude (ATTI) Mode.

In ATTI Mode, the drone is still flown based on input via the remote controller in the same manner as compared to GPS Mode. While the vertical distance of the drone can still be maintained by the onboard automation, the position of the drone will not be automatically locked. In contrast to the example cited earlier when the aircraft was flown in GPS Mode, when a drone pilot moves only the pitch control up in ATTI Mode, the aircraft may veer off its track laterally as it moves forward in response to the control input. This poses a much greater challenge to the drone pilots as their control input will need to be very accurate, precise and very dynamic as the aircraft reacts and responds to environmental conditions such as gusts. In order for the aircraft to maintain the desired track (i.e. straight ahead without any lateral deviation), the pilot will also need to apply a suitable roll input to compensate for the lateral drift. Similarly, when all the flight controls are neutralised, the aircraft may not be able to maintain its hovering position in ATTI Mode. Instead, it will drift from its intended position according to the wind conditions. In order for the drone to hover is a fixed position in ATTI Mode, the pilot will have to continuously apply input to the remote controller as the aircraft is swayed by the environmental elements.

1.2 Differences Between Manned and Unmanned Pilots’ Training Requirements

With the removal of pilots from the flying machine, different hazards, which in some ways are greater than those of manned aircraft, are introduced. These novel hazards are not addressed by traditional training regimes of the manned pilots (McCarley and Wickens 2005; Hayhursy et al. 2006). Seated in the cockpit behind the flight controls with an array of panels and displays, a manned aircraft pilot is intimately aware of the surroundings, as well as the state of the aircraft. Cues indicating the aircraft performance and possible failures such as visual and aural alerts, vibrations and smells, are readily available to the manned pilots without the need for a transmission media. On the other hand, when operating RPAS, information pertaining to the performance, orientation, motion and system states of the aircraft become very limited to the drone pilots as they need to be sensed by the on-board automation before being sent through the data transmission medium. In addition, drone pilots are stripped of all vestibular and proprioceptive stimuli, essentially rendering them to operate in “sensory isolation” (Van Erp and Van Breda 1999; McCarley and Wickens 2004; Dalamagjidis et al. 2012; International Society of Air Safety Investigators 2015).

Various technologies have been harnessed to enhance and improve the situational awareness of the drone pilot and the controllability of the aircraft. For example, being physically separated from the platform, it is very difficult for drone pilots to detect that the aircraft is encountering turbulence. However, the turbulence can be detected by the aircraft automation and sent to the ground control station or remote controller via the wireless transmission medium. The information can then be conveyed to the pilot through, for example, the vibrating of the remote controller to create awareness of the turbulence (Calhoun et al. 2002). Auditory alerts and visual indicators that are normally available to pilots of manned aircraft can also be presented to the drone pilots as a method of alerting operators to system failures enabling better human performance as compared to using only visual indication to reflect systems status (Dixon et al. 2003; Wickens 2010).

Although automation such as automatic navigation, remote controlling in GPS Mode and the transmitting of sensory information to the drone pilots has enabled RPAS to be operated relatively easily, it cannot be depended upon solely to ensure that safe and reliable operation is always maintained due to the possibilities of malfunctioning automation and degraded GPS signal. The availability, accuracy, and timeliness of this sensory and system information is heavily dependent on the automation software which currently are not being demonstrated to or certified by any aviation authorities to be sufficiently reliable.

In additional to the on-board automation and sensors’ limitations in their reliabilities, the quality and timeliness of the data presented to the drone pilot on the ground will also be constrained by the bandwidth and quality of the communications link between the drone and ground control station or remote controller. (McCarley and Wickens 2004). Data link bandwidth limits, for example, will limit the temporal resolution spatial resolution and field of view of the visual displays presented to drone pilots on the ground and may adversely affect the judgment and decision-making process of the pilot (Van Erp 1999). Besides the quality of the data, there is always a time delay from the transmission of the data to the time they are received by the drone pilot on the ground. This further compounds the difficulty of the drone pilot in receiving up to date and accurate information and status of the aircraft in order to maintain control (Gawron 1998). Other than causing drone pilots to always receive slightly outdated information, data transmission delays reduce the time available for the pilots to process the information and respond with the most appropriate control input as fast as possible.

Rogers et al. (2004) and Tvaryanas et al. (2006) found that as high as 68% of the accidents and incidents involving unmanned aircraft can be linked to the lack of situation awareness. Although the drone automation, GPS signal and the quality of the communication link are usually very reliable, pilots should refrain from over-relying on full functioning automation to operate their aircraft. The ease of operating a drone in GPS Mode and the seemingly reliable (at least most of the times) automation can easily lead pilots to excessively trust and over rely on the automation. This over trust in automation can in turn slowly and eventually erode their skills required to manually operate the aircraft (Parasuraman and Riley 1997). If the drone pilot’s skills do get eroded and when the situation arises that require the pilot to quickly take control of the aircraft without full automation (i.e. in ATTI Mode), they may not be able to assess the situation quickly enough and take the most appropriate recovery actions competently or confidently. Instead, the pilots in these situations may find themselves being left “out of the loop” and not be able to regain safe control of the aircraft (Billings 1991; Wickens and Hollands 2000; Mouloua et al. 2001; Sharma and Chakravarti 2005).

Regardless of the status of on-board automation, integrity of GPS signal received and quality of data transmission link, drone pilots remain responsible for the safe operation of their aircraft. So, it is important that pilots are able to competently regain safe operation of the aircraft via the lowest level of control, i.e. the remote controller and operating in Attitude (ATTI) Mode, when necessary.

In a study conducted by the United States Department of Defence, the accident rate of Unmanned Aerial Vehicles (UAV) can be as much as 100 times higher compared to that of manned aircraft (Department of Defence 2001, Schaefer 2003). A significant percentage of the UAV accidents has been attributed to human errors which in several cases can be attributed to inexperience (Williams 2004; Damalagjidis et al. 2012) and operating the unmanned aircraft via remote controllers (Williams 2004; McCarley and Wickens 2005; Williams 2006).

Human factors dissimilar from those normally experienced by pilots operating manned aircraft, including but not limited to sensory deprivation and motion (or the lack of it) inconsistent with the attitude of the aircraft being controlled, place unique physical and mental demands on the drone pilot (McCarley and Wickens 2005; ICAO 2011). One of the key difficulties that drone pilots may face when operating the aircraft is that there is an inconsistent mapping between the movement of the remote controller and the relative response of the aircraft, especially when a part of the aircraft other than its tail is facing the pilot. For example, when the aircraft nose is facing the pilot handling the remote controller, an input to roll the aircraft to the left will cause the aircraft to appear to roll to the right from the perspective of the pilot. This inconsistent mapping of the drone’s movement with respect to the pilot is a violation of the human factors principle of motion compatibility and may place high cognitive demands on the drone pilot (Wickens and Holland 2000; McCarley and Wickens 2005).

As drones transit into ATTI Mode, either due to the degradation of automation, loss of GPS signal or intentionally in order to fit certain types of operation, the difficulty in controlling the aircraft accurately and precisely increases tremendously as the position of the aircraft will be subjected to deviation caused by environmental factors such as wind. However, drone pilots are still expected to maintain safe operation by maintaining visual contact with the aircraft and safely manoeuvre the drone via the remote controller (Stevenson et al. 2015).

It may be advantageous for a set of flight profiles to be identified as relevant and important for drone pilots to demonstrate satisfactorily during their practical assessments in order for them to be recommended by flight examiners for the granting of the PfAW by the CAA. This way, a more standardised scope of the practical assessment of drone pilots with minimum variance be- tween assessments conducted by different flight examiners can be established.

2.1 Identifying Current Practice for Attitude (ATTI) Mode Assessment

In order to solicit information on how the assessment of drone pilots operating in Attitude (ATTI) Mode is currently being performed in the United Kingdom (UK), a survey form was sent to the entire population of 20 National Qualified Entities (NQEs – flight examiners) authorised by the Civil Aviation Authority (CAA) CAA 2015a) in an attempt to find out what are the typical flight profiles they require drone pilots to perform during the practical assessments they conduct. However, only two of the 20 NQEs responded to the survey. Since the sample size cannot be representative of the NQE population, the two responses were not used and this research looked to the commercial drone operators in the UK as the source of information.

The UK CAA publishes a list comprising of drone operators that have been approved to perform commercial or official drone operations in the UK. All of the 1557 drone operators listed (at the tie of this study) would have been subjected to at least one practical assessment by a NQE in partial fulfillment of the requirements to be granted a Permission for Aerial Work (PfAW). A similar survey was therefore sent to the entire population of CAA-approved drone operators in the UK. The drone operators were asked to recall and report the flight profiles that they had been asked to demonstrate in ATTI Mode during their practical flight assessments with the NQEs. There is no limit to the number of flight profiles each drone operator may report. Every reported flight profile is recorded as one count. Similar flight profiles, however, are grouped together and the total number of reports would be recorded. For example, if one operator reported a flight profile as flying in a circular path around an object and another operator reported a flight profile as flying a square path around a tree, these two profiles would be grouped together as a single flight profile as ‘flying around an object’ and two counts would be recorded.

The flight profiles consolidated based on the survey responses were then sorted according to the number of times they were mentioned in the survey.

A total of 31 responses were received from the survey sent to CAA-approved drone operators. From the responses, 18 different flight profiles were reported to have been asked to be performed in ATTI Mode by drone pilots during their practical flight assessments. The list of all the flight profiles reported are shown in Table 1 and sorted in decreasing number of times they were reported. The top five most common flight profiles identified from the survey were used as a basis to conduct the second survey.

As listed in Table 1, the most common flight profile reported by drone operators was ‘general handling’. As this is a very generic and vague profile to determine its effectiveness of, three flight profiles that had been grouped together under the ‘general handling’ profile were used to solicit input from the participants of the second survey where participants will be asked to rate the effectiveness of various flight profiles. These three profiles are ‘following a line’, ‘following a route’ and ‘recovering from wind’. In summary, the flight profiles that are included in the second survey are as follow:

• Following a line,

• Following a route,

• Figure-of-8,

• Recovering from wind,

• Circling an object,

• Drone orientated other than tail facing the pilot (hereafter referred to as ‘non-tail-facing pilot’)

• Landing.

2.2 Rating Effectiveness of Identified ATTI Mode Flight Profiles

A second survey was conducted where participants were asked to rate the effectiveness of the seven flight profiles identified from the first survey. For each flight profile, a short video clip of a SUA being flown in accordance to the profile was produced. In each video, an inset was included to illustrate to the survey participants the corresponding input and coordination required from the drone pilot.

A snapshot of one of the video clips is shown in Fig. 2.

Snapshot of a video clip from the second survey showing a drone flying various flight profiles with inset illustrating the corresponding pilot input and coordination’s

Full size image

At the beginning of the second survey, the participants were provided with an explanation of the various inputs of the remote controller used in the survey and an explanation of what ATTI Mode means and entails. After watching each video, participants were then asked to rate the effectiveness of the flight profile when used to assess the competence of drone pilots.

It was decided not to send this survey to the targeted populations in the first survey, i.e. NQEs and CAA-approved drone operators in the UK, so as to avoid any prejudiced or biased responses. For example, a drone pilot who is unable to proficiently perform a particular flight profile included in the second survey may deliberately rate that corresponding profile as being not effective at all, regardless of its actual effectiveness. Instead, post-graduate students in the UK were invited to participate in this survey. This sample group is considered to be neutral and unbiased, thus providing a more objective evaluation of the survey questions.

2.3 Statistical Analysis of Survey Data

A total of 28 fully completed responses were received from the second survey. These responses were first tested for normality using the Kolmogorov-Smirnov test. It was found that the participants’ responses for all the flight profiles were normally distributed. From the histograms, it was also observed that the distributions of the effectiveness ratings for the ‘circling an object’ and ‘landing’ flight profiles were negatively skewed.

The mean effectiveness ratings of the flight profiles, as shown in Table 2 in descending order of mean effectiveness, indicate that the ‘landing’ profile (mean = 4.14, SD = 1.01) was rated the most effective profile whereas the ‘following a line’ profile (mean = 2.68, SD = 0.86) was considered the least effective.

The data was then subjected to the Mauchly’s Test of Sphericity, which indicated that the assumption of sphericity had not been violated (χ2(20) = 24.689, p = 0.217).

Thus, no correction for the degree of freedom was required.

The mean effectiveness ratings of the seven flight profiles were then tested to investigate if their differences are significant by an ANOVA with repeated measures test with sphericity assumed. The test results revealed the means were statistically significantly different (F6, 162 = 13.17, p < 0.005).

A post-hoc pairwise comparison of the means of the effectiveness ratings using the Bonferroni correction was further conducted. Results of the comparison revealed that the significant differences exist only between the flight profiles listed below. Details of the pairwise comparison results are shown in Table 3.

• Following a line and Figure-of-8 (p < 0.05)

• Following a line and Circling an object (p < 0.05)

• Following a line and Non-tail-facing pilot (p < 0.05)

• Following a line and Landing (p < 0.05)

• Following a route and Circling an object (p < 0.05)

• Following a route and Non-tail-facing pilot (p < 0.05)

• Following a route and Landing (p < 0.05)

• Figure-of-8 and Landing (p < 0.05)

• Recovering from wind and Circling an object (p < 0.05)

• Recovering from wind and Non-tail-facing pilot (p < 0.05)

• Recovering from wind and Landing (p < 0.05).

The test for normality of the data consolidated from the second survey showed that the ratings of the effectiveness of all the flight profiles surveyed were normally distributed. In addition, visual examination of the histograms revealed that the distributions for the ‘circling an object’ and ‘landing’ flight profiles were negatively skewed. One possible explanation for the negative skew of the flight profile ‘circling an object’ is that not only does the drone pilot need to maintain the aircraft within visual range while accurately controlling its flight path just like any other flight profiles, the object that is being flown around needs to remain constantly within the field of view of the optical camera attached to the drone. This means that the drone pilot will have to allocate additional cognitive resources to another critical task of maintaining the target within the camera’s view at all times. Since this flight profile requires additional cognitive resources attending to an additional task (i.e. monitoring the video imagery sent from the drone camera) when compared to the other flight profiles, the successful execution of the ‘circling an object’ flight profile may, on the average, be considered as a very highly effective flight profile to assess the competency of a drone pilot. This may have led more participants to rate this profile as highly effective, resulting in the negative skew of the histogram.

The other flight profile with a negatively skewed histogram was the ‘landing’ profile. Incidentally, the ‘landing’ flight profile also scored the highest mean effectiveness rating (mean = 4.14, SD = 1.01). In the event of a drone transiting to ATTI Mode due to, for example, a malfunction of automation, a drone pilot may choose to terminate the flight as soon as practicable, thus not be required to perform any further complicated flight profiles such as following a predetermined route. However, the aircraft almost al- ways has to be landed safely. The fact that drone pilots are expected to always land their aircraft safely in ATTI Mode may have influenced most of the survey participants to rate the ‘landing’ flight profile as the most effective flight profile among those that are included in the second survey, resulting in the ‘landing’ flight profile having a negatively skewed histogram and also scoring the highest mean effectiveness.

The ‘following a line’ flight profile has been rated as the least effective flight profile (mean = 2.68, SD = 0.86) to assess the competency of drone pilots. One possible explanation for this is that this profile may have seemed to be very easy to the survey participants since they are not able to fully experience the challenges involved to ensure that the drone maintains the planned line. These challenges may include environmental conditions such as the sun glare and wind. Also, the survey participants may also have opined that ‘following a line’ profile was similar to, and may be thus considered as a subset of, the profile ‘following a route’. As a result, the ‘following a line’ profile is rated as the least effective, and possibly even be considered irrelevant if the ‘following a route’ profile is being considered, such as it is in this survey. The effectiveness of some of the flight profiles was statistically found to be not significantly different. For example, the flight profiles ‘figure-of-8’ and ‘non-tail-facing pilot’ are not significantly different from each other. However, the controls and coordination required to execute both these flight profiles can be perceived to be rather different. This may lead NQEs to require a drone pilot to demonstrate these two flight profiles in a single practical assessment session. The effect of this, at the very least, may be a waste of time and effort for drone pilots to perform two flight profiles that are not statistically significantly different in demonstrating their competency. At the other end of the spectrum, if all the flight profiles that NQEs require drone pilots to execute are not significantly different, for example only the ‘figure-of-8’ and ‘non- tail-facing pilot’ flight profiles were asked to be demonstrated during the practical assessments, the evaluation of the latter’s competence may not be sufficiently comprehensive.

With the knowledge of the mean effectiveness and significant difference, or the lack of it, between flight profiles, NQEs can better design a practical assessment that is more effective by employing profiles that are highly effective and significantly different. For example, the ‘landing’ profile may be always included in the practical assessment since it has been rated as the most effective profile. It may then not be necessary for the second and third most effective flight profiles, i.e. ‘circling an object’ and ‘non-tail- facing pilot’, to be performed during the assessment since these two profiles had been found to be not statistically significantly different from the ‘landing’ profile.

While all the remaining flight profiles were found to be statistically different from the ‘landing’ profile, there is no significant difference between ‘figure-of-8’ and ‘following a route’, between ‘figure-of-8’ and ‘recovering from wind’, nor between ‘figure-of-8’ and ‘circling an object’. Thus, the ‘figure-of-8’ profile may be considered to be included in the practical assessment of drone pilots instead of ‘following a route’ and ‘recovering from wind’, since it has been rated as the most effective among these three flight profiles.

Although the ‘following a line’ profile has been rated as the least effective flight profile, it was found to be statistically significantly different from both the ‘landing’ and ‘figure-of-8’ profiles. Thus, NQEs may also consider including the ‘following a line’ flight profile in the practical assessment of drone pilots in order to assess a wider range of skills and competency.

Accordingly, it is considered that a practical assessment of drone pilots requiring candidates to demonstrate the flight profiles ‘landing’, ‘figure-of-8’ and ‘following a line’ in ATTI Mode can most comprehensively assess drone pilots in the most effective and efficient manner.

Currently, National Qualified Entities (NQEs) are requiring drone pilots to demonstrate a very wide array of flight profiles when assessing their competency as a prerequisite to the latter being granted a Permission for Aerial Work (PfAW) by the Civil Aviation Authority (CAA). This large variance of practical assessments is due to the lack of guidance on the recommended flight profiles that NQEs may stipulate drone pilots to perform during their practical assessments. This may result in a waste of time and effort, as there is a possibility that only flight profiles that are not significantly different would be asked to be performed. In such cases, the assessment of the drone pilot’s competency may also be not sufficiently comprehensive to assess a wider range of skills and competency.

CAA-approved drone operators in the UK have been asked to describe the flight profiles they had been asked to demonstrate during their practical assessments through a survey. The effectiveness of the reported flight profiles to assess SUA pilots’ competency in operating their unmanned aircraft were then rated by post-graduate students in the UK through a second survey. Statistical analysis of the responses from the second survey reflected the mean effectiveness of each flight profiles to assess the competency of drone pilots and also revealed that not all the flight profiles are significantly different from one another.

With this information, NQEs may consider selecting flight profiles that are highly effective and significantly different from one another to be performed by SUA pilots during practical assessments. A possible set of flight profiles that fit these conditions comprises of the flight profiles ‘landing’, ‘figure-of-8’ and ‘following a line’. By designing practical assessments of SUA pilots based on this principle, time and effort would not be spent on flight profiles that are basically testing similar skill sets. More importantly, the assessment of the SUA pilots’ competency may be more comprehensive.

4.1 Further Research

In order to identify a set of flight profiles that drone pilots are currently required to perform in Attitude (ATTI) Mode during their practical assessments, a survey was first distributed to ask SUA operators in the United Kingdom (UK) what were the flight profiles they had to perform during their practical assessment with a National Qualified Entity (NQE). Out of the 18 types of flight profiles consolidated, the top five (which were eventually expanded to seven) were used in a second survey to solicit the effectiveness of each profile in assessing a drone pilot’s competency. In doing so, the opportunity is lost in evaluating the remaining 13 flight profiles which, although currently used less often by NQEs, could well be more effective than the ones analysed in this research. There is also the possibility that a new flight profile could be designed to be more effective in assessing drone pilots’ competency than all those that had been identified. The scope of this research could possibly be expanded to analyse all 18 flight profiles that had been reported to be required by NQEs to be performed during the practical assessment. Also, new flight profiles may be designed and included in the evaluation.

The NQEs and CAA-approved drone operators in the UK were not asked to participate in the second survey in order to avoid prejudice and bias in the collected data. Instead, the sample group of the second survey consisted of post-graduate students in the UK. While this sample group is able to provide an independent and fair response, their relative inexperience in drone flight operations may affect their perception of the input, coordination, and difficulty level of the flight profiles presented to them in the survey through the video clips. Similar future research could be performed with sample groups that include professionals having relevant drone expertise who will still be able to appraise the survey questions without prejudice or bias. An example of such a sample group may by representatives from the CAA.

In addition to asking the survey participants to rate the effectiveness of the flight profiles presented to them, future surveys may also include a free text field for participants to provide the reasons for their ratings and their comments, if any. With this additional information, the explanations of the histogram distribution and analysis of the data may be further supported and substantiated. For example, although it has already been shown statistically that the effectiveness of ‘following a line’ and ‘following a route’ is not significantly different and a possible explanation was discussed, comments from the survey participants may further reinforce the statistics results and analysis.

Much effort was put into this research to help the survey participants understand the challenges involved in operating the drone in ATTI Mode using the remote controller. In order to allow survey participants to fully appreciate the difficulties and skills involved, it may be considered in future research for participants to operate a drone, either a real one or using a simulator, instead of watching a video clip.