Who’s in charge?

1.1 Control and responsibility between driver and vehicle

Figure 1

Automation levels and control transition spectrum (adapted from [4]).

Historically, the allocation of functions to a machine has been motivated by the wish to relieve the human from monotonous and/or strenuous tasks [18]. Initially, the primary aim behind the introduction of driving automation was to address traffic safety problems by supporting the prevention of accidents [9]. While there is a great potential for these technologies to increase traffic safety and reduce crashes, it is important that driving automation systems are designed and integrated in such a way that drivers are supported in building a mental model of the systems capabilities and limitations. Accordingly, an accurate mental model is crucial for a safe interaction with an DAS, since it reflects the driver’s knowledge about the systems purpose, its function and possible driving modes [14].

Lee and Seppelt [10] have summarized the challenges of designing automated systems, stating that they come down to a technologically focused approach, disregarding the human operator who reacts to changes in feedback, changes in tasks and task structure, and the cognitive and emotional responses. All in all, discussions regarding appropriate levels of automation and the transition of control between operator and automation argue that the automation level must be adjusted with regard to the operator’s involvement, designed for collaboration with and the building of appropriate trust in the automated system [8], [10], [11]. Specifically, in the context of driving, the allocation of control becomes critical, bearing in mind the dynamic and continuous nature of the driving task [5], [7].

Flemisch et al. [4] therefore proposed a taxonomy for levels of automation, taking into consideration the authority to change control allocation. Their model describes different levels of automation in general, depicting a range from manual control, through low automation and semi-automation, to high automation and finally full automation. Further, Flemisch et al. [4] illustrate how control can be transferred between the human and the system, mapping the control authority and level of involvement.

Figure 1 illustrates how the human has control authority only at one end of the spectrum and the machine only at the other end. It is noted that within the different levels of automation, control allocation can vary and must be defined in terms of the system functions and the responsibility that users have for the task. The arrows indicate who has the authority to pass on control. According to Flemisch et al. [4], the human is allowed to change control allocation in both directions, while the machine is only allowed to propose a change in control, but not actually implement the change. One important differentiation Flemisch and colleagues [4] make with their description is that even though different regions of control authority can be identified for both human and machine, each actor has certain abilities and therefore not every control allocation might be possible. Instead, they propose context-sensitive control allocation, based on whether the human operator or the machine has the ability to handle different situations. This distinction is particularly relevant in the context of driving automation. This dynamic shift of control between driver and vehicle depending on the context makes the assessment of control authority between driver and vehicle more complex. Therefore, it is important to distinguish actual control from the control that the driver perceives he/she has. For this reason, the term ‘perceived control’ will be used throughout this paper to refer to the driver’s perception of control of the driving task.

Additionally, perceived control influences the drivers’ perception of their responsibility for the driving task [4]. However, there are many factors influencing when we feel responsible; particularly in the context of driving, there are many laws and regulations that define the realm of responsibility and suitable actions for the driver. Therefore, it is helpful to divide responsibility into subjective responsibility and objective responsibility. Subjective responsibility refers to the perceived responsibilities of the driver while objective responsibility is described in the instruction manual or from a legal point of view [4]. This paper focusses on subjective responsibility, which is defined as the driver’s perceived moral and legal obligations that are present in the context of automated driving.

The complexity of a vehicle offering multiple levels of automation can cause a number of issues if drivers do not understand their role. The interaction between driver and vehicle must therefore work satisfactorily if the automation is to contribute to increased traffic safety and to a positive user experience. Thus, it is of considerable importance to address the perceived control and responsibility that drivers have when using automated technologies in order to support the development of driving automation systems that clarify mode awareness and driver responsibility for the driving task under different conditions and in different driving modes.

Monk [12] differentiated between two types of mode awareness. First, the awareness of the existence of different levels of automation and second, the awareness of the currently active mode. Both types of mode awareness are particularly interesting for the focus of this paper, as they give conclusion about the users’ understanding of the automated vehicle and their responsibility for the driving task during the interaction with a vehicle offering several levels of automation.

1.2 Research questions and hypothesis

This study investigates how driver perception of control influences their feeling of responsibility for the driving task in a vehicle with multiple levels of automation, i. e. level 2 and level 4 automation. Data from interviews conducted after an on-road driving session was used to understand the factors influencing users’ feelings of responsibility and their understanding of the different driving modes. It is assumed that:

1. Perceived control influences the driver’s responsibility over the driving task.

2. Perceived control influences the driver’s mode awareness.

2.1 Equipment

2.2 Participants

The study involved 20 participants, 11 females and 9 males, whose ages ranged from 22 to 62 years (Mean = 42, SD = 14). They were recruited and reimbursed through a local agency. All participants had to be holders of a valid driver’s license, and they had to drive a car equipped with Common Cruise Control (Driver Assistance System which maintains a steady set speed) and an automatic gearbox. All the participants were frequent drivers and all but one commuted to work by car daily.

2.3 Study design and procedure

During the drive all 20 participants experienced both modes. However, while the participants could choose to engage with the level 2 system whenever they wanted, they could only use the level 4 system when they received a prompt, indicating that the necessary traffic congestion conditions were fulfilled, and the system was ready. The study consisted of three phases during which data was collected, applying a mixed methods approach. This comprised personal interviews and on-road observations, and the interviews were analysed for this paper.

Phase 2

The driving session took approximately 90 minutes and included highways and urban areas in the San Francisco Bay area (see Figure 2), during which the participants were encouraged to try out the functions as much as they wanted and to think out loud, i. e. verbalizing their thoughts about the interaction with the system. All the drives were conducted during rush hour, either in the morning or the evening, to ensure both congested traffic and free-flowing traffic for all the participants.

Figure 2

Route for the observation study in the San Francisco Bay area.

2.4 Coding and analysis

As a first step, the interviews were transcribed verbatim. From the transcribed material, statements were extracted in which participants described the control they perceived in each driving mode, and whether they always felt responsible for the driving task. A thematic analysis was then conducted using an inductive coding approach, in order to explore how the participants experienced their control and responsibility. The statements were coded into two groups: 1. factors affecting the perceived responsibility, i. e. statements explaining why participants felt high or low levels of responsibility; and 2. links between perceived control and responsibility, i. e. statements discussing the connection between control and responsibility. The statements regarding why participants felt high or low levels of responsibility were further analysed by dividing the statements based on whether they related to the level 2 or level 4 system, and how they perceived their responsibility, in order to identify and compare factors that had a positive or negative impact on the participants perceived responsibility when using each system.

3.1 Perceived responsibility

Based on the answers to the interview questions, driver perception of whether the driver or the system was responsible for the driving task during the two different driving modes was analysed. This resulted in categorization of the perceived responsibility into high and low responsibility and identification of factors influencing that perception.

The level 2 system is a form of assistance and requires the driver to be in control and responsible for the driving task at all times. The level 4 system, on the other hand, completely takes over the driving task under certain conditions so the driver is not responsible for the driving activities during that period. When enquiring about their responsibility during use of the two systems, the preferred outcome would be that the objective responsibility and subjective responsibility match. The results, however, showed a discrepancy resulting in only two drivers whose perceived (subjective) responsibility matched the objective responsibility, i. e. level 2 – driver responsible; level 4 – vehicle responsible. The majority of participants felt responsible for both levels and there were indications of ambiguity over who had control during which driving mode. A summary of the factors influencing the drivers’ perceived responsibility during usage of the level 2 and level 4 system is presented in Table 1.

The feeling of responsibility was partially motivated by the fact that the drivers were still in the driver’s seat and that they had the ability to activate and deactivate the system, or simply the fact that a driver is always responsible for the vehicle and what happens around it. This points to a more moral and legal accountability that drivers see and accounts for this feeling of responsibility. Other reasons why drivers felt responsible was the amount of control they had over the driving task. In level 2, for example, the feeling of responsibility was perceived because they were still steering and monitoring, although some participants explained that the steering assistance was so good, they thought they did not need to intervene. Hence the ambiguity of responsibility in this driving mode. A further indicator for responsibility was the hands-on-wheel icon, which accompanied by auditory feedback tells the driver that they have to steer the vehicle in case they let go of the steering wheel. However, this was overlooked by many and often only recognized after the audible warning chimed.

3.2 Linking responsibility and control

The ambiguity and uncertainty most drivers experienced about their responsibility were influenced by the level of control drivers perceived to have over the vehicle. While the ability to activate and deactivate the systems at any time enhanced their feeling of control and responsibility, the perception of the system being extremely capable and not needing human input decreased their feeling of responsibility. The illustration in Figure 3 provides an overview of how the participants perceived the systems’ control of the driving task and how this related to whom they felt was responsible.

Figure 3

How perceived control affects responsibility in level 2 and level 4 driving modes.

Especially regarding the level 2 system, the participants expressed a high level of trust in its performance, leading them to feel less responsible. This perception was due to the system taking over acceleration, braking and maintenance of distance to vehicles in front, as well as the steering support, which led many participants to feel as if they were sharing control with the car. In some cases, the drivers felt as if they were competing with the car for control as the steering support gave them a feeling that the car wanted to take over control and that they could therefore disengage. Figure 3 illustrates how perceived control affects responsibility and highlights the fact that the participants were ambiguous about the perceived control in level 2 driving mode. As indicated by the vertical arrows, if the system was perceived to have high control, the driver was perceived to be less responsible and vice versa. The participants expressed that they interpreted the system’s feedback as though it was controlling some of the driving task, but it was difficult to interpret how much of the driving task was handled by the system and how much control they themselves were supposed to have.

When the level 4 driving mode was activated the steering wheel was locked and the vehicle completely took over the driving task, which caused heightened ambiguity as to who had control. Most drivers perceived the system to have high or full control over the driving. However, there was a big difference in how the drivers perceived their responsibility. Many still felt responsible even if they did not feel like they were in control owing to lack of trust, and also because as the driver they were able to activate and deactivate the system. This led to a mismatch, as illustrated by the diagonal arrow in Figure 3. Some drivers felt responsible even though the system was perceived to have high control, saying that they always had responsibility when driving and also because they still had the ability to turn off the system.

In summary, in level 2 it seems to be difficult to judge the degree to which the system controls the driving task, while in level 4 drivers had difficulty in letting go of the feeling of responsibility. Ultimately, control allocation and authority during the different driving modes caused confusion, which led to the perceived subjective responsibility not matching with the objective responsibility. Such confusion is risky and indicates a lack of mode awareness or understanding of the systems.

Perceived control influences driver responsibility over the driving task

From the results it seems that perceived control affects perceived responsibility in different ways for the different driving modes. In level 2, the high control that users assume the system to have, and the good system performance, led to the conclusion that they were not responsible for the driving task. This connection between the level of perceived control and responsibility is partly supported by Siebert et al. [19] who found that perceived responsibility was influenced by participants’ perceived ability to generally control the vehicle. The factors identified through the interviews show that aspects like the system’s steering support and feedback through icons are direct influencers of the perceived control that users feel they have. Furthermore, the results suggest that there is a conflict within the feedback that drivers receive. Specifically, the hands-on-wheel icon competing with the system offering advanced support, for instance steering, accelerating and braking, which causes ambiguity over control authority. The same observation was reported by Wilson et al. [20], who found that a hands-on-wheel system alone is not adequate to maintain driver engagement, as there were other factors which indicated they could disengage from the driving task, thus contributing to mode confusion. These assumptions are supported by Flemisch and colleagues [4], who describe the triple bind between responsibility, control and authority and point out that the balance between these concepts needs to be designed with consistency in mind across the different automation levels in order to avoid mode ambiguity. Thus, suggesting a transparent system design regarding the nature of the driver’s role is essential in supporting understanding of control authority and therefore driver responsibility in the different driving modes.

Even though control is a major indicator, it is not the only factor supporting drivers’ understanding of their responsibility for the driving task. In level 4 it can be seen that when the system exerts a high degree of control it supports users in understanding that they are not responsible for the driving task, even though other factors nevertheless made them feel responsible. These factors were not technical or cognitive but rather underlying and indirect influencers, which can be described as legal, ethical or individual, and need further investigation. The results show that some drivers felt in control even though they understood that they were not in charge of the driving task, because they had activated the function or because they could take over at any time. However, this also made it unclear for some drivers where their responsibility lies and whether they could disengage completely or were still responsible for monitoring the system, for example.

Perceived control influences the driver’s mode awareness

The results suggest that control and system understanding are highly interconnected, as the perceived control communicates to the driver what their role is during the act of driving and therefore supports an understanding of the system modes. However, prior results showed that when describing the two driving modes, the respondents were not able to really differentiate between the different modes based on the perceived control as there was considerable ambiguity [13]. These findings further indicated that users’ understanding of driving modes is dependent on the extent of control they perceive. This was also reported by Endsley [3] who during a six-month trial of a vehicle with a level 2 system noted several accounts of mode confusion based on driving mode changes that offered no clear feedback to the driver. Consequently, one may assume that if users have to interact with a vehicle featuring multiple levels of automation which are not clearly distinguishable to them, they will not be able to develop an accurate mental model. The ambiguity of perceived control over the system can therefore result in mode confusion. This implies that mode awareness is the result of a correct mental model, making users’ understanding of the system absolutely critical.

Mode awareness clarifies responsibility

The lack of mode awareness can lead to inadequate and unsafe usage strategies, which will result in serious consequences. Frequent reports of drivers misusing driving automation systems by not paying attention to their driving, or assuming they can release the steering wheel during partially automated driving modes, show the implications of an ambiguously designed system: complacency and over-trust [1], [3]. Looking back at Monks’ definition of mode awareness, there seems to be the need to broaden that definition by including not only the awareness of the existence of different levels of automation, or the awareness of the currently active mode [12], but also the understanding of what each driving mode is capable of. The understanding of who is in charge of what, in each driving mode, means users understand their responsibility for the driving task at any given point in time.