[go: up one dir, main page]
More Web Proxy on the site http://driver.im/
Next Article in Journal
Mobility-Based Multi-Hop Content Precaching Scheme in Content-Centric Vehicular Networks
Previous Article in Journal
Open-Source Telemedicine Platform Based on WebSockets for Management of Biosignals
You seem to have javascript disabled. Please note that many of the page functionalities won't work as expected without javascript enabled.
 
 
Search for Articles:
Title / Keyword
Author / Affiliation / Email
Journal
Article Type
 
 
Advanced Search
 
Section
Special Issue
Volume
Issue
Number
Page
 
Journals
Electronics
Volume 13
Issue 22
10.3390/electronics13224366
electronics-logo
Submit to this Journal Review for this Journal Propose a Special Issue
► Article Menu

Article Menu

share Share announcement Help format_quote Cite question_answer Discuss in SciProfiles thumb_up
...
Endorse textsms
...
Comment
first_page
settings
Order Article Reprints
Font Type:
Arial Georgia Verdana
Font Size:
Aa Aa Aa
Line Spacing:
Column Width:
Background:
Article

Quantifying the Remote Driver’s Interaction with 5G-Enabled Level 4 Automated Vehicles: A Real-World Study

by
Shuo Li
*,
Yanghanzi Zhang
,
Simon Edwards
and
Phil Blythe
School of Engineering, Newcastle University, Stephenson Building, Claremont Road, Newcastle upon Tyne NE1 7RU, UK
*
Author to whom correspondence should be addressed.
Electronics 2024, 13(22), 4366; https://doi.org/10.3390/electronics13224366
Submission received: 16 October 2024 / Revised: 31 October 2024 / Accepted: 5 November 2024 / Published: 7 November 2024
(This article belongs to the Special Issue Advanced Technologies in Intelligent Transport Systems)
Browse Figures
Figure 1
<p>Level 4 automated vehicle (<b>left</b>) and the teleoperation system (<b>right</b>).</p> "> Figure 2
<p>The trail route of the connected and automated logistics.</p> "> Figure 3
<p>The remote driver is in the “monitoring” driving condition in the Level 4 automated vehicle teleoperation workstation.</p> "> Figure 4
<p>The remote driver is in the “disengaged” driving condition in the Level 4 automated vehicle teleoperation workstation.</p> "> Figure 5
<p>Illustration of motor readiness time.</p> "> Figure 6
<p>Illustration of decision-making time.</p> "> Figure 7
<p>Fixation duration heat map when remote driver is in the “monitoring” condition (Gaze filter-Tobii I-VT, Radius 30 px, Scale max value 2.20 s, warmer colours-red and yellow indicates areas where the remote driver focused their gaze the most, cooler colours-green shows areas of less visual attention).</p> "> Figure 8
<p>Fixation duration heat map when remote driver is in the “disengaged” condition (Gaze filter-Tobii I-VT, Radius 30 px, Scale max value 2.20 s, warmer colours-red and yellow indicates areas where the remote driver focused their gaze the most, cooler colours-green shows areas of less visual attention).</p> "> Figure 9
<p>Fixation duration heat map when remote driver is in teleoperation mode (Gaze filter-Tobii I-VT, Radius 30 px, Scale max value 2.20 s, warmer colours-red and yellow indicates areas where the remote driver focused their gaze the most, cooler colours-green shows areas of less visual attention).</p> ">
Versions Notes

Abstract

:
This real-world investigation aimed to quantify the human–machine interaction between remote drivers of teleoperation systems and the Level 4 automated vehicle in a real-world setting. The primary goal was to investigate the effects of disengagement and distraction on remote driver performance and behaviour. Key findings revealed that mental disengagement, achieved through distraction via a reading task, significantly slowed the remote driver’s reaction time by an average of 5.309 s when the Level 4 automated system required intervention. Similarly, disengagement resulted in a 4.232 s delay in decision-making time for remote drivers when they needed to step in and make critical strategic decisions. Moreover, mental disengagement affected the remote drivers’ attention focus on the road and increased their cognitive workload compared to constant monitoring. Furthermore, when actively controlling the vehicle remotely, drivers experienced a higher cognitive workload than in both “monitoring” and “disengagement” conditions. The findings emphasize the importance of designing teleoperation systems that keep remote drivers actively engaged with their environment, minimise distractions, and reduce disengagement. Such designs are essential for enhancing safety and effectiveness in remote driving scenarios, ultimately supporting the successful deployment of Level 4 automated vehicles in real-world applications.

1. Introduction

In recent years, connected and automated vehicles have gained significant attention, with governments across the world increasing their funding and setting legal frameworks for research, development, and the safe deployment of this technology [1,2]. They are widely recognised as having the potential to bring substantial benefits to society. One of the key advantages is their potential to improve fuel efficiency and reduce emissions, which would make an important contribution to environmental sustainability, combating climate change and leading to economic benefits [3,4,5,6]. Additionally, by reducing emissions, the connected and automated vehicles could play a key role in transportation decarbonization and potentially facilitate a greener economy [3,4,5,6,7]. Safety is another major benefit wherein connected and automated vehicles could potentially make a significant impact [8,9,10]. Human error is the leading cause of road accidents, and by removing or reducing human error, vehicle automation has the potential to greatly enhance road safety [11,12], with automated vehicles predicted to reduce crashes by nearly half on average [12]. Advanced sensors, artificial intelligence, and real-time data exchange enable automated vehicles to react faster and more accurately than human drivers, potentially preventing collisions and saving lives [13,14,15]. Beyond safety and environmental benefits, connected and automated vehicles could also help reduce traffic congestion [16]. By communicating with each other and with traffic infrastructure, connected and automated vehicles can potentially lead to shorter travel times, especially in areas with heavy traffic, and make daily commutes more efficient and less stressful for road users [17,18,19]. Moreover, connected and automated vehicles could facilitate social inclusion and improve accessibility for people with limited mobility, such as older people and individuals with disabilities [20,21,22,23]. By offering tailored transportation solutions, these vehicles can provide greater independence and freedom for these user groups, enabling them to access services, work, and social activities more easily and independently, which potentially not only improves their quality of life, but also enables them to be more fully integrated into society [20,24,25].
Connected and automated vehicles could be classed into six levels according to their different functionalities, operational design domains, as well as the requirement of human–vehicle interactions [26,27]. A widely used classification is the definition of vehicle automation developed by SAE International (Society of Automotive Engineers) [28]. Among the different levels of automation, the Level 4 automation systems have unique and innovative features, as they can perform self-driving without human input within their operation design domains [28,29,30]. In contrast to lower-level automation systems, such as those classified under SAE Levels 0 to 3, which generally are able to remain fail-silent in the event of system limitations [28], Level 4 automated vehicles introduce a significant advancement in safety and operational resilience. While systems at the lower levels often rely on human intervention when encountering limitations or malfunctions [28,31], Level 4 AVs are capable of automatically starting fail-safe and fail-operational protocols. These protocols enable the Level 4 automated vehicle to either continue functioning at full capacity, operate at a reduced level, or transition into a secure state, depending on the nature of the issue [26,27,28,32]. One significant solution implemented as a fail-safe for Level 4 automated vehicles is the concept of remote driving. In this system, the automated vehicle can be controlled remotely by a specially trained remote driver using a teleoperation workstation [26,27,33]. Such fail-safe ensures that, in the event of unforeseen circumstances or system limitations, a skilled human driver can take control of the Level 4 automated vehicle, enhancing safety and operational reliability.
This study is among the earliest real-world research assessing remote drivers’ behaviour when teleoperating automated vehicles. It offers valuable insights into the unique challenges and demands that remote drivers face in real-world tasks, contributing significantly to the broader research field. Moreover, this paper aims to introduce a methodology for assessing remote driving performance in real-world scenarios, establishing a foundational framework that can inform future research. The evidence gathered here also serves as a practical resource for developing training programs, advancing technology, and refining operational protocols for the remote driving of automated vehicles.

1.1. Related Work and Research Gaps

Recent studies have investigated teleoperation in vehicle automation from various angles. One important study by Goodall et al. [34] reviewed the legal aspects of vehicle teleoperation. A model was developed to estimate how many remote drivers would be needed to manage large fleets of automated vehicles. Their findings suggested that many jobs held by professional drivers in the USA could be replaced by automated vehicles operated by these remote drivers. This research emphasises the need for governments to update their policies and consider the role of teleoperation, as it could greatly support the advancement of vehicle automation. Another area of research examines the human–machine interface (HMI) used in teleoperating automated vehicles. Kettwich and colleagues tested a prototype HMI designed for SAE Level 4 automated vehicles [33]. This system included various screens, such as video displays, detail and disturbance screens, a map screen, and a touchscreen interface, and required thirteen users from public transport control centres in Germany. The feedback on this prototype was largely positive, suggesting it could effectively assist remote operators. Additionally, some studies focus on defining and classifying teleoperation concepts. Majstorovic and his team conducted a systematic review to categorise different teleoperation strategies that could be used as backup solutions during critical situations or when automated vehicles face operational limitations [35]. They identified six main categories: Direct Control, Shared Control, Trajectory Guidance, Waypoint Guidance, Interactive Path Planning, and Perception Modification. Furthermore, Bogdoll and colleagues surveyed terms related to remote human input systems for automated driving [36], proposing a clear definition to help reduce confusion in the field of vehicle teleoperation. Also, Zulqarnain and his team addressed one of the biggest challenges in teleoperation—latency issues between the teleoperation workstation and the automated vehicles [37]. Their research offers insights into choosing the best locations for teleoperation workstations to minimise delays. Moreover, Li and colleagues qualitatively investigated remote drivers’ needs and requirements when remotely controlling a 5G-enabled Level 4 automated vehicle. Their findings revealed that key support needs among remote drivers include improved field of vision and enhanced motion feedback from the workstation [27]. This research offers valuable insights for developing safe and user-friendly teleoperation systems in vehicle automation. Additionally, the same research team examined user requirements for remote driving from a public end-user perspective, focusing on the new driver–automation–remote driver interaction in Level 4 automated vehicles [26]. The results revealed that end-users want clarity on how remote drivers operate vehicles and expect them to avoid multitasking and distractions. They emphasised the importance of remote drivers being qualified and having local knowledge to address potential issues like road layouts and traffic rules. End-users prefer dedicated remote drivers and believe a review system is essential for evaluating services and addressing liability concerns during remote operation [26].
Although the previous research has examined remote driving from a wide range of perspectives, several research gaps remain significant. The following are examples.
  • There has been limited research quantifying remote driver behaviour and performance in real-world scenarios, limiting the understanding of how remote drivers interact with automated vehicles in practical situations in the real-world settings;
  • There has been limited research exploring the factors potentially affecting remote drivers’ performance and behaviour, such as distractions and the implications of out-of-loop driving; understanding how these factors impact remote operation is crucial for developing effective teleoperation systems.

1.2. Research Aim

To add new knowledge and fill the above knowledge gap, the aim of this study was twofold, as follows.
  • To propose a new methodology to quantify and assess remote drivers’ performance and behaviour when operating automated vehicles remotely in real-world settings;
  • To explore remote drivers’ attention and behaviour when interacting with the 5G enabled Level 4 automated vehicles in real-world conditions, with a particular focus on investigating the effect of mental disengagement on remote drivers’ takeover performance and behaviour in Level 4 automated vehicles.

2. Materials and Methods

2.1. Level 4 Automated Vehicle

This study investigates Level 4 automated vehicles powered by 5G technology, developed by a UK-based company specialising in vehicle automation. The vehicle was retrofitted from an existing Terberg electric tractor unit. The primary objective was to test and demonstrate the operational capabilities of a 5G-enabled autonomous delivery system in a real-world setting, specifically focusing on the autonomous delivery of goods using a 40-tonne truck between Vantec UK and Nissan Motor Manufacturing UK in Northeast England. The system primarily consists of a modified Terberg electric heavy goods vehicle (HGV) and a 5G-enabled teleoperation workstation, as shown in Figure 1.

2.2. The Trail Route and Outline of the Trial

The trail route is shown in Figure 2. The green pin marks the route starting point, while the red pin marks the route finishing point. Sections highlighted in green indicate automated driving. Those points, marked with yellow arrows, show where the automation system requires input from the remote driver. In such cases, the remote driver assesses the situation and makes a strategic ‘GO’ or ‘NO GO’ decision. Areas highlighted in red indicate potential remote driving, where the automation system hands control over to the remote driver, who then operates the vehicle via the 5G-enabled teleoperation system. In such cases, it follows the following sequence: The automated driving system encounters a system limitation, the vehicle then pulls to a stop, and the vehicle notifies the remote drivers to take over the vehicle control.

2.3. Study Overview

The 5G-enabled Level 4 automated vehicle (L4 AV) developed within this project facilitates automated driving classified as SAE Level 4 automation (SAE J3016). According to SAE definitions, Level 4 automation refers to a system capable of managing all elements of the dynamic driving task, even in instances where a human driver fails to respond adequately to intervention requests [28]. An important use case of the Level 4 automated vehicle involves the transition of control of the automated vehicle during critical scenarios that exceed the automation system’s capabilities. In these situations, the automated driving system will come to a halt and enter ‘fail-safe’ mode, allowing the 5G-enabled teleoperation system to be operated by a specially trained remote driver [27]. Consequently, the use cases examined in this study primarily concentrate on the interaction among three entities during the control transition process of the L4 AV: the automated driving system, the teleoperation system managed by the remote driver, and a human safety driver present in the vehicle [26,27].

2.4. Experimental Design

In alignment with the overall aim of the study, which was to quantify the effect of mental disengagement on remote drivers’ performance and behaviour when operating Level 4 automated vehicles remotely in the real-world, the methodological framework adapted for this investigation was a one-way repeated measures experimental design. This design facilitates within-subjects comparisons, ensuring that participants were exposed to different experimental conditions to observe potential changes across these conditions [38]. The independent variable is termed “disengagement,” which operates as a within-subjects factor. The “disengagement” variable consists of two distinct levels, each representing a different cognitive and behavioural state in relation to remote drivers’ interaction with the automated driving systems, as well as the teleoperation workstation. The two levels are as follows.
  • Baseline Condition–Monitoring driving (constantly monitoring the automated system driving);
  • Experimental Condition–Disengaged (disengaged from monitoring the automated system driving).
The “baseline” condition in this study is defined as “monitoring driving” when the remote driver is not directly operating the automated vehicle remotely, as illustrated in Figure 3. In this condition, the remote driver is deliberately physically disengaged from any manual control input, as controlled by enabling their hands to be removed from the steering wheel of the teleoperation workstation. The remote driver is equipped with a communication headset to maintain a continuous line of contact with the on-board safety driver presented in the Level 4 automated vehicles. Throughout the entirety of the “baseline” condition, the remote driver is tasked with maintaining an uninterrupted, vigilant focus when the automated system is performing automated driving, and they are not distracted by any non-driving related activities. The remote drivers were tasked to promptly respond to any system requests, such as takeover requests and strategic GO or NO GO decision-making interventions.
In contrast, the “experimental" condition, as illustrated in Figure 4, is defined as the “disengaged” condition. As in the “baseline” condition, the remote driver is physically disengaged from any manual control input, as controlled by enabling their hands being removed from the steering wheel of the teleoperation workstation. However, a key differentiating factor in the “disengaged” condition is the introduction of a distracted task. The remote driver, while still wearing the communication headset to interact with the safety driver onboard the automated vehicle, is engaged in a non-driving related activity when they are not operating the vehicle remotely. Specifically, the remote driver is assigned a reading task that involves using a hand-held tablet, which shifts their attention away from the operation of the automated vehicles. This task was selected because it would effectively control disengagement status among remote drivers. It is difficult for researchers to control and monitor whether remote drivers actually shift their attention away from operating and monitoring the automated vehicle when simply instructed to do so, especially without implementing any specific control tasks. Utilizing a reading task with a hand-held device could ensure that remote drivers are both mentally and physically disengaged from the driving environment, as well as the controls of the teleoperation workstation. This reading task has been widely employed in research focused on the transition of control in automated vehicles and has been shown to be effective in fostering disengagement [20,31,39]. During this period of automated driving, this experimental setting could enable the remote drivers’ cognitive load to be divided, potentially creating a state of partial disengagement from the driving task itself. Despite this, they are still required to remain sufficiently aware to respond promptly to any requests issued by the automation system for intervention, such as takeover and strategic decision commands.

Measurements

To quantify remote drivers’ behaviour, several measures were adopted, as shown in Table 1. To begin with, the concept of motor readiness time is used to quantify the response of a remote driver in relation to different requests issued by the automation systems, including takeover requests and strategic decision-making interventions. This time measurement is defined as the duration between the moment the remote driver receives the necessary authorisation to proceed with the request to the point when the remote driver makes a move, such as a hand movement or verbal communication to take over vehicle control or to initiate a ‘GO’ or ‘NO GO’ decision. Drivers’ response time to events/incidents plays a vital role in assessing the safety and effectiveness of transport technologies and the in-vehicle systems [40]. And it has been widely adopted by previous research assessing drivers’ interactions with automated vehicles [20,21,41,42,43,44]. In the context of the remote driving of the Level 4 automated vehicle in the real-world settings, understanding this reaction time is essential for assessing the overall effectiveness and safety of remote driving, as it directly impacts the remote driver’s ability to respond to dynamic road conditions and unforeseen events in the real world. Furthermore, it can provide insights into the reliability and effectiveness of a wide range of designs of connected and automated logistics, such as user interface design and communications protocols.
In addition, the construction of decision-making time is adopted as another measurement to quantify how fast a remote driver’s decision-making is when receiving the necessary authorisation to proceed with a request by the automation system. It is defined as the duration between the moment the remote driver receives the requisite authorization to proceed with a request to take over and the time point when the remote driver is making a decision to ‘GO’ or ‘NO GO’/start to operate the vehicle remotely. Understanding the decision-making time potentially informs the development of more advanced teleoperation of automated driving that prioritises not only operational efficiency, but also the physical and cognitive performance of the remote driver. This potentially contributes to the development of a robust framework for evaluating interactions among remote drivers, the automated systems, and the teleoperation system.
Finally, remote drivers’ gaze behaviours were collected to visualise their attention. The visualisation of remote drivers’ attention is important in assessing safety and usability within automated driving systems. Eye tracking enables a comprehensive understanding of how remote drivers allocate their attention, identifying which aspects of the driving environment capture their focus and revealing potential distractions. This is important for assessing the remote driver’s cognitive load, allowing for the development of interventions to minimise distractions and improve the overall safety of the teleoperation of the automation systems.
All the measures were collected using Tobii Pro Glasses 2, which is an effective research tool and has been proven to be able to generate useful research data [45].

3. Results

3.1. Motor Readiness Time

In this study, motor readiness time was adopted as an important measurement to evaluate how fast a remote driver responds to system requests while operating the Level 4 automated vehicle on the teleoperation workstation. Specifically, this measurement captures the time from the moment the remote driver receives an authorization request—typically prompting them to either make a strategic decision or completely take over control of the vehicle—to the point when the remote driver exhibits a detectable response, such as move-hands, or a verbal response. In the context of the remote operation of automated vehicles, this motor readiness is important for ensuring timely and effective interventions during periods of vehicle automation when remote control becomes necessary.
Figure 5 visually presents the concept of motor readiness time in this study, representing the time gap between the issue of the system request and the remote driver’s initial response. The results showed that, in the “monitoring” condition, where the remote driver was required to constantly monitor the automated vehicle’s operation, a mean motor readiness time of 2.916 s was observed. In contrast, during the “disengaged” condition, where the remote driver was not actively monitoring and was distracted by a reading task, a significantly longer motor readiness time of 8.225 s was recorded. This notable increase in reaction time, amounting to a difference of 5.309 s, indicates the substantial delay happening when the remote driver’s attention is diverted from the monitor task when the Level 4 automated vehicle is performing self-driving.

3.2. Decision-Making Time

In this study, decision-making time was adopted as a key performance indicator to assess the speed and efficiency of remote drivers in making critical operational decisions when operating the Level 4 automated vehicle. The decision-making time, as illustrated in Figure 6, is defined as the time interval between the moment the remote driver receives the system request—an authorization request requiring intervention, such as a request to proceed or take over vehicle control—to the moment the remote driver initiates a decisive action. This action includes initiating a ‘GO’ or ‘NO GO’ command, or physically starting the process of operating the Level 4 automated vehicle remotely. The decision-making time is a fundamental measure of the remote driver’s ability to process situational information and execute an appropriate response under varying levels of engagement and cognitive workload in the Level 4 automated vehicles. The experimental results showed substantial differences in decision-making time across different conditions of remote drivers’ mental status. Under the “monitoring” condition, where the remote driver was actively monitoring the vehicle’s automated operations, an average decision-making time of 6.111 s was recorded. In contrast, in the “disengaged” condition, where the remote driver was distracted by a reading task on a tablet, a significantly longer decision-making time of 10.343 s was observed. This marked a notable difference of 4.232 s between the two conditions. The marked difference in decision-making times reflects the profound influence of disengagement and cognitive distraction on the remote driver’s ability to promptly and effectively process and respond to operational demands. In the “monitoring” condition, the remote driver’s sustained attention and vigilance likely led to quicker comprehension of the vehicle’s status and enabled more efficient decision-making. In contrast, the “disengaged” condition, wherein the remote driver’s situational awareness was reduced, led to a prolonged decision-making process, suggesting that cognitive disengagement from the driving loop significantly affected the remote driver’s ability to assess and respond to critical situations in a timely manner.
The above showed that the disengagement led to delays in both the motor readiness time as well as decision-making time. A possible explanation is that when the remote driver is distracted by the reading task while the automation system is performing automated driving, the reduced situation awareness, divided attention, and multitasking may lead to a slow reaction and decision-making. In addition, the “disengagement” condition was achieved via the remote driver hand holding a tablet. So, when the remote driver was suddenly required to step in, the remote driver had to put down the tablet, which led to further delays in terms of reaction and decision-making.

3.3. Remote Driver’s Visual Attention While Interacting with the 5G CAL

The remote driver’s visual attention was analysed using eye-tracking heat maps, which provide a detailed visualisation of how visual focus is distributed across the stimulus environment. Eye-tracking heat maps are an effective tool for revealing the concentration of visual attention by highlighting areas of interest and indicating where participants focus their gaze most intensely [46,47]. In the context of this experiment, heat maps were employed to illustrate the distribution and intensity of the remote driver’s gaze during two conditions: the “monitoring” condition (where the driver is constantly monitoring the vehicle’s operation) and the “disengaged” condition (where the driver’s attention is distracted by a non-driving reading task). An important aspect of this analysis is the fixation duration, which measures the length of time the remote driver’s gaze remains fixed on a specific area. Fixation duration is a widely used and well-established eye-tracking metric, commonly used to measure the subject’s cognitive processing load [48,49]. Research has demonstrated that longer fixation durations are correlated with greater mental task demands, as participants need to devote more cognitive resources to interpreting and processing information [48,49]. In the context of driving, increased fixation durations have been observed during hazardous situations, suggesting higher mental workload as drivers process and evaluate potential threats [50,51,52]. Therefore, fixation duration serves as an ideal measure for this study, as the experimental design involves cognitive manipulation—achieved by distracting the remote driver with a reading task—to induce disengagement. For this experiment, duration-based heat maps were generated for both the “monitoring” and “disengaged” conditions, which enables a comparative visual representation of the remote driver’s gaze behaviour for these conditions. These heat maps were created based on absolute fixation durations, with the colour scale adjusted to a maximum duration of 2.2 s, where deep red represents the areas of longest fixation. This allows for a clear depiction of how visual attention shifts in response to varying levels of cognitive engagement.
As shown in Figure 7, the heat map corresponding to the “monitoring” condition reveals a broad distribution of the remote driver’s visual attention across the monitoring environment. Specifically, the driver’s gaze was spread widely over the different display monitors, but there was a distinct concentration of high fixation durations on the visual representation of the road ahead. This pattern suggests that, when constantly monitoring the automated driving system, the driver is allocating substantial cognitive resources to maintaining situational awareness of the road environment, as indicated by the long fixations in this critical area.
The following figure (Figure 8) illustrates the remote driver’s visual attention patterns during the “disengagement” driving condition, wherein the driver was distracted by a reading task while the automated driving system was driving the vehicle. In this condition, the remote driver’s visual attention was spread across both the monitors displaying the driving environment and the tablet used for the reading task. High fixation durations were distributed between these two focal points. This reflects the divided attention of the remote driver.
Compared to the “monitoring” condition, in which the driver’s attention was primarily focused on the road ahead, the “disengaged” condition showed increased heat map activity across both the monitors and the tablet. This shift indicates a slightly higher mental workload in the “disengaged” condition, consistent with the effects of cognitive distraction.
Figure 8 provides a detailed illustration of the remote driver’s visual attention patterns during teleoperation mode, wherein the remote driver controls the automated vehicle remotely via the 5G-enabled teleoperation workstation. As shown in Figure 9, there is a notable increase in fixation duration compared to both the “monitoring” and “disengagement” conditions. This increase in fixation duration, which represents prolonged focus on specific visual areas, reflects a marked increase in the remote driver’s cognitive workload during teleoperation. It could be interpreted as though, in teleoperation mode, the remote driver is actively engaged in navigating and controlling the vehicle, which requires constant attention to multiple sources of information, such as road conditions, vehicle status, and environmental information. This level of engagement is more demanding compared to the “monitoring” and “disengaged” conditions, where the remote driver merely oversees the automation system. The increased fixation duration observed during teleoperation suggests that operating the vehicle remotely led to higher cognitive demands.

4. Discussion

The results showed that, compared to constantly monitoring during driving, mental disengagement led to a slowed motor readiness time for the remote driver when required by the automation system to step in, with a difference of 5.309 s. This finding is in accordance with previous studies that found disengagement from driving affects reaction times [53,54,55] and suggests a profound impact of cognitive disengagement on the remote driver’s response efficacy. In particular, the extended motor readiness time among remote drivers in the “disengaged” condition underscores the risks associated with driver distraction and reduced situational awareness, which could critically impair their ability to promptly assume control of the vehicle in situations necessitating immediate intervention [45,56,57,58,59,60,61]. This highlights the importance of maintaining a certain level of cognitive readiness for remote drivers even when they are not controlling the vehicle remotely. Such delays could pose safety risks in real-world applications, where the timely execution of a ‘GO’ or ‘NO GO’ decision is essential for mitigating potential risks and ensuring smooth and cost-effective vehicle operation. Thus, these results underscore the need for further exploration into methods for reducing distraction-related behavioural impact on operators in remote driving contexts, with a particular focus on enhancing situational awareness and minimising cognitive disengagement during automated vehicle operations. Regarding the decision-making time, the results showed that compared to constantly monitoring driving, mental disengagement leads to slowed decision-making time for the remote driver when required by the automation system to respond and make decisions, with a difference of 4.232 s. This delay in decision-making when the remote driver is disengaged has important implications for the safety and reliability of remote operations of vehicle automation, especially in critical situations where rapid intervention may be required. The difference of 4.232 s of decision-making time observed in the “disengaged” condition could prove detrimental in time-sensitive scenarios, where swift decision-making is essential to prevent accidents or ensure the safe continuation of automated vehicle operations. These findings highlight the need for strategies to mitigate the effects of distraction on decision-making in remote drivers, such as the implementation of advanced driver warning systems or cognitive support mechanisms to maintain a higher level of situational awareness during remote driving tasks [27]. Further research is warranted to explore the cognitive and environmental factors that contribute to decision-making delays and to develop solutions aimed at optimizing decision-making efficiency in remote driving contexts [26,27].
These findings are in line with previous research that found disengagement in the driving loop led to delays in reaction time and decision-making [55]. They could be interpreted by the three-level model of situation awareness [62], which clearly describes the cognitive processes involved in understanding and reacting to dynamic situations. Endsley’s model emphasizes three stages of situational awareness (SA): perception, comprehension, and projection [62]. In the context of the present study, in the “monitoring” condition, the remote driver was likely able to more effectively perceive key elements of the environment (Level 1 SA), comprehend their meaning in relation to the vehicle’s performance and the driving environment (Level 2 SA), and project future states based on this understanding (Level 3 SA), facilitating rapid motor readiness time and decision-making. However, in the “disengaged” condition, the remote driver was kept out of the driving loop while engaged in the reading task; a failure to adequately perceive or comprehend relevant information about the vehicle and the driving environment would slow down the decision-making process, as the remote driver would struggle to build an accurate mental model of the situation. This breakdown in situational awareness is particularly concerning in remote operations, where timely responses to dynamic conditions are critical for safety. Moreover, the cognitive load experienced during disengagement may have affected the driver’s ability to maintain higher levels of SA, aligning with research that links multitasking and distractions with reduced cognitive capacity for complex tasks [63,64]. This further highlights the importance of designing remote driving systems that support continuous engagement and reduce cognitive distractions, to ensure that remote drivers maintain a high level of situational awareness and are prepared to react promptly and effectively to evolving operational demands. In addition, prolonged reaction times in remote driving can result in slower responses to real-time events, creating gaps in traffic that reduce the effective the number of maximum vehicles per hour per lane (VPHPL) [65]. Therefore, delays in remote driver response times could not only compromise safety, but also reduce road capacity, potentially leading to reduced road capacity and, thereby, significant inefficiencies in traffic flow.
The virtualisation of the remote driver’s attention between the “monitoring” and “disengaged” conditions supports the notion that the reading task significantly influenced the driver’s ability to maintain focused attention on the vehicle’s operational displays. The increased heat (Figure 7) observed in this condition suggests that the remote driver’s cognitive load was significantly increased, likely due to the cognitive demands of dividing attention between monitoring the automated vehicle’s status and processing the information presented on the tablet. This multitasking dynamic appears to have introduced an additional layer of cognitive workload, which, in turn, affected the driver’s ability to efficiently respond to driving-related stimuli. This could be explained by previous research suggesting that safe multitasking while driving depends on selecting non-driving related tasks that complement the specific demands of driving at a given moment [66]. For instance, when fully engaged in challenging tasks like navigating city traffic [66,67], it is critical to focus entirely on driving. However, in this study, the remote driver’s task was equally demanding, making continuous focus on driving operations essential for maintaining performance and safety. This visual pattern of divided attention (Figure 7) correlates with the delayed motor readiness time and extended decision-making time observed in the “disengaged” condition, as previously reported in the study. The delayed responses can be attributed to the cognitive burden imposed by the reading task, which diverted the driver’s attention away from critical driving information and slowed the processing of the information related to the automated vehicle and driving environment. A possible explanation for these findings could be the cognitive limitations associated with multitasking [63,64,66]. When the remote driver was engaged in the reading task, their attention was continuously shifting between the tablet and the vehicle’s monitoring systems, thereby increasing the cognitive load. This constant shifting between tasks not only increased the mental demand, but also likely impaired the driver’s ability to maintain sustained focus on the driving environment. The divided attention, coupled with the cognitive load induced by reading, contributed to slower decision-making and motor readiness times, further highlighting the risks of cognitive disengagement in remote drivers of automated vehicles. These results suggest that distractions such as reading and other non-driving tasks can significantly compromise the remote driver’s ability to respond quickly and accurately to operational requests by the automation system. This highlights the need for strategies to mitigate cognitive distractions and manage mental workload in remote driving systems to ensure optimal performance and safety.
The virtualisation of remote driver’s attention when controlling the vehicle remotely via the teleoperation workstation showed a marked increase in their fixation duration, which reflects a significantly higher workload. This finding aligns with established research on fixation duration, which suggests that longer fixations are typically associated with more complex cognitive processing and higher task demands [68,69]. In the context of the remote driving of automated vehicles, the need to process visual data from multiple monitors, interpret sensor information, and issue control commands in real time likely contribute to the significant increase in cognitive workload. The remote driver is required to integrate and process a variety of information while making critical decisions about the vehicle’s movements at the same time, which could result in prolonged fixation durations as the driver allocates more mental resources to these demanding tasks. Moreover, the increased fixation duration in teleoperation mode highlights the complexity of remote vehicle control, particularly in comparison to the relatively lower cognitive demands of the “monitoring” and “disengagement” conditions. In the “monitoring” condition, the remote driver’s role is largely passive, mainly monitoring the automation system driving the vehicle. However, teleoperation mode places the remote driver in a more cognitively intensive position, wherein continuous focus and rapid decision-making are critical to the safe and effective remote operation of the automated vehicle. The substantial increase in fixation duration during teleoperation could also suggest potential implications for fatigue and stress [70], which may negatively affect the driver’s ability to maintain optimal performance over time. This highlights the importance of developing strategies to manage cognitive workload in remote driving operations, such as improving human–machine interfaces (HMIs) or incorporating assistive technologies to reduce the mental burden on the remote driver [27].

5. Conclusions and Future Work

Vehicle automation has the potential to deliver significant social, economic, and environmental benefits. Before fully autonomous vehicles became widely available, remote operation empowered by 5G networks could serve as a critical transitional technology, enabling the safer and more efficient oversight of automated vehicles in complex and unforeseen situations [26,27,33,34,36,71,72].
This study proposed an innovative method to quantify remote drivers’ behaviour and performance when interacting with the Level 4 automated vehicle in the real-world settings. In addition, this study adopted eye-tracking technology to contribute new knowledge to the field of the remote operation of Level 4 automated vehicles by addressing key knowledge gaps in the understanding of remote drivers’ performance and behaviour. The study emphasises the critical role of visual and cognitive engagement in the remote driving of Level 4 automated vehicles. Results showed that distraction and multitasking significantly increase response latency and impair the decision-making of the remote driver, potentially raising safety concerns. Eye-tracking data provided valuable insights into how visual attention and cognitive load are distributed in different driving conditions. Teleoperation mode demands higher cognitive effort from the remote driver due to the need for real-time decision-making. The key findings are summarized as follows.
  • Compared to constantly monitoring driving, mental disengagement (achieved by distraction via a reading task on a tablet) leads to slowed motor readiness time from the remote driver when required by the Level 4 automated driving system to step in, with a difference of 5.309 s, which highlights the risks associated with divided attention during remote driving;
  • Compared to constantly monitoring driving, mental disengagement leads to slowed decision-making time from the remote driver when required by the Level 4 automated driving system to step in and make decision, with a difference of 4.232 s, emphasising the detrimental effect of distraction on the timely intervention of remote drivers;
  • Compared to constantly monitoring driving, mental disengagement was found to shift the remote driver’s attention away from the road, the diminished focus from which compromises the driver’s ability to maintain situational awareness;
  • Compared to constantly monitoring driving, mental disengagement leads to increased cognitive workload for the remote driver;
  • When the remote driver is controlling the vehicle remotely via the teleportation system, it resulted in higher cognitive workload compared to the “monitoring” and “disengagement” conditions.
The implications of this study for the UK and global automated vehicle industry are significant, particularly when the importance of integrating remote driving technologies into the ecosystem of automated driving has become widely recognised [26,27,34,36]. The findings highlight the potential impact associated with the cognitive disengagement of remote drivers, such as delayed response times and impaired decision-making when remote drivers are distracted. In urgent scenarios where rapid intervention is required, even minor delays could potentially lead to serious safety issues. For the vehicle automation industry, this underscores the need to explore solutions and develop systems that minimise remote driver distractions and manage cognitive workload effectively. It also calls for improved human–machine interfaces and advanced driver warning systems to ensure that remote drivers can maintain optimal workload and situational awareness so that they can respond promptly and effectively. As the UK pushes forward with 5G-enabled vehicle networks and teleoperation technologies [73], the evidence-based knowledge gained from this study could be essential for facilitating safer and more reliable remote driving systems.
While this study has contributed valuable insights into the impact of distraction on remote drivers operating Level 4 automated vehicles, there are still limitations that potentially suggest areas and directions for future research. Firstly, this study’s experimental design and findings are based on the specific use case of deploying automated vehicles and remote driving technologies within the logistics sector. This focus allows for a detailed exploration of the operational challenges, response times, and cognitive demands unique to logistics applications. Future research should expand to explore and quantify remote driving performance across diverse use cases, such as automated public transportation, automated emergency services, and automated passenger car ride-hailing services. These use cases present distinct operational demands, environmental factors, and driver response requirements, which could yield different insights and performance metrics for remote driving. This broader examination will help build a more comprehensive understanding of remote driving effectiveness and safety across various real-world scenarios. Secondly, the research is at the cutting edge of its field and is one of the first studies of its kind conducted in the real-world setting. As such, the availability of fully trained and qualified remote drivers was limited, as the technology and training infrastructure are still in the early stages of development and deployment. However, this study is at the cutting edge of remote driving research and represents one of the earliest attempts to evaluate remote driving behaviours in a real-world setting, rather than a purely theoretical or simulated environment. This real-world approach provides unique and innovative insights into the challenges and requirements faced by remote drivers when conducting real-world tasks, adding significant value to the wider research landscape. Additionally, a key aim of this paper is to propose and establish a methodology for evaluating remote driving performance under real-world conditions, providing a foundation for future studies and research, as well as offering real-world, evidence-based knowledge that can guide the development of training programs, technology, and operational protocols as the field of vehicle automation remote driving progresses. By working with the existing limitations, this study contributes essential groundwork, bridging theoretical research and practical application in an emerging and rapidly evolving domain of the remote driving of automated vehicles. This restriction further highlights the pioneering nature of the study, but also underscores the need for future research to involve a larger and more diverse pool of remote drivers to enhance the reliability and generalisability of the results. A larger sample would also allow for deeper statistical analysis, including an exploration of individual differences in response to distractions. In addition, in the present study, the non-driving task used to distract the remote driver involved reading from a tablet. Although this task effectively created a state of cognitive distraction and has been widely used by previous studies of vehicle automation, it represents just one of many potential distractions that remote drivers may encounter in various real-world scenarios. Future research should, therefore, investigate a broader range of non-driving tasks to evaluate their impact on the remote driver’s behaviour, attention, reaction time, and decision-making. For example, activities such as eating, drinking, and using mobile phones may present different levels of cognitive load and physical engagement, potentially leading to varied effects on performance [25,26]. Investigating the influence of these additional tasks would provide a more comprehensive understanding of how various forms of distraction affect remote driving operations in the real-world scenarios. Another point relates to the environmental conditions in which the remote driver operated. The experiments were conducted in a setting with some level of ambient noise, which may have influenced the driver’s performance. While the present research reflects real-world conditions, the experimental environment may be less controllable compared to lab-based ones. To more accurately quantify the effects of distraction and disengagement on the remote driver’s behaviour, future research could be conducted in a more controlled environment with minimal external noise. A quieter environment would allow for the isolation of specific variables related to distraction and cognitive load, providing a clearer picture of how the remote driver interacts with the automation system without the confounding influence of environmental factors. Moreover, the design of the human–machine interface (HMI) in this study involved visual and auditory signals for communication between the remote driver and the automation systems. Future research should explore the use of additional or alternative modalities, such as tactile or vibration-based feedback, to enhance the effectiveness of the HMI. Multimodal interfaces could provide a more robust means of communication, especially in situations wherein one sensory channel may become overloaded or ineffective (e.g., in a noisy environment where auditory signals might be less noticeable for remote drivers). Exploring how different combinations of visual, auditory, and tactile signals affect remote driver performance could lead to innovations in HMI design, making remote operation safer and more effective. Furthermore, the present study found that the remote driver’s mental workload was significantly higher when controlling the vehicle remotely compared to when they were in monitoring driving mode. This increased cognitive load during active control may contribute to fatigue, reduced situational awareness, and delayed response times among remote drivers. Future research should investigate potential measures and strategies for reducing mental workload in remote driving scenarios. This could involve the development of advanced user interfaces that adapt to the remote driver’s cognitive state, the integration of assistive technologies like AI-based decision support, the use of predictive algorithms that support automated vehicle decision-making to reduce the cognitive demands placed on the remote driver, or integrating the Cooperative Intelligent Transport Systems, which enable the vehicle to communicate with infrastructure and provide the remote driver with timely and context-aware warnings and informational alerts [74]. Such strategies and measures could potentially contribute to a better understanding of how to manage and mitigate mental workload and distractions, which will be critical for maintaining long-term performance among remote drivers and ensuring the safety and reliability of remote driving operations.
Overall, the present study highlights the importance of quantifying human–machine interactions in automated vehicles in urgent situations [74], as well as the effectiveness of using eye-tracking data to examine operators’ situations awareness within these system [75]. It has pointed out several important directions for future research, which could lead to the development of more comprehensive guidelines and technologies for optimizing remote driver performance. Our next step potentially involves adopting more complex routes with mixed traffic conditions to better understand how remote drivers navigate real-world scenarios that involve interacting with various road users. We also plan to modify the design of teleoperation workstations based on the findings of this study, including ergonomic improvements, improved feedback systems, and features that facilitate better situational awareness for remote drivers. Finally, we are also planning to explore the best practices for training remote drivers that incorporate the findings of current research, which would potentially equip remote drivers with skills needed to maintain situations awareness and manage attention in critical situations.

Author Contributions

Conceptualisation, S.L., Y.Z., S.E., and P.B.; methodology, S.L., Y.Z., S.E., and P.B.; software, S.L. and Y.Z.; validation, S.L., Y.Z., S.E., and P.B.; formal analysis, S.L. and Y.Z.; investigation, S.L., Y.Z., and S.E.; resources, S.L., Y.Z., S.E., and P.B.; data curation, S.L. and Y.Z., writing—original draft preparation, S.L. and Y.Z.; writing—review and editing, S.L., Y.Z., S.E., and P.B.; visualisation, S.L. and Y.Z.; supervision, S.L., P.B., and S.E.; project administration, S.L., Y.Z., S.E., and P.B.; funding acquisition, S.L., Y.Z., S.E., and P.B. All authors have read and agreed to the published version of the manuscript.

Funding

This study was funded by the UK Centre for Connected and Autonomous Vehicles (CCAV) and Innovate UK V-CAL project (10039732); the UK Centre for Connected and Autonomous Vehicles (CCAV) and Innovate UK SAMS project (10041570); and the UK Department for the Digital, Culture, Media and Sport (DCMS) 5G-Enabled Connected and Automated Logistics (CAL) project.

Institutional Review Board Statement

This study was approved by the Ethics Committee of Newcastle University (protocol code 33357/2023 and date of approval 25 May 2023).

Data Availability Statement

The data are available on request to the corresponding author.

Conflicts of Interest

The authors declare no conflicts of interest.

References

  1. HM Government. Connected & Automated Mobility 2025: Realising the Benefits of Self-Driving Vehicles in the UK. 2022. Available online: https://assets.publishing.service.gov.uk/media/62ff438c8fa8f504cdec92df/cam-2025-realising-benefits-self-driving-vehicles.pdf (accessed on 1 October 2024).
  2. Tyers, R. Automated Vehicles Bill [HL] 2023–24. Parliament. 2024. Available online: https://commonslibrary.parliament.uk/research-briefings/cbp-9973/ (accessed on 1 October 2024).
  3. Dichabeng, P.; Merat, N.; Markkula, G. Research Journeys to Net Zero; Routledge: London, UK, 2024; pp. 148–159. [Google Scholar]
  4. Bhardwaj, C.; Axsen, J.; Crawford, C. Simulating long-term emissions from private automated vehicles under climate policies. Transp. Res. D Trans. Environ. 2023, 118, 103665. [Google Scholar] [CrossRef]
  5. Taiebat, M.; Brown, A.L.; Safford, H.R.; Qu, S.; Xu, M. A review on energy, environmental, and sustainability implications of connected and automated vehicles. Environ. Sci. Technol. 2018, 52, 11449–11465. [Google Scholar] [CrossRef]
  6. Jiang, H.; Hu, J.; An, S.; Wang, M.; Park, B.B. Eco approaching at an isolated signalized intersection under partially connected and automated vehicles environment. Transp. Res. Part C Emerg. Technol. 2017, 79, 290–307. [Google Scholar] [CrossRef]
  7. Tsakalidis, A.; Gkoumas, K.; Pekár, F. Digital transformation supporting transport decarbonisation: Technological developments in EU-funded research and innovation. Sustainability 2020, 12, 3762. [Google Scholar] [CrossRef]
  8. Campbell, M.; Egerstedt, M.; How, J.P.; Murray, R.M. Autonomous driving in urban environments: Approaches, lessons and challenges. Philos. Trans. R. Soc. A Math. Phys. Eng. Sci. 2010, 368, 4649–4672. [Google Scholar] [CrossRef]
  9. Lee, D.; Hess, D.J. Regulations for on-road testing of connected and automated vehicles: Assessing the potential for global safety harmonization. Transp. Res. A Policy Pract. 2020, 136, 85–98. [Google Scholar] [CrossRef]
  10. Sohrabi, S.; Khodadadi, A.; Mousavi, S.M.; Dadashova, B.; Lord, D. Quantifying the automated vehicle safety performance: A scoping review of the literature, evaluation of methods, and directions for future research. Accid. Anal. Prev. 2021, 152, 106003. [Google Scholar] [CrossRef]
  11. Haghi, A.; Ketabi, D.; Ghanbari, M.; Rajabi, H. Assessment of human errors in driving accidents; analysis of the causes based on aberrant behaviors. Life Sci. J. 2014, 11, 414–420. [Google Scholar]
  12. Wang, L.; Zhong, H.; Ma, W.; Abdel-Aty, M.; Park, J. How many crashes can connected vehicle and automated vehicle technologies prevent: A meta-analysis. Accid. Anal. Prev. 2020, 136, 105299. [Google Scholar] [CrossRef]
  13. Khayyam, H.; Javadi, B.; Jalili, M.; Jazar, R.N. Artificial intelligence and internet of things for autonomous vehicles. In Nonlinear Approaches in Engineering Applications: Automotive Applications of Engineering Problems; Springer: Berlin/Heidelberg, Germany, 2020; pp. 39–68. [Google Scholar]
  14. Bathla, G.; Bhadane, K.; Singh, R.K.; Kumar, R.; Aluvalu, R.; Krishnamurthi, R.; Kumar, A.; Thakur, R.N.; Basheer, S. Autonomous vehicles and intelligent automation: Applications, challenges, and opportunities. Mob. Inf. Syst. 2022, 2022, 7632892. [Google Scholar] [CrossRef]
  15. Fu, Y.; Li, C.; Yu, F.R.; Luan, T.H.; Zhang, Y. A survey of driving safety with sensing, vehicular communications, and artificial intelligence-based collision avoidance. IEEE Trans. Intell. Transp. Syst. 2021, 23, 6142–6163. [Google Scholar] [CrossRef]
  16. Aria, E.; Olstam, J.; Schwietering, C. Investigation of automated vehicle effects on driver’s behavior and traffic performance. Transp. Res. Proc. 2016, 15, 761–770. [Google Scholar] [CrossRef]
  17. Festag, A. Cooperative intelligent transport systems standards in Europe. IEEE Commun. Mag. 2014, 52, 166–172. [Google Scholar] [CrossRef]
  18. Cohen, T.; Cavoli, C. Automated vehicles: Exploring possible consequences of government (non) intervention for congestion and accessibility. Transp. Rev. 2019, 39, 129–151. [Google Scholar] [CrossRef]
  19. Gruyer, D.; Orfila, O.; Glaser, S.; Hedhli, A.; Hautière, N.; Rakotonirainy, A. Are connected and automated vehicles the silver bullet for future transportation challenges? Benefits and weaknesses on safety, consumption, and traffic congestion. Front. Sustain. Cities 2021, 2, 607054. [Google Scholar] [CrossRef]
  20. Li, S.; Blythe, P.; Zhang, Y.; Edwards, S.; Xing, J.; Guo, W.; Ji, Y.; Goodman, P.; Namdeo, A. Should older people be considered a homogeneous group when interacting with level 3 automated vehicles? Transp. Res. Part F Traffic Psychol. Behav. 2021, 78, 446–465. [Google Scholar] [CrossRef]
  21. Li, S.; Blythe, P.; Guo, W.; Namdeo, A.; Edwards, S.; Goodman, P.; Hill, G. Evaluation of the effects of age-friendly human-machine interfaces on the driver’s takeover performance in highly automated vehicles. Transp. Res. Part F Traffic Psychol. Behav. 2019, 67, 78–100. [Google Scholar] [CrossRef]
  22. Voinescu, A.; Morgan, P.L.; Alford, C.; Caleb-Solly, P. The utility of psychological measures in evaluating perceived usability of automated vehicle interfaces–A study with older adults. Transp. Res. Part F Traffic Psychol. Behav. 2020, 72, 244–263. [Google Scholar] [CrossRef]
  23. Robertson, R.D.; Woods-Fry, H.; Vanlaar, W.G.; Hing, M.M. Automated vehicles and older drivers in Canada. J. Saf. Res. 2019, 70, 193–199. [Google Scholar] [CrossRef]
  24. Young, K.L.; Koppel, S.; Charlton, J.L. Toward best practice in Human Machine Interface design for older drivers: A review of current design guidelines. Accid. Anal. Prev. 2017, 106, 460–467. [Google Scholar] [CrossRef]
  25. Li, S.; Blythe, P.; Guo, W.; Namdeo, A. Investigation of older drivers’ requirements of the human-machine interaction in highly automated vehicles. Transp. Res. Part F Traffic Psychol. Behav. 2019, 62, 546–563. [Google Scholar] [CrossRef]
  26. Li, S.; Zhang, Y.; Blythe, P.; Edwards, S.; Ji, Y. Remote driving as the Failsafe: Qualitative investigation of Users’ perceptions and requirements towards the 5G-enabled Level 4 automated vehicles. Transp. Res. Part F Traffic Psychol. Behav. 2024, 100, 211–230. [Google Scholar] [CrossRef]
  27. Li, S.; Zhang, Y.; Edwards, S.; Blythe, P.T. Exploration into the needs and requirements of the remote driver when teleoperating the 5G-enabled level 4 automated vehicle in the real world—A case study of 5G connected and automated logistics. Sensors 2023, 23, 820. [Google Scholar] [CrossRef]
  28. J3016_202104; Taxonomy and Definitions for Terms Related to Driving Automation Systems for On Road Motor Vehicles. Society of Automotive Engineers: Warrendale, PA, USA, 2016.
  29. Shladover, S.E. Connected and automated vehicle systems: Introduction and overview. J. Intell. Transp. Syst. 2018, 22, 190–200. [Google Scholar] [CrossRef]
  30. Soteropoulos, A.; Mitteregger, M.; Berger, M.; Zwirchmayr, J. Automated drivability: Toward an assessment of the spatial deployment of level 4 automated vehicles. Transp. Res. A Policy Pract. 2020, 136, 64–84. [Google Scholar] [CrossRef]
  31. Li, S.; Blythe, P.; Guo, W.; Namdeo, A. Investigation of older driver’s takeover performance in highly automated vehicles in adverse weather conditions. IET Intell. Transp. Syst. 2018, 12, 1157–1165. [Google Scholar] [CrossRef]
  32. Vom Dorff, S.; Böddeker, B.; Kneissl, M.; Fränzle, M. A fail-safe architecture for automated driving. In Proceedings of the Design, Automation & Test in Europe Conference & Exhibition, Grenoble, France, 9–13 March 2020; IEEE: New York, NY, USA, 2020; pp. 828–833. [Google Scholar]
  33. Kettwich, C.; Schrank, A.; Oehl, M. Teleoperation of highly automated vehicles in public transport: User-centered design of a human-machine interface for remote-operation and its expert usability evaluation. Multimodal Technol. Interact. 2021, 5, 26. [Google Scholar] [CrossRef]
  34. Goodall, N. Non-technological challenges for the remote operation of automated vehicles. Transp. Res. A Policy Pract. 2020, 142, 14–26. [Google Scholar] [CrossRef]
  35. Majstorović, D.; Hoffmann, S.; Pfab, F.; Schimpe, A.; Wolf, M.-M.; Diermeyer, F. Survey on Teleoperation Concepts for Automated Vehicles. In Proceedings of the IEEE International Conference on Systems, Man, and Cybernetics (SMC), Prague, Czech Republic, 9–12 October 2022; pp. 1290–1296. [Google Scholar]
  36. Bogdoll, D.; Orf, S.; Töttel, L.; Zöllner, J.M. Taxonomy and Survey on Remote Human Input Systems for Driving Automation Systems. In Proceedings of the Future of Information and Communication Conference, San Francisco, CA, USA, 3–4 March 2022; pp. 94–108. [Google Scholar]
  37. Zulqarnain, S.Q.; Lee, S. Selecting Remote Driving Locations for Latency Sensitive Reliable Tele-Operation. Appl. Sci. 2021, 11, 21. [Google Scholar] [CrossRef]
  38. Charness, G.; Gneezy, U.; Kuhn, M.A. Experimental methods: Between-subject and within-subject design. J. Econ. Behav. Organ. 2012, 81, 1–8. [Google Scholar] [CrossRef]
  39. Li, S.; Blythe, P.; Zhang, Y.; Edwards, S.; Guo, W.; Ji, Y.; Goodman, P.; Hill, G.; Namdeo, A. Analysing the effect of gender on the human–machine interaction in level 3 automated vehicles. Sci. Rep. 2022, 12, 11645. [Google Scholar] [CrossRef] [PubMed]
  40. Stevens, A.; Quimby, A.; Board, A.; Kersloot, T.; Burns, P. Design Guidelines for Safety of In-Vehicle Information Systems; Project report PA3721/01; TRL Limited: Berks, UK, 2002. [Google Scholar]
  41. Gold, C.; Happee, R.; Bengler, K. Modeling take-over performance in level 3 conditionally automated vehicles. Accid. Anal. Prev. 2018, 116, 3–13. [Google Scholar] [CrossRef]
  42. Zhang, B.; de Winter, J.; Varotto, S.; Happee, R.; Martens, M. Determinants of take-over time from automated driving: A meta-analysis of 129 studies. Transp. Res. Part F Traffic Psychol. Behav. 2019, 64, 285–307. [Google Scholar] [CrossRef]
  43. Zeeb, K.; Buchner, A.; Schrauf, M. Is take-over time all that matters? The impact of visual-cognitive load on driver take-over quality after conditionally automated driving. Accid. Anal. Prev. 2016, 92, 230–239. [Google Scholar] [CrossRef]
  44. Körber, M.; Gold, C.; Lechner, D.; Bengler, K. The influence of age on the take-over of vehicle control in highly automated driving. Transp. Res. Part F Traffic Psychol. Behav. 2016, 39, 19–32. [Google Scholar] [CrossRef]
  45. Niehorster, D.C.; Hessels, R.S.; Benjamins, J.S. GlassesViewer: Open-source software for viewing and analyzing data from the Tobii Pro Glasses 2 eye tracker. Behav. Res. Methods 2020, 52, 1244–1253. [Google Scholar] [CrossRef]
  46. Luan, X.; Zhao, B.; Liu, D.; Liu, X.; Tong, X.; Li, K. A lightweight heatmap-based eye tracking system. In Proceedings of the International Conference on Computer Communications and Networks (ICCCN), Athens, Greece, 19–22 July 2021; pp. 1–9. [Google Scholar]
  47. Menges, R.; Kramer, S.; Hill, S.; Nisslmueller, M.; Kumar, C.; Staab, S. A visualization tool for eye tracking data analysis in the web. In Proceedings of the ACM Symposium on Eye Tracking Research and Applications, Palm Beach Gardens, FL, USA, 6–8 November 2000; pp. 1–5. [Google Scholar]
  48. Skaramagkas, V.; Giannakakis, G.; Ktistakis, E.; Manousos, D.; Karatzanis, I.; Tachos, N.; Tripoliti, E.E.; Marias, K.; Fotiadis, D.I.; Tsiknakis, M. Review of eye tracking metrics involved in emotional and cognitive processes. IEEE Rev. Biomed. Eng. 2021, 16, 260–277. [Google Scholar] [CrossRef]
  49. Marquart, G.; Cabrall, C.; De Winter, J. Review of eye-related measures of drivers’ mental workload. Procedia Manuf. 2015, 3, 2854–2861. [Google Scholar] [CrossRef]
  50. Lohani, M.; Payne, B.R.; Strayer, D.L. Review of psychophysiological measures to assess cognitive states in real-world driving. Front. Hum. Neurosci. 2019, 13, 57. [Google Scholar] [CrossRef]
  51. Underwood, G. Visual attention and the transition from novice to advanced driver. Ergonomics 2007, 50, 1235–1249. [Google Scholar] [CrossRef]
  52. Lazaro, M.J.; Yun, M.H.; Kim, S. Stress-level and attentional functions of experienced and novice young adult drivers in intersection-related hazard situations. Int. J. Ind. Ergon. 2022, 90, 103315. [Google Scholar] [CrossRef]
  53. Radlmayr, J.; Gold, C.; Lorenz, L.; Farid, M.; Bengler, K. How traffic situations and non-driving related tasks affect the take-over quality in highly automated driving. In Proceedings of the Human Factors and Ergonomics Society Annual Meeting, Adelaide, Australia, 17–19 November 2014; Volume 58, pp. 2063–2067. [Google Scholar]
  54. Eriksson, A.; Stanton, N.A. Takeover time in highly automated vehicles: Noncritical transitions to and from manual control. Hum. Factors 2017, 59, 689–705. [Google Scholar] [CrossRef]
  55. Li, S.; Blythe, P.; Guo, W.; Namdeo, A. Investigating the effects of age and disengagement in driving on driver’s takeover control performance in highly automated vehicles. Transp. Plan. Technol. 2019, 42, 470–497. [Google Scholar] [CrossRef]
  56. Stanton, N.A.; Chambers, P.R.; Piggott, J. Situational awareness and safety. Saf. Sci. 2001, 39, 189–204. [Google Scholar] [CrossRef]
  57. Endsley, M.R. Measurement of situation awareness in dynamic systems. Hum. Factors 1995, 37, 65–84. [Google Scholar] [CrossRef]
  58. Gugerty, L.J. Situation awareness during driving: Explicit and implicit knowledge in dynamic spatial memory. J. Exp. Psychol. Appl. 1997, 3, 42. [Google Scholar] [CrossRef]
  59. Cunningham, M.L.; Regan, M.A. Driver distraction and inattention in the realm of automated driving. IET Intell. Transp. Syst. 2018, 12, 407–413. [Google Scholar] [CrossRef]
  60. Horberry, T.; Anderson, J.; Regan, M.A.; Triggs, T.J.; Brown, J. Driver distraction: The effects of concurrent in-vehicle tasks, road environment complexity and age on driving performance. Accid. Anal. Prev. 2006, 38, 185–191. [Google Scholar] [CrossRef]
  61. Bellet, T.; Cunneen, M.; Mullins, M.; Murphy, F.; Pütz, F.; Spickermann, F.; Braendle, C.; Baumann, M.F. From semi to fully autonomous vehicles: New emerging risks and ethico-legal challenges for human-machine interactions. Transp. Res. Part F Traffic Psychol. Behav. 2019, 63, 153–164. [Google Scholar] [CrossRef]
  62. Endsley, M.R. Toward a theory of situation awareness in dynamic systems. Hum. Factors 1995, 37, 32–64. [Google Scholar] [CrossRef]
  63. Strayer, D.L.; Watson, J.M.; Drews, F.A. Cognitive distraction while multitasking in the automobile. Psychol. Learn. Motiv. 2011, 54, 29–58. [Google Scholar] [CrossRef]
  64. Zhang, Y.; Li, S.; Blythe, P.; Edwards, S.; Guo, W.; Ji, Y.; Xing, J.; Goodman, P.; Hill, G. Attention Pedestrians Ahead: Evaluating User Acceptance and Perceptions of a Cooperative Intelligent Transportation System-Warning System for Pedestrians. Sustainability 2022, 14, 2787. [Google Scholar] [CrossRef]
  65. Wiseman, Y. Autonomous vehicles will spur moving budget from railroads to roads. Int. J. Intell. Unmanned Syst. 2024, 12, 19–31. [Google Scholar] [CrossRef]
  66. Nijboer, M.; Borst, J.P.; Van Rijn, H.; Taatgen, N.A. Driving and multitasking: The good, the bad, and the dangerous. Front. Psychol. 2016, 7, 1718. [Google Scholar] [CrossRef] [PubMed]
  67. Neyens, D.M.; Boyle, L.N. The effect of distractions on the crash types of teenage drivers. Accid. Anal. Prev. 2007, 39, 206–212. [Google Scholar] [CrossRef] [PubMed]
  68. Goh, K.N.; Chen, Y.; Lai, F.; Daud, S.; Sivaji, A.; Soo, S. A Comparison of Usability Testing Methods for an E-Commerce Website: A Case Study on a Malaysia Online Gift Shop. In Proceedings of the 2013 10th International Conference on Information Technology: New Generations, Las Vegas, NV, USA, 15–17 April 2013; pp. 143–150. [Google Scholar]
  69. Negi, S.; Mitra, R. Fixation duration and the learning process: An eye tracking study with subtitled videos. J. Eye Mov. Res. 2020, 13. [Google Scholar] [CrossRef]
  70. Naeeri, S.; Kang, Z.; Mandal, S.; Kim, K. Multimodal analysis of eye movements and fatigue in a simulated glass cockpit environment. Aerospace 2021, 8, 283. [Google Scholar] [CrossRef]
  71. Vreeswijk, J.; Habibovic, A.; Madland, O.; Hooft, F. Remote Support for Automated Vehicle Operations. In Road Vehicle Automation 9; Meyer, G., Beiker, S., Eds.; Springer International Publishing: New York, NY, USA, 2022; pp. 149–164. [Google Scholar]
  72. Zhang, T. Toward automated vehicle teleoperation: Vision, opportunities, and challenges. IEEE Internet Things J. 2020, 7, 11347–11354. [Google Scholar] [CrossRef]
  73. GOV.UK. Policy Paper UK Wireless Infrastructure Strategy, Department for Science, Innovation & Technology. 2023. Available online: https://www.gov.uk/government/publications/uk-wireless-infrastructure-strategy/uk-wireless-infrastructure-strategy (accessed on 1 October 2024).
  74. Naujoks, F.; Mai, C.; Neukum, A. The effect of urgency of take-over requests during highly automated driving under distraction conditions. Adv. Hum. Asp. Transp. 2014, 7, 431. [Google Scholar] [CrossRef]
  75. Liang, N.; Yang, J.; Yu, D.; Prakah-Asante, K.O.; Curry, R.; Blommer, M.; Swaminathan, R.; Pitts, B.J. Using eye-tracking to investigate the effects of pre-takeover visual engagement on situation awareness during automated driving. Accid. Anal. Prev. 2021, 157, 106143. [Google Scholar] [CrossRef]
Electronics 13 04366 g001
Figure 1. Level 4 automated vehicle (left) and the teleoperation system (right).
Figure 1. Level 4 automated vehicle (left) and the teleoperation system (right).
Electronics 13 04366 g001
Electronics 13 04366 g002
Figure 2. The trail route of the connected and automated logistics.
Figure 2. The trail route of the connected and automated logistics.
Electronics 13 04366 g002
Electronics 13 04366 g003
Figure 3. The remote driver is in the “monitoring” driving condition in the Level 4 automated vehicle teleoperation workstation.
Figure 3. The remote driver is in the “monitoring” driving condition in the Level 4 automated vehicle teleoperation workstation.
Electronics 13 04366 g003
Electronics 13 04366 g004
Figure 4. The remote driver is in the “disengaged” driving condition in the Level 4 automated vehicle teleoperation workstation.
Figure 4. The remote driver is in the “disengaged” driving condition in the Level 4 automated vehicle teleoperation workstation.
Electronics 13 04366 g004
Electronics 13 04366 g005
Figure 5. Illustration of motor readiness time.
Figure 5. Illustration of motor readiness time.
Electronics 13 04366 g005
Electronics 13 04366 g006
Figure 6. Illustration of decision-making time.
Figure 6. Illustration of decision-making time.
Electronics 13 04366 g006
Electronics 13 04366 g007
Figure 7. Fixation duration heat map when remote driver is in the “monitoring” condition (Gaze filter-Tobii I-VT, Radius 30 px, Scale max value 2.20 s, warmer colours-red and yellow indicates areas where the remote driver focused their gaze the most, cooler colours-green shows areas of less visual attention).
Figure 7. Fixation duration heat map when remote driver is in the “monitoring” condition (Gaze filter-Tobii I-VT, Radius 30 px, Scale max value 2.20 s, warmer colours-red and yellow indicates areas where the remote driver focused their gaze the most, cooler colours-green shows areas of less visual attention).
Electronics 13 04366 g007
Electronics 13 04366 g008
Figure 8. Fixation duration heat map when remote driver is in the “disengaged” condition (Gaze filter-Tobii I-VT, Radius 30 px, Scale max value 2.20 s, warmer colours-red and yellow indicates areas where the remote driver focused their gaze the most, cooler colours-green shows areas of less visual attention).
Figure 8. Fixation duration heat map when remote driver is in the “disengaged” condition (Gaze filter-Tobii I-VT, Radius 30 px, Scale max value 2.20 s, warmer colours-red and yellow indicates areas where the remote driver focused their gaze the most, cooler colours-green shows areas of less visual attention).
Electronics 13 04366 g008
Electronics 13 04366 g009
Figure 9. Fixation duration heat map when remote driver is in teleoperation mode (Gaze filter-Tobii I-VT, Radius 30 px, Scale max value 2.20 s, warmer colours-red and yellow indicates areas where the remote driver focused their gaze the most, cooler colours-green shows areas of less visual attention).
Figure 9. Fixation duration heat map when remote driver is in teleoperation mode (Gaze filter-Tobii I-VT, Radius 30 px, Scale max value 2.20 s, warmer colours-red and yellow indicates areas where the remote driver focused their gaze the most, cooler colours-green shows areas of less visual attention).
Electronics 13 04366 g009
Table 1. Dependent variables.
Table 1. Dependent variables.
MeasurementsData TypeUnit
Motor readiness timeContinuousSeconds
Decision-making timeContinuousSeconds
Vitalization of gaze behaviour NominalN/A
Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content.

Share and Cite

MDPI and ACS Style

Li, S.; Zhang, Y.; Edwards, S.; Blythe, P. Quantifying the Remote Driver’s Interaction with 5G-Enabled Level 4 Automated Vehicles: A Real-World Study. Electronics 2024, 13, 4366. https://doi.org/10.3390/electronics13224366

AMA Style

Li S, Zhang Y, Edwards S, Blythe P. Quantifying the Remote Driver’s Interaction with 5G-Enabled Level 4 Automated Vehicles: A Real-World Study. Electronics. 2024; 13(22):4366. https://doi.org/10.3390/electronics13224366

Chicago/Turabian Style

Li, Shuo, Yanghanzi Zhang, Simon Edwards, and Phil Blythe. 2024. "Quantifying the Remote Driver’s Interaction with 5G-Enabled Level 4 Automated Vehicles: A Real-World Study" Electronics 13, no. 22: 4366. https://doi.org/10.3390/electronics13224366

APA Style

Li, S., Zhang, Y., Edwards, S., & Blythe, P. (2024). Quantifying the Remote Driver’s Interaction with 5G-Enabled Level 4 Automated Vehicles: A Real-World Study. Electronics, 13(22), 4366. https://doi.org/10.3390/electronics13224366

Note that from the first issue of 2016, this journal uses article numbers instead of page numbers. See further details here.

Article Metrics

Back to TopTop