Human Factors Within the Transportation Security Administration: Optimizing Performance Through Human Factors Assessments

1.1 Session Overview

Often, difficult human factors problems in operational environments are addressed by appealing to the peer-reviewed literature in human factors, cognitive psychology, industrial-organizational psychology, or other related fields. However, it is not always clear that the results from that open literature generalize to the particular operational environments to which they are applied. This session describes just such a series of situation; specifically the duties and stresses experienced by Transportation Security Officers (TSOs) and Behavior Detection Officers (BDOs) at the airport passenger security checkpoint. Over the last several years, the Human Factors team in the TSA’s Office of Security Capabilities has been supporting and funding research on human factors problems specific to TSOs and BDOs. The papers presented in this session describe experimental methods intended to honor the specific constraints experienced by Officers at the checkpoint from stimuli and methods of stimulus presentation to procedures. First, however, is a description of the TSA, the Human Factors Program and its goals, and some of the specific human factors problems encountered in the airport operational environment.

1.2 History and Mission of TSA

The Office of Homeland Security (OHS) was established under the White House in immediate response to the September 11, 2001 terrorist attacks. Two months later, with the passage of the Aviation and Transportation Security Act, the United States Congress created the Transportation Security Administration (TSA) under the Department of Transportation. In order to further align TSA with the nation’s security needs, Congress reassigned TSA to the Department of Homeland Security (DHS, formerly OHS) when DHS officially became a Cabinet-level department in March of 2003.

Since its inception, TSA has worked to protect the nation’s transportation systems and ensure freedom of movement for people and commerce. While TSA’s mission covers all modes of passenger and commercial transportation, including maritime, rail, intercity and commuter bus, and trucking, among others, TSA’s presence in aviation security draws disproportionate attention both from the general public and from foreign terrorist groups. The remainder of this paper will focus on TSA’s work in the aviation security domain.

1.3 TSA’s Approach to Aviation Transportation Security

To counter the diverse and sophisticated threats devised by hostile adversaries, TSA has adopted a layered approach to aviation security. The layered approach has the advantage that it is both redundant, providing multiple opportunities to identify the same, specific threat before it reaches an airplane, and comprehensive, allowing TSA to detect a wide variety of cues or abnormal behaviors that indicate an adversary’s intent to deceive or do harm.

Each layer of security is a system that requires collaboration among a number of elements, including some combination of technology or technologies (such as scanning or imaging devices, automated algorithms, software tools, local and networked databases, and other forms of technology), human beings with different roles and organizational affiliations (security personnel, passengers, analysts, and commercial airline crew members and representatives, among others), and/or trained animals. While a variety of technological innovations have made this layered approach possible, human actors ultimately are the components of each system that are responsible for acquiring, integrating, interpreting, and making decisions based on the information that the other elements of the system provide. Despite this critical role in determining how efficiently and effectively transportation security systems can operate, the human factor is the least well understood and has often been one of the last considerations in the push to develop new ways to address an evolving threat landscape.

1.4 Human Factors Engineering at TSA

TSA’s Office of Security Capabilities (OSC) has a team of engineering psychologists and operations analysts with subject-matter expertise in airport security. This team, also called the human factors team, is dedicated to determining how human performance is impacted by the design, implementation, and effectiveness of equipment, human interfaces, software and algorithms, training, policies, and procedures in transportation security systems. The human factors team makes recommendations to TSA stakeholders on how to optimize existing systems for human performance while also influencing human-centered requirements for new procedures and technologies prior to acquisition. To support these efforts, the human factors team works with national laboratories, private companies, universities and other DHS components such as DHS’s Science and Technology Directorate (DHS S&T) to conduct assessments that will provide valid and reliable data to assist in improving human performance. Although this paper cannot elaborate on many activities and programs due to security sensitivities, the sections below will provide a non-sensitive overview of OSC’s efforts to support TSA’s front line officers in a number of ways.

2.1 Overview of Human Factors Concerns at TSA

The availability and implementation of technology to screen passengers has increased dramatically since the inception of TSA. Rolling out new or modified versions of tools or procedures can substantially impact how efficiently TSOs are able to integrate the multiple sources of information required to screen passengers effectively and efficiently. Managing the varying levels of cognitive workload that have resulted from new tools is a high priority for the TSA. Sensory and perceptual load impact the TSOs’ decision-making, vigilance, and attention, all of which are of the utmost importance when working within an airport checkpoint environment. TSA understands the need to develop techniques to optimize attention allocation; logical reasoning; pattern recognition and classification; visual search and visual memory strategies; and problem solving for TSOs which should include physiological, behavioral, cognitive, and environmental assessments. TSA seeks to maximize human-in-the-loop performance by increasing throughput of screened items or persons; increasing screeners’ ability to accurately resolve alarms through improved higher hit rates and lowered false alarm rates; decreasing the time to make a determination; and decreasing the number of secondary searches of carry-on items.

The sections that follow will describe components of one of the most visible and most critical components of TSA’s layered security approach, the airport checkpoint, and some of the specific human factors challenges in maximizing the security and efficiency of checkpoint operations.

2.2 Checkpoint Technologies and Personnel

When progressing from the public side to the sterile side of an airport security checkpoint, the first TSO that many passengers encounter is the Travel Document Checker (TDC). The TDC verifies and cross-checks critical information in the traveler’s identification document (ID) and boarding pass. At most airports, electronic Boarding Pass Scanning Systems (BPSSs) assist the TDC in verifying the status of boarding passes. In addition, TSA is currently testing and rolling out Credential Authentication Technology (CAT) systems that will assist TDCs in authenticating IDs and eventually will allow the TDC to cross-check the passenger’s identity against a list of known travelers for that day at that airport.

Passengers then proceed to a lane where they must place carry-on items onto the conveyor to be scanned through an Advanced Technology (AT) X-ray system. In many cases, a divestiture officer will direct passengers to divest particular items into a bin to be scanned through the X-ray. Passengers then pass through either a walk-through metal detector (WTMD) or an Advanced Imaging Technology (AIT) body scanner designed to detect anomalies that might indicate a failure to divest all property necessary to enter the sterile area. If these systems provide certain types of anomalous readings, checkpoint officers will pat-down the passenger to determine whether the passenger failed to divest an item or whether the system produced a false alarm.

As the passenger is being scanned by either the WTMD or the AIT, a TSO operating the X-ray reviews the image of the passenger’s carry-on and divested items. On some X-ray units, automated detection algorithms alert the TSO to areas of concern within the image. X-ray operators can clear or confirm these system alarms, and they also can annotate an image to indicate their own regions of concern within the image. If after final review an X-ray operator clears an image, he or she sends the bag down the exit conveyor where it is picked up by the passenger. The passenger then redresses and proceeds to the gate. If the X-ray operator suspects the image, he or she pulls the suspected item for further inspection by a dynamic officer before the item reaches the passenger-accessible area. The dynamic officer then searches the suspected item with the help of Explosive Trace Detection (ETD) technology. For certain types of passengers and in certain situations, ETDs can also be applied to the hands of passengers as an additional security measure.

2.3 Specific Human Factors Concerns at the Checkpoint

The demands of the checkpoint work environment create a diverse set of human factors challenges. Some duties, such as X-ray image review and checking travel documents, are repetitive tasks in which TSOs must monitor and search cluttered images or documents for subtle, ultra-low frequency threat items. Such tasks can cause eye strain, fatigue, and susceptibility to rapid decrements in performance over time on task and to mind-wandering (i.e. task disengagement). To combat these deleterious effects, officers at the checkpoint rotate through duty positions (TDC, divestiture officer, AIT/WTMD operator, X-ray operator, and dynamic officer) at somewhat regular intervals.

While somewhat short, regular duty cycles can help TSOs to maintain vigilance and focus on the task at hand, they can come at a potential cost. X-ray image analysis is a particularly difficult task, and individuals’ ability to perform this task well can vary substantially, even among veteran, well-trained officers. On the other hand, duties such as the TDC and dynamic officer positions require TSOs to interact with passengers on a regular basis. Simplified duty cycles in which all TSOs at the checkpoint rotate through all positions guarantee that, at times, individual officers will not be optimally matched for the particular position they are performing based on their individual profile of knowledge, skills, and attributes.

Training potentially can help reduce the skill gap between the lowest performing and highest performing officers at a given duty position; however, staffing constraints often make it difficult for airports to provide TSOs with one-on-one training opportunities with experts. The most consistently available opportunities for training often are self-paced, computer-based training modules. Traditional computer-based training tools, however, have not always been designed to make a detailed assessment of individual officers’ overall skill level and specific areas of weakness. Accurate TSO performance assessment, as it relates specifically to performance on the floor during regular operations, is a prerequisite for determining where TSOs’ true training needs are in the first place but has proved to be quite challenging in practice. Establishing and obtaining good performance metrics both during training and during normal operations at the checkpoint is critical to narrowing the skill discrepancies across TSOs.

As TSA has been funding research in human factors for a number of years, the papers included are only a small subset of the research that has been conducted. However, this sample should provide a reasonable perspective on TSA’s attempt to incorporate data from well-designed studies into their decision-making about procedures, technologies, and training methods. (The reader is referred to two papers in another session of this conference, “Domain General and Domain Specific Expert Visual Search” for descriptions of another current TSA project [1, 2].

3.1 Exploring Cognitive Load and Cognitive Fatigue in Behavior Detection Officers

As TSA moves to Risk-Based Security (RBS), one focus of screening activity at the checkpoint is evaluating passenger behavior as well as screening their belongings for threat items. The officers who perform this duty are called Behavior Detection Officers, or BDOs. Initially modeled after the Israeli behavior detection program, and modified for appropriateness within the American culture and Constitution, the BDO program trains its officers to monitor, in real time, behaviors of passengers as they approach and move through the airport security checkpoint. Officers have a list of behaviors they are watching for, and have thresholds of behavior which, when crossed, require additional intervention by either the BDO or a Law Enforcement Officer. The difficulty in the job is that the list of behaviors can be long and watching for these critical behaviors in a crowd of people can be fatiguing, causing decrements in the BDO’s performance.

The goal of the project described by Kittinger and Bender [3] is to perform Cognitive Task Analyses (CTAs) on two versions of the BDO Standard Operating Procedure (SOP) to identify areas that cause difficulty for Officers and then to begin to identify areas for which that cognitive load and cognitive/physical fatigue can possibly be reduced. The project will start by augmenting existing Job Analyses and then to design and conduct Cognitive Task Analyses using BDOs from representative airports around the country. Results will provide additional information on ways to make BDOs more effective, and information on how to actually measure cognitive load and fatigue in the field.

3.2 Designing a Method for Quantifying Pat-Down Effectiveness

While TSOs are able to rely on assistance from a variety of technological tools for most current checkpoint security measures, passenger pat-downs are a decidedly ‘low-tech’ solution for resolving AIT or WTMD anomalies. TSOs performing pat-downs must completely rely on their hands, their eyes, and their judgment to decide whether passengers have fully divested all personal property. To date, the same has been true for training TSOs how to perform proper pat-downs – expert instructors provide qualitative feedback to TSOs during training regarding various aspects of how the TSOs performed the pat-down. These pat-downs often are performed by the TSOs either on the instructors themselves or on other TSOs during training. The instructors have no quantitative or objective measures or tools to refer to when providing feedback to the TSOs; the feedback is based solely on what the instructors see and feel. Furthermore, some passengers may require modified pat-downs based on physical disabilities or chronic pain – current training provides no clear criterion for instructors to determine whether TSOs have modified their pat-down appropriately for any given physical malady.

To address these limitations in current pat-down training methods, the Matteson and colleagues [4] at Johns Hopkins University Applied Physics Laboratory (JHUAPL) are developing the Pat-down Accuracy Training Tool (PATT) for TSA. PATT is a set of anatomically realistic mannequins fitted with pressure sensors that provide objective performance measures to trainees and instructors during pat-down training (see REF TO APL’s PAPER). PATT will provide a level of objective feedback that is not currently possible and has the potential to offer new-hire TSOs, as well as TSOs at airports who are approaching annual performance exams, an opportunity to practice pat-downs and receive expert feedback even when instructors or other TSOs might not be available.

3.3 Comparing Visual Search in TSOs and BDOs

Visual search is a key component of the jobs performed by both TSOs and Behavior Detection Officers (BDOs). While there is a growing body of knowledge surrounding the nature of TSO visual search at the X-ray, less is known about how similar that task is to the task of BDOs, who are searching a dynamic field (i.e., passengers moving through the checkpoint) for transient signals (behavioral indicators). Spain and colleagues [5] present a description of method for building on existing research covering similarity between TSO and novice performance in basic visual search tasks to compare BDO performance on these same tasks. The first experiment Spain and colleagues describe involves comparisons between TSOs and BDOs on a task in which subjects search for perfect Ts amidst a field of Ls and approximate Ts (where the vertical bar is offset from the middle). Prior research has illuminated differences between TSOs and novices on this task (accuracy, speed of search termination).

A second experiment focuses on identifying personality characteristics that are predictive of BDO visual search performance (e.g., Big 5 personality, Patriotism) that have been predictive of TSO performance in previous empirical work.

3.4 Adaptive Methods for Training Visual Search Skills

One of the key duties of a TSO is the X-ray image analysis. In order to optimize performance, Hale and colleagues [6] designed, developed and tested and adaptive training system for use both in the laboratory and in the field. This paper outlines the process utilized to design and develop a simulation-based training system to address identified gaps through the integration of process-level measures of individual performance to personalize training, with the goal of enhancing training effectiveness and efficiency. The “ScreenAdapt” prototype’s goal is to enhance TSOs ability to detect threats quickly and accurately using behavioral and eye tracking data that records TSO’s eye movements during training sessions. Eye tracking data provides insight into performance errors, identifying scan vs. recognition errors, as well as providing visual feedback of scan path and focus areas.

After action review feedback strategies summarize performance process and outcome measures relative to targeted training goal(s), and provide suggested next training steps such as focused training on specific deficits using exposure or discrimination training. Further, individualized image sets are generated based on identified performance deficiencies or inefficiencies, providing endless combinations of image components to avoid image repetition.

3.5 Identifying Cognitive and Psychological Traits Predictive of TSO Performance on Non-X-Ray Duties

TSA employs approximately 55,000 people as Transportation Security Officers (TSOs) to perform a variety of critical security duties as they screen almost 650 million air passengers each year. One of the most studied of these duties is the X-ray operator, for which the TSO has to scan X-ray images of all carry-on bags searching for threat items. However, the X-ray is not the only security duty TSOs perform at the checkpoint. TSO duties include the Travel Document Checker (TDC), the Divest Officer (DO, who prepares passengers and their bags for moving through the relevant imaging technologies), the Walk-through Metal Detector (WTMD), the Advanced Imaging Technology (AIT), and the Dynamic TSO (D-TSO, the person who performs pat-downs, bag checks, etc.). Despite these duties being a lion’s share of TSO time at the checkpoint, relatively little research has been done on characteristics that make individual officers successful at these various duties.

Emmanuel and colleagues [7] describe a project with the goal identifying the cognitive and personality characteristics that predict TSO effectiveness at the non-X-ray tasks at the checkpoint. Unlike the X-ray, these other duty stations have an added requirement for success – customer interaction. That is, when they are performing in these roles, TSOs must strike a balance between keeping passengers happy, calm, and compliant without compromising the overall security posture of the checkpoint.

By augmenting two existing job analyses through three TSO subject matter expert workshops, Emmanuel and colleagues identified a list of critical job duties and competencies that were rated as both frequent and critical for the successful performance of these non-X-ray duties. They then identified a group of 32 normed and validated personality and cognitive measures that correspond with these critical TSO characteristics/competencies including Five Factor inventories, measures of the dark side of personality (e.g., narcissism), patriotism, matrix reasoning problems (akin to Raven’s Progressive Matrices), and integrity. Their plan is to collect data from ~ 400 current TSOs and small sample of New Hire TSOs from eight airports on this battery of measures and will be using exploratory and confirmatory factor analysis to test the various hypotheses and to norm the measures for this specific population. The criterion variables to be used include a number of performance measures TSA currently collects on each TSO including annual recertification scores, practical skills evaluations, disciplinary actions, awards, absenteeism, etc.

Over the last several years,TSA has begun an empirically rigorous program for measuring TSO and BDO performance in the field. This effort has provided a respectable and growing body of data that is used by decision-makers to craft policy, training, and to guide the purchase and use of technologies at the checkpoint. Researchers include individuals in academia (Jeremy Wolfe, Steve Mitroff), private companies (Design Interactive, Leigh Fisher, Deloitte, RTI International) and national laboratories (Sandia National Laboratories), enabling TSA to investigate TSO and BDO performance from a variety of perspectives, using a variety of methods, at a variety of information sensitivity levels, yielding a more complete picture of the factors that influence Officer effectiveness, passenger experience, and security posture.