Clinical Research Staff Perceptions on a Natural Language Processing-driven Tool for Eligibility Prescreening: An Iterative Usability Assessment

2.1. Setting and sample

The evaluations were conducted remotely through Zoom. Twenty clinical research staff with experience in Alzheimer’s disease and other related dementias (ADRD) clinical research were recruited through CUIMC Irving Institute for Clinical and Translational Research Clinical Research Resource, research team’s professional network, and word of mouth. Evaluators had to be 18 years old, able to read and communicate in English, worked as clinical research staff (e.g., clinical research coordinator, research assistant, research nurse) in the last 12 months, and worked in ADRD clinical research for at least three months. Demographic variation sampling was used to reduce situational uniqueness and attain a comprehensive understanding of the user experiences.²¹

2.2. Features of Criteria2Query

C2Q is an NLP-driven tool that facilitates human-computer collaboration by translating free-text eligibility criteria to standards-based cohort definition queries and allowing user-defined modifications through an editable user interface (Fig. 1).¹⁰ It enables clinical research staff without database query experience or informatics training to transform eligibility criteria into clinical database queries for cohort identification and tailors the query to the eligibility criteria of their study.⁹ C2Q has the following properties: an editable user interface with functions to prioritize or simplify the eligibility criteria text to query for potentially eligible participants; accessible and portable cohort Structured Query Language (SQL) query formulation based on the Observational Medical Outcomes Partnership (OMOP) Common Data Model (CDM) version 5; and real-time cohort query execution with result visualization. For evaluation purposes, we connected C2Q to the publicly available Medicare Claims Synthetic Public Use Files SynPUF_1K containing 1,000 synthetic patient cases.²²

2.3.1. Cognitive walkthrough with a think-aloud protocol

We conducted a cognitive walkthrough to evaluate how users perform tasks with little to no formal instruction using C2Q.^23.24 The evaluators completed four steps:²³ 1) identifying an end goal; 2) inspecting available actions; 3) selecting one of the correct actions as the next step leading to the end goal, and; 4) evaluating system feedback on the progress toward accomplishing the end goal. A think-aloud protocol is a commonly employed method that asks evaluators to verbalize their thoughts as they navigate the system interface.²⁵ Analysis of the verbalization supports exploring how users perceive the system and why particular interface components are easy to use.²⁶

2.3.2. Post-Study System Usability Questionnaire (PSSUQ)

The PSSUQ assesses users’ perceived usability with a computer system in a scenario-based context.¹⁹ With a Cronbach’s alpha of .94, the 16-item survey has a three-factor structure: system usefulness, information quality, and interface quality.¹⁹ The items are rated on a seven-point Likert scale from “strongly agree” (1) to “strongly disagree”(7), with lower overall scores indicating better user satisfaction.²⁷

2.3.3. Independent usability severity rating

Two health informatics experts in the team (RS, CW) independently rated the severity of each usability violation to determine the priority of C2Q refinement for the next evaluation cycle. Problem severity was assigned based on three factors: frequency of the problem, the potential impact of the problem on the user, and persistence of the problem.²⁸ Using these factors, the problem severity was rated according to the Nielsen 5-scale rating²⁸ as follows: (0) not a problem at all, (1) cosmetic problem only, (2) minor usability problem, (3) major usability problem, or (4) usability catastrophe.

2.4. Data collection methods and procedures

The study involved one 1-1.5-hour online zoom meeting. Evaluators were granted remote access to navigate the study team’s computer screen with a beta version of C2Q. Using a purposely selected Alzheimer’s disease (AD) research protocol from ClinicalTrials.gov (NCT04619420; Fig. 2) to assess burden (estimating an average of one minute to modify a criterion⁴), the evaluators completed ten tasks step-by-step (Supplementary Table 1) without any specific guidance on how to complete the tasks. Evaluators were prompted to describe their thought processes as they attempted to complete a task. Occasionally, we provided prompts when the evaluators were unable to complete a task.

We monitored and logged the evaluators’ interface interaction and recorded the audio and on-screen activity. All audio recordings of the usability evaluation sessions were transcribed verbatim. In addition to PSSUQ, evaluators completed a demographic survey and a computer literacy evaluation (Table 9), a validated 22-item (Cronbach’s α = .92) with a Likert-type scale survey scored 1-7; a higher score indicates higher computer literacy.²⁹ Evaluators received a $50 digital Amazon gift code for participation. After five clinical research staff completed evaluation, a list of the unique usability problems was distributed for the independent usability severity rating. There were four iterations of C2Q system refinement (five evaluations for each iteration).

2.5. Data analysis and rigor

Sessions were analyzed using audio and video recordings, verbatim transcriptions, and field notes. Comments, silence, and repetitive actions were reviewed and evaluated to determine potential usability issues.³⁰ Comments were categorized as positive or negative characteristics and recommendations. The severity of the identified unique usability issue was determined by averaging the severity scores assigned to the problem. Usability issues that scored 3.5 (0-5, with 5 being more severe) and higher were prioritized and addressed during system refinement. The PSSUQ, demographic survey, and computer literacy evaluation results were analyzed to calculate the descriptive statistics. We explored the differences in usability based on sex, age, job experience, and position.

Further, a directed deductive content analysis³¹ of the transcriptions of the cognitive walkthrough was performed. Transcripts were reviewed for accuracy and uploaded to Dedoose software.³² A thematic codebook was created and drawn from the adapted Organizational Framework for Intuitive Human-computer Interaction constructs: seeking user goals, performing well-learned tasks, and determining what to do next.¹³ The framework conceptualizes the constructs as pie slices representing the user’s required cognitive activities in intuitive human-computer interactions (Fig. 3).

The framework constructs became the main themes into which meaning units would be coded (Table 1). The subthemes were further categorized as metacognition and user knowledge.

Two coders (BI, CD) independently coded the data. Memos related to codes and coded segments were documented to keep track of the developments in the data analysis. Discrepancies in the themes or coding were discussed among the team until a consensus was reached. To ensure that the analysis reveals the clinical research staff’s experiences in eligibility prescreening, member checking was done throughout data collection by asking participants if the findings from the participants of the previous evaluation cycle matched their experience.³³ Maximum variation through our sampling strategies allowed a vast range of views and perspectives to be considered, confirming the data’s conformability and credibility.³⁴ Data saturation was reached once no new themes emerged.

3.1. Evaluator Characteristics

Twenty evaluators completed the usability evaluation. The mean age was 33.45 years (range 24-46), 75% female, 35% White, 50% Hispanic, and 60% above undergraduate level. Most (45%) of the evaluators worked as clinical research coordinators, 85% worked as clinical research staff for two years or longer, and 75% worked in ADRD research for two years or longer. Majority (80%) of the evaluators manually review the EHR to identify potentially eligible participants. Half of the evaluators spend two hours or more a week, and most (65%) spend ten minutes or more prescreening one potentially eligible participant. Evaluators scored 6.36 on average on the computer literacy questionnaire (range 5.58-7), indicating high computer literacy. Table 2 summarizes the evaluators’ characteristics by cycle.

3.2. Usability evaluation

The system’s usability was rated poorly in the first cycle (3.18; SD: 1.24) but surpassed the benchmark (overall score: 2.82; system usefulness: 2.80, information quality: 3.02; interface quality: 2.49)³⁵ across all the subscales and overall usability in the second cycle (Table 3). Evaluators without database query experience rated the usability slightly better (n=10; mean: 2.45; SD: 0.81) compared to those who have experience (n = 10; mean: 2.50; SD: 1.16).

Though the system achieved the highest usability during the third cycle, it continued to demonstrate high usability during the final cycle (2.26, SD: 0.74). Supplementary Table 2 details the rating of the unique usability issues identified in each cycle. Forty-four usability issues were prioritized and addressed during system refinement (Supplementary Table 3). The number of usability issues decreased by 56% from Cycle 1 to Cycle 4.

3.3. Qualitative Analysis

With 1,802 codes, 14 subthemes were identified and classified into three themes: seeking user goals, performing well-learned tasks, and determining what to do next. The number of codes increased from Cycle 1 to 3 and decreased in Cycle 4 (Fig. 4). Performing well-learned tasks has the highest number of codes. Certain terms were removed from the query, such as “memory box score” (because this is not commonly documented in the EHR), “positive tau PET results” (because the result may not be available in the EHR and only a limited number of patients had the procedure), “progressive subjective decline” (because it is unclear how to characterize progressive), and “medications that affect the central nervous system” (because it is too broad). A detailed list of the proportion by cycle of the subthemes within the themes is detailed in Fig. 5.

Exemplars are provided in Table 4, and a comprehensive list of exemplars by theme is available in Supplementary Tables 4-6.

4.1. Importance of human-computer collaboration

C2Q was developed to leverage computer efficiency with human intelligence in transforming free-text clinical research eligibility criteria into executable cohort queries. It integrates the domain expertise of clinical research staff into the process of identifying entities from the eligibility criteria text with minimal or no database query experience. Vague, subjective, or complex eligibility criteria may need human determination, hence will not be included in the query but will be further assessed once the potentially eligible participant is referred.³⁹ Domain expertise is necessary to recognize and simplify the complexity of the eligibility criteria text, as previously reported.³⁹

Evaluators with more experience in AD modified the automated criteria parsing results, demonstrating that the breadth of content knowledge and background is necessary to maximize the system functionalities. Conversely, less-experienced evaluators typically included more criteria in the query and retained the automatic concept mapping. This may be due to a lack of knowledge of the nuances involved in eligibility prescreening and which criteria are feasible to query in the EHR.⁶ This is consistent with the previous usability evaluation of C2Q’s editable user interface with clinical research coordinators highlighting domain knowledge as a key differentiating factor that influences user experience.¹⁰ Regardless of the user’s modification approach, the system’s ability to allow changes iteratively based on the output supports cohort optimization until the desired cohort definition is attained.

4.2. Considerations for designing NLP-driven systems for eligibility prescreening

The recommendations addressed a broad range of issues, including system usability, interface design, and additional functionalities to improve system intuitiveness and the overall user experience. Understanding the end-user goal will help direct system refinements.⁴⁰ The recommendations in Table 5 can reduce the costs of implementing a system that does not meet user needs or standards.

The evaluators prefer a more inclusive system for eligibility prescreening. Due to the inherent complexity of eligibility criteria (e.g., underspecified requirements, temporality), a prescreening tool should provide flexibility in eligibility criterion text content modifications at any phase of the system use.⁵⁰ Concept mapping was found to be challenging. Multiple options of recommended concepts could be overwhelming and may prompt users to choose the first option or decide not to map any concepts, regardless of the level of experience.

The evaluators highlighted the importance of the accuracy of the query output. While tools like C2Q will not eliminate manual patient chart review, it will reduce the number of patient charts the clinical research staff needs to review and consequently focus their thorough chart review efforts on the most likely eligible candidates.⁵¹ The evaluators also pointed out that knowing the characteristics of the ineligible patients and what criteria precluded them from the study is helpful to “flag” the criteria that are not definitive of exclusion. The prioritization of potentially eligible participants in eligibility prescreening has increased screening efficiency.³

The evaluators underscored the importance of unstructured clinical notes in eligibility prescreening, especially the narrative results of imaging tests and clinician’s visit notes. With up to 80% of EHR data stored as free text (e.g., progress notes, discharge summaries),⁵² NLP is instrumental in advancing the current state of eligibility prescreening. Concerns about abbreviations and misspellings were also expressed in how the system identifies them and how their presence in the EHR might affect the query. This echoes the need to advance NLP methods to extract relevant information from the EHR and recognize named entities in free-text eligibility criteria, focusing on a domain-specific language to facilitate a more comprehensive electronic eligibility prescreening.^53,54 The evaluators also raised concerns surrounding how the system handles protected health information and upholds ethical considerations in terms of confidentiality and consent to be contacted. The initial contact with potential participants and the permission for this contact are critical components of study recruitment that are highly variable between institutions and have significant ethical implications.⁵⁵

4.3. Strengths and limitations of the present study

The iterative study design has limitations and is accompanied by an acceptance of the need for further system refinement. In real-world applications of electronic eligibility prescreening tools for clinical research, the quality and availability of clinical documentation is an important issue.^12,56 Though the evaluation involved a synthetic database that could limit the full use of C2Q, it provided insight into the domain experts' strategies in eligibility prescreening and how clinical documentation affects clinical research staff’s electronic eligibility prescreening. While the usability scores improved, it is important to consider that the system’s accuracy in identifying potentially eligible participants is crucial for the evaluators – which may affect the perceived usability⁵⁷ – but its investigation was out of the scope of this study.

The results may not be generalizable to all clinical research in other disease domains, despite the study sample's diversity in domain expertise. Future studies are warranted to further test the generalizability of these findings in diverse clinical domains. The evaluation was conducted in English and with evaluators who currently live in the US; hence, the results cannot be generalized to other languages or cultural settings. We acknowledge that interacting with C2Q in a remote virtual environment posed particular challenges in the system interaction due to potential differences in the operating systems used by the evaluator and the researchers and the reliability of the internet connection, which may have impacted the evaluator’s perception to the usability of the system.⁵⁸ Finally, because the data obtained in this mixed-methods study was cross-sectional in design, future longitudinal study is needed to assess the long-term impact of adopting C2Q on clinical research staff workload efficiency and recruitment outcomes.

4.4. Clinical and research implications

Exploring the perception of clinical research staff in other disease domains is needed to comprehensively understand the system’s usability depending on the use case.⁵⁸ More research is required to investigate the impact of the implementation of NLP-driven eligibility prescreening tools by the type of study (e.g., pharmaceutical trial, longitudinal study) and other disease domains (e.g., rare diseases, cancer). Further system development that involves human-in-the-loop machine learning models to improve named entity recognition, concept mapping, and criteria simplification could potentially address the concerns identified in this evaluation (e.g., the system-recommended concepts, inaccuracy in the automated parsing result). This underlines the importance of training clinical researchers to develop literacy and competency in this methodology. There is a need for more research on the types of training needed by clinical research staff and the efficacy of various training strategies to recruitment outcomes. Systems like C2Q provide an opportunity to identify and enroll participants of diverse backgrounds eligible for clinical research but not necessarily seen by specialists conducting trials.