research-article

Open access

Facilitating Counselor Reflective Learning with a Real-time Annotation tool

Authors:

Laura DabbishAuthors Info & Claims

CHI '23: Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems

Article No.: 493, Pages 1 - 17

https://doi.org/10.1145/3544548.3581551

Published: 19 April 2023 Publication History

All formats PDF

Abstract

Experiential training, where mental health professionals practice their learned skills, remains the most costly component of therapeutic training. We introduce Pin-MI, a video-call-based tool that supports experiential learning of counseling skills used in motivational interviewing (MI)through interactive role-play as client and counselor. In Pin-MI, counselors annotate, or “pin” the important moments in their role-play sessions in real-time. The pins are then used post-session to facilitate a reflective learning process, in which both client and counselor can provide feedback about what went well or poorly during each pinned moment. We discuss the design of Pin-MI and a qualitative evaluation with a set of healthcare professionals learning MI. Our evaluation suggests that Pin-MI helped users develop empathy, be more aware of their skill usage, guaranteed immediate and targeted feedback, and helped users correct misconceptions about their performance. We discuss implications for the design of experiential training tools for learning counseling skills.

1 Introduction

Almost half of the U.S. population will experience a mental disorder in their lifetime [25]. Only about 34-45% of individuals, however, actually seek the help they need [25]. Commonly cited barriers to help-seeking include: living in a rural area, self and perceived stigmatization to seeking help for mental health problems, holding negative attitudes towards seeking help, or having negative past experiences with mental health professionals [25]. In the US, there is also a severe lack of mental health professionals such as counselors and psychotherapists: one in five counties in the US reports a shortage [75]. Training mental health professionals is a resource-intensive process, and mental health professionals face challenges in the process of learning counseling skills. The cost of experiential learning for professional counselors is the most expensive element of training programs. Experiential learning, in this case, refers to the actual practice of the learned therapy skills. Because it often requires one-on-one supervision and is therefore difficult to scale, it is the most expensive component of counselor training [33, 80].

To address this challenge, we present Pin-MI, a web-based application combining real-time annotation and peer feedback to support counselors’ reflective learning. Users engage in role-play sessions via a video call as client and counselor and take part in reflection and discussion. Users annotate, or “pin”, notable moments in their role-play sessions in real-time when counselors exhibit effective use of a counseling skill or there is opportunity for improvement. This design aimed to improve reflection-in-action by providing an easy-to-use tool that can be used while a skill is being practiced, as well as creating a reference point for later reflection. Following the role-play, these pins are reviewed individually and then jointly discussed to facilitate reflective learning. We describe and substantiate the design of Pin-MI and present the results of a qualitative evaluation with healthcare professionals to understand the benefits and opportunities Pin-MI provides users. In our evaluation, we found that role-playing helped users develop empathy, annotation helped users be more aware of their skill usage, the discussion component guaranteed targeted and immediate feedback, and peer feedback helped users correct misconceptions. At the same time, we also found that peer feedback could encourage or discourage users and that annotation had varying degrees of impact on users’ mental efforts. We further discuss these findings and potential design implications.

2 Background

2.1 Psychotherapy skills and training

Psychotherapy is a common method employed by psychologists and counselors to treat mental disorders. Despite the approaches taken by different schools of therapy, such as Cognitive Behavioral Therapy, the success of psychotherapy is frequently based on a variety of common factors. For instance, these factors include building therapeutic alliance, developing collaboration and consensus on goals, expressing empathy, displaying positive regard and affirmation [56].

It is estimated that the common factors account for about 30% of improvement in psychotherapy patients, and specific techniques of different schools for about 15% [56]. Motivational Interviewing (MI) is considered a general talk-therapy skill [37, 44, 66] and commonly used in conjunction with other clinical interventions [24, 65, 67]. In fact, the most common self-identified orientation in psychotherapy is eclectic or integrative psychotherapy [54]. Therapists who take the eclectic or integrative approach often learn about and integrate different concepts and skills from a variety of psychotherapy schools and use them together to treat their clients [54]. Outside of clinical therapists, therapy skills are also used by healthcare professionals such as nurses and doctors. In this context, the trained skills are used not only as a counseling technique but as a way to elicit medical history and encourage treatment compliance from their medical patients [73].

However, training psychotherapy is a challenging process. There is a clear lack of research on what is effective in psychotherapy training [10, 16]. However, past research has identified potential gaps as well as important factors to focus on when training psychotherapy. Experiential components that emphasize the practice of learned skills have been highlighted by various researchers. For instance, Boswell and Castonguay [10] suggests that training programs should include experiential components throughout the course, instead of simply focusing on didactic approaches, as techniques that appear to be quite simple in a textbook could turn out to be complex to use in a real clinical setting. Similarly, a 2015 meta-review of past psychotherapy training research suggests that instead of focusing on broad and abstract conceptual models in psychotherapy, training should focus on monitoring outcomes and feedback [16]. Despite the importance of experiential components, training of therapy skills overwhelmingly focuses on lectures, workshops, readings, and homework [45].

Even though clinical supervision is commonly implemented as an important experiential component of training, it is often a one-on-one procedure and difficult to scale [80]. It requires significant time and effort to train supervisors, observe clinician practice, gather feedback on performance, and conduct sessions. These costs are presumed to be one of the most expensive investments in supporting effective treatment implementation [33]. Such resource-intensity has brought about the need for developing more efficient, scalable, lower cost training methods [78]. Despite such need, psychotherapy training has been slow in adapting to the multitude of technologies that could potentially aid and benefit the training process. In the past, technology has often been extremely limited in psychotherapy education, mainly serving as a secondary component to traditional didactic components such as lectures [9]. More recently, researchers have started exploring other methods where technology could be integrated into the training process [5]. The author presented a variety of scenarios where technology such as internet and video recordings were used in conjunction with traditional training methods. However, these methods are still rather limited in their approaches, and the author also suggested that further research is needed in order to assess how effective they are at improving the training process.

2.2 Reflective Learning

In education and learning science, researchers have extensively explored the role of reflection for learning. Kolb [41] described learning as the process of transforming experience into knowledge: the student studies their concrete experience, abstracting and generalizing this experience, coming up with theories and testing these theories on new experiences. Experience forms the basis for the learning process. However, in order for learning to take place, reflection must happen. Boud et al. [11] defined reflection as the “activity in which people recapture their experience, think about it, mull it over and evaluate it (p.19)” and identified it as an important element in adult learning. The opportunities for learning through reflection come from one’s positive or negative perceptions towards their situation, namely being satisfied or disillusioned about the outcome of their experience and the subsequent observation of this experience [11].

The role of reflection is important in education because a deep level of reflection can create transformative effects on the learner by generating deeper understanding of why something happened the way it did, therefore leading to subsequent learning and behavior change [29, 68, 72]. Past literature on how reflections happen detailed a variety of conceptual models of the stages one goes through in order to properly reflect on experiences. For instance, Kolb [41] described 4 stages of the learning cycle, and Bain’s 5R framework suggested 5 levels of reflection [4]. More grounded in HCI work, Fleck and Fitzpatrick [29] proposed a five-stage model for reflection: revisiting the learned materials, revisiting with explanation, exploring relationships with learned knowledge, transformative reflection that leads to change of practice or further understanding, and critical reflection that expands beyond the current example. The authors highlighted the idea of transformative reflection, the stage that leads to significant learning and understanding. Even though reflection itself can be useful, simply triggering reflections does not always lead to the transformative effect and therefore fails to provide the users with deeper understanding or knowledge to improve their actions or behavior [48].

To better understand how to support this type of transformative reflection, Slovák et al. [72] adapted Schön’s theory of reflective practicum within the HCI context and provided a design framework to support transformative reflections with technology. The reflective practicum aims to create a safe environment where students can practice what they have learned free from actual consequences they might have faced if these actions were to happen in the real world [68]. Slovák et al. [72] proposed three important components for supporting transformative reflections: explicit, social, and personal. The explicit component refers to directly reconstructing an experience for learning, such as role-plays. For the social component, it is important to provide a supportive learning environment and resources such as peer and expert feedback. Lastly, the personal component relies on the learner’s personal ability and motivation to learn [72].

On the other hand, for learning in experiential context, past literature also suggested encouraging reflections-in-action to support experiential learning. Schon [68] described reflection-in-action as thinking about one’s action in the moment, not retrospectively, and planning for the next action accordingly [82]. An et al. [2] further interpreted this concept in the HCI context and proposed a series of design principles that can help support reflection-in-action: providing information that informs the user’s current action, providing users with the agency to interpret their performance, allowing the system to be used while performing an action, and encouraging simple actions that do not require a large amount of mental resources. Based on their design principles, the authors presented a case of using lights in the classroom to remind teachers in real time to reflect on their teaching practices [2].

These findings can be further extended into the field of therapist skill development. Bennett-Levy [7] proposed a model of counselor skill-development which emphasizes the importance of reflection, closely aligned with traditional learning science research [12]. Various follow-up studies demonstrate that reflection is an effective tool for skill development after initial training [23, 69].

However, effective reflection remains challenging for two reasons. First, it requires the novice counselor’s ability to guide themselves to pinpoint and work through their mistakes. This guidance is frequently provided by more experienced practitioners [19]. This is evident in traditional clinical counseling training where clinical supervision plays an important role in skill development [46].

At the same time, theories of reflection have emphasized the importance of the experiential nature of reflections [7, 11]. Properly capturing the experience for the learner in order to have an accurate record of their behaviors and the outcome is essential to reflection. However, conventional reflective practices generally rely on summary and recall-based activities such as journals, logs, and portfolios [6, 47, 51]. These summary-focused activities might not properly capture the important details and thoughts that might occur during the experience being reflected on. At the same time, recalling thoughts and behaviors post-hoc is often inaccurate [71]. [27] provided support for immediate reflections for clinical training: students preferred reflection immediately after a challenging session to analyze their behavior compared to an overall reflection after a few weeks.

2.3 Peer feedback

Giving and receiving feedback is considered an important aspect of learning for students. Although self-assessment allows the student to assess their level of understanding, and therefore identify their learning needs, feedback provides a means for obtaining external evaluation of their learning [34]. Good feedback not only stimulates students to reflect on their actions, but also provides directions for them to further grow. However, there is a high demand for feedback yet limited ways it can be provided. Prior research has explored filling the demand for feedback using peer feedback [70].

Peer feedback has been demonstrated to be an invaluable process for learning. The feedback and assessment provided by the peers can serve as a guide for the students to evaluate and reflect on their work [13]. Additionally, not only receiving peer feedback but also providing the feedback by reviewing and reflecting on other students’ work has considerable benefit for learning [26]. A study comparing the performance of undergraduate students who either provided or received written peer feedback in the context of a writing task found that both providing and receiving feedback led to similar improvements of writing performance [39].

Peer feedback also provides a reliable way for students to receive feedback that is similar in quality to what a teacher would provide. In a meta-analysis carried out by Falchikov and Goldfinch [28] across 48 studies, overall grades given out by students and teachers had a correlation of 0.69, highlighting that grades given out by students are relatively consistent with what the teacher would have given. Hovardas et al. [38] also found that in terms of written feedback, peers provided feedback that is similar in structure and content to expert feedback.

The effectiveness of peer review has been explored in the context of undergraduate nursing education. Carey et al. [15] found that “peer-assisted learning” stimulates student engagement in clinical practice and helps cultivate a collaborative environment. Similarly, Ravanipour et al. [62] demonstrated that nursing students found peer-assisted learning provided a much more in-depth learning experience with less stress compared to receiving feedback from a more authoritative figure like the teacher, and it helped them practice the skills they need before they go into formal employment.

However, providing quality feedback is important. Overall, past research has suggested a few rules of thumbs for providing good feedback: First of all, feedback receivers should be provided with the chance to discuss the feedback received with the provider. Second, feedback should be explained in detail. Simply pointing out something as good or bad is not enough to provide information for the student to benefit from the feedback. Third, feedback should contribute to the continued improvement and learning for the student. Lastly, feedback should be provided in a timely manner, immediately after the evaluated action if possible [76].

Fig. 1:

2.4 Annotation

Annotation has been a tool commonly used for learning and for different purposes. For instance, Novak et al. [57] discussed the case of using “social annotation”, which refers to using annotation as a collaborative tool in higher education, namely using sticky notes, highlights, comments linking to electronic resources to support easy online information sharing. The authors found that using annotation tools to supplement the learning process resulted in better reading comprehension, critical thinking, and meta-cognitive skills in students. Aside from using annotation as a collaborative tool, other research also investigated annotation in the context of reflection. For instance, Mirriahi et al. [49] used an annotation tool for undergraduate performing arts students to reflect on their recorded performances and found that the annotation tool improved engagement in the reflection process. The authors also found that familiarity with the annotation process improved the subsequent reflections and recommended annotation to be an integral part of reflective learning.

Moreover, using annotation tools to support reflections, Colasante [21] developed a system for students learning to teach physical education. The system allowed the students to upload recordings of their teaching sessions and mark specific points with different tags and comments for self-reflections as well as peer feedback. The authors found that students enjoyed the feedback received as well as the ability to review their own teaching, despite some concerns about the quality of the feedback. The benefit of using annotation tools for reflection comes about because annotation helps provide a connection between observed actions and the intents and reasoning behind these actions [63].

In the clinical context, annotation has been mostly used for diagnostic purposes. For instance, Orfanos et al. [58] used an annotation tool to analyze schizophrenic group therapies in order to identify interpersonal interactions that occur within the sessions. However, the potential of annotation tools has not been fully explored in clinician education and skill development. For instance, Cattaneo et al. [17] highlighted the lack of exploration of annotation tools for analyzing nursing practice. Hulsman and van der Vloodt [40] conducted a preliminary study using an annotation tool that allows medical students to upload their sessions and identify positive and negative events in their communications with clients for self-directed as well as peer feedback. However, there was no conclusive evidence on whether this tool supported the students in developing their clinical skills. Two similar tools have been recently designed and evaluated in the context of therapist training: mPATH and CORE-MI both presented tools where users can analyze session recordings and annotate events that occurred during the therapy sessions [43, 53].

Fig. 2:

The aforementioned annotation tools still rely on post-hoc recall and reinterpretation of past events. For instance, mPATH and CORE-MI rely mostly on post-hoc analysis of the events. Even though these tools provide invaluabe insights for the learner and supervisors, they employ rather reflection-on-action, instead of reflection-in-action, similar to traditional annotation tools. This reliance on recall might not lead to accurately capturing decision making in the moment. To address this issue, just-in-time captures of essential moments in a training session could potentially increase the accuracy and targeting of the reflective learning process. The use of just-in-time captures for caretakers of children with autism in Hayes et al. [36] demonstrates the potential for aiding post-hoc recall. In an evaluation, they showed that the tool aided caretaker recall and behavioral analysis.

In addition, to provide support for reflection-in-action, it is important to make effective usage of the opportunities when the learner actively engages in the practice of a new skill. Arakawa and Yakura [3] presented a tool, INWARD, which used a machine learning algorithm to facilitate reflection. The tool identified significant moments that happened during a recording by detecting cues such as gaze and gestures. Even though this tool provided easy opportunities to automatically capture important moments for later reflection, the users do not actively engage in the process of reflecting on their behavior during the capture, but rather rely on the algorithm’s identification. In comparison, more recently, Rasmussen et al. [61] proposed a new method of assisting reflection by incorporating real-time annotation and tagging in order to capture and record spontaneous moments in design sessions. The authors found their system was able to capture important design sparks that happened in these sessions. Even though these two implementations both highlighted the utility of just-in-time captures, they only focus on recording spontaneous moments and do not support the reflective learning process.

3 Design

Pin-MI is a web-based application that allows two participants to role-play as therapist and client in a mock therapy session through a video call. Participants can use the tool to mark important moments in the conversation and reflect on these moments individually as well as discuss these moments later with their peer. In summary, there are three important elements to a Pin-MI session: 1) role-playing with real-time annotation, 2) reflection, and 3) peer-feedback.

These three elements are closely aligned with the three elements of the reflective practicum outlined by Slovák et al. [72], namely explicit, personal, and social. We construct the explicit component from Slovak’s model using the interactive role-play, providing an environment where skills can be safely practiced. The assisted self-reflection provides scaffolding for the personal component by allowing the user to further assess their own abilities for learning. The users are also provided with support and learning resources by being able to connect with their peers and get feedback and suggestions from them, aligning with the social component. Furthermore, to facilitate reflection-in-action, we complement the role-playing process with easy-to-use real-time annotation.

Fig. 3:

3.1 Session flow overview

The typical process that a user goes through during a Pin-MI session is as follows: the user starts with a ten-minute role-play session with a peer through the video call function that is embedded in the Pin-MI system. During this role-play video call, both the user and their peer would have a button on the bottom left corner of their video call interface (see figure 1). Both users can click on the button to mark significant moments in the conversation that they might want to reflect on later. After this process, both users have ten minutes to individually write their reflections for each pin. There are short prompts that users fill out throughout this process. Lastly, the pair of users will be reconnected with each other on video, discuss and compare their individual reflections for each pin, and collaboratively analyze the pins and share their opinions with each other.

3.2 Real-time annotation

The first design element is intended to provide therapists with a low-effort tool to quickly annotate their sessions while they are happening. For the pinning stage, both parties engage in a therapist-client role-play session and have a clickable button (a pin) that can be used to mark a moment in the conversation where they feel the person in the therapist role used an MI skill well, used on poorly, or could have used a skill (see figure 1). We added the ability for the mock client to pin moments during the role-play and pinning stage as a result of feedback during user testing. This ultimately allows users to review moments not only pinned by themselves, but by their peer. The moments that are pinned are recorded in the system with their timestamp. After role playing and pinning for ten minutes, the role-play session ends and the pair enters the next stages where they can use the pinned moments to facilitate reflection and discussion.

Chen et al. [18] investigated challenges that novices faced when learning therapeutic skills and suggested two windows for intervention for supporting MI learning. The first window is intervention in a session during which the skills are being practiced. The authors found that therapists tend to be able to identify their challenges but not necessarily understand the best way to solve them [18]. This aligns with the design principle suggested by An et al. [2] outlined in the previous section: the authors suggested designing easy-to-use tools that can be used while the user practices their skills. Therefore, annotation, or "pinning", provides a quick and easy way for users to reliably identify significant moments in their conversation.

The goal of the pinning process is to help users take quick notes in real time while reducing cognitive burden. Real-time note-taking requires a significant amount of cognitive effort [60]. The urgency of identifying important information in real time and noting it down taxes working memory [60], particularly when the annotator is simultaneously engaged cognitively complex tasks like planning and executing a conversation and trying to learn and practice newly acquired skills. Therefore, we designed the pins to replace note-taking in order to reduce the mental load yet still give practitioners an efficient way to later reflect on their experience. In contrast to taking notes in real time, the pinning process only requires participants to identify the significance of a specific moment, which they can later navigate to in order to review and reflect.

In user testing, we found that people sometimes forgot to pin during the session or realized there was a moment they wished they could have pinned to discuss with their peer. Even with the reduced demands of pinning, people still had a lot on their mind during the mock session, such as steering the conversation, phrasing an open-ended questions, or cultivating change talk. We added the ability for users to add pins during the self-reflection phase immediately prior to the discussion so they could remain primarily focused on the conversation during the role-play.

3.3 Assisted self-reflection

The main goal of this stage is to provide assisted self-reflections for users to properly look back at, analyze, and learn from their experience. During this stage, users can hear the recording of their conversation and receive a transcript of their conversation in order to aid the review and reflections. They users are able to see the pins that they and their partner made during the session in red (figure 2). For each pin, we provide a series of questions to assist the user in reflecting on the pins: 1) why they pinned particular moments, and 2) what MI skills were used (figure 3). The user has ten minutes to finish reflecting on the pins individually and then enter the last discussion stage.

During assisted self-reflection, we also include the pins that a user’s partner made in order to encourage them to preemptively conjecture what their partner might have been thinking during the moments pinned and to facilitate the discussion of these pins.

Fig. 4:

In order to support effective peer feedback in the discussion stage, we implemented several strategies in the assisted self-reflection stage suggested by Cook et al. [22]. These strategies include allowing time for self-reflection before peer reflections as well as providing guided activities for feedback prioritization. As previously discussed, in Pin-MI, we implemented the features to follow these strategies by first asking users to go through the self-reflection and using questions to assist and guide their reflections. For feedback prioritization, we asked participants to categorize the pin as either a strength or opportunity to highlight to the users important areas to improve. At the same time, we encouraged users to discuss moments where both users made a pin in close proximity in time to one other, as this signals a critical moment where something is noticed by both users.

3.4 Peer feedback

After finishing the self-reflection, both users are directed to the discussion stage to give and receive peer feedback. In the discussion stage, the users are reconnected with each other for ten minutes. We designed this stage to follow the conversation in a timely manner to ensure the participants still have a fresh memory of the mock session and are able to receive immediate feedback. Past research has pointed out that the immediacy of feedback is essential for improving performance of learners [42, 76].

During the discussion stage, the users are still able to view all the pins that were made during the session and re-listen to the recording if needed. However, compared to the self-reflection stage, they also get to see the notes that their partner wrote for each pin (figure 4). The discussion is also structured with the same questions that guide them through their self-reflection of each pin: what happened, what skills were used, and what could be improved upon. Both users can share their perspective for each of the pins during the session and ask for clarifications and feedback. As pointed out by prior literature, detailed explanation of feedback is important [76]. We believe use of the reflection questions should enrich the content and details of the feedback.

The pins themselves also serve as a guide so that the discussion is focused on important moments and specific instances instead of broad generalities. The additional perspective of the client peer can be useful. Users reported during user testing they would often employ an MI skill without necessarily realizing it. Through the feedback process, participants develop explanations for their peers, and through the creation of these explanations,their own understanding of MI can be enhanced, deepening the connection between didactic learning about MI and their own experience applying it [81]. Lastly, the discussion process allows both parties to clarify any questions they might have regarding the pins and thoroughly discuss the feedback they received, consistent with recommendations from the literature [76].

Lastly, the consideration of the two-sided usage of the pins during the session as well as discussion were aimed to also provide valuable experience for not only the participant role-playing as the therapist, but also the participant role-playing as the client. For the client, pinning critical moments and providing feedback on the therapist’s performance can support learning through the observation of others’ work [79]. By observing a therapist’s behavior as the client, peers can gain insight into strengths or opportunities they would not otherwise reflect on in their own practice.

4 Implementation Notes

The Pin-MI web application is implemented in HTML, JavaScript, and CSS and uses the React JavaScript library¹ for building UI components. In particular, it uses the Vonage Video API² to set up and record end-to-end encrypted video conferencing sessions between participants and the Speech-to-Text API from Symbl.ai ³ to automatically transcribe participants’ conversations in real time. The production version of the web application is hosted on Netlify⁴, which works on all major browsers, including Google Chrome, Microsoft Edge, Mozilla Firefox, Apple Safari, etc.

For the back-end, Pin-MI uses Google Firebase⁵ for utility functions, real-time database, and user authentication. In addition, it leverages the Amazon Simple Storage Service (Amazon S3)⁶ for securely archiving and retrieving recorded role-play and discussion sessions.

Table 1:

Participant ID	Occupation title	MI experience
1T	Behavioral Health Specialist	Proficient
1C	Behavioral Health Specialist	Proficient
2T	Learning Architect	Novice
2C	Behavioral Health Specialist	Proficient
3T*	Behavioral Health Specialist	Novice
4T	Social Worker	Expert
4C	Behavior Health Coach	Novice
5T	Transplant Case Manager	Novice
5C	Social Worker	Novice
6T	Complex Case Manager	Novice
6C	Behavioral Health Specialist	Expert
7T	Case Manager	Expert
7C	Social Worker	Novice
8T	Social Worker	Novice
8C	Social Worker	Novice
9T*	Social Worker	Novice
10T*	Behavioral Health Specialist	Expert
11T	Transitional Care Coordinator	Novice
11C	Outpatient Nurse Coordinator	Novice
12C*	Social Worker	Expert
13T*	Health Coach	Novice
14T	Health Coach	Proficient
14C	Health Coach	Expert
15T	Health Coach	Proficient
15C	Psychologist	Expert

Notes: For MI experience, we considered novice to have 1 year of experience with MI or less, proficient to have 1-3 years of experience, and expert to have 3+ years of experience. Participants marked with * had a researcher as their partner.

Table 1: Participant information, including occupation and experience with Motivational Interviewing

5 Evaluation

5.1 Procedure

In order to understand how the key design elements of Pin-MI (pinning, reflection, and peer feedback) affected the user’s learning experience, we conducted a qualitative study with 15 pairs of users. In 5 sessions, only one participant was available. In these cases, one experimenter role-played either the client or therapist by following a general script. In total, there were 10 pure dyads, and 5 individual participants who were paired with a researcher, for a total of 25 naive participants. Participants were recruited from two healthcare providers through word of mouth. All participants received some level of basic MI training through MI training courses provided by their organization and were encouraged to use the MI approach in dealing with their patients. Participants’ experience with MI is listed in Table 1 below. Before each session, participants provided consent and filled out a questionnaire on their MI experience. In each session, the pair engaged in a role-play session. Participants were randomly assigned to take either the therapist or client role. The client was asked to talk about a change they wanted to make using inspiration from a real-life experience (e.g. exercising more, eating healthier, etc). The therapist was simply told to help the client who was having a difficult time in life. The participants went through all stages of the Pin-MI platform: a 10-minute role-play, a 10-minute self-reflection, and a 10-minute discussion. After participants finished using Pin-MI, each participant was interviewed by a researcher on their experience. We used semi-structured interviews to understand their experiences during the session. We conducted interviews immediately after the Pin-MI session, structured as a post-interaction think-aloud protocol. Participants were asked to reflect on specific memorable experiences in the Pin-MI session.

Table 2:

Pinned moment in role-play	Self-reflection	Discussion transcript
7T: So your motivation is your son and his level of energy. And tell me a little bit about what might be some of the smaller steps. You said you’re drinking the water, which is great. And that could be one thing since you’re already successful in that way that we could build upon. And one of the questions that I have for you is, are you able to see yourself as that person who is keeping up with your son and able to run around with him? 7C: Absolutely. There’s no reason why I can’t do that.	7T notes: Look over the fence, what does it look like to be successful. 7C notes: She gave me small goals to start with instead of one large goal, it makes it feel more manageable. Giving ideas of what a ’fun mom’ could look like and attainable goals that don’t seem so overwhelming.	7T: I was trying to do the look over the fence. What, what does it look like to be successful to give you an idea to, for your goal, to be more personable, to have a better idea as to... ’cause if we have a better idea as to actually why we’re doing something, what we want to accomplish, it becomes a little bit more personable and meaningful. 7C: And doing small goals as opposed to one large goal, it makes it seem more manageable and helps you kind of reflect on your life.

Table 2: Data from Pin-MI session 7, pinned moment at 6:17

5.2 Analysis

5.2.1 Pinned moments.

Each pair of participants generated on average 10 pins per session across the 15 sessions. Pins varied in the amount of detail provided in the self-reflection or discussion session so we could not include all pins in our analysis. Some pins had no self-reflection content or discussions associated with them, making it difficult for us to understand why a participant might have pinned that specific moment. In addition, some sessions included a researcher role-playing as either the client or therapist. Therefore, for our analysis, we only analyzed pins that were actively discussed by participants in peer feedback sessions where no researcher role-played either the therapist or the client. This resulted in 50 pins.

We took a content analysis approach to analyzing the self-reflection notes and transcript of the participants discussing the pinned moments. Two researchers first generated a set of codes based on the pins. The two researchers then independently coded 10 pins to establish inter-coder reliability. After reaching 70% agreement, the two researchers continued to code the entire dataset.

5.2.2 Interviews.

We conducted semi-structured interviews with the participants after the sessions concluded. All interviews happened online over a Zoom call. The interviews were recorded and transcribed using an automatic transcription service. A researcher then double checked the transcription against the original recording to ensure the accuracy of the transcriptions. In the interviews, we asked participants to recall the most notable pinned moment they remembered and explain the thought process behind each pin. With this approach, we were able to identify specific pinned moments that were mentioned during the interview. We took a thematic analysis approach to the interviews. Two researchers first independently open-coded half of the interviews. Then the researchers presented the set of open codes to the research group. The research group discussed and finalized the list of codes. The researchers then coded the rest of the interviews using the list of codes. The group then used affinity diagramming to, discuss and group different quotes together in order to generate higher-level insights. We edited the interview quotes for readability by taking out filler words and filling in missing context. Finally, as described previously, we identified pinned moments that were connected with each specific interview in order to gain insight and context about the participants’ feedback. In presenting results we combine data from the pinned moments and the interviews.

We excluded the participants who role-played with a researcher from the analysis of the pins because in these sessions, pins made by the researcher are not valid data from the users. However, in our analysis of the interviews, we chose to include these participants’ experience. We believe the overall Pin-MI user experience can still be fully experienced by the participant despite having joined the session with a researcher. The researcher followed a general prompt and did not provide any additional information that might differ from what a typical MI practitioner would provide. We believe the experience of these users is still valid, as the session would be comparable to a session with an MI expert.

Table 3:

Pinned moment in role-play transcript	Discussion transcript
6T: Hi, how are you? And what brings you in today? 6C: Hi, I am doing Okay. I was really hoping for some support. I am having a difficult time with my job and it’s affecting my relationship, and I was hoping to get some feedback and maybe some strategies to help manage all the stress that it’s causing me. 6T: Yes, yes. Absolutely. So what I hear is that you’re under a lot of stress and it’s affecting your everyday life between your work and your relationships, and you’re looking for some support and some coping strategies. Is that correct? Okay, and can you tell me a little bit about when this began?	6T: I think we (the therapist and the client) agreed with open-ended questions, reflective listening. I think that is what worked well...Oh yes, I have it as a great open-ended question...So I was just trying to achieve establishing that rapport, like the first contact with getting the client to open up.

Pinned moment in role-play transcript

Discussion transcript

6T: Hi, how are you? And what brings you in today? 6C: Hi, I am doing Okay. I was really hoping for some support. I am having a difficult time with my job and it’s affecting my relationship, and I was hoping to get some feedback and maybe some strategies to help manage all the stress that it’s causing me. 6T: Yes, yes. Absolutely. So what I hear is that you’re under a lot of stress and it’s affecting your everyday life between your work and your relationships, and you’re looking for some support and some coping strategies. Is that correct? Okay, and can you tell me a little bit about when this began?

6T: I think we (the therapist and the client) agreed with open-ended questions, reflective listening. I think that is what worked well...Oh yes, I have it as a great open-ended question...So I was just trying to achieve establishing that rapport, like the first contact with getting the client to open up.

Table 3: Data from Pin-MI session 6, pinned moment at 1:04

6 Results

6.1 Role-playing helps users develop empathy

Regardless of the role they played, participants felt communicating with each other about their experience allowed them to develop more empathy for their clients, and feel a deeper connection with them as humans. Practitioners described typically having very little chance to listen or understand what the client is thinking in their regular day-to-day clinical practice. In Pin-MI, by combining the discussion with the role-play, participants were able to not only put themselves into the role of the client, but also explain how it felt to be in the client’s shoes. This process allowed them to learn to be more empathetic to their clients in real life.

In clinical psychology, empathy is one of the core skills in counseling and the development of an effective patient-therapist relationship [52]. Being able to relate to the client and have authentic interaction is essential to developing an empathetic relationship with each other. Using Pin-MI gave participants the chance to express their feelings and reasoning behind their statements. In table 2, we present data in session 7, including the role-play transcript, self-reflections, and the discussion transcript to further demonstrate how Pin-MI encouraged empathetic responses from the participants.

In this excerpt, the therapist was able to accurately pinpoint what element was most important to the client’s behavior change and make use of the client’s positive feelings and emotions. The client in the session mentioned a few times that their son was a big motivating factor in considering a healthier lifestyle. In the therapist’s discussion with the client, the therapist’s response highlighted their attempt to consider the client’s perspective, sensing what might feel like a success and what might be motivating the client to feel this sense of success.

This behavior illustrates the concept of associative reasoning in an empathetic response: to make an empathetic response, one should be able to recognize not only what the client says but also the important emotional connection identified through the client’s words [35]. In this case, the therapist recognized the emotional impact the client’s child had and made use of this element in their response, therefore encouraging the client. This attempt was appreciated by the client. In the self-reflection presented in column two, the client talked about the impact of the therapist’s framing of the behavior change around their son and expressed that this positive framing reduced the anxiety associated with the potential behavior change. The client in their own reflection highlighted how the idea of a "fun mom" is motivating, corroborating the therapist’s attempt at guiding the conversation using what they felt might be the most important emotional connection in the client’s life.

For the client, taking this alternative perspective, they get to consider what an empathetic response might have felt like compared to one that might be a procedural canned response. Moreover, the therapist in this case also explained their thought process and how they understood the emotional impact their statement might have on the client, further providing information and potential learning opportunities for the client. 7C later in the interview commented on this experience and how it helped them develop empathy:

“It just puts you in the position of our clients and trying to get to know them and not sound like a robot and building on the information that they give you and, just trying to relate to them as a person, not, not part of your job.” -7C

6.2 Pinning helps users be more aware of their skill usage

Overall, participants reported that Pin-MI helped them be more aware of when and how they used MI skills. Participants were able to recognize the moments they used an MI skill and understand their strengths and weaknesses through self-reflection. This awareness and active thinking about their skill usage sets a foundational stage for reflective learning, according to Donald Schön’s concept of “reflection-in-action” [68, 82]. Schön described reflection-in-action as thinking about one’s action in the moment and planning for the next action accordingly. In this example below, 6T was able to recognize their skill usage, having awareness of their actions, and think about their next actions accordingly in those moments:

“So I was trying to just pin where I heard motivational interviewing skills as well as where I could have maybe built on those skills as well, where I might have had an opportunity to use different ones... So I think it just made me aware of using the MI techniques and the different communication styles and because you were pinning and in that certain spot where you felt like you were using like the skills or where you thought that you could improve on.” -6T

The therapist talked about the following pin that we present in table 3 as an example of being aware of their skill usage. In this example, the client gave a brief summary of what they were going through. The therapist was able to take in this information, reflect on the client’s statement, and ask another open-ended question to understand the client’s situation further. In this example, the therapist was aware of the skills being used and was able to understand and recognize their reasoning behind the actions. 6T described later in the interview:

“So the first pin that I did, I pinned because I had asked the client what had brought them in and used open-ended questions, and also reflective listening and the summarizing as well. I was trying to say, well, I heard you, I heard that this is what you’re saying. And just to try to repeat that back to them.” -6T

This illustrates the important thinking process that happened during the skill usage, prompted by the pinning action. The client made the pin in the moment when they realized what was happening in their mind. This recognition further prompted them to plan their action accordingly. This action presented more opportunities during the later stages for a deeper analysis and reflection.

Table 4:

Pinned moment in role-play transcript	Discussion transcript
15C: For me, the gum seems to work best [for quitting smoking], and I have had better luck with the gum than with the patch. So that definitely helps. I know that helps so like I think that’s something that I am probably going to do again, or think I would. And then the time I got the longest was probably three years ago when I told a bunch of people, and so something tells me that maybe that should be a part of it too. 15T: So what you have learned so about yourself is that the nicotine gum is more effective than the patch. And you have also learned that sharing this intention to quit with other individuals has helped you. Talk a little more about what role are these folks playing that is so helpful to you.	15C: So it was a reflection followed by an open-ended evocative question. And that is always like an excellent way to pivot is to capture what was just said in a reflection and then ask for additional detail. And I think that was the one, if I’m not mistaken where I was talking about involving other people, and that’s an excellent thing to ask more about. And so I thought that the question, the open-ended question you used was perfect. I thought that was great.

Pinned moment in role-play transcript

Discussion transcript

15C: For me, the gum seems to work best [for quitting smoking], and I have had better luck with the gum than with the patch. So that definitely helps. I know that helps so like I think that’s something that I am probably going to do again, or think I would. And then the time I got the longest was probably three years ago when I told a bunch of people, and so something tells me that maybe that should be a part of it too. 15T: So what you have learned so about yourself is that the nicotine gum is more effective than the patch. And you have also learned that sharing this intention to quit with other individuals has helped you. Talk a little more about what role are these folks playing that is so helpful to you.

15C: So it was a reflection followed by an open-ended evocative question. And that is always like an excellent way to pivot is to capture what was just said in a reflection and then ask for additional detail. And I think that was the one, if I’m not mistaken where I was talking about involving other people, and that’s an excellent thing to ask more about. And so I thought that the question, the open-ended question you used was perfect. I thought that was great.

Table 4: Data from Pin-MI session 15, pinned moment at 2:51

6.3 Pinning guarantees that feedback is both targeted and immediate

Participants reported that they really valued how they can target specific moments in the conversation and focus on the important moments in reflection and discussion. This is evident from session 15. For this pair, the client was an MI trainer who had extensive experience training MI novices. 15C emphasized the strength of Pin-MI, especially in trying to provide guidance for users who are trying to learn MI skills. Comparing it to their own experience in the past, they explained how Pin-MI allowed them to provide targeted feedback in a much more immediate manner after the session.:

“If I’m working with a psychology intern, and she just recorded a session. You will then listen to the entire session after it’s done. And she may be doing kind of informal pins, like while she’s taking notes while listening to that session. But from my perspective, I’m not listening to [everything], and we’re only really going back in discussing things that she has identified as issues from the session. I think what this [Pin-MI] offers is just, it’s more real time, it’s happening in the moment. And it gives me the opportunity in a role-play or a real-play to give feedback.” -15C

The excerpt presented in table 4 illustrates the value that pair 15 found in Pin-MI. In this excerpt, the client talked about wanting to quit smoking and identified several past strategies that they have employed that worked for them. The pin was marked as a strength because the therapist demonstrated skill in using reflections and then exploring the role of social support in more detail.

By pinning this moment, the pair was able to identify an important learning moment, capturing usage of an important MI skill, and elaborate on why that statement was impactful. In this case, the takeaway from the therapist’s action was the successful usage of reflection skill combined with the question, which lead to the client’s further elaboration. The immediate discussion presented in column 2 also ensured that whatever feeling and feedback the client had was conveyed to the therapist. These examples illustrate the value of pins in providing immediate and targeted feedback to the users. This specificity of feedback was appreciated by the therapist 15T in this pair. 15T in the interview talked about the feedback they received:

“I noticed that my client was very accurate and pinpointed in [a way that is] almost coding, coding the session... So he was very able to pluck out the components, and I was trying to cognitively feel where he was and stay with him and guide him and use MI.” -15T

6.4 Discussion helps users correct misconceptions about their performance

In addition to helping participants develop empathy, Pin-MI also allowed participants to correct misconceptions about their performance in the session. We observed this theme in both the content analysis of the pins as well as the interviews. In 10% of the pins, the therapist was uncertain about how their statement affected the client. In our interview and analysis of the discussions, we found that the client sharing their perspective helps rectify these misconceptions. This is further discussed below for a pinned moment from session 2, presented in table 5.

Table 5:

Pinned moment in role-play	Self-reflection	Discussion transcript
2C: Hi, how are you? 2T: Hi, I am doing well. Thank you. How are you today? 2C: I’m okay. 2T: Okay. Yeah, [pause] just okay? 2C: I am just, okay. 2T: Just okay. [pause] So what can be better?	2T notes: I think I could have started the interview a little more decisive and confident in the way I approached the beginning of the conversation. 2C notes: [The therapist] asked more information on why I said okay and asked what can be better. He seemed interested in wanting to know more.	2T: I think I could have started the interview a little more decisive and confident in the way I approached the beginning of the conversation... I pinned that because I wasn’t happy with the way I started the interview. I thought I could have come out sooner. 2C: I think that the first sessions are typically a little bit like that, you’re just kind of feeling it. I don’t think there was an issue on my end... I like it when I said, okay, you didn’t just let it go. You were like, okay, why just okay. So I think that sometimes, we kind of just moved past it and get straight to the meat and potatoes where even those the first introduction can be important. And I’m glad that you kind of ask more about that. Sometimes I tend to not get more into from the introductions. People say, hey, I’m alright. And I just kind of move forward. So you remind me to kind of look more into people when they just say they’re okay.

Table 5: Data from Pin-MI session 2, pinned moment at 0:17

In this session, the pair started the conversation with small talk. However, the therapist had a few pauses in between their questions. The therapist marked the pinned moment as an opportunity for improvement due to these hesitations, but the client focused on the more positive aspect of the therapist’s ability to follow up. This discrepancy in how the two sides perceived the conversation is revealed in the discussion transcript. For novice learners, it is common to have doubts about one’s ability to perform [77]. In this case, the therapist overestimated the impact their pauses had on the client. The feedback from the peer client provided an alternative perspective and guided the therapist to focus on the positive aspect of the conversation.

The therapist and client in their interview further discussed this pinned moment. Interestingly, in the discussion and the therapist’s personal notes, the therapist focused lack of confidence. However, in the interview, when asked about a pin that made a strong impression, the therapist mentioned this pin, and instead of talking about the negative aspect they originally focused on, the therapist elaborated on their strength:

“I think when she was even saying it right up front, she’s just, okay. And then I kind of probed her to elaborate on why she said she was just okay... and even she had kind of made a comment after the fact that that was a really good question. I felt the same as well. I pinned that because I thought, okay, I’m on the right track. I’m getting her to open up. And I am asking her some probing questions there that are open-ended, make her do that analysis. So that really triggered me to pin that, to go, okay, I was onto something there. And I thought that was very, very helpful.” -2T

This example highlights Pin-MI’s value in perspective sharing between the peers. The discussion presents the learners with elements of the conversation or skill practice that could have otherwise been overlooked without the alternative perspective. The therapist initially focused on a negative element in the conversation, their lack of confidence, but overlooked the positive element, how probing the client encouraged the client to open up and showed their empathy. By correcting this misconception, the therapist was able to gain insight into what was more important for the client and reflect on how they could further improve the probing question and obtain more information from the client.

6.5 Peer feedback can encourage or discourage users

Overall, participants said they appreciated the opportunity to discuss their session with a peer and reported valuable learning experiences from the discussion. Most of the feedback and discussion focused on positive elements in a session, with 70% of the pins labeled in the Pin-MI sessions were strengths. In addition, the most common behavior in the peer discussion was the client identifying a strength of the therapist. This positive feedback not only provides valuable information on what the therapist was doing well but also can give novice therapists more confidence in their ability to execute the skills they learned. For instance, 13T commented on the positivity of feedback:

“I really enjoyed that. It gave me encouragement and made me feel, in some cases, more qualified or more professional than what I felt.” -13T

13T continued to talk about the value of receiving peer feedback and the importance of this feedback in enabling them to continue to improve their skills.

“When there’s conversation or notes that someone else took, because we know we are not in a real client coach relationship, that you do have feedback from someone, a colleague, who is also, taught or learned in this situation that we’re exploring. So that feedback is very important in honing skills and, and improving... I want to be analyzed by a peer... Because one of the things we hate about audits are that somebody’s auditing our stuff that doesn’t know our style of work.” -13T

This example points out the importance of context and shared experience between the peers and how the receiver of the information could benefit more when the feedback is provided by someone who has context of what is going on, knowledge of counseling, and/or exposure to and knowledge of MI skills.

However, even though participants seem to appreciate feedback, some participants pointed out that receiving negative feedback can be discouraging. In the quote below, 11T idicated that hearing criticism can sometimes feel like invalidation of their effort and cause anxiety:

“I think we all wanna believe that we are doing the best we can. And when it’s pointed out that maybe you could be doing something better, for me, at least my normal tendency is to, I don’t know, I think just get anxious and perhaps not fully hear what’s being said.” -11T

The conversation that 11T mentioned is included in table 6. 11T referred to a specific moment that the client pinned as an opportunity for improvement. The criticism was centered around the therapist’s statement “I’m so proud of you”. The therapist identified the reasons why they believe the moment could be improved: by saying “I’m so proud of you”, the therapist focused on their own evaluation and feelings towards the client instead of the client’s behavior. The therapist also included a personal experience to serve as an example to potentially lessen the impact of the statement by showing that they have committed the same error in the past.

Table 6:

Pinned moment in Role-play transcript	Discussion transcript
11C: I had a baby four months ago. So just like during pregnancy, I ate differently. I was pretty restrictive on my diet before pregnancy, and I kind of loosened up on it during and so like now that I am not pregnant again, going kind of back to the more restrictive diet has been hard. It’s been kind of nice to not be having to follow that restrictive eating for a while there. So I think I got out of habit. 11T: Absolutely, you just had a baby. So you remember that it was like nine months of change in your body and in your hormones and how you feel about yourself. It would be, it would be unwise to think that such a quick change could happen now, and I am so proud of you for taking those baby steps to making those healthy changes for you and for your child.	11C: It was good to use an affirmation there. I think that it, you could take out the I’m so proud part, and that it might actually make it even more effective for me. Since it would be more about me and less about how you are feeling about something that I did... I know personally, when I’m doing MI I fall into that like trap where I’m like almost cheer-leading and saying, I’m so proud of you. And so it’s something like I’ve been trying to be more aware of for myself. So when I heard it, it just made me think like, oh, maybe could we just remove the I’m so proud and, and then it might even be more powerful. 11T: I like that. I think that makes complete sense. Just to remember that this hour is about you and not really about, I shouldn’t have the reaction. I’m there to listen and offer suggestions, but not to like reflect back emotions to you. Good, good catch.

Pinned moment in Role-play transcript

Discussion transcript

11C: I had a baby four months ago. So just like during pregnancy, I ate differently. I was pretty restrictive on my diet before pregnancy, and I kind of loosened up on it during and so like now that I am not pregnant again, going kind of back to the more restrictive diet has been hard. It’s been kind of nice to not be having to follow that restrictive eating for a while there. So I think I got out of habit. 11T: Absolutely, you just had a baby. So you remember that it was like nine months of change in your body and in your hormones and how you feel about yourself. It would be, it would be unwise to think that such a quick change could happen now, and I am so proud of you for taking those baby steps to making those healthy changes for you and for your child.

11C: It was good to use an affirmation there. I think that it, you could take out the I’m so proud part, and that it might actually make it even more effective for me. Since it would be more about me and less about how you are feeling about something that I did... I know personally, when I’m doing MI I fall into that like trap where I’m like almost cheer-leading and saying, I’m so proud of you. And so it’s something like I’ve been trying to be more aware of for myself. So when I heard it, it just made me think like, oh, maybe could we just remove the I’m so proud and, and then it might even be more powerful. 11T: I like that. I think that makes complete sense. Just to remember that this hour is about you and not really about, I shouldn’t have the reaction. I’m there to listen and offer suggestions, but not to like reflect back emotions to you. Good, good catch.

Table 6: Data from Pin-MI session 11, pinned moment at 6:11

Connecting this example back to the earlier quote by 11T, it is likely that the therapist felt hurt when a positive statement such as “I am proud of you” was pointed out as an opportunity for improvement. 11T explained that although they wanted to do their best, their attempt was not positively received. This discrepancy between their own positive evaluation of their performance and the criticism received from their partner potentially created a feeling of invalidation. In this case, the therapist felt they were in a vulnerable situation in which they received feedback that felt like a judgment of their ability. Later in the interview, the client described an experience where they felt more comfortable receiving feedback by observing others and hearing how others can improve, thereby learning vicariously. This contrast between direct and vicarious feedback suggests that while participants can value peer feedback, but less so when the feedback felt personal. They learned that even though they are trying to do their best, there is still room for improvement.

6.6 Pinning reduced mental effort for most users

One of the design goals of Pin-MI was to reduce the mental energy usually required of note-taking in a traditional clinical setting by introducing a simpler method of marking critical moments in a conversation. A majority of participants expressed that pinning was easy and fit into the flow of the conversation without being disruptive. For instance, 6T described the experience as easier than note-taking, allowing them to pay attention to the conversation while knowing they have the chance to go back and review the content later:

“[It is] very easy to pin so that you’re not having to take notes, and you can really be focused and listen on what the other person is saying and then just pin it and go back later and, and listen to it for additional thought.” -6T

Others, however, felt they had to actively think about pinning, pausing their train of thought while talking or listening to the client. We believe judgements about mental effort may in part be due to participants’ familiarity with motivational interviewing. For instance, 11T, who was an MI novice, described their experience:

“It’s hard to pin when you’re doing the talking. I think it, it would’ve been easier for the person who was listening to do the pinning. Cause when you’re doing the interviewing, you’re thinking about what are they answering me? How am I responding? And it wasn’t until you made that mention about pins that I was like, oh yeah, I need to pin too. So for me, that was just a lot of things to juggle.” -11T

Another participant, 7C, talked about the difficulty in using the pins due to her relatively little experience with MI and nervousness throughout the experience:

“It was kind of hard thinking about when I had to pin and then trying to pay attention to what the person is saying to you. I guess it’s not like a natural thing, where you wanna listen and, and take note of something, but then to remember to push the pin, it’s a very easy thing, but it’s just not like something natural that you would do... And, because MI is not something that I’ve done for very long. I just started this position in August. So this is all new to me.” -7C

In contrast, more experienced participants reported having the mental capacity to do more than just pinning and wanted to write down notes for the pins. For instance, in the example below, 14T pointed out that they wanted to take brief notes for each pin to reference later on.

“Maybe, actually, even though if I’m pinning, whatever timestamp, that I thought was important, I feel like maybe actually writing my notes in real time as well. Just to have that, to reflect back on, along with the pin just to see if I’m really understanding correctly or heard her correctly.” -14T

Note-taking is part of therapeutic professional practice and requires more mental effort than simply pinning as a form of annotation, which might be more familiar to practitioners with greater experience. Therefore, this type of multitasking might be challenging mostly to novices who do not have enough practice of the skill yet. This difference in how participants react to the mental resources required of pinning suggests the value of further exploring how to adjust the design for trainees at different experience levels.

7 Discussion

7.1 Cost and benefit of real-time annotation

Our results highlight important trade-offs in applying reflection-in-action in experiential training systems. Reflection-in-action is beneficial because users are actively thinking about their action in the moment. At the same time, our results suggest that reducing mental workload is an important design challenge for reflection-in-action experiential training systems, particularly for novice trainees. With Pin-MI we found that people experienced varying levels of mental effort when pinning during role plays depending on their prior experience with MI. Novice therapists in our study talked about how lack of experience with MI skills or the novelty of the annotation tool that we designed made it hard to annotate and carry on a conversation at the same time.

Future work on experiential training systems could explore how machine intelligence could scaffold reflection-in-action and how integrating intelligence in different ways may impact learning. Machine learning algorithms can identify MI skills [74], meaning the system could isolate moments in a conversation where MI skills are being applied. Experiential training systems could prompt the user to pin or notify that an MI skill is being used. Automating aspects of the annotation process could impact the learning benefits of reflection-in-action, if for example annotation is fully automated and reflection only happens post-hoc.

Designs for reflection-in-action may need to be adaptive to user skill levels. We found in our study there were users who felt annotation overwhelmed their available cognitive resources, while users with more experience desired even more detailed annotation capabilities. This raised an interesting point: earlier we discussed the design suggestions by An et al. [2]. This work suggested designing low-effort actions that the learner engages in while performing a learned action to trigger reflection-in-action. The authors described their design as a passive reminder for reflection instead of an active procedure that the teacher needs to undertake. This provides an interesting comparison to our design, which focuses on active annotation. Even though it is a simple button press, it was interesting that this simple action still appeared to take a toll on some learners’ mental resources, yet not others.

This difference highlights the importance of exploring the extent to which an action can be considered low-effort for its users, and whether an active process triggers more active reflection. Future work should explore designs for flexible annotations to support reflection-in-action for different skill levels. With flexible annotations, novices could use simpler annotations like pins, while more experienced users could have the chance to expand on them (e.g., with tags or notes). Very experienced users could input a free response for each pin to document additional details. Expanding the range of possibilities for the pinning action, in terms of what kind of annotation users can add for each pin, could flexibly support users of different levels.

Reflection-in-action also needs to fit with existing reflective practices for a profession or domain. In our setting, note-taking is an integral part of therapy [59], and we found that therapists with more experience expressed wanting to take notes alongside pins. In Pin-MI, pins were focused on improving therapist behavior instead of focusing on an analysis of the client and their problems. Users were asked to pin when a skill was executed well or poorly by the therapist instead of identifying moments where clients shared key information or were able to make a breakthrough. Pins then are different from the type of notes that would occur in a typical therapy session, which focus on what the client needs and the details of their situation. In designing a training tool for therapeutic practice we must consider the role of note-taking and how annotation for reflection connects to note-taking that serves as a reference. Note-taking about clients and referencing those notes during sessions are skills in and of themselves, require their own training, and could benefit from annotation and reflection just as MI skills do.

Finally, the question remains of how reflection-in-action compares with and could influence post-hoc reflection given its cognitive overhead. Literature on reflection suggests the value of introspective moments in which students can focus on reflection [68, 82]. The nature of our design, however, requires quick on-the-spot thinking during skill execution. Follow-up work should examine whether quick reflections-in-action, as in our design, enhance therapist skill acquisition for therapists with different levels of experience and whether these quick in-action reflections strengthen and deepen the reflections that happen post-hoc.

7.2 Ensuring quality feedback

Feedback provides a helpful supplement for reflection. However, it is important to further understand its impact. As our results demonstrate, providing constructive feedback without discouraging the receiver is a delicate task. In Pin-MI, we implemented features to ensure good feedback quality based on prior literature. We ensured the immediacy of feedback as well as the opportunity for peers to collaboratively discuss the feedback. We also provided guiding questions to suggest feedback content. Our design could further scaffold the feedback content. Literature on peer feedback has suggested methods for its enhancement, e.g., balancing positive and negative comments, posing questions that foster reflection, no use of naive words like “good” and “nice”, and presenting external examples [31]. Each recommendation points to potential design directions.

Interestingly, principles of good feedback were evident in a lot of the feedback that our users provided. For instance, participants would use their own past experiences as examples when pointing out an instance where the therapist could improve. Participants frequently referred to mistakes they made in the past in order to soften the impact of a critical statement and establish more common ground with the feedback receiver. A remaining problem is how to ensure consistency in users’ ability to provide quality feedback for each other.

In order to ensure consistency of feedback, one design idea is a feedback constructor. A step-by-step process could be implemented that users would encounter when reflecting on each pin. This constructor could provide questions for users to answer in each step, guiding them to create feedback that conforms to best-practice recommendations. After this reflection process, the generator could output a paragraph of feedback based on the user’s answers. For instance, the user could first be asked to write one strength they or their partner exhibited during the pin, followed by one area for improvement. They could then be asked to write down some past experiences related to this area for improvement, and then lastly, they could write a thought-provoking question for themselves or their partner on possible ways to improve. A similar process is described by Cambre et al. [14]. The authors designed a tool where peer reviewers would fill in answers to prompts which ask the reviewer to evaluate different aspects of their peer’s work. This ensures consistency of feedback across reviewers. However, the lack of standardization across sessions could be a challenge. In therapy, a conversation could branch in different directions based on the client’s readiness to change, and the skills involved may vary.

Moreover, we believe another action that can positively impact quality of feedback is assisting the users with the interpretation of their data. In recent works in personal informatics, researchers highlighted the importance of providing users with insights and context to aid understanding of their performance and thus stimulate better reflections [8]. Similar works explored different methods of visualization and explanation of the data [20, 50]. In Pin-MI, we only provided the users with a one-dimensional representation of their data and relatively little information that helps them assess their own performance and their partner’s performance in the context of MI skill usage. We believe a potential direction for future work is providing more information and context to the users. For instance, this information could take the form of presenting the users with how many pins they make compared to their peers, tallying the number of opportunity pins and comparing this against the number of strength pins, or even presenting comments of all pins together, enabling easier summarizing of the most common mistakes in each session.

7.3 Vicarious learning as the client

Experiential learning as a therapist versus a client may have different influence on learning. In our design, the two sides of the role-play have very distinct functions. The therapist role-player has the opportunity to practice and reflect on their skills. The client, however, is mostly the observer and feedback provider. The client has less opportunity to actively use therapeutic skills, raising the question of whether this experience is as enriching for the client as it is for the therapist. This is an important question to consider since in role-play situations, half of the time one party acts as the client.

Learning benefits for the client role-player are not clear from past literature on role-playing in psychotherapy and medical education. Frequently, role-play learning benefits are reported for students in the aggregate without differentiating distinct roles [30, 55], or in other cases, role-play is conducted using standardized clients, i.e., hired actors that serve as the patients [1, 64]. We reported cases where clients benefited from role-playing and reflecting by simply observing and providing feedback on the therapist’s actions. This observation closely aligns with the concept of vicarious learning, which refers to learning from observing someone else’s behavior rather than through one’s own performance [32].

Future work should examine the learning benefits of experiential learning and reflection as a client versus a therapist. In Pin-MI, the client not only observed the therapist’s behavior in the role-play; they also observed the therapist’s reflections and evaluations of their own actions. Moreover, client reflections frequently focused on the therapist’s statements and their impact: what skills the therapist used and how their statements impacted the client. They were not only engaging in reflective behavior as the client but also vicariously learning through the therapist, analyzing the therapy skills that were used in the session. This reflection could potentially benefit the client in the session even though they are not the party actively practicing therapy skills.

8 Conclusion and Limitations

In this work, we present the system Pin-MI, an interactive tool that supports counselors’ experiential learning with real-time annotation and reflection. Using a qualitative approach, we tested the application with 15 pairs of healthcare professionals who have learned and need to use Motivational Interviewing in their work. We found that Pin-MI was able to help users develop empathy and be more aware of their skill usage; it also guaranteed immediate and targeted feedback and helped users correct misconceptions about their performance. We found that Pin-MI required varying levels of mental effort for different users. Additionally, the peer feedback process can encourage or discourage users, which has potential design implications as discussed.

At the same time, our work is not without limitations. The first limitation is the specific focus of Motivational Interviewing in our work. We chose to focus on Motivational Interviewing skills in our evaluation to ensure the consistency between participants’ approach to role-play conversations. However, we believe that future researchers can benefit from experimenting with other counseling styles and skills. Second, our research is qualitative in nature, aimed at understanding the details of the Pin-MI experience. Lastly, the inclusion of researchers in some of sessions could have potentially altered the sessions. Even though we believe that our study adequately captured the overall experience of Pin-MI, we consider a controlled quantitative experiment aimed at evaluating user’s improvement in learning and performance to be an appropriate next step.

Footnotes

https://reactjs.org/

https://www.vonage.com/communications-apis/video/

https://docs.symbl.ai/docs/speech-to-text

https://www.netlify.com/

https://firebase.google.com/

https://aws.amazon.com/s3

Supplementary Material

MP4 File (3544548.3581551-talk-video.mp4)

Pre-recorded Video Presentation

Download
74.32 MB

References

[1]

Michael A Ainsworth, Linda P Rogers, John F Markus, Neil K Dorsey, Thomas A Blackwell, and Emil R Petrusa. 1991. Standardized patient encounters: a method for teaching and evaluation. Jama 266, 10 (1991), 1390–1396.

Abstract

1 Introduction

2 Background

2.1 Psychotherapy skills and training

2.2 Reflective Learning

2.3 Peer feedback

2.4 Annotation

3 Design

3.1 Session flow overview

3.2 Real-time annotation

3.3 Assisted self-reflection

3.4 Peer feedback

4 Implementation Notes

5 Evaluation

5.1 Procedure

5.2 Analysis

5.2.1 Pinned moments.

5.2.2 Interviews.

6 Results

6.1 Role-playing helps users develop empathy

6.2 Pinning helps users be more aware of their skill usage

6.3 Pinning guarantees that feedback is both targeted and immediate

6.4 Discussion helps users correct misconceptions about their performance

6.5 Peer feedback can encourage or discourage users

6.6 Pinning reduced mental effort for most users

7 Discussion

7.1 Cost and benefit of real-time annotation

7.2 Ensuring quality feedback

7.3 Vicarious learning as the client

8 Conclusion and Limitations

Footnotes

Supplementary Material

References

Cited By

Index Terms

Recommendations

Use of Agent Prompts to Support Reflective Interaction in a Learning-by-Teaching Environment

Using a video annotation tool to enhance student-teachers’ reflective practices and communication competence in consultation practices through a collaborative learning community

Tools to Support Data-driven Reflective Learning

Comments

Information

Published In

Sponsors

Publisher

Publication History

Check for updates

Author Tags

Qualifiers

Conference

Acceptance Rates

Upcoming Conference

Contributors

Other Metrics

Bibliometrics

Article Metrics

Other Metrics

Citations

Cited By

View options

PDF

eReader

HTML Format

Login options

Full Access

Figures

Other

Share

Share this Publication link

Share on social media

Affiliations