From User Perceptions to Technical Improvement: Enabling People Who Stutter to Better Use Speech Recognition

Voice Assistants (e.g., Amazon Alexa, Apple Siri, and Google Assistant). These systems listen for spoken questions or commands. For example, “What is the weather?” or “Set a timer for five minutes.” A typical user flow is as follows, where a simplified view is shown in Figure 1. A user may initiate a query by saying a wake word, such as “Alexa”, “Hey Siri”, or “OK Google.” This may also be called invocation. They then start vocalizing their command. As the user speaks, an ASR model starts to transcribe their words and simultaneously an endpointer model—also called end-of-speech detector—detects whether the user has finished speaking.¹ A Natural Language Understanding (NLU) model then takes the transcribed phrase and any metadata from the ASR and identifies the user’s intent, which is then sent to the Decision Making component to perform the appropriate action. Based on the user’s query, the VA may respond with information, perform a command, ask the user a follow-up question (i.e., “is the alarm request for AM or PM”), or notify the user that the query could not be understood.

There are alternative ways of interacting with a VA. Instead of using a speech-based invocation mechanism on a phone or tablet, a user may hold down a button on the side of the device (“Hold to Speak” on iOS) or squeeze the side of the phone (some Android devices). VAs can also be invoked using accessibility features such as Accessibility Menu on Android or Assistive Touch on iOS. Instead of speaking, a user can type a command with a keyboard using “Type with Alexa” or “Type to Siri.” This bypasses the ASR component and may directly process the input text through NLU.

Dictation Systems. These systems provide an alternative to using a keyboard by converting a user’s speech to text for tasks such as composing text messages, notes, or emails. A common use case is to verbalize a text while in a car. Typically, a user starts dictation by manually clicking a button on a device, instead of via a wake word, and they finish by manually clicking a button again, instead of relying on an endpointer. An ASR model may transcribe their speech in real-time and visualize the text if the device has a screen. A punctuation model may add grammatical structure to spoken words, like adding commas and periods. Finally, if the ASR model yields low confidence in a predicted word, this word may be highlighted on screen along with suggestions for replacement. A user can then manually revise their statement or click on low confidence words for suggestions.

To refine speech technologies for PWS, we focus on endpointer and ASR models independently. Endpointer models are optimized to balance the trade-off between cutting someone off too early and having a long delay after their speech, and implicitly or explicitly rely on cues such as how much silence has elapsed since the last word (e.g., [44, 45]). These models are largely optimized on data from people who do not stutter, and are tuned to respond to the user with as low latency as possible. For PWS, as described by Bleakley et al. [13] and in Sections 3 and 4, endpointer models commonly cut off before the user is done speaking, especially during blocks where there may be a long pause or gasp, or even partial word repetitions where the vocalization may be indecipherable.

At a high level, ASR models (e.g., [15, 32, 36, 71]) commonly consist of an acoustic component that encodes low-level phoneme or phone-like patterns, a language component that acts as a prior over word sequences, in some cases a pronunciation component that describes the phonetic pronunciation of words (e.g., for hybrid ASRs [36, 41, 59]), and a decoder that may map these together and generate candidate transcriptions. For PWS, we hypothesize that sound, syllable, or partial word repetitions fed into the acoustic component may have lower confidence phone- or phone-like outputs, as these models may not be trained with large amounts of dysfluent speech. Similarly, the language component may not be trained with word or phrase repetitions, meaning these repetitions may have lower likelihood and be misrecognized. Furthermore, interjections mid-word or partial word repetitions may cause errors when paired with a pronunciation dictionary and decoder, as there may be spurious phone-like units interspersed with known words.

It is worth noting that while VAs and Dictation often rely on the same underlying ASR system, a VA’s model may be tuned to work better for a smaller set of assistant commands [31], whereas Dictation may be biased towards free speech from arbitrary topics. In addition, for VAs, an NLU model (e.g., Chen et al. [16], which is used in this paper) may take an erroneous transcription and map it to the correct action using context and error cues. This means that individual ASR errors may play a more important role in Dictation than for VA tasks.

Technical work on improving speech assistants for PWS has focused on ASR models [8, 23, 31, 35, 50, 51, 61], stuttering detection [43], dysfluency detection or classification [22, 40, 42, 48, 56], clinical assessment [11], and dataset development [12, 37, 42, 55]. Shonibare et al. [61] and Mendelev et al. [50] investigate training end-to-end RNN-T ASR models on speech from PWS. Shonibare et al. introduces a detect-then-pass approach that incorporates a dysfluency detector where audio frames with dysfluencies are ignored entirely by the RNN-T decoder. Examples in the paper demonstrate how this approach can be used to ignore some partial-word repetitions and blocks. Alharbi et al. [2, 3] focused on stuttered speech from kids that incorporates the structure of repetitions and other dyfluencies into an augmented language model that is better at including dysfluencies in a transcription. In our work, we focus on solutions, like Mitra et al.’s [51], which can be applied on top of existing recognition systems and do not require as much data as end-to-end solutions. Their VA-oriented approach was to optimize a small set of ASR decoder parameters on stuttered speech, such that the system is biased towards common VA phrases, and was effective in removing dysfluencies such as repetitions in speech. One limitation of that work is that it was only applied to speech from 18 PWS, so in this paper we validate results on our much larger 91 person study.

Screened participants filled out the survey online using a commercial survey website. The survey took 30 to 60 minutes to complete and included the following four primary sections with up to 77 questions depending on conditional logic.

Many of these questions had multiple choice responses and open-ended text input to further explain responses. For questions related to the endpointer and other technical material, layperson descriptions like “How often does a VA cut off your speech” were used. See the Appendix for the list of survey questions discussed here.

Participants were recruited through social media, referrals from speech-language pathology clinics, and word of mouth. To participate, individuals had to self-identify as having a stutter, after which they signed an Informed Consent Form and were screened by a speech-language pathologist (SLP) who listened to their connected speech patterns. Stuttering characteristics such as frequency, duration, and distribution of dysfluency types can vary greatly across individuals [64], thus we aimed to recruit a broad range of participants with different stuttering severities. While there are many clinical severity assessments (e.g., [57, 70]), we chose the Andrew & Harris (A&H) Scale [6] which can be performed entirely using audio recordings. The SLP performed this assessment by examining dysfluency rates in audio recordings of a participant reading a passage, talking impromptu about a topic of their interest, and speaking with other individual(s). The result was a grading of mild (0-5% of words have dysfluencies), moderate (6-20% of words), or severe (> 20% of words) stuttering.

With the A&H scale, 12 participants were rated as having a mild stutter, 31 as moderate, and 18 as severe. When asked about whether people understood their speech, almost all participants (93.4%, n=57/61) reported that friends and close contacts ‘always’ or ’usually’ understand their speech, however this was lower for acquaintances (73.8%, n=45) and strangers (67.2%, n=41). Many individuals noted that their speech qualities change day-to-day (65.5%, n=40) or throughout the day (55.7%, n=34) and on top of typical dysfluencies such as blocks, prolongations, and repetitions, their voice may be strained (42.6%, n=26) or breathy (16.4%, n=10). Most individuals noted that speaking requires ‘some’ physical effort (67.2%, n=41) with fewer responses of ‘very high‘ (11.4%, n=7) or ‘very low‘ (9.8%, n=6).

In terms of demographics, there was an even gender split between men (50.8%, n=31) and women (49.2%, n=30), with no participants reporting another gender. Most participants were under 40 years old: aged 18-19 (n=2), 20-29 (n=28), 30-39 (n=21), 40-49 (n=3), 50-59 (n=4), 60-69 (n=2), and 70-79 (n=1). Individuals identified as white (n=38), black or African American (n=12), Hispanic (n=5), Asian (n=2), American Indian or Alaska Native (n=1), or preferred not to answer (n=3). In terms of technology use, all participants owned smartphones, with most having an iPhone (93.4%, n=57) and four having an Android phone. Over half (59.0%, n=36) had smart speakers in the home: 27 had an Amazon device, 5 had a Google device, and 4 had an Apple device. For other technology, 58 participants reported having a computer, 35 had a tablet, and 27 had a smartwatch or fitness tracker.

For closed-ended questions, we report primarily on descriptive statistics including counts and proportions. We have included a few comparative statistical tests for high-level questions of how often participants use VAs vs. dictation, and whether there is an impact of stutter severity ratings on usage. In doing so, we selected tests that are appropriate for ordinal data: Mann Whitney U and Kruskal Wallis tests. For open-ended questions, we conducted two open coding passes on each question and report on the prominent themes identified in this analysis to help contextualize the quantitative results.

Most participants were familiar with speech recognition systems (VA: 100%, n=61; DS: 90.2%, n=55), although more had used VAs (96.7%, n=59) than dictation (73.8%, n=45). Overall, among participants who were familiar with the given technology, VAs were used significantly more often than dictation (Mann-Whitney U test²: U = 1937, n₁ = 59, n₂ = 53, p = .019). As shown in Figure 2, about half of the participants familiar with VAs used one on a daily (32.2%, n=19/59) or weekly basis (20.3%, n=12/59), whereas fewer used dictation on a daily (15.1%, n=8/53) or weekly basis (24.5%, n=13/53). This VA usage rate is somewhat lower than recent estimates for the general population in the US that suggest 57% use VAs daily and 23% do so weekly [53], perhaps reflecting the accessibility barriers faced by PWS.

Because past work has shown that stuttering severity affects speech recognition accuracy [51], we further examined the relationship between A&H severity ratings and reported dictation and VA usage. These results, shown in Figures 3 indicate no clear patterns based on graded severity. Kruskal Wallis tests did not reveal significant impacts of A&H severity level (i.e., mild, moderate, and severe) on reported frequency of use for either VA usage or dictation usage.³ This lack of significance may be due to low statistical power related to our sample size and variability in how people rate their frequency of use, but we return to additional considerations in Section 5.

Turning to how and why participants adopted VAs and dictation, we first examined what platforms they used. Among people with experience using VAs or dictation, phones were the most common platform (VA: 84.7%, n=50/59; DS: 88.9%, n=40/45) followed by smartwatches (VA: 25.4%, n=15; DS: 24.4%, n=11) and car infotainment systems (VA: 25.4%, n=15; DS: 20.0%, n=9). Smart speakers were also relatively common for VA use (47.5%, n=28) but not used as much for dictation (9.0%, n=4)—which is expected given that smart speakers are often communal devices and have fewer dictation use cases. Ten or fewer participants had used speech on tablets or traditional computers for both VAs and dictation.

For why they use speech technology, a majority of participants find speech recognition systems to be at least “somewhat” useful; see Table 1 a for detailed utility ratings. For those surveyed that use VAs, common use cases generally mirror those from the general population [4]. For VA users (Table 1 b), participants most commonly reported listening to music, making phone calls, asking general questions, checking the weather, setting alarms and timers, and sending messages. For dictation users (Table 1 c), the common use cases were writing text messages, taking notes, writing email, and dictation within a web browser. However, about a quarter of participants familiar with each of the two speech technologies felt that they were ‘not very’ or ‘not at all’ useful. Overall, reasons were split between participants who saw no inherent value in VAs (e.g., “I haven’t felt a need for them in my life” [P2-51]) and those who encountered accessibility challenges due to stuttering (e.g., “[...] I often have to repeat myself so many times for it to understand me that it’s quicker and easier for me to just use my hands [...]” [P2-64]). We examine these and other challenges in more detail in the following sections.

To understand the range of challenges PWS face when using VAs, we asked about what factors prevented participants who were familiar with VAs from using them more often. The most common were ‘voice assistants cut off my speech before I finish talking’ (61.0%, n=36/59) and ‘voice assistants don’t understand my speech well enough’ (57.6%, n=34). Indeed, as shown in Figure 2 (b), a majority of participants (54.2%, n=32) felt VAs only recognized their speech ‘sometimes’ or worse. As a result, many participants (37.3%, n=22) reported ‘often’ or ‘always’ attempting to revise what they say due to inaccuracies in speech recognition. Other concerns, however, were not purely technical. Twenty-two participants (37.3%) reported not wanting other people to hear them talking to the VA, reflecting a social consideration that past work has also identified [13]. As well, many participants (22.0%, n=13) cited that it takes ‘too much physical effort for me to speak.’ Only six participants (10.2%) responded that perceived utility of voice assistants prevented them from using the technology more often.

Concern with endpointing—being cut off too soon—is typically more relevant to VAs than dictation, and was identified as a challenge in Bleakley et al.’s [13] interviews on VA use with 11 PWS. Our data demonstrates the pervasiveness of this problem: almost half of participants stated that VAs cut them off ‘always’ (3.3%, n=2) or ‘often’ (37.7%, n=23), while an additional 31 participants (50.1%) reported being cut off ‘sometimes’ or ‘rarely.’ For the 57 participants who had at least some experience being cut off, when asked if they knew why it happened, by far the most common explanation was that there was a pause or block in their speech and the system thought they had finished speaking (77.2%, n=44 of 57). Being cut off could also have consequential impacts, such as a poor user experience described by [P2-42]: “The [smart speaker] which I use most often does not like any hesitations in speech. It assumes the slightest pause means the person is done speaking. So I get cut off often. It creates a sense of having to rush which makes my speech worse.” And, as [P2-77] described, impacts can also be emotional: “I block a lot on my name and vowel words so when I take time to talk it is frustrating when they cut me off [...] I have to deal with people often trying to speak for me and having a voice assistant doing that because I can’t have one who understands my speech rhythm makes me not want to use it even more.” We address endpointing errors in Section 4.2.1 and significantly reduce the problem on our dataset of speech samples.

Some smartphones (e.g., Apple iPhone) offer an alternative to automatic endpointing, where the user can push down a button (physical or virtual) while speaking and only release it when done. Twenty-seven participants (44.3%) reported using a device that supports this functionality. Of those 27, almost all found being able to press a button and speak to be useful: 51.9% (n=14 of 27) ‘extremely useful,’ 18.5% (n=5) ‘very useful,’ and 29.6% (n=8) ‘useful.’ Reasons provided for this utility were most commonly that it eliminates being cut off, but some participants also explicitly referred to a psychological impact of feeling less rushed or worried, such as: “I can fully get my phrase in without worrying about blocking” [P1-62] and “I don’t feel rushed to finish what I need to say” [P1-45].

Another potential issue unique to VAs is invocation. Most VAs use a wake word like “Alexa,” “OK Google,” or “Hey Siri” for invocation, which can be difficult for some PWS to use. Almost half of participants reported that these wake words ‘always’ (11.5%, n=7) or ‘often’ (36.1%, n=22) worked for them. But, problematically, 17 participants (27.9%) reported wake words only working ‘sometimes’ and 8 participants reported ‘rarely’ or ‘never’ (13.1%); the remaining participants had not used a wake word. While we did not ask participants to elaborate on their use of wake words, a handful of responses to other open-ended questions referred to challenges with wake words, including [P2-1] who stated “Problem with wake words is my biggest frustration. I would want to change it to something I can say easily”. While popular consumer systems do not allow for arbitrary wake words, some offer multiple options (e.g., Alexa has at least five options and Google Assistant has two). Additionally, a few participants mentioned that pressing a button to talk (as described above) eliminated issues with wake words by allowing an alternative explicit signal that the user is ready to talk. [P1-28] noted “It’s [Hold/Squeeze to Speak is] useful on my phone to bypass the lag that occurs when [the VA] does not recognize me using the wake phrase”, and [P1-73] uses it “Because it knows when to start listening and when I’m done.”

As with VAs, we aimed to broadly understand what factors impact dictation adoption for PWS. When asked what reasons prevent them from using dictation more often, 56.4% (n=31) of the 55 participants who were familiar with dictation noted that ‘dictation tools don’t understand my speech well enough.’ Of those who had used dictation, a majority (59.1%, n=26 of 44 who answered this question) felt the ASR only recognized their speech accurately “sometimes” or less (Figure 2, right). As a result of these errors, participants reported attempting to revise or restate what they say, with about half of participants doing so ‘always’ (6.7%, n=3/45) or ‘often’ (42.2%, n=19), and an additional 35.6% (n=16) doing so ‘sometimes.’ Finally, and similar to VAs, participants also commonly cited non-technical reasons for not using dictation more, most notably the physical effort it takes to speak (32.7%, n=18 of the 55 familiar with dictation) and not wanting other people to hear them dictate to the device (30.1%, n=17). Additionally, 11 participants (20.0%) reported that low utility prevented them from using dictation more often, and 12 (26.7%) felt that they use more refined language when they type than when they speak.

In contrast to VA systems, which predict the most likely command based on a spoken phrase, with dictation the user’s speech is transcribed verbatim. This difference is reflected in participants’ experiences with dictation errors. When asked an open-ended question about what types of errors tend to occur with dictation, by far the most common response, mentioned by a majority of participants, was errors in the ASR transcription (e.g., “Message contains different words than I intended” [P2-5]). Some participants (13.3%, n=6) also mentioned being cut off, although this was a much less common error than with VAs because many dictation systems do not use an automatic endpointer. A few participants mentioned poor performance with punctuation, proper nouns, and slang. Finally, some participants mentioned specific stuttering characteristics and challenges, including their repetitions being displayed in the transcribed text (“Misunderstood due to repetition or actually repeats word” [P2-46]), the impact of blocks (“words that I have a block in the middle of will turn into a strange phrase” [P2-51]), and interjections being shown, as described by [P2-77]: “because I have blocks and use filler words, the fillers are listed on the text message all my disfluencies are written down and when I read over the text it makes no sense and I have to type it myself instead because I will never be fluent all the time.”

The findings above suggest that technical improvements in invocation, endpointing, and ASR accuracy should be welcomed by many PWS and ultimately increase adoption of speech technologies by this population. However, future adoption also depends on factors beyond technical improvements, such as social concerns and the physical difficulty of speaking, emphasizing that speech input may not be relevant to all users even with substantial accuracy improvements.

To begin to understand what future adoption might look like, we asked participants to imagine speech technologies that can accurately understand their speech. Within this context, most participants were at least ‘somewhat‘ interested in VAs (86.9%, n=53/61): ‘extremely’ (n=25), ‘very’ (n=13), ‘somewhat’ (n=15), ‘not very’ (n=7), ‘not at all’ (n=1) and similar for dictation (Figure 3, right). Positive responses reflected earlier feelings about VAs and dictation, notably increasing efficiency, ease and convenience, being able to do actions hands-free (including while driving), and generally having confidence in the system. For example, [P1-66], who currently uses VAs at least monthly and has not tried dictation wrote, “I would be extremely interested in this, especially knowing it’s something I don’t have to worry about the speech input being able to understand me. It would remove my hesitancy and frustration with my stuttering from the equation.”

However, the negative and neutral reactions to this future technology scenario indicated a preference for typing, seeing no need for speech input, and value depending on context (e.g., “It would depend on the device” [P1-64], who uses VAs weekly and dictation less than monthly). This preference for typing by some participants also arose in other responses throughout the survey, reflecting in some cases the physical effort required to speak. For example, [P2-73] said, “It’s a nice thought but it’s still easier for a person who stutters to type most of the time. Unless our hands are filthy or tied up” and [P2-87] “It still requires effort to speak even if it understand me perfectly, and I am good at controlling them manually.” In addition, 9.8% (n=6) stated ‘voice assistants don’t seem useful.’ while [P2-51] said, “I’m not sure. As someone who stutters, and just situationally in my life, I can’t imagine dictation regularly being a more convenient option than typing.”

Additionally, social factors may influence future adoption, a consideration touched on in past work with PWS [13]. While the survey did not explicitly ask participants about the social context of speech technologies, as already noted, many participants felt that having to speak in front of other people limited their willingness to use speech technology. For example, [P2-1] wrote “[...] I would be reluctant to use voice commands in public because of my speech impediment”, and [P1-58] wrote, “I still would not use it at work. I would use it in spaces where I am alone, with my spouse or family.” Some participants also commented on how anxiety affects their stuttering, which translated to speech technology use in different ways. [P2-88], for example, expressed that “...before I am about to use voice recognition I get a bit nervous because I think that I have to speak without stuttering for it to work”—a concern that could be alleviated with improved ASR. At the same time, some participants expressed that speaking to devices reduces anxiety compared to speaking to other people, including: “I don’t stutter much when talking to myself so I usually don’t experience this problem [dictation errors]” [P2-15] and “I usually stutter in high anxiety situations not when I’m by myself” [P2-70]. Depending on the individual person and the context of use, these social factors could prove to be barriers to adoption.

An endpointer model identifies when the user stops speaking, and must balance the desire for a low truncation rate (i.e., what percent of utterances are cut off too early) with the desire for a minimal delay after speech (i.e., time from the end of the utterance to when the VA stops listening). Our base model is trained on completed utterances from the general population, and predicts the end of a query using both auditory cues like how long the user has been silent as well as ASR cues like the chance a given word is the final word of an utterance. For each input frame (time window), the model outputs the likelihood of utterance completion. Once the output exceeds a defined threshold, the system stops listening and moves on to the next phase of processing.

Baseline endpointer. The baseline likelihood threshold from the model we used was set such that 97% of utterances from the general population data are endpointed correctly (i.e., truncation rate of 3%). When evaluated on our data from PWS, which is likely poorly represented in general population data, a much higher portion of data is truncated early. Across the 41 Phase 2 participants,⁶ the baseline endpointer model truncates on average 23.8% of utterances (SD=19.7, Median=16.8, IQR=29.0); see Figure 4(a). Truncation rates also vary substantially per person—for 7 of 41 participants over 50% of utterances are cut off early. This high truncation rate reflects the survey findings, where a majority of participants reported that early truncation was a key issue.

Our hypothesis, based on the literature and understanding of stuttering, is that blocks, which are often expressed as inaudible gasps, cause most endpointing errors. We validated this using the dysfluency annotations by computed Spearman’s rank correlation⁷ and found truncation rates are significantly positively correlated with rates of blocks (r(39) = .64, p < .001), part-word repetitions (r(39) = .44, p = .004), and interjections (r(39) = .48, p = .002). Correlations with prolongations (r(39) = .17, p = .301) and whole word repetitions (r(39) = .18, p = .271) are not statistically significant.

Tuned endpointer threshold (Intervention #1). To improve the VA experience for PWS, we investigate performance of three new thresholds that could reduce truncations for people with different rates of dysfluent speech. Specifically, using the Phase 1 data, we compute three new, higher threshold values that target an average 3% truncation rate for participants with different levels of A&H severities: a mild threshold based on the 25 Phase 1 participants with mild ratings, a moderate threshold based on the 18 moderate participants, and a severe threshold based on the 7 severe participants. By definition, increasing the threshold will always reduce or at worst maintain the same truncation rate as a lower threshold, at the cost of incurring a longer delay before the system responds. We evaluate these thresholds on the Phase 2 participant data.

As shown in Figure 4(a), the new thresholds substantially reduce the truncation rate for PWS compared to the baseline. Even the smallest threshold increase, “mild”, reduces the truncation rate to a per-participant average of 4.8% (SD=6.4, Median=1.5, IQR=6.5), while the moderate threshold achieves our goal of under 3% on average (M=2.5%, SD=3.6, Median=0.8, IQR=3.1). Table 2 shows these results averaged across all Phase 2 utterances and includes metrics to capture how much delay occurs after the user finishes speaking. Here, P50 and P95 refer to the 50^th (median) and 95^th percentile delay between when a speech utterance ends and when the system stops listening. This analysis shows that the improvements in truncation rate come with a modest median delay of an additional 1.2 seconds over the baseline in the mild case and a 1.7 second increase in the moderate case. While the severe threshold is successful for 99.2% of utterances, it causes a median delay of over 3 seconds. We return to these tradeoffs in the Discussion.

Next, we evaluated baseline ASR performance and compared the results to an approach that tunes the ASR decoder using dysfluent speech. For this analysis, we use the Apple Speech framework [7] and report on results from an ASR model trained on voice assistant tasks and speech from the general population. For completeness, we also examined baseline performance with a model trained on dictation tasks; the pattern of results was similar and thus results are omitted for clarity.

Our primary evaluation metric is Word Error Rate (WER), as widely used within the speech recognition community. Word Error Rate is computed by counting the number of substitutions, insertions, and deletions in the transcribed text, and dividing by the total number of intended words. For example, the intended phrase “Add apples to my grocery list” may be spoken “A-(dd) a-(dd) add apples to my grocery list” and be recognized as “had had balls to my grocery list”. With two insertions and one misrecognized words, the WER is 50.0%.⁸ For some experiments, we also look at Thresholded WER, as used in work by Project Euphonia [33, 65], to assess VA performance for people with speech disabilities. This is computed as the percentage of utterances, per person, with WER below 10% or 15% (as specified). These values have been suggested as potential minimums for VAs to be useful depending on domain. Lastly, we look at Intent Error Rate (IER), which captures whether the VA carries out the correct action in response to an utterance. To compute Intent Error Rate, we run the ASR output on the NLU model from Chen et al. [16] which also relied on models used by the Apple Speech Framework. ⁹

Baseline ASR. Table 3 shows baseline ASR performance. Across participants in both phases, the average baseline WER is 19.8%, which is much higher than the ∼ 5% reported for consumer VA systems [62, 68, 69]. The WER distribution is also highly skewed, with many participants having low WERs but also a long tail of participants with much higher WERs. For people with mild A&H severity, the average WER was 4.8%, which is similar to what is expected for people who do not stutter, whereas moderate and severe had average WERs of 13.6% and 49.2%, respectively. Moreover, 84.0% of participants with mild severity had a WER of less than 10% (i.e., Thresholded WER) and the average Intent Error Rate for this group was 4.9%, suggesting that many people with mild A&H severity ratings would likely be able to use off-the-shelf VA systems, which echos the VA usage reported in survey presented in Figure 3(left).For moderate and severity grades, Intent Error Rates are 7.3% and 18.4% respectively. Overall, this analysis both confirms the ASR accuracy difficulties described in the survey, as well as reflects the varied experiences that survey participants reported in how well VAs understood them (Figure 2).

To understand how different types of dysfluencies affect WER, we examined the Phase 2 data, which includes detailed dysfluency annotations (see Section 4.1 for rates of each dysfluency type). Note that Phase 2 only included participants with moderate and severe A&H ratings, so on average has somewhat higher WER (25.4%) than the full set of participants (bottom row of Table 3). For these Phase 2 participants, we found high Spearman’s rank correlations⁷ between WER and part-word repetitions (r(39) = .85, p < .001) and between WER and whole word repetitions (r(39) = .60, p < .001). The correlations were not significant for prolongations (r(39) = .3, p = .056), blocks (r(39) = .21, p = .181) or interjections (r(39) = .15, p = .347). This indicates that part-word and whole word repetitions tend to increase word error rates, and that blocks, prolongations and interjections have less of an impact even if they are frequent.

Among all errors the ASR system made in Phase 2, 80.9% were word insertions, 17.5% substitutions, and only 1.6% deletions. The high rate of word insertions has a strong correlation with part-word repetitions (r(39) = .73, p < .001) followed by whole word repetitions (r(39) = .60, p < .001). This echoes survey reports that part-word or whole word repetition can lead to misrecognized words being inserted in their transcription. A trained speech-language pathologist characterized insertion errors from part-word repetitions and found that in many cases insertions come from individual syllables being recognized as whole words (e.g., the first syllable in “become”, vocalized as “be-(come) be-(come) become”). In contrast, a sound repetition on /b/ in “become” is less likely to lead to word insertions. Furthermore, some people demonstrated part-word repetitions between syllables in multi-syllabic words, such as the word “vocabulary” may result in spurious insertions, such as “vocab cab Cavaleri.”

ASR Decoder Tuning (Intervention #2). Consumer ASR systems are commonly trained on thousands of hours of speech from the general public; an amount far larger than can likely be obtained from PWS. However, an initial investigation by Mitra et al. [51] on 18 PWS has shown it may be possible to tune a small number of ASR decoder parameters to improve performance for PWS. Here, we validate this approach on our larger dataset and find even greater gains when tuning on our 50 participant Phase 1 subset. While we defer to that paper for details, in brief the approach increases the importance of the language model relative to the acoustic component in the decoder and increases the penalty for word insertions. These changes reduce the likelihood of predicting extraneous low-confidence words often caused by part-word repetitions and bias the model towards more likely voice assistant queries. We used Phase 1 data to tune these parameters and report results on Phase 2.

The average WER for Phase 2 participants jumps from the baseline of 25.4% to 12.4% (SD=12.3, median=6.1, IQR=13.1) with the tuned decoder, which is a relative improvement of 51.2%. A Wilcoxon signed rank test shows that this improvement is statistically significant. See Table 4 for more metrics and to understand how WER improves as a function of dysfluency types. For example, the WER lowers (improves) in 43.2% of utterances with part-word repetitions and increases (worsens) 3.9% of the time. For ASR tuning, WER improves the most in utterances that have whole word repetitions, part-word repetitions, and interjections and least for those with prolongations or blocks.

According to our survey, PWS are often displeased when seeing repeated words, phrases and filler words in their dictated notes and texts. To address this issue, we refine the transcribed text using two strategies. First, we look at filler words such as “um,” “eh,” “ah,” “uh” and minor variations. Many of these filters are not explicitly defined in the language model (i.e., by design “eh” is never predicted), however, in practice, short fillers are frequently transcribed as the word “oh.” As part of our approach, we remove “oh” from predictions, unless “oh” is used to represent the number zero. We considered others such as “like” and “you know,” which are common in conversational speech, but they did not appear as fillers in our dataset, likely because the utterances tend to be short and more defined than free-form speech. Second, we remove repeated words and phrases. This is more challenging because words may naturally be repeated (e.g., “We had had many discussions”). In our refinement approach, we take all adjacent repeated words or phrases in a transcript and compute the statistical likelihood that they would appear consecutively in text using an n-gram language model that is similar to [34]. If the probability is below a threshold,¹⁰ i.e., \(\mathrm{P}({\rm\small {SUBSTRING}}_1, {\rm\small {SUBSTRING}}_2) < \tau\), then we remove the duplicate. Note that these two strategies—interjection removal and repetition removal—can be applied to any ASR model output, thus we evaluated our dysfluency refinement approach in combination with both ASR models from Section 4.2.2: the baseline model and the model with the tuned decoder.

As shown in Figure 4(b), dysfluency refinement reduces WER for the Phase 2 data compared to both the baseline and tuned ASR models. The average WER goes from a baseline of 25.4% to 18.1% (SD=19.2, median=6.1, IQR=22.8) after dysfluency refinement—a 28.7% relative improvement on average. This improvement is significant with a Wilcoxon signed rank test (W=839, Z=5.43, p<.001). As shown in Table 4, across the entire Phase 2 dataset, 64.7% of utterances that contain whole word repetitions see a WER improvement and only 0.3% regress. For those with interjections, WER improves in 30.9% and regresses in none of the utterances.

Applying dysfluency refinement to the output from the tuned ASR model further reduces WER, from on average 12.5% with the tuned model alone to 9.9% (SD=8.8, median=5.3, IQR=10.3) for the tuned model plus dysfluency refinement. Compared to the baseline ASR model, this tuned model plus dysfluency refinement combination is an average 61.2% improvement across participants. Wilcoxon signed rank tests show that these improvements are statistically significant both when appended to the baseline ASR model output (W=741, Z=5.37, p<.001) or appended to the tuned ASR model output (W=595, Z=5.08, p<.001). The percentage of participants with WER < 10% increases from 48.8% to 65.9% and the average Intent Error Rate improved from 10.4% (SD=10.8, median=5.3, IQR=13.4) to 5.4% (SD=5.8, median=3.1, IQR=7.2). Such improvements may enable a VA to be usable for many of our participants when the baseline is not; for example, P2-21’s WER improves from 40.4% (baseline) to 13.0% (tuned ASR + dysfluency refinements) and IER improves from 19.1% (baseline) to 4.6%.

The above analyses assess the effectiveness of three changes to an existing speech recognition system that require little data from PWS to implement. At the same time, there has been substantial progress in general speech recognition models in recent years, with some accounts claiming WERs of 5% or less using consumer voice assistants [69]. Theoretically, these general improvements could also result in improvements for PWS.

To understand how these changes over time impact performance on dysfluent speech, we use the Apple Speech framework to run all Phase 1 and 2 data through archived ASR models that had been publicly available between Fall 2017 and Spring 2022; these experiments were conducted in Summer 2022. Figure 5 shows WER across all utterances in the dataset every 6 months across the five years. The utterance-weighted WER was 29.5% for the Fall 2017 model, fell consistently for the following eight time periods, and ended at 19.9%. This is a 32.5% relative reduction in WER from the start to the end of the 5-year time span. Differences at each time point may be attributed to what data was used in training, the convolutional architecture, and/or the language model.

For Phase 2 utterances, we further examined changes in how specific types of dysfluencies manifest in WER performance between the Fall 2017 to Spring 2022 models. For part-word repetitions, the WER improved 43.5% of utterances (worse 12.2%), those with whole word repetitions 46.8% (worse 14.5%), prolongations 36.9% (worse 10.3%), blocks 36.0% (worse 9.4%), and interjections 65.8% (worse 10.4%). The improvements with part-word repetitions are especially interesting, because these errors are more challenging to correct using our strategies. See the Discussion (Section 5) for further implications of these findings.