Beyond Repairing with Electronic Speech: Towards Embodied Communication and Assistive Technology

Starting in the early 20th century, Husserl and his student Heidegger began the philosophical enterprise of phenomenology – they both sought to ground and understand the human experience of the world [32]. However, the conception of embodiment features most strongly in the subsequent phenomenology of perception developed by Merleau Ponty [32, 55]. In particular, Svanæs [81] presents Merleau-Ponty’s influence on theories of embodied interaction and relevance to human-computer interaction (HCI) research. In summary, Merleau-Ponty’s framework broke with Cartesian mind-body dualism, arguing that we inhabit our bodies, living with them and through them in a complex social world [26, 27, 55, 81]. He distinguished two notions of the body – the objective body: our size, weight etc. and the lived body: the body through which we act and experience the world [2, 55, 81].

Notably, Merleau-Ponty’s lived body postulates that world experience is fundamentally composed of bodily experience and not just mental cognition [55, 81]. Indeed, daily we live, breathe and experience all manner of subtleties within our lived bodies [2]. Subsequently, Merleau-Ponty’s work has strongly influenced more recent enactivist phenomenologies [88]. Enactivist theory argues that cognition arises from the dynamic interaction between the lived body and the environment [88]. Enabling the possibility for extended minds and that cognition can even depend on environmental resources [15, 80].

Clark and Chalmers [15] illustrated this extended mind thesis (EMT), through a famous thought experiment involving low-tech assistive technology. Within the thought experiment, a fictional character – Otto – has Alzheimer’s, and thus carries an assistive technology in the form of a notebook, which he annotates with critical information to manage amnesia [15, 18]. Just as we regard internal brain processes as constitutive of memory Clark and Chalmers [15] argue that we too should regard Otto’s notebook as a constituent part of his memory [18].

Originally introduced by Ihde [41, 42], post-phenomenology builds upon the contribution of phenomenology by focusing more strongly on the role of technological artifacts and their relation to human experience. Based on the philosophies of Ihde, Verbeek originally taxonomised the different relations technology has on human experience [89, 90]. With the first four based on Ihde’s work, Verbeek argued for ultimately seven relations to technology: embodiment, hermeneutic, background, alterity, cyborg, immersion and augmentation [89, 90].

More recently, Rapp [70] has extended this framework to constitute human relations with wearable technologies. Depicted in Figure 1, his framework of extended intentionality implores designers to look beyond the external properties of wearable technologies and more readily consider internal aspects of interaction [70]. Rapp argues that wearable devices can serve as extensions of our motor and conscious intentionality and can even alter how we relate to the world [70]. In much the same way, assistive technologies can extend intentionality and actively mediate users world relations. Indeed, first-person internalistic experience of using assistive technologies reveals that assistive technology can alter people’s bodily action (e.g., access), environmental experience (e.g., stigmas) and cognition (e.g., self-identity).

Embodied frameworks of human cognition and human relation to technology can serve as a rich conceptual framework for designing and developing assistive technologies as exemplified by Figure 2. Ultimately, ensuring that assistive technologies better support a more holistic view of human conscious experience. Merleau-Ponty [55] originally discussed the embodiment of assistive technologies and relation to conscious perception whilst interacting with environments. He described the first-person experience of a blind man and cane as follows, “the pressures on the hand and the stick are no longer given; the stick is no longer an object perceived by the blind man, but an instrument with which he perceives. It is a bodily auxiliary, an extension of the bodily synthesis” [55, 81]. Accordingly, perception involves the whole body and the cane has become an embodied part of his corporeal schema – facilitating the experience of environments [55, 81].

More recently, Toombs [83] phenomenological account of living with a disability has reinforced these embodied relations with assistive technologies. She reflects on her own experiences as a wheelchair user, “For the person who routinely uses a wheelchair the device becomes part of the body. One intuitively allows for the width of the wheels when going through a doorway; one performs the necessary hand/arm movements to move forwards and backwards without thinking about it... Thus when a stranger pushes my wheelchair without my permission, it is invading my personal bodily space”. For Toombs [83], the wheelchair becomes a part of her body and the tangible hand/arm gestures for interaction become second nature – not requiring significant conscious thought.

Elsewhere, Butcher [12] has described her experience of using her wheelchair in a similar vein, “my wheelchair really is a part of my body; I spend every minute of every day in it, and when it breaks, I am stripped of my independence and my autonomy”. Indeed, her wheelchair becomes a part of her body, intentionality and cognitive experience of the world. In Butcher’s own words – “The characteristics of my wheelchair, the way it moves and the things it allows me to do, are indicative of something alive and human, not a lifeless object” [12]. These phenomenological accounts and Figure 2, emphasise the importance of designing for a more extended view of cognition and experience. Whereby, much closer attention is paid to end-users internal experience as the assistive technology actively mediates their cognitive experience and relations to the world.

Accessibility scholarship has repeatedly emphasised the importance of engaging end-users and stakeholders directly within the design process as essential for developing technologies that will effectively support people living with disabilities [51]. Critically, participatory design provides a method and vision for involving the users of assistive technology directly within the design process [51]. Indeed, due to the embodied relation users have with their assistive technologies it is critical to support end users participation in the design of the assistive technologies – particularly as the technology can serve as an extension of their lived experiences and intentionality.

Researchers have adapted and developed methods that hone on the role of the body to support understanding and communication during co-design activities. For instance, Wilson et al. [92] developed tangible design languages to support people with aphasia’s voice in design. In particular, their rich design activities ranged from short direct tasks to group activities and even testing of high-fidelity prototypes to limit cognitive abstraction [92]. Aware of people with aphasia’s difficulties with verbal communication they supported tangible making activities which considered embodied styles of communicating and critique (e.g., the use of expressive bodily gestures) [92].

Elsewhere, Spiel and Angelini [79] used customised co-design methods to support accessible engagement, collaboration and critique of technology including using the communities preferred communication and cultural conventions. Furthermore, they also recommended acting out with technology – as an effective means for the communication of embodied emotions and mental models [79]. Finally, the tangible and physical action of co-design or self-building assistive technologies has been found to empower co-designers living with disabilities. In particular, Hamidi et al. [33] noted that the participatory design, tangible fabrication and customization of DIY AAC devices fostered bodily empowerment and design for the unique and multidimensional experience of living with a disability in Western Kenya.

In total, we have held 14 co-design activities at the Roberta Williams Speech and Language Therapy Clinic across a 14 month period beginning in July 2022. Co-design sessions ranged between: 12–19 people living with aphasia (avg: 13.7), 3–6 speech and language therapists (SLTs) (avg: 4) and 1 family member (avg: 0.3). Approximately, sessions took 1–2.5 hours. Ethical approval for all research was granted by the King’s College London Health Faculties Research Ethics Subcommittee.

Aphasia is an acquired language impairment most commonly caused by stroke [4, 7] but can be the result of any damage to the language centres of the brain. It can affect reading, writing, speech and comprehension [4, 7]. Often, aphasia is a publicly invisible disability, affecting approximately one-third of stroke survivors yet less than 10% of the population know of the condition [17]. Typically, people with aphasia’s communication abilities vary significantly [4]. For instance, some people with aphasia might find speaking more challenging than writing or vice versa. The number of people living with aphasia will likely increase due to ageing global populations [17] and increasingly people with aphasia face social exclusion and barriers to long-term speech and language therapy [11, 37, 52]. People with aphasia face individualised challenges with technology [56] particularly technologies which remove tangible, physical and embodied interactions [14, 43, 48, 49].

For people with aphasia dedicated hardware and subscription apps remain the most common AAC intervention. Typically, dedicated AAC devices are symbol-based with touchscreen, eye-gaze or manual switches as input e.g., Dynavox, Liberator and Lingraphica [20]. Yet, there has been a substantial growth in mobile and tablet-based AAC applications – the Tavistock Trust for Aphasia provides an extensive repository of iOS and Android options [54, 84]. Most AAC devices take the form of an electronic lexical or symbol-based dictionary, which present navigable grids of images, symbols or text e.g., Proloquo2Go and TouchChat [56]. Most AAC for aphasia operates as an SGD in which the user composes messages before it is then read-aloud by the device¹.

The vignettes were derived from reflective analysis of the researchers extensive engagement with older adults living with aphasia and by following the processes outlined in previous work: [8, 63, 93]. Each vignette is drawn from three data sources: (1) 25 hours of coded video footage from co-design sessions, (2) ethnographic experiences of the first-author’s 300 hours spent observing/volunteering at the aphasia clinic and (3) coded data from exhaustive field notes documenting lived experiences. These data sources were collectively analysed enabling: identification of key quotes from transcripts and clear identification of people’s unique contextual/bodily communication strengths – this categorised data was then thematically analysed to craft vignettes [16]. For each vignette, we then applied the embodied framework to generate key AAC design insights and share relevant co-developed AAC from people living with aphasia. Overall, we employed a qualitative methodological approach, analyzing our data inductively. Vignettes have been very helpful for synthesising the large amounts of field data – supporting triangulation of observations, identification of quotes and themes. The vignettes seek to show the real, unique and embodied experiences of people living with aphasia and not just present abstract representations of people’s communication experiences.

We briefly outline three co-design techniques we frequently employed throughout sessions to develop more embodied assistive technologies. These techniques draw on embodied and phenomenological frameworks of cognition especially Rapp’s [70] internal aspects of interaction. Indeed, these techniques helped support our understanding of co-designer’s internalistic thought-styles whilst co-designing assistive technologies. Ensuring our co-designers had an empowered influence on technologies that would mediate their world relations, shape their cognitive abilities, sense of identity and eventual experience of different environments.

We present four vignettes of communication strengths and challenges from people living with aphasia to elucidate different communication experiences, embodied perspectives and subsequent key insights for AAC devices. All names are pseudonyms.

The embodied framework recognises the importance of AAC working in unison with the: (1) brain, (2) body, (3) environment – yet in these areas AAC has several design shortcomings. For our co-designers, archetypal AAC that generates speech would replace their dialogue and communication strengths. Plus, introduce established problems with AAC/SGDs into their lives including: learning demands [50], exerting operational pressures [13], mobility problems with heavy AAC form-factors [21, 25], public stigmas [58], slow sender-receiver styles of communication [28] and more. In contrast, the embodied model has disrupted prevailing modes of thinking and provided novel design insights that look beyond traditional AAC functionality. We synthesise these insights into three recommendations for future AAC research.

Researchers must be more cognisant that personal identity and experience can be shaped by assistive technology [64, 77]. Detrimentally, assistive technologies can be paternalistic – undermining both users bodily agency and confidence [64, 77]. For instance, previous research found that children with severe, speech and physical impairments (SSPIs) had to resort to rejecting their AAC to ascertain bodily agency and communicate non-verbally [39]. Elsewhere, large AAC tablet form factors were found to hinder users own resourcefulness and agency to travel [19, 21]. Instead, assistive technologies should embolden users view of themselves and enhance their personal and bodily agency. Equally, assistive technologies should be designed for periods of non-use and respectful of the pre-existing abilities of their user.

With the increasing prominence of AI-centered solutions, researchers must be reminded that we are not designing technology to repair bodies [74]. Indeed, AI may improve the naturalness of AAC speech synthesis – yet AAC users can often already communicate competently leveraging their body and the environment [71]. Therefore, a human-centered and interdependent direction must still remain the core focus of future research [6]. Furthermore, it is important to design assistive technologies that more strongly consider the role of the environment and context of usage [6, 75]. Devices may function but cause the user public discomforts and thereby limit overall agency. For example, Parette and Scherer [65] interviewed deaf and hard of hearing (DHH) users that actively avoided using hearing aids due to social stigmas. Whilst, co-design sessions undertaken by Profita et al. [68] noted DHH community members were keen to customise hearing aids and cochlear implants to support personal style and self-expression in public environments. Consequently, assistive technologies could look to embrace users personal style as fashionable technologies [69] or adapt dependent upon environmental context [21].

Embodied assistive technologies serve as an enhancing bodily extension: extending human abilities intuitively, supporting low-latency interactions and ensuring minimal cognitive demands [45]. Previous research has noted that AAC devices can often have too high learning demands – undermining users self-belief and personal confidence [9, 13, 39, 40, 50]. In contrast, we discourage from assistive technologies that burden users with effortful mental tasks in public domains. Therefore, we support the design of assistive technologies that are more immediately intuitive – naturally extending intentionality of the body, engendering feelings of competence and enhancing experience intuitively. Previously, Theil et al. [82] extended deaf-blind users motor intentionality by affording immediate novel sensations for communication through a wearable vest. Meanwhile, Boyd et al. [10] extended conscious intentionality through smart glasses, which discreetly supported the atypical speech prosody of adults living with autism. Participatory design methodologies centered on the internalised thoughts, interactions, competencies and feelings of the user are helpful for co-designing these more intuitive and embodied assistive technologies [79].

Under the more holistic embodied framework, the body and environment play a significant factor in human cognition. Equally, the framework recognises the potential for intimate relations with assistive technologies – viscerally serving as an extension of cognition. In light of these factors, caution should be applied regarding technologies that strongly dis-inhibit from our bodies or seek to replace traditionally more tangible forms of intuitive interaction with physical environments. Especially given the emergence in artificial intelligence (AI) (e.g., [5, 29, 86]), videoconferencing (e.g., [61, 72]) and virtual reality (VR) (e.g., [44, 60, 94]) technologies for communities with disabilities. At all times, the focus must remain on supporting the people and communities rather than the improvement of technology. Previously, Obiorah et al. [62] have emphasised the importance of technologies which seek to augment rather than automate and replace – thereby preserving empowering communal, cultural and social rituals for their communities.