Cultivating Spoken Language Technologies for Unwritten Languages

Ong’s seminal work on Orality and Literacy [48] interrogates the technology of writing and unpacks the ways in which taken for granted aspects of experience are deeply affected by writing and how these do not generalise to primarily oral—rather than literate—contexts and cultures. The subtleties of Ong’s theory are not without critique, especially because of the way it frames signed languages and primarily oral cultures as dependent on or inferior to spoken languages and literate cultures, respectively [8, 15, 26]. However, ICT4D researchers have engaged with the broader arguments of Ong’s work to surface a range of pertinent and practical design implications: for instance, by drawing attention to the ways in which oral thought relies on repetition; how information is structured through additive narratives; and, how abstract categories and complex information hierarchies should consequently be avoided when designing user interfaces in such contexts [67]. Goody’s argument that written language is not merely speech transcription but a mode of thought reproduction [27] is pertinent to our research too. Consider this paper as an example, which not only contains tabular structures (for instance, the author table at the beginning of this paper, which lists names above affiliations and contact details) and hierarchical ordering (sections, subsections, lists, etc.), but also includes subordinate clauses (such as this one) that exemplify this mode of thinking. According to Goody, knowledge and representation are two sides of the same epistemological coin. This has far-reaching implications, as what we might call the epistemology of the written word also influence the metaphors that structure our experience and thinking (see [35]). Consider how Bidwell juxtaposes the literal and figurative translation of “are we walking together?” with “are we on the same page?” in reporting her insights of deep design research with rural, oral communities in the Eastern Cape of South Africa [4]. These consequences are not limited to conversation and writing. User interfaces extend and re-produce written thought too [82], just as databases—dominant cultural forms [40] of literate societies across the ‘hyperdeveloped’ world [72]—are driven by relational, tabular, and hierarchical data structures [20].

Against this backdrop, Speech interfaces in particular have been identified as promising technologies to broaden digital participation of illiterate and semi-literate populations across the Global South [83], especially for those speakers of languages that, like Gormati, do not have a native script. Here Vashistha and Raza [77] give a useful overview of almost 15 years of research, innovation, and impact of Interactive Voice Response (IVR) and Interactive Voice Forum (IVF) applications that—through the widespread adoption of mobile phones—“have found applications in diverse domains and have profoundly impacted underserved communities in low-resource environments” [77, p. 570]. Both IVR and IVF systems are accessed by calling a (typically subsidised) phone number, and then navigating a menu of options using speech commands or keypad numbers. An agricultural extension IVR system, for instance, might allow callers to listen to announcements by pressing 1 or saying ‘announcements’. Rather than purely disseminating information, IVF applications provide callers with the ability to record messages, and in the case of Patel et al.’s Avaaj Otalo [50], allow them to post questions and listen to the questions and responses of other farmers and agricultural extension workers. This social element of IVF platforms has driven the success of related systems such as CGNet Swara (for citizen journalism) [45], Polly (for entertainment and jobs) [58], and Sangeet Swara (for social media) [75].

IVR/IVF research has investigated the efficacy of and caller preference for key press vs speech command input modalities (e.g.,  [50]), but generally do not engage with the nuances of Ong’s orality theory and recognise the ways in which IVR/IVF systems impose a hierarchical ordering system. Vashistha and Raza mention that for caller-generated content it is “difficult to automate categorisation” [77, p. 596] and how such categorisation is necessary for users and IVF operators/moderators to find and access content.

IVR systems in particular [57], as well as smart-speaker installations [60], have been leveraged to develop digital language resources that many minoritised languages lack. Involving and empowering minoritised language speakers in the crowdsourcing of transcriptions has been a further area of HCI research and user interface innovations [60, 76, 78, 79]. However, these systems have to date targeted regional languages with more established writing systems.

Our approach of documenting and engaging community members around their farming practices resonates with the Digital Green initiative that leverages peer learning and participatory video to record small-scale farmers across India on effective practices such as vermicomposting or fertiliser application [25]. However, content that is shared through the organisation’s agricultural extension platform, while popular and impactful, is stored on a database and accessed through a webpage with a hierarchical navigation system (e.g., by language; by category; by sub category; etc.) that favours larger, regional languages that are more commonly written. Particularly when working in indigenous, marginalised and minoritised contexts, Science and technology studies scholars Verran and Christie alert us to the misplaced dichotomy between traditional (i.e., oral) and modern (i.e., literate) cultures:

They key challenge, then, is to devise appropriate and flexible ways of arranging, storing, and finding digital content that are usable for those working within nonmodern cultures and where this “becomes a site, a time and place where young and old, with their varying competencies, work together [ ...] in ways that can empower and educate the young while recognizing older people as knowledge authorities” [80, p. 74].

HCI researchers have partnered with remote, marginalised, or indigenous communities across the globe to design digital technologies to “enact culture in the digital age” [74, p.16] to address specific problems communities face [5] while cultivating sensibilities to cultural and addressing sticky representational issues [44, 70]. Designing with an Aboriginal community, Soro et al. developed a community-notice board with support for both oral and written storytelling, bi-lingual content and different representations of time [70]. Working with Sámi people of the circumpolar north, Moradi et al. designed a web-based digital archive of cultural heritage materials, where a tension emerged around border(less) maps [44]. And in South Africa Bidwell et al. designed two iterations of a community audio repository, to create and share recordings with access to shared tablets, to address “the difficulty local Xhosa people have in communicating between villages” [5, p.227]. While the interface to record audio remained unchanged across the iterations, community members found it difficult to find specific recordings, so the revised interface allowed users to tag a recording with photos and record short, annotating abstracts about the recording. The interface to find a recording displays photo tags alongside the recording and autoplays annotations as the user scrolls through their list of recordings. Difficulty in finding voice recordings, in the form of WhatsApp voice messages, was experienced by both Xhosa participants in South Africa and Marathi participants in Maharashtra, India, in Reitmaier et al.’s study; supporting textual search of voice messages was identified as key opportunity for Automatic Speech Recognition (ASR) systems [59].

Those working in the fields of language documentation [7, 21] and ICT4D [16] have highlighted the pertinence of research ethics. The studies presented in this paper were approved by an institutional review board at Swansea University. We also follow best practices within ICT4D research that emphasise long-term engagement and reciprocity [16], and working with local researchers and organisations [16]; and, the practices of linguistic research [7, 21] into unwritten languages, which proposes to establish informed consent orally [21], as well as to place linguists in control of operationalising storage and access to collected data [7]. In following Brereton et al.’s advice and configuring our research approach for reciprocity and engagement, we supported community members when they visited us in the city or when they asked for support or information not directly related to our research [9].

We also consulted ethnographic accounts of Banjara culture and language elsewhere in India [11, 46] and report on these in the next section. We identified linguistic resources² such as word lists [43] and multi-lingual dictionaries [33]. However, we found that community practices, particularly surrounding writing and transliterating, diverged from ethnographic descriptions and so we did not draw on written resources in developing our system. This also means that our development approach is more likely to be applicable to other contexts where language is spoken, not written, and where knowledge practices are not- or less-influenced by writing systems.

Every morning and at different times throughout the day, we went for walks to the surrounding fields to observe, learn, and participate. By walking with our hosts into the fields, we also followed in the metaphorical footsteps of Gupta, who has been walking with rural communities in India to uncover and share grassroots innovations on topics that range from farming, sustainable conservation, animal husbandry to cooking and recipes [29]. Along the way we stopped to greet and chat with other community members. We asked what they were doing and observed them carry out their work in the field: ploughing, weeding, watering, planting. We captured glimpses of such encounters through photos, and videoed community members demonstrating certain aspects of their work to serve as aide-mémoire and later as a means of communicating these experiences with the wider research team, distilled and presented through slide decks. These activities were not directed towards a particular purpose, but were invaluable later to situate design activities. For instance, we developed an intuition of when people were communicating freely and when the language barrier was creating confusion. We mostly chatted in Hindi, but when we sensed confusion, our hosts would translate. Farming is as intrinsic to Banjara culture as the Gormati language, and our hosts were eager teachers of both. They said that if we stayed a few more weeks we would be able to speak Gormati, as we already were picking up certain greetings, phrases, and names of crops.

Both men and women, old and young, work in the fields. Cotton is the primary cash crop; millet, corn, lentils, chillies, and onions are grown for subsistence. Branches cut from trees planted around field boundaries are harvested as tree-hay and fed to cattle and goats along with stalks from corn or millet. Oxen are used to pull carts and for ploughing. Cow’s milk is usually sold, but goat milk is served with chai. We mostly ate chapatis made of millet with lentils, prepared by women over wood fires.

When it got too hot (~33 °C), we returned to the courtyard and rested under a neem tree. The weather, cycles of day and night, as well as the needs of animals and crops—rather than the clock—created the rhythm of quotidian activities. Both in the fields and in the courtyard, Banjara women often chant poems and sing songs. And as more community members learned that we were interested in Banjara culture and the Gormati language, they would approach us to capture video of them singing a song. When it got dark, and during periods of down time, we composed some of our own songs inspired by the sonority of the voices we heard throughout the day.

Corroborating our direct experiences with ethnographic [11, 46] and linguistic [10] accounts of Banjara culture and language elsewhere in India, we were struck by how these frequently mention transliteration through regional scripts (e.g., using the widespread Devanagari script in Maharashtra). This contrasts with a key finding of our engagements with the community: that Gormati was spoken, rather than transliterated. In fact, the linguistic landscape of the community contained very little writing.⁸

Through discussions with the wider research team across Design, HCI, Linguistics, and NLP disciplines we decided not to pursue lines of inquiry that utilised or surfaced transcriptions or transliterations, mostly because of limited transliteration practices in the community, but also because we wanted our approach to be adaptable to other contexts. Respecting oral practices, rather than imposing tranliteration or writing systems, was a key implication of this phase of research.

These discussions were scaffolded around slide deck presentations containing images and videos we recorded and co-produced while in the community. These slide decks showed the thandas we visited, farming practices we observed, and expressed and communicated glimpses of what we learned about everyday life and the use of Gormati in the community (see [52]).

Our discussions also focused on what Harper [30] refers to as a “marriage of purpose” between users and machines that is sensitive to community needs and context, but is also anchored in an understanding of how technology works and what it might realistically deliver – an equally important consideration given the opportunities and hype that currently surround AI [23].

Given current barriers to digital participation especially for older generations and the increasing adoption of smart phones by younger generations (after the installation of a 4G telephone mast), the key finding of our ethnographic engagements throughout phase 1 was that speech and language technologies could have a role to play in the digital expression and sharing of Banjara insight and culture.Through our engagements we also established trusting relationships with community members. Not only had community members proven to be willing (and patient) Gormati teachers, but upon our departure they also expressed a wish for us to return again and establish a nascent partnership. From the technology side, we had to be realistic about what we could deliver, but also needed data to develop Gormati language technologies. Through discussions across the research team, we agreed on a series of high-level guiding principles:

To alleviate some of the technical complexities of the project and to reduce the amount of required data, given also that there are so few digital language resources in Gormati, we decided to focus data collection to a couple of specific domains or topics (e.g., farming). Rather than collecting open-ended unconstrained speech, that one might use when chatting with a friend, we posited that focusing on specific domains would also involve a smaller subset of potential words (~100), and that gathering 30 hours of speech data within these domain constraints, would provide enough word repetition to develop a basic language model. To implement these guiding principles and to tighten feedback loops, we also decided that members of the broader research team should participate in future community visits .

Returning to the community for five days, as a team of two researchers (one Hindi speaking; one non-Hindi speaking), we settled into the familiar rhythm of accompanying our hosts into the fields and engaging with people as we went along. We again slept on the rooftop of the main house in the courtyard and benefited from being immersed in context and surrounded by community members. We consulted and clarified with our hosts whenever questions emerged. During downtime we wrote up notes, which we also shared with the wider research team, and continually discussed, refined, and reflected on our plans and methods.

Here, we found that the metaphor of how young children pick-up phrases through repeated exposure useful, and would draw on this metaphor to explain how spoken language technologies can make mistakes that can seem childish. We also set up a voicemail box that community members could call and contribute recordings of the different things they did to care for their plots, animals, plants, and equipment. We thought initially that it could serve as a spoken diary that fulfils the domain-constrained vocabulary requirement, but that it could later be queried by community members, for instance, if they needed to remember when something happened. However, when we presented this idea to workshop participants they did not express interest in keeping, and being able to query, such a diary and, to our initial surprise, reported no difficulty in remembering things. However, on post-hoc reflection our surprise likely says more about how we equated memory with written or calendar records, which again shows the deep level at which writing restructures consciousness (see [48, p. 95]).

In preparing for the workshops we worked with one of our hosts to create a consent recording in Gormati that explained that we would be using the data to create an interactive Gormati-language speech-driven application for them. We also created slide decks of the photos we took of different crops and tools, but discussed how we wanted to avoid common approaches to voice user interface development that seek to detect the presence of a predefined keyword (e.g., ‘cotton’) or keyphrase (e.g., ‘watering cotton’) [e.g., 56, 66]. We were wary of reifying photographs into objects and keywords or phrases. On this topic specifically, Ong reminds us that “an entirely oral language which has a term for speech in general [ ...] may have no ready term for a ‘word’ as an isolated item, a ‘bit’ of speech” [48, p. 60]. During the workshops, we instead asked community members to play an adapted form of a Wittgensteinian language game [84] and narrate the doings associated with the things (see [32]) pictured.

So we recorded participants narrating those ‘doings’ – the steps involved in growing the crop or operating the tool pictured in each slide. Here we found a generational divide, whereby older men and women would feel confident in narrating at length, but younger generations were far briefer. Later on we used these narrated slide decks to showcase how an information retrieval system might work. This was done, for instance, by asking participants to say something similar to what they just recorded for the photo of a ox-drawn wagon and then accessing the slide containing that photo through a keyboard shortcut and playing back their recorded narrative. It was difficult to create ‘clean’ recording environment as the courtyard represents a nexus of both (noisy) activities and (inter)family relations. People had to come and go, so we worked with people when they had time and showed interest, but also did not keep them longer if they had other things to attend to. Participants in workshops found it difficult to imagine use-cases or domains other than farming. They did however mention finding songs and accessing religious ceremonies, which typically also included singing.

We discussed and reflected on our in-situ activities with the wider team and decided to steer away from sung content because this is an application area that would likely be too complex for the capabilities of a system in such a resource-constrained context. Given that community members had observed us recording community narrations in the fields and workshop participants had practised creating photo narrations, we decided to utilise mobile digital storytelling software⁹ as a data-collection probe, replicating the process we had started on our laptops. We trained four younger community members on how to use the data-collection probe (see fig. 3), following the process we trialled and refined during earlier workshops. We created a slideshow template on each phone, which contained 30 photos of crops, animals, and equipment.

We encouraged younger generations, who were more adept at operating their Android smartphones, to help older people to record narrations, a practice which ICT4D researchers refer to as ‘intermediation’ [61] and which Verran and Christie identify as a site of inter-generational collaboration [80]. To make the process easier for data-contributors, we also explained that it is helpful to record similar content for the same photos; again using the metaphor of a child learning words and phrases through repeated exposure. We loaned a phone to a young lady, to ensure that female voices are represented, as these are often missing from IVR datasets [77]. The three young men used their own Android devices. We paid data-contributors and erred on the side of generosity to ensure that the amount was appropriate and commensurate, and also covered airtime expenses: 4000 rupee (~$50) per contributor.

We showed data-contributors how to export narratives from the digital storytelling software and how to share these with the research team. Making this step explicit made sure that we did not accidentally collect data that was not intended for us (e.g., if they were using the software for other purposes). We shared the consent recording from the earlier workshop with the data-contributors and explained how they needed to obtain consent if they recorded someone we had not already obtained consent from earlier.

We stayed in contact with the data-contributors and host family after leaving the community and collated data in batches. While the digital storytelling app exports made it easier to link recorded audio to specific photos across multiple phones and data-contributors, the order of photos in our presentation template meant that data-contributors were creating more narrations for photos that appeared towards the start of the slide deck and fewer for photos that appeared towards the end. In total we collected 3h43m of spoken-language annotations of 30 photos (see fig. 4).

Meanwhile, in the lab, we (NLP researchers & linguists) reviewed technical literature about spoken content retrieval to find a solution that would allow us to respond to this challenge and build an interactive demonstrator system using only 3h43m of training data.

The most straightforward method for spoken content retrieval is keyword search . In this method an automatic speech recognition (ASR) system, trained on manually transcribed data, is used to generate several alternative transcripts of each recording. These transcriptions of all recordings are then used to build a search index, which can subsequently be used to search for a given keyword or keyphrase [2]. The small amount of training data poses a central challenge to building a high-quality ASR system. Applying a multilingual phone recogniser trained on data from well-resourced languages [24, 39], can remedy this problem.

We decided to split our information retrieval system into two components: a phone recogniser and a ranker¹⁰. We used a multi-lingual phone recogniser trained on transcribed speech data from well-resourced languages to transcribe queries and captions into phone sequences. We then trained the ranker to predict how these phone sequences correspond to each photo. Through this decoupled architecture, we were able to rapidly prototype our system and bootstrap the voice user interface with almost zero hours of spoken content in the target language. Furthermore, it allowed us to quickly and iteratively update the system, in anticipation of community members contributing more data during phase 3 of the work.

When we were satisfied with the performance of the IR system in lab conditions, we deployed it as an API so that it could be used and evaluated in the Banjara community. For this purpose, we also created a probe to interact with the IR system through the API, implementing this as an Android app. The interface (see fig. 5) was kept deliberately simple: after speaking a query the user is shown the top-ranking photo result as generated by our ranker. The user can then rate the result using either the green checkmark or red X-mark buttons, after which the next photo in the ranked list is displayed. After rating all results the user can create a new query.

Before arriving in the community, we ran an evaluation trial with a community member who had recently migrated to the nearest major city (Mumbai). We brought a phone with the probe app installed as well as printouts of the 30 photos listed in fig. 4. We asked the participant, who was previously a data-contributor, to complete two tasks:

On the second day of our visit we recruited eight community members (4M, 4F; aged 20–50), who had not been part of data-contribution during phase 2, to experiment with and evaluate the system. We decided to conduct evaluation sessions inside one of the homes, back-to-back, and all on the same day to minimise community members consulting with one another about the task and how they created queries. We went through ethics and consent with participants as well as introducing the system, what it does, and how it operates. We kept the corpus of photos out of view from participants and then showed an individual photo from the corpus, asking participants, as in our earlier evaluation trial, to say a query that would bring up that photo in the app. We then rated the returned photo results using the rating buttons on the photo results screen (see fig. 5). After completing the rating step, we showed participants the next photo and asked them to record a query for it.

In some cases we had to retry a photo query: because participants were still thinking about what to say after we had already started recording; because someone had entered the room and was talking while recording; because the goat tied to the front of the house was bleating to be fed; or, because host family members brought in chai. We noted these interruptions by recording comments on the results screen.

In articulating her alternative account of the relations between plans and situated action in the context of scientific research, Suchman found that the experimenters’ expertise lay not in strict adherence to plans and protocols but in being able to continually adapt by drawing on plans as a resource for action [73, p. 185]. During the evaluation sessions, we too found it necessary to adapt. Drawing on the trust and intuitions we formed over the course of three visits, we could sense that early participants were getting tired during evaluation sessions, especially since we had to ask them to repeat queries to compensate for interruptions by people or animals. So, we decided to accommodate participants by excluding photos with less than two minutes of audio moving forward, and hence limiting our evaluation to 19 photos. However, in the moment we mixed up two of the photos and excluded the photos of ‘Chillies’ (437s) and ‘Wild Berries’ (282s) by accident and included the photos of ‘Stove’ (60s) and ‘Dried Dal’ (18s) instead. The green columns of fig. 3 show the 19 photos that formed part of the evaluation dataset, while the red columns indicate the long tail of photos we decided not to trial because they had so little data associated with them. A further accommodation we made was to show the photos on a laptop, because older participants had difficulty seeing the printed photos inside the dimly lit homes. Changing this on the fly, we could no-longer rely on simply shuffling the photos in the deck to randomise the order, nor rely on removing cards from the deck to keep track of which ones we had shown to participants. While most photos were shown to between five and eight participants, three photos (‘Dried Dal’, ‘Corn’, & ‘Stove’) were only shown to one or two participants. Despite occasional interruptions and unanticipated accommodations, however, overall we settled into a steady rhythm of querying and rating the returned photo results.

After returning from the community, we iteratively cleaned the data generated during the system evaluations by first removing those queries which contained an audio comment to mark them as excluded (e.g., for testing, needing to be retried, etc.) or was inaudible because the microphone was blocked. Next we marked the remaining 99 queries which contained audible speech for inclusion. Figure 6 shows the distribution of results of the 99 queries that participant-evaluators created on the app in response to being shown one of 19 different photos. The dotted blue line indicates our target of returning the corresponding photo as a top-5 result. Across the 19 photos there were 73 queries that returned the corresponding photo on the app as a top-5 result. The remaining 26 queries did not meet our target threshold.

Subsequently, we recruited a community member to assist with translating a random sample of 40 queries into Hindi using voice recordings. We further transcribed and translated the Hindi audio into English.

Consider our worst results for ‘Pomegranate’ (19^th) and ‘Onion’ (18^th). In the ‘Onion’ example, the participant formulated a query surrounding the replanting of field, as the photo showed a freshly cultivated field that had recently been planted out to onions (see fig. 5). It is likely that the IR model picked up the words surrounding fields and planting, which would overlap with the annotations of the millet photo, which was incorrectly returned as the top result for that particular query. The ‘Pomegranate’ example also featured a descriptive query about how the fruit on the bush of the photo looked ready for harvest and selling at market. However, the photo of ‘Wheat’ was returned as the top result. Although these queries produced outlying results, other queries for pomegranate and onions produced top-5 results on five and three occasions, respectively. This style of fluid and descriptive querying is furthermore representative of the larger query dataset, which did not contain keyword or keyphrase queries (e.g., for ‘the pomegranate photo’).

We leveraged the interactive demonstrations that our probe afforded to engage and involve eleven community members (7M, 4F; 18–68) in the co-creation of media retrieval use-cases that are more appropriate in oral contexts. These occurred during two workshops across two days, one focused on current information seeking practices and the other on uncovering use-cases that support and extend these practices with more useful content than the photos of the current probe. Before, between, and after workshops we experimented with different content generation approaches as well as refining the ways of collecting audio annotation data to drive the IR system with the same participants.