From Inanimate Object to Agent: Impact of Pre-beginnings on the Emergence of Greetings with a Robot

Focusing exclusively on the moment at which the robot appears¹ to participants for the first time, HRI studies and datasets collected in controlled or natural settings can, at first glance, be sorted into two general categories.

The two types of HRI pre-beginnings described above make relevant an earlier distinction made by J. J. Gibson in his ecological psychology, regarding the way humans may appear on the social scene, and gradually achieve participant status in the pre-beginnings of encounters. In co-present encounters in relatively uncluttered spaces, co-participants usually get into a greeting position progressively, relying on the way they move, their gaze and gestures to continuously coordinate their getting-together, and make relevant interactional moves such as distant greetings [42]. Gibson calls this type of appearance a “coming into sight” [24]. This is the most common configuration in co-present encounters. He opposes to this another type of appearance, in which the other person seems to materialize or come to life suddenly in the situation, as when someone hidden by features of the local environment suddenly becomes visible, which Gibson calls “coming into existence” [24] to allow for the “pop-up,” quasi-instantaneous character. Other examples of “coming into existence” would be the initial connection in a video call [50], or in a co-present encounter, someone sleeping who suddenly wakes up after being approached. Because of its suddenness, the exact moment of “comings into existence” can be difficult to anticipate, the causes and underlying processes for such a “coming into existence” are not apparent, and finally, the interactional status and competence of the potential co-participant at the moment of its “coming into existence” can be uncertain.

Gibson's distinction may be highly applicable to HRI, for it now appears that in the pre-beginnings of encounters of the first type of studies mentioned above, more or less prepared subjects have to deal with a robot that “comes into existence,” while in studies of the second type, the robot may seem to “come into sight” and allow for some form of embodied mutual co-ordination in the pre-beginning phase. However, in most cases, studies neither analyze nor clarify the state of the robot when participants see it [4] or enter in physical co-presence with it. Indeed, as “most experimental studies only start when the human is already placed in the appropriate starting position in front of the robot” [33], methodology sections rarely cover the observable behavior of the robot when participants encounter it. HRI experiments display an orientation to the “opening” phase of the interaction as the first relevant moment and tend to set aside the “pre-beginning” phase, although, depending on the scenario, it may be crucial to the way subjects and robots achieve some form of co-participation status. These very first seconds, during which the robot appears to participants, are often, so to speak, “off-the-record” in the data that ends up being collected.

Here, we will analyze a relatively common HRI experimental setup in which subjects are brought in the presence of a robot that suddenly animates and “comes into existence.” Thereby, it imitates the interaction opening delays regularly observed in laboratory or “in-the-wild” human-robot interaction studies, where robots can require time before springing to life after they stand in co-presence with a human. We will show how this is consequential with respect to the way in which openings unfold, in which some moves such as greeting and waving may become interactionally relevant, and, ultimately, in which the robot emerges as a social agent. We conclude that pre-beginnings are not anecdotal or peripheral issues with respect to HRI experiments but should be thought about and designed as an integral part of those.

Ethnomethodological Conversation Analysis (EMCA) [22, 58] is a micro-sociological approach that studies the temporal unfolding of events in an interaction to understand, on a moment-to-moment basis, how participants’ “actions—turns-at-talk, embodied actions, or complex multimodal moves—are produced and recognised as performing meaningful social actions” [6, 96]. EMCA offers methodological tools for understanding what is treated as publicly relevant by participants in a given situation—among the potentially inexhaustible number of features of this situation describable from an external perspective—without imposing the researcher's perspective on the data: It adopts an emic point of view rather than an etic point of view [96]. In sum, EMCA will consider “as phenomena only those practices of members which are used by them to produce, accomplish, sustain, reproduce, recognize, and give account of, to and for themselves, social order” [70].

This angle of analysis treats social order as a product of the local organization of participants: that is, as continuously maintained (or modified) through their local accomplishments [22]. EMCA multimodal approaches rely on the minute analysis of video recordings [29] and of their detailed transcriptions to identify the fine-tuned temporal unfolding [57] of participants’ actions with a much higher degree of precision than mere observation would allow [57]. The in-depth analysis of large collections of data can subsequently reveal recurring patterns in the procedures and methods through which social order is accomplished [28, 66].

When applied to HRI more specifically, EMCA is suited to identify what, in a robot's multimodal behavior (its talk, gestures, flashing LEDs, sounds signals, motor noise, etc.) [61] is treated by involved participants as social actions: that is, as actions that make relevant “a set of potential next actions” [96]. This becomes apparent when these behaviors from the robot are responded to (and in a certain way) by co-present participants, in the following turns-at-talk. In other terms, EMCA's micro-analytic level of description can be leveraged to explore what emerges as pragmatically consequential when humans’ and robots’ actions intertwine and respond to each other (often in ways unforeseen by designers [60]), i.e., to retrospectively unpack how humans and robots co-constructed the interactions [68] that ended up being captured on camera. Therefore, researchers drawing on EMCA will focus on behaviors produced by a robot that are made publicly recognizable and accountable (as an action of a certain type, e.g., an offer, a question) by co-present humans, as these humans are immersed in a situated activity, where they face specific practical problems [31]—e.g., whether to respond to a robot producing a “waving gesture” at the start of an interaction and, if so, how to respond to it. In this sense, the methodological tools of EMCA can be mobilized by researchers faced with the issue of analyzing robots’ and humans’ micro-adjustments over time as something else than an inscrutable “black box” [31].

A result of this approach is to study the “sociality of robots” independently from mental representations. Our analytical focus will be exclusively limited to what is oriented to by participants as an “action” from the robot, i.e., the way in which some of its “behaviors” are responded to or, alternatively, what actions from the robot are accountably displayed as absent by humans. In particular, an ethnomethodological and conversation analytic methodology does not aim at establishing how the robot is otherwise mentally represented [26] by the participants (as a social agent, as an object, as a human, as an animal, as a new ontological category…) or if they engage in pretense towards it [87]. Unless it is made observably accountable, it makes no difference for the status occupied by the robot in the interaction that participants are “behaving as” or “behaving as if” this robot is a “subject with internal states and perceptual experiences” [87]. Similarly, this analytical perspective is independent from the set of questions related to whether participants' actions involve a mental representation of the robot as they interact with it, or if these behaviors are produced as part of a non-representational “mindless coping” with the situation [14].² In other words, EMCA takes an agnostic stance regarding cognition [37, 54], as it is interested in participants’ (humans or robots) “observable and hearable conduct […] at the interactional surface” [66].

After laying her coat on the chair, the participant gazes at the robot and positions herself in front of it, facing it (L.1). Standing at 0.8 meter from the robot, she is comparatively close to it in regard to other participants (average initial distance = 1.3 meter; SD = 0.26). She immediately initiates a first greeting pair and an address term (“Pepper,” L.2) and maintains her gaze, body orientation, and posture—slightly leaning towards the robot—during the next 3.6 seconds. After this standstill, she produces a lateral head tilt for 3.8 seconds (L.3), while maintaining her gaze towards the robot's eyes. A few seconds later, the robot directs its face towards the human's eyes, establishing mutual gaze. This quick adjustment of the robot's head is associated with motor noise and plastic sounds (L.4). These so-called “consequential sounds” are known to be regularly oriented to by participants in human-robot interactions [20, 85, 94]. The robots’ arms also start shaking lightly (L.4), which will persist until the end of the experiment. After 3.5 seconds of this mutual gaze, the participant raises one eyebrow (L.5) and, after a second of silence, produces a new greeting (“hello Pepper,” L.6). This action appears to account for the robot's continued silence and, in particular, to orient to the robot's alignment with the human's gaze as creating expectations for a next action on its part. However, the participant's greeting is immediately followed by a greeting from the robot (L.7). This implies that, in this fragment,⁶ the robot's greeting turn is sequentially positioned as a second pair part responding to the participant greeting—who, indeed, does not achieve a new greeting in return (L.8), suggesting that she orients to the robot's “bonjour” as a reaction to her own. After a short pause, the robot then starts to produce a greeting wave (L.8), with which the participant aligns by producing a similar wave. Once the robot starts to retract this waving gesture, the participant immediately begins to lower her own arm and finishes her retraction simultaneously with the robot.

The previous fragment constitutes one of only three openings in our corpus where a greeting is initiated by the participant before the robot was activated (from an etic perspective). All along the interaction, the robot is constructed as a co-present interlocutor, even when it is not moving yet. Besides the “bonjour” that the robot produces after her greeting, the participant orients to every “activation step” displayed by the robot (physical co-presence, mutual gaze, and wave) as opportunities to produce a first greeting. Thus, as soon as she is positioned in front of the robot, she treats the situation as a “framework in which a greeting sequence is relevant” [59] and the robot as able to react to the production of a greeting: by tilting her head⁷ (L.3) and by raising one eyebrow (L.5), she makes accountable the non-answer of the robot to the first pair parts she produced and she displays expectations for a reaction.

Remarkably, even though the participant's last verbal greeting (which took place immediately before the robot's vocal greeting) ultimately positioned her as having “initiated” the greeting sequence (L.6 & 7), she instantly re-positions the robot as the “anchor” [83] or, more generally, as the speaker initiating new sequences. Indeed, after the robot greets her back, the participant stays silent (L.8) and does not use her position as a first speaker to self-select [76] and to initiate a new sequence. This silence by the participant leads the robot to produce an “interlocked” turn [82] that combines both its response to her “hello” and its initiation of a greeting wave (L.8). By staying silent, the participant thus provides the robot with the adequate position to initiate subsequent sequences [82] and, later, to initiate the topic of the interaction (not in transcript).

After positioning himself in front of the robot, the participant does some self-grooming [42] as he readjusts his clothes (L.1). Standing at 0.8 meter from the robot, he stands closer to it than most participants. After the robot orients its face towards his eyes (L.2), with the matching motor noise and squeaking plastic sounds, the participant gazes back at it and produces a tongue smack (“tsk,” L.4) followed by a first greeting (“hello,” L.6), interrupting his self-grooming (L.5). He then takes a step forward while staring at the robot and, after 2.1 seconds of silence, takes a step back to his original position (L.7). After 2.8 additional seconds of silent mutual gaze, the robot utters a greeting term (“bonjour,” L.8) as the participant just finished taking his arm outside of his pocket. The participant briefly extends his hand toward the robot (L.9), then retracts it and produces a return greeting (L.10). The robot starts to raise its arm to prepare a waving gesture (L.10, image 2.6). When this gesture has not reached its apex yet, the participant starts to extend his own hand towards the robot (L.11, images 2.7 and 2.8); however, once the robot's arm becomes fully extended and starts the waving motion, the participant redirects his arm to produce a waving gesture (L.11, image 2.9). He simultaneously achieves a new verbal greeting and smiles (L.12).

In this fragment, physical co-presence is not treated as sufficient to initiate the interaction—unlike the previous example. The robot is not oriented to as a conversational partner from the very start. This is especially visible through the production of “self-grooming” by the participant, usually displayed during the approach between two interactants [30]. However, the status of the robot in the interaction shifts after the establishment of mutual gaze. The participant's interruption of his self-grooming (L.5), his greeting (L.6), and the step forwards he takes (L.6) accentuate a shared inner space [42] and display the expectation of an imminent action from the robot. This reconfiguration results from the “crucial analytic distinction” [59] made by the participant about what the gaze from the robot is projecting: It is not oriented to as a merely automatic “gaze tracking,” nor as a “mere look” [45, 59], but as a look projecting the initiation of an upcoming action. The participant's expectation is not met, however, as he goes back to his original spot. Mutual gaze, therefore, constitutes the first “breaking point” after which the robot becomes (momentarily) present as a potential interlocutor. As Reference [66] notes in the case of young children interacting with a toy robot, even “little sequential phenomena of the robot's timely conduct” in relation to participants’ actions can have a critical impact on the “categorization and re-interpretation” [66] of this robot. In our fragment, the redirection of the robot's gaze towards the participant after a silence is treated as a meaningful social action.

Incidentally, two reconfigurations could be observed in this participant's gestures. First, L.9, he extends his hand towards the robot immediately after its first verbal greeting, before retracting this hand and producing a return “bonjour.” The cancellation of his tentative gesture and, instead, his production of a verbal greeting, appear to constitute alignments with the robot's (then verbal) mode of greeting. Later, as the robot starts to visibly raise its arm as part of its waving gesture, the participant's response gesture shifts from an apparent “handshake” gesture to a clearly observable wave (L.11 to L.12; images 2.4 to 2.7). These two episodes display quickly evolving interpretations of what action the robot is projecting during its first greeting and, later, during the preparation of its waving gesture. It highlights an online monitoring of the robot by the participant [68], which allows him to reconfigure his embodied course of action to align with the robots’ co-occurring action.⁸

After entering the room, the participant positions herself in front of the robot, at a higher distance than most participants. She starts to swing from one leg to the other while looking around the room (L.1). After a few seconds, the robot gazes at the human's face and, doing so, produces motor noise and squeaking plastic sounds (L.2). The participant instantly stops her swinging movement and initiates mutual gaze with the robot, while raising an eyebrow (L.3). She maintains this posture for the next 4 seconds of mutual silence, and even after the robot utters a first greeting (“bonjour,” L.4). Because of a momentary processor overload, the robot's waving gesture takes 3.7 seconds to be triggered after its “bonjour,” unlike the rest of the corpus where it took on average 3 seconds. After 3.5 seconds of silence, possibly orienting to the silence and the stillness of the robot as offering a response slot, the participant softly whispers a first greeting (L.6) with a rising vocal pitch—right before the robot starts its wave. Once the robot initiates its waving gesture (L.6), the participant glances towards its waving arm (L.7) then utters a new greeting (“bonjour,” L.8). This new greeting is uttered out loud and with a final continuing intonation, while the participant widens her smile (L.8, image 3.4).

In this fragment, the participant orients to the first greeting of the robot as sequentially equivocal [35]. Her first greeting (whispered, delayed, with a rising pitch) displays uncertainty regarding what the robot's greeting is projecting: The robot's “bonjour” is not oriented to by the participant as clearly constituting the first part of an adjacency pair that should be completed by a return greeting. The design of her first greeting thus appears to question the status of the interaction—and even the existence of a “stepwise process of mutual adjustments” [67]. Conversely, the participant's second greeting turn appears to confirm “what is going on” [6] as an “exchange of mutual greetings”: Uttered out loud and simultaneous with a widened smile, it orients positively⁹ to the robot's gesture as initiating a second greeting sequence.

This supports an interpretation where each greeting produced by the participant accomplishes a different task [55]. The first greeting, uttered 3.5 seconds after the robot's own greeting, mainly checks the availability of the robot and its ability to perceive and react to the human's relevant actions¹⁰—a form of “device testing” [71]—whereas the second greeting is a clear ratification of the start of the co-present interaction: It is a “sociability practice” [55]. Consequently, we observe a form of inertia in this fragment: The inanimate object that the robot is first oriented to requires interactional work (lasting over several seconds) to be replaced by a conversational agent. The first greeting term produced by the robot does not immediately nor automatically institute it as a conversational partner that can be greeted back.

The participant positions herself in front of the robot, facing it. However, standing at 1.8 m from it, she is among the two participants who positioned the farthest. She looks around the room (L.1) for a few seconds, then switches her gaze to the robot. At the same time, the robot's head aligns with the participant's eyes, which triggers motor and plastic noises. This pose is silently maintained for the next six seconds by the participant and the robot (L.3). Then, the robot produces the greeting term “bonjour” L.4 while opening its arms, with its palms facing the ceiling. The participant produces no response, whether vocal or gestural. After 2.3 seconds of silence, the robot initiates a “wave” (L.5). When the wave reaches its apex (i.e., the robot has fully raised its hand), the participant produces a first verbal greeting (“bonjour,” L.6) and a smile at the same time as she initiates a waving gesture—fast enough to catch up with the robot's own gesture. The participant's wave stays synchronized with the robot's wave, then stops immediately after the robot starts retracting it (L.7): 200 milliseconds after the robot's arm starts to lower, the participant also starts to retract her arm—as in fragment 1. Like the overwhelming majority of our corpus, this participant's gaze focuses on the robot as soon as it moves its head to track her gaze—but she does not immediately produce a speaking turn. The lasting silence, mutual gaze, and “consequential sounds” are not oriented to as initiating a “slot” where to self-select. Even after the robot utters a “bonjour,” she returns no greeting and maintains her previous pose and gaze for the next few seconds.

However, once the robot starts a waving gesture, the participant silently observes its arm rise during the action's preparation. She then abruptly produces her own wave—which catches up with the robot's gesture—and simultaneously produces a smile and a verbal greeting (L.6). The speed of this return wave seems to indicate that the participant orients to the robot's gesture as, either, producing a normative obligation to achieve a return greeting, or, alternatively, as upgrading a normative obligation to respond that she would have previously failed to observe. In particular, based on the numerous occurrences of this situation in our corpus, we suggest that, in this fragment, the participant's hasty first greeting displays her alignment as normatively expected at an earlier point in the interaction. That is, she orients to the robot's wave as a second greeting sequence, which reinforces the conditional relevance attached to its first vocal first greeting (“bonjour”), to which she did not answer. The robot's wave is treated as making accountable the participant's non-answer to this first greeting sequence.¹¹

After entering the room and dropping her bag on the ground, the participant gazes at the robot (L.1). As she starts approaching it, she produces a speaking turn involving a deictic reference to the robot as “this thing” (“ce truc,” L.2)—referring to it in the third person—and qualifies it as “weird.” This comment displays most of the typical properties of “self-talk” [41]: It is achieved while the participant is leaning forward, her body not oriented towards the robot, and part of it is uttered while looking at the ground, in a low voice. As a consequence, the participant does not manifest any expectation for an answer; her comment does not open a “conversational sequence” [41]. Once her approach is complete, the participant stands at 1 m from the robot, closer than the average initial distance of 1.3 meter for all participants. After a silence of 3.6 seconds, the robot shifts its gaze towards her eyes—doing so, it produces soft motor noises and squeaking plastic sounds (L.4). The participant reacts with a short laugh (L.6). After a new silence of 4 seconds, the robot initiates a first greeting (L.8) and opens its arms—palms towards the ceiling. The participant produces another laugh, more audible and longer than her previous one (L.10). This laugh is continued during most of the hand wave of the robot and is followed by an in-breath when the robot's arm starts to retract (L.10).

The actions of this participant highlight a double orientation to the robot, as both normatively neutral and unable to react to human actions. First of all, she treats the robot as an autonomous script whose verbal utterances imply no normative obligation to be responded to, even after it produces a greeting term: No conditional relevance emerges from the behaviors produced by the robot in the course of the interaction. Yet, simultaneously, the participant orients to the robot as unable to respond to (or to perceive) her own actions. The “self-talk” (L.2) or laughs (L.6 and 10) she produces in front of the robot do not manifest any expectation for an answer, and the absence of reaction from the robot is not visibly made accountable (unlike fragment 1). Her turns are not “recipient designed” to be registered as “inputs” in response to which the robot would produce, reconfigure [25], or interrupt speaking turns. In other words, the robot is never “characterized as being able to perform reciprocal interactions” [91], which is a prerequisite for the existence of a “social encounter” [91]. Neither the robot nor the human imposes a normative order on the other: There is no observable “sequence organization” [82] that exerts a constraint on their actions. Using the previously mentioned terminology from Reference [37], this participant is merely “using speech” but not “talking” with the robot, in the sense where “talking” would imply to produce “actions that are discoverable within a normative order” [37] and to assume “other participants to be able to perceive them as actions within that order” [37].

Additionally, no facework is achieved by the participant as, in particular, her comments are not fully whispered (L.2) and her laughs are achieved audibly and visibly while standing right in front of the robot. Her first utterance (“this thing is really weird”) also explicitly characterizes the robot as an object and refers to it in the third person. As a whole, semantic content and sequential organization mutually reinforce to characterize and position the robot as a non-agent: The participant establishes herself as the spectator of a pre-recorded monologue, whose performer is not socially present with her in the room.

Last, we note that even though this participant approached the robot more than average (i.e., stood at less than 1.3 meters) this unusual proximity was part of a sequence where she scrutinized and commented on the robot, treating it as an inanimate object instead of an interactant. This is unlike participants in fragments 1 and 2—even though they also stood unusually close to the robot—for whom this proximity displayed a treatment of the robot as an ongoing (fragment 1) or imminent (fragment 2) interactant. This connects with the general observation that, on an individual level, the distance at which a participant stood from the robot was meaningful only in connection with a sequential context.