OpenMic: Utilizing Proxemic Metaphors for Conversational Floor Transitions in Multiparty Video Meetings

E.T. Hall’s notions of fixed and semifixed features have inspired several HCI works on co-located collaboration [28, 29, 30, 43, 51]. The notions articulate how features such as walls (fixed) and furniture (semifixed) in the environment serve to frame our opportunities for social interaction [35]. Prior work has used them as entities for triggering interface responses through gradual engagement as a function of proximity  [28, 48, 49], incorporated tables and walls in cross-device sharing techniques [29], and articulated how furniture is used for proxemic transitions [30]. Video-mediated communication research has emphasized the socially configuring role of office furniture in video-conference room design [43, 57]. This line of work has inspired the idea of the Virtual Floor in OpenMic, which “furnishes” the virtual 2D space with a fixed feature to which participants can orient their attention.

E.T. Hall’s work is best known for the concept of proxemic distance zones, i.e., people move in close proximity to others to show intention to engage with them, and increase the distance to others when signaling to disengage [35]. These spatial patterns have been shown to re-occur consistently in VR spaces with 3D avatars [78], and even in virtual 2D spaces. For example, the relative position between 2D avatars provides means for serendipitous encounters by “bumping” into each other [44].

Interestingly, other perceptual cues have shown to impact people’s perception of proxemic distance in video-mediated communication  [18, 27]. Grayson and Coventry  [27] found that in video-mediated communication, image size was a key factor in determining impressions of distance in photographs: the larger an image is, the closer it appears, regardless of the object-to-background ratio. Furthermore, during instructional videoconferencing, Ellis [18] found that perceived proximity to the teacher affected the student’s performance in the course and attitude towards the teacher.

Prior work on media spaces has proposed to design office spaces and their communication hardware to provide social and spatial cues and investigate the utility of proxemic cues such as gaze direction and bodily orientation [65, 66], and/or apply spatial (proxemic) cues to connect the context of office space layouts in the workplace [13]. While these did not explicitly articulate their rationales in terms of proxemics, later works started to use proxemics to frame remote meeting interactions [57, 67], such as the media space of spatially aligning remote spaces (e.g., [55, 57]) and granting blended spatial consistency (e.g.,  [57]). Several systems have explored the gaze cues for mediating floor control in video communication [58] or visualizations [36, 75, 79].

In previous research, various methods have been proposed to compensate for the absence of real proxemic cues in distributed collaborations.  [32, 37, 63, 72]. For example, Montage [71, 72] provided teleproximity for distributed groups as a mutual approaching glance fades in on others’ virtual workstations, enabling individuals to peek into each other’s offices. VideoWindow [20] designed "artificial proximity" to maintain informal interaction by directly translating physical proximity to a virtual environment. MirrorSpace [63] constructed proximity as an interface to provide smooth transitions between peripheral awareness and intimate forms of communication. PêLe-MêLe [32] supports different degrees of engagement by gradually shrinking and drifting videos towards the center of the screen. FluidMeet [37] operationalizes proxemics for supporting private messaging and calls during multiparty video meetings by enabling different levels of interaction with others based on nested virtual interpersonal distances. However, this line of work utilizes proxemic cues for mediating interpersonal communication rather than individual or group conversation dynamics that include speakers, addressees, and listeners.

Researchers have also explored the use of VR systems for mediating remote social interactions as the spatiality of VR allows more natural 3D proxemic relations [10, 12, 25, 78, 80]. Social VR provides opportunities for creating unique virtualized proxemic relations by manipulating properties of interpersonal relations such as the relative scale of avatars [80]. A few works have studied the relationship between turn-taking and embodiment in VR via conversation analysis [11, 12]. We share their interest in understanding how embodiment can be used to manage turn-taking; albeit focusing on the manipulation of video feeds instead of 3D avatars.

A shared 2D virtual frame of reference can be used to establish proxemic cues without custom hardware setup. Some systems enable the blending of multiple video feeds to create a shared hybrid space [31, 52]. MirrorBlender [31] supports continuous repositioning, resizing, and blending of video feeds in a shared 2D interface using the principle of WYSIWIS (What You See Is What I See [68]). Unlike MirrorBlender, which supports blending of physical spaces in hybrid meetings, we focus on support for turn-taking in fully virtual meetings and more dynamic manipulation of person and task space. Most relevant to our work are the tools that support the free manipulation of video feeds, avatars, or UI elements to grab others’ attention in a shared 2D frame of reference [1, 5, 6, 7, 62]. Several recent videoconferencing (VC) tools enable similar experiences, such as proximity-based social interactions with avatars to trigger bubbles of conventional video-conferencing (e.g., Gather Town [1], Wonder.Me [7]), and/or repositionable video feeds to provide social awareness of parallel conversations with the proximity of everyone’s video with ad-hoc subgroup conversations (e.g., SpatialChat [5], Sprout [6]). Most of these interfaces provide some notion of virtual fixed/semifixed features, e.g., abstract rectangles or depictions of chairs or roundtables, that then provide a bounded space for scoping the video and audio channels to subgroups of currently online users (e.g., Remo [4]). The role of fixed/semifixed features in these interfaces is to support approach and leave-taking in groups [24] or larger online crowds [44]. Instead, our goal is to provide such spatial features for mediating turn-taking within groups that have already been formed.

Turn-taking describes the flow of participation among speakers in a conversation over time. It is through taking turns on the conversational floor that humans engage in dialogic collaborative problem-solving. The conversational floor is an “dynamic and socially negotiated space” where individuals have the opportunity to make contributions [17, 19]. Hence, examining the turn-taking structure is essential for understanding the interplay of various voices in a group. We are interested in how speaking turns shuffles among speakers proposed by the analytical tool called the participation-shift framework [21, 22]. The p-shift framework [21, 22], based on the work of Goffman  [23], assigns the roles of the speaker, target (addressee), and third party (unaddressed recipient) to the participants in a conversation. The framework introduced four participation shifts¹ that accounts for possible micro turn-taking patterns, thereby illustrating the role of different participants in shaping the conversation and how turns are transferred from one speaker to another (Figure 2).

In HCI research, most explorations into distributed collaboration employing videoconferencing (VC) or shared media space have primarily focused on pairs  [38, 39, 40, 47, 59]. Research has also delved into how the medium impacts the user and conversational behaviors  [16, 66]. However, with larger group sizes, meetings tend to be more moderated [26] and dominated by a few individuals [73]. Prior studies quantified turn-taking behaviors to compare face-to-face interactions with videoconferencing, examining various factors such as spoken characteristics (e.g., backchannels and overlaps) [56]), the influence of visual information [66], the provision of spatial cues [13, 65, 66], and head-turning and gaze cues [14]. A prominent example is Sellen’s study, which compared three VC systems with different levels of visual information shown to the users and revealed significant differences in how floor control and simultaneous talk were handled, particularly in comparison to same-room conversations [66]. For example, the study showed that same-room conversations exhibited a greater frequency of interruptions and fewer formal floor.

These findings remain relevant today; the conventional user interfaces for video-conferencing, i.e., Gallery View and Speaker View in Zoom [8], remain largely similar to those in Sellen’s study in 1995. However, with the recent emergence of 2D virtual spaces (with interpersonal proximity features) such as Gather [1] and Teamflow [9], it is timely to investigate whether such mechanisms can support remote users in handling who is speaking and/or who is being addressed. Our work aims to explore turn-taking patterns during video meetings to understand which participation-shift patterns  [21, 22] emerge with OpenMic.

Before the study, the research team demonstrated how to use OpenMic, including moving video feeds and sharing screens, and how OpenMic’s Moderated and Freeform mode works to users. This was done via video calls with a live remote demonstration. Participants were then asked to try OpenMic on their web browser and were encouraged to freely explore the interface to get familiar with OpenMic.

The first task was to introduce themselves to the group. After a brief introduction, the eight participants were split into two subgroups of four in two breakout rooms.

After joining the breakout room, the participants started an icebreaker game called “One truth and two lies.” The moderator instructed each participant to think of three statements about themselves: two must be true statements, and one must be false. Others, as audiences, could ask questions about the statements. After three rounds of questions, they discussed and decided which one was the lie. We chose this task to resemble a meeting where one speaker controls one’s presence on the Virtual Floor, regarded as the floor holder and others are supporting through audience participation and other short interjections. We particularly observed dynamic use of video position and scale in the conversation.

Participants will be assigned to discussion sessions with a popular team-building task called Lost at Sea [2]. We assigned a Google sheet with survival items to each group and provided the group and individual sub-sheets. After receiving their scenario, participants were given up to ten minutes to rank their items individually in their individual sub-sheet in the order of most helpful to least useful for survival.

The study sessions were screen recorded. During the study, two researchers collected field notes to capture interesting episodes of turn-taking behaviors. We observed the participants’ behaviors, such as how they communicated non-verbally through the video movement (via re-positioning and re-scaling) and how they approached or retracted from the Virtual Floor for participation shifts. We conducted a top-down (deductive) thematic analysis on all the notes. We first discussed preliminary themes that aligned with the three proxemic metaphors (fixed/semifixed features, perceived proxemic distance, and conversational transitions) and then grouped the notes into themes related to these. When revisiting the video data, we focused on episodes (see Figures 9–17) that are specifically related to these themes (see themes below). Two researchers (the ones who collected the field notes) were involved in the thematic analysis. They also cross-referenced the focus-group findings with the key episodes from the video analysis in order to re-review and further refine the notes and key events from the recorded video with the three main theoretical themes. The findings were finally discussed and organized by all researchers on the team.

Our observations show that the Virtual Floor provides a visual separation of the 2D virtual space that differentiates how participants behave and perceive each other on and off the Virtual Floor in terms of their perceived personal space.

On the Virtual Floor, we saw indications that participants embody their video feeds as their personal space, which is consistent with the findings of Grønbæk et al. [31]. For example, during the switch to a new presenter group (Figure 8), participants on the Virtual Floor tended to move closer to the boundary to leave space for others. The focus group interview confirmed this analysis of how participants interpreted the Floor as an open space and indicated respect for the personal space of one another: “I consciously thought about open space for more people, and we do not need to cover all the space, we may leave some spaces for others to join. ” “...resemble personal space, in the real world, would you stand on top of someone? No, you wouldn’t.”

Off the Virtual Floor, however, we observed that participants tended to stay at the initial position of where they appear upon entrance with two or three participants overlapped with each other (Figure 9A:top-left). They did not move their video feeds unless they intended to take turns (Figure 9B).

This observation is corroborated by the focus group interview, when one participant recommended supporting a transition from the structured state of “taking seats in the dimmed area or around it [the periphery of the floor]” to the unstructured state in OpenMic so that participants perceived that they can benefit from “whenever you want to take a turn, you can enter the floor and re-sizing our video.”

This observation further indicates that the Virtual Floor provides a clear visual separation to indicate where the shared attention is. Because the floor area in the presentation scenario of Figure 8 and Figure 9 serves the function of a stage for groups to present in front of others, personal space becomes important to have clear views of each presenter, but less so for the audience, who is not the center of attention.

We found the differences in the way participants enter and leave the Virtual Floor by the group size, particularly in how the boundary mediated their turn-taking behaviors. Even when the breakout groups used a Moderated Mode, they usually kept themselves on the boundary and maintained the conversation flow for a long duration. For instance, Figure 10A illustrates an example in a four-person group when three participants kept maintaining their position on the boundary for several minutes after their audio got approved by the speaker. In contrast, Figure 10B shows an example that in a larger eight-person group, a participant near the moderated boundary kept entering and leaving the Floor four times to actively engage in the conversation. During the focus group interview, a participant mentioned how they interpreted the space on and around the Virtual Floor and used it for turn-taking in group conversations: “I thought [that] when you take the floor [it is] like when you stand up in the classroom.” To further illustrate, the participant described his experience in the classroom that he would sit back down when he did not contribute to the conversation.

On-Floor participants made consistent use of their malleable video feeds for taking the turn - both re-sizing and re-positioning on and around the Virtual Floor for explicit communication with other members.

The following two episodes illustrate two different ways of taking the turn during the group’s presentation: (1) moving to the center area of the Virtual Floor; (2) slightly enlarging themselves near the edge by moving closer to the floor center. Vignette 1 (Figure 11) shows an episode where a participant shifted the "stage" of Virtual Floor by first putting himself as the next speaker by moving his video feed to the center of the Floor, and he moved to top peripheral area after finishing his turn (Figure 11C):

Vignette 1 (Figure 11): Eight-Person Group, Presentation. Group 1 entered the Floor to start their group presentation, and each of them placed them at the edge of the Floor. There was a short silence before anyone wants to start (A). P1 (yellow) self-selected as the next turn-taker. He started the Moderated Mode and moved his video to the Floor center to present his group’s idea (B); After P1 finished his presentation, he moved himself to the upper-right position (C).

Another way can be illustrated in Vignette 2 (Figure 12) that a participant enlarged himself a bit to answer the question by an audience member during the Q&A session:

Vignette 2 (Figure 12): Eight-Person Group, Q&A session. There was an audience question by P2 during the Q&A (A); P2 withdrew his video after asking the question. P3 from the presenter group (red) moved himself to the side (and then enlarged himself a bit) to answer the question (A); P2, as the audience, moved himself to the boundary (approved by speakers), saying “But...”, while he saw the intention of another audience (P4) with yellow rings around his video feed (B-C). P2 then said “you first” and withdrew (C).

These episodes show how on-Floor participants use the size and position of their video feeds by shifting between the center and peripheral area with their concurrent “self-selection” mechanism. The Virtual Floor as a stage to configure multiple co-speakers and align and re-configure their malleable video positions was also of significant utility to participants to enhance participation and grab attention from both other speakers and off-Floor audiences.

While resizable and repositionable video feeds enabled on-Floor participants to grab attention, off-Floor participants made consistent use of their relative position to communicate with different subsets of the group, e.g., address a group, a specific speaker and maintain awareness of other audiences’ behavior.

Often, off-Floor participants took the floor using the shortest path to the Virtual Floor (e.g., Figure 13A). This shows a turn-claiming pattern in which audiences reversely address the whole presenter group via the boundary of the Virtual Floor, and one or two of the co-presenters would respond to the question (e.g., Figure 10). We also observed some cases in which off-Floor participants move to a specific position of the boundary to have a conversation in relation to someone (i.e., address a specific person) on the Floor. This shows a turn-usurping pattern when one of them takes the floor and addresses a specific speaker to continue or adds a point of another turn-taker. An example was illustrated in Vignette 3 (Figure 13) where one audience member (P3) prominently moved her video feed right next to the boundary near the current speaker (P1) to take the turn of what P1 addressed to another audience member (P2), and built up on the conversational threads between them (P1 and P2):

Vignette 3 (Figure 13): Eight-Person Group, Q&A session. Right after the speaker (S1 in green) finished his presentation, he moved himself to the corner of the Virtual Floor (top-right); the auditor (A1 in red) drew his video feed to the moderated boundary and asked a question (A); S1 then responded to him ending up with “Does it make sense?”. Another audience (A2 in yellow) moved herself from the bottom-left to top-right (farther path but closer relative position), saying “well, an adjacent question to [A1]’s question...” (B).

The finding shows how the relative position around the Virtual Floor and the proximity to other participants’ (especially a speaker) video feeds provide participants with more opportunities to express their intention to others. The relative position (being able to enact closer interpersonal distances) also aided communication as the user can point their video feeds to a specific individual as an indicator to address a person.

As illustrated in prior sections, speakers (on the floor) and auditors (off the floor) made use of the re-positioning extensively for turn-taking and role switch between central and passive speakers in the Survival Task (eight-person group) for group presentation and Q&A tasks.

Here, we also observed two ways of how speakers used video resizing to convey intentions of turn-claiming³ in four-person groups. First,one episode (Figure 16B-C) shows how decreasing video size can indicate intent to give the floor to others: the participant slightly moved towards the corner to invite others to take the floor. Second, as an opposite example, the episode in Vignette 4 (Figure 14A) shows how the speaker subtly enlarged the video as a nonverbal form of conveying the intention to the whole group to reach an agreement. It then shows how this interaction invites other group members to enter the Virtual Floor to engage in the conversation (Figure 14B-C).

Vignette 4 (Figure 14): Four-Person Group, Group discussion. S1 (red) tried to hand over the floor verbally with “I guess if everyone agrees, we can put 1 and 4 and the seat cushion, right? That would work?” Other group members responded by nodding their heads (A). He then moved himself closer to the Floor center and hence subtly enlarged his video feed on the Floor , saying “yeah? Or I will just put it in the sheet” (A-B); Other group members started to enter the Virtual Floor and address their ideas. A1 responded “Yes, I agree; also, water would be the second choice, and for the third, we kind of divided on that.” (C).

During the focus group, participants reported mismatches between their understanding of the conversation state: “I nodded my head a lot during the tasks mostly because I thought that the speaker was gonna continue his statement and ideas so that I would not interrupt, [...] but later he enlarged himself, and it was obvious that it could be my turn to say something.”

One reason for this mismatch is that the request was interpreted differently by other group members as a confirmation rather than a handover request as participants further commented on the process of addressing the whole group is sometimes a hard-to-tell intention in video-mediated communication: “we can explicitly mention someone’s name, when you what to address a specific person. [...] When someone asked the whole group, it is often hard to tell when the turn goes next in the video meetings. Did the speaker finish his turn? [...] In this case, OpenMic did a great job.” Another possible explanation regarding this challenge of addressing the whole group has been found in prior work [66] that people are relatively polite in video meetings when taking the turn and less tenacious about holding the floor, which can be potentially alleviated by adjusting the size of video feeds to mediate their concurrent talk in OpenMic.

In the presentation configuration, speakers are on the Virtual Floor, whereas the audience is distributed around it, which supports workspace awareness during Q&A: crossing the boundary shows clear intention to talk, which helped prevent potential confusion regarding who is next to speak. For example, Fig 12 B-C illustrates such an example where the audience member withdrew his video feed as he saw the approach of another audience member to the boundary and understood his intention to take the floor.

The support for multiple simultaneously shared screens distributed around the Virtual Floor helped mediate turn-taking. We will show two episodes in four-person groups that show interesting ways of using screens to handle turn-taking conflicts. For instance, Figure 15 details a workspace organization that shaped the given group’s turn-taking patterns during artifacts sharing, where one participant maintained the central role of leading the discussion while other members momentarily jumped in to the conversation to question her choices (without showing their own list). However, Vignette 5 in Figure 16 shows an episode that represents a conversational transition from a configuration that enabled item-to-item discussion of comparing their individual selections around each other’s shared sheets (Figure 16A) to a configuration supporting quick shifts between individuals presenting (Figure 16B-C).

Vignette 5 (Figure 16): Four-Person Group, Group discussion One participant was presenting his selection of items, followed by a quick round of discussion about the top item and comparison between each other’s decision and sheets (all the participants moved their screens to the floor to point out their rationales over the item). They then found it a bit hard to come to an agreement. One participant proposed to go through it one by one (A); Participants then put off their screens. One participant (green) claimed the turn to be the first to speak and moved his screen to the center (B). After he finished his turn, he slightly moved his video feed towards the bottom-right corner, saying “I will take myself off the screen so some of you can take it” before he leave the floor (B,C).

The above episode shows that the re-positioning of screens prevents potential conflicts regarding who is taking the next turn to speak. On the contrary, the current implementation of sharing screens in full-screen ( standout ) also caused issues regarding how to position person videos within the limited display space. The episode in Vignette 6 (Figure 17) shows that when screen-sharing needed to cover the entire Virtual Floor, users struggled with how to position themselves to avoid occluding the content of discussion. Vignette 6 (Figure 17): Four-Person Group, Group discussion Once each member of a four-person group had presented their own ranking of the survival task, there was a three-minute discussion during which each group member was on the Floor, yet none of them shared screens or moved their video feed (A), and they viewed their dual screens and discussed their group’s decision aloud. Finally, after a discussion about an item, S1 was trying to confirm the choice with his group members, when he looked back to the primary screen on his laptop: “Okay, then I will put the third one as the shaving mirror...wait, I will share my screen.” After the screen was shared, the origin position of S2 and S3 appeared to cover the shared screen. They then moved themselves to other places (A,B). Later, S3 avoided covering the display by shifting to the upper-left edge (C).

Video resizing served as a way to grab attention, which extends the concept of Perceived Proxemic Distance [18, 27] from static configuration of engagement to gradual transitions between levels of engagement. Aside from configuring their video feeds to the center, a more frequent way to enhance participation in a conversation is to subtly enlarge their video feeds. A reverse pattern is that users may slightly make themselves smaller when they want to indicate that their speaking turn was finished.

While Perceived Proxemic Distance is originally concerned with the impact of video size on interpersonal relations [18, 27], we found that the relative position between people’s video feeds also appears to serve as a proximity cue to grab others’ attention. What is especially interesting about this, is that in terms of turn-taking, it may provide a substitute for mutual gaze [36, 75], which we use as a primary cue for addressing individuals when face-to-face. Modern meeting tools such as Gather [1] and Wonder [7] use proximity (i.e., the relative distance between videos) to determine the social group boundary and support approach and leave-taking [24]. With OpenMic, we have demonstrated the novel possibilities of using these same interaction principles but for the purpose of conveying clearer intentions between who is talking to whom within a conversation.

The patterns of use show that video resizing is often used when grabbing the attention (lowering perceived distance) of an entire group, whereas moving one’s video feed closer in relative position to another video feed means lowering the perceived distance to a specific person (which is useful for addressing a specific individual). In Gibson’s terms of turn-taking [21], this then means that video resizing provides support for turn-claiming (i.e., addressing a group), whereas relative video positions provide support for turn-receiving (i.e., addressing an individual).

We designed OpenMic to support turn-taking in small to medium-sized group conversations (4-10 people), and the user study only investigated the use of OpenMic using 4- and 8-person groups. While the concept of using proxemic metaphors for conversational floor transitions is not limited to smaller meetings, the system may need further design iterations to support larger group meetings. For example, the off-floor space may not be sufficient to accommodate videos of a large audience, and the floor boundary for moderated conversation may cause challenges in understanding the order of bystanders lined up for participation. A further study after design iterations with larger meetings will provide an understanding of how a video meeting interface designed using proxemic metaphors can support the transitions across groups of audience and speakers. Moreover, OpenMic mainly focuses on the proxemic metaphors of person space and digital task space. Future work can utilize and explore the effect of proxemic metaphors on turn-taking for physical task spaces such as conversations around shared physical artifacts [38].