InflatableBots: Inflatable Shape-Changing Mobile Robots for Large-Scale Encountered-Type Haptics in VR

The design of our inflatable body structure is heavily influenced by Vine Robots [20]. The system employs a polyethylene membrane, which is stored in a spool format at the center of the base. The height of the inflatable module is changed based on a motorized spool, which controls the release and retraction of the polyethylene tube. This spool-centric design allows a substantial height change, spanning from 40 cm to 200 cm (shown in Figure 3), with a compact base dimension of 25 cm diameter circle. The fan-based inflation mechanism allows fast and dynamic shape change with a speed of 10.4 cm/sec.

The inflatable component is primarily constructed from a vinyl sheet with 0.02 mm thickness. The fabrication process requires cutting the sheet to the specified length and subsequently heat-sealing to fabricate the tube. Upon inflation, the tube expands to a diameter of 25 cm. For our design, a tube length of 450 cm is utilized to achieve a height of 200 cm, given that the tube typically requires over twice its length to reach the intended height. One end of the tube is stored in a motorized spool, while the opposite end is securely anchored to a base plate. This approach guarantees the inflatable structure’s durability and operational efficiency.

The base plate is created with a 7 mm cardboard that constructs a circular shape for the inflatable base and space to attach two mobile fans. The vinyl tube is attached to this circular shape with a diameter of 25 cm. It also has an alignment shaft, which prevents any tangling of the spool when it is extended and to enhance the overall robustness of the structure (see Figure 2 (c)). A key consideration is the diameter of the hole; if too small, the structure becomes unstable due to restricted airflow. The spool is controlled by a DC motor (Bringsmart JSMG028A12V80), which has 20 kg · cm torque and 80 RPM. This motor is controlled via a microcontroller (ESP32) with motor drivers (TB6612). The motorized spool is powered with a mobile battery (NiMH battery, 12 V, 1.8 Ah) (see Figure 4). Each module’s assembly takes about 30 minutes, with the cost per actuator being under 1,100 USD, including an omni-directional mobile robot.

The height of each actuator is determined by the motorized spool, which is controlled by a main computer. The computer communicates wirelessly with a microcontroller. The control mechanism is an open-loop system, based on the rotation time of the DC motor. The DC motor rotates at 80 RPM, and it can feed the 10.4 cm of vinyl tube per second. Based on this, we track and control the current height. To control the motorized spool, the main computer dispatches commands to the microcontroller, prompting the spool to rotate and effectuate the desired change.

Each robot has two portable fans (AINOUT, 22V high output fan for work clothes). These fans are capable of producing a high airflow rate with 8.6 cm/sec measured by Sanwa supply anemometer CHE-WD1. While our inflatable design is independent of and compatible with both fan-based and air pump actuations, our experiments highlighted some challenges associated with the latter approach. Mobile air pumps, despite their ability to provide more robust inflation, tend to be slower and less portable. For instance, when we experiment with a mobile air pump (Yasunaga air pump, AP-40P), it took a considerable amount of time to inflate the structure (e.g., 1 minute), which is not ideal for dynamic haptic interactions. On the other hand, powerful air compressors could offer a faster inflation time, but their larger size and high power requirements are not suitable for mobile form factors, requiring a tethered setup with a pneumatic tube connecting to the compressor. After evaluating the advantages and disadvantages, we decided to choose fan-based actuation, prioritizing its fast transformation and compact design.

Our mobile robot base employs an omni-directional mobile robot (Nexus Robot 4WD Mecanum Wheel Robot 10011 [46]). Each robot measures 36 cm x 40 cm and it can move seamlessly in any direction, thanks to its omni-directional wheels at the maximum speed of 58.5 cm/sec. These robots are equipped with reel-based inflatable structure and portable fans, as mentioned above. Once assembled, the basic module occupies a horizontal footprint of 36 cm x 40 cm and minimum stands 40 cm tall.

Position tracking for each robot is achieved using the HTC Vive tracker 3.0, which is attached to a base plate. The entire interaction area is monitored by four external HTC Vive Base Station 2.0 lighthouses, capable of covering expansive areas measuring 3.5 m x 2.5 m (shown in Figure 5).

Our system integrates efficient path planning with real-time body tracking, ensuring that the movements of all robots are synchronized with the user’s hand movements. By tracking and predicting potential touchpoints, we guide the robots to meet the user’s hands in a timely manner. Utilizing the RVO algorithm [65], we determine the direction in which the robot should move during each time step. Given the omni-directional capabilities of our robot, it can directly navigate to its target position. Robot control commands are executed through another microcontroller (ESP32), which controls the robot. The robot’s speed is adjusted based on a PID control.

Our VR scenes are created and rendered using Unity (version 2021,3,10f1) and Steam VR (version 1.27.5). The main computer, a Windows machine equipped with a CPU of Intel Core i7-8086K, a GPU of NVIDIA GeForce RTX 3060, and 32 GB RAM, runs Unity and streams the content to a VR headset (HTC VIVE Pro). For applications, we also use pre-defined 3D models from several Unity assets, such as Toon Farm Pack.

Unity also communicates with ESP32 microcontrollers through Bluetooth protocol. Our software first receives a robot’s tracking data through HTC Vive tracker 3.0, and then visualize the robot’s position in the Unity scene (See Figure 5). The software simultaneously stores the current height of each inflatable structure, rendering the current height by changing a semi-transparent cube on top of the robot. The VR scene is rendered through HTC VIVE Pro via connected cable. When the user’s position or hand moves, the system prompts the robots to dynamically adjust its position and height accordingly. To do this, the system first measures the height of the virtual object that the user encounters, then actuate the inflatable to the target height.

To approximate the current height of the surface, the system utilizes vertical ray casting. This technique gauges the height of the virtual contact point based on the robot’s position. Given a virtual entity or surface in Unity space, we cast a ray vertically from a robot’s position. The distance between the ray’s origin and its intersection point provides the estimation of the current height.

We create the interaction area (3.5 m x 2.5 m) by room setup of Steam VR. Using this area, the system can get target heights for each object and surfaces in the scene. The system tracks the user’s hand with the HTC Vive tracker 3.0. As the user’s hand moves in 3D space, the robot repositions within this target zone. If the identified targets are more than the number of available robots, the system optimizes robot placement based on the proximity. When multiple robots are available, they can collaboratively cover an extensive zone.

The methods to measure each criteria is as follows: 1) Robot and Inflatable Speed: We determined the speed of the robot and inflatables by analyzing video footage of the movement alongside a ruler for reference. 2) Inflatable Extension Accuracy: We gauged the accuracy of the inflatable’s extension by activating the motor for ten distinct durations. For each duration, we measured its height in a similar way as the first one. 3) Position and Orientation Precision: To assess the accuracy in position and orientation, we documented the deviations in distance and angle between the robot’s actual position and its intended target. We report the average errors of ten trials. 4) Pushing Force Measurement: We utilized Imada’s dual-range force sensor (with a precision of 0.001N) to measure force. Specifically, we recorded the peak impact force exerted against the sensor when the InflatableBots was propelled over a 10 cm distance. 5) Latency Measurement: We evaluated the latency at each stage of the process, including Bluetooth communication and path planning. This was achieved by comparing timestamps from the initiation of each event to its conclusion. We repeated this process ten times. 6) Tracking Robustness: To gauge the robustness of our tracking system, we assessed the error rate when the tracking is lost within 1 minute.

Table 1 shows all of the technical measurements for each criterion. The table provides a summary of the results for metrics (1) through (6). Regarding the sixth metric, the average latencies recorded were 1.4 ms for Bluetooth communication and 1.0 ms for path planning computation. In total, the control loop’s latency measures 111 ms (maximum fps in unity with HTC VIVE Pro is around 90). Lastly, for the fifth metric, over 10 trials, the system lost tracking approximately once every 0.27 seconds.

One of the fundamental capabilities is rendering multiple stationary objects. For instance, users can haptically interact with trees and plants situated at different locations (Figure 8 (a)). As users navigate, the robot dynamically move itself to haptically represent these objects in real-time. By coordinating multiple robots, these robots can reach to the object in a timely manner.

Another key interaction technique is to engage with moving virtual objects. Imagine virtual animals like dogs and pigs roaming a farm; users can touch and pet them (Figure 8 (b)). The coordinated efforts of multiple robots enable the haptic representation of several moving objects simultaneously.

InflatableBots can also haptically render large continuous surfaces by synchronizing vertical shape transformation with horizontal movements (see Figure 7). This allows users to touch large objects like cars and buildings (Figure 8 (c)). As a user slides their hand over a surface, the robot adjusts its position and height to simulate the object’s surface. Rapid shape transitions also let users feel surfaces with steep curves, such as the arched back of a horse.

Another unique feature is to render objects that are changing its shape in real-time. For instance, when a user touches a virtual sleeping bear, they can sense its rhythmic breathing. The system can also simulate objects like a window being lowered, so that when the user opens the window, it shows up the window by changing its height (Figure 8 (d)).

Finally, by leveraging its durability, users can also interact with handheld tools, as InflatableBots can withstand intense interactions like hitting and knocking. For example, by using a variety of tools, from sticks and swords to hammers, users can strike an object with a hammer, engage in a whack-a-mole game, practice drumming with drumsticks (Figure 8 (e)), or hit the enemy with a sword. The system remains resilient even when subjected to forceful hits.

The first exploration is full-body sports, which is now a critical application example using room-scale VR setup. InflatableBots allows users to haptically interact with height-changing moving objects (e.g. airborne elements). For example, the InflatableBots’s height-changing and mobile capabilities can simulate a VR tennis game where the parabolas of tennis balls in different flight paths are haptically rendered. While the haptic sensation of touching the balls may not be perfect due to the shape and vinyl material (as discussed), such real-time, large-scale, and multiple physical feedback can significantly improve the user’s experience, engagement, and body control. Furthermore, inflatable structures additionally enable safe and robust features, such as supporting repeatedly striking balls with a racket (Figure 9) or punching sandbags (Figure 10). Such safe physical interactions is vital for VR users who are fully immersed in virtual sport fields, and robust features make the device easily reusable.

The second exploration is to render 3D structures of a room by highlighting the high inflatable structure of up to 200 cm. When supposing a walkthrough in a darkroom or maze, various type of room structures such as walls and slopes can be haptically rendered (Figure 11), helping the user’s spatial understanding. Unlike ZoomWalls[71], the InflatableBots can simualte handrail of slopes and higher/lower walls.

Utilizing the flexibility of height-changing and the inflatable top’s shape versatility, the InflatableBots can be used as a physical stand for various tools. For example, just like the robotic ergonomic assistant [14] for room-scale VR, users can install a physical canvas on InflatableBots which allows them to keep their hands (Figure 12) at a comfortable height, while enhancing accuracy in illustration activities in room-scale VR. Various tools can be considered, such as cameras, notebooks, auditory equipment, and so on. By placing a bar or plate across two InflatableBots, more complex shapes can be rendered, such as a hurdle or a bar of limbo game. Furthermore, while an additional force-sensing mechanism is required, the inflatable structure itself can effectively be used as a full-body input device (e.g., joysticks).

We recruited 12 participants (age: 20-24 years old, 4 females and 8 males) from our university who have experience with VR headsets. The total duration of the study was about two hours per participant. They received payment of about 20 USD for their participation according to the university’s regulations.

We used Unity scripts to create and render our experimental VR world and an HTC VIVE system to track an HMD, hand-held controllers, and trackers affixed to the moving robots. Figure 13 shows the 2.5 m x 3.5 m tracking area, the HMD, and the InflatableBots prototype with two trackers. The rendered virtual world was the same size as the tracking area. For safety reasons, besides the experimenter, an assistant constantly monitored the participant to ensure no collisions with the InflatableBots, walls, or other objects, and was prepared to press the emergency stop button if necessary (although this never occurred).

Due to the sensors’ potential limitations, there might be a positional discrepancy between the visual content and tactile stimulus rendered with the InflatableBots. Participants were instructed to point out if they felt any inconsistency so that the experimenter could correct it by manually adjusting the robot’s parameters to avoid any misalignment issues significantly affecting the reality assessment.

This task was designed to evaluate the reality of InflatableBots’ haptic rendering for different shapes, hardness, and surface textures of objects. We compared six types of objects under the following three haptic conditions: no use (mid-air), an actual object (ground-truth), then use of InflatableBots. Figure 14 shows the six objects used in task 1: a cushion, an exercise ball, a rough ball, a table, a stuffed bear, and a plant, all of which have soft (elastic) properties with different shapes, hardness, and surface textures.

When the VR world started in one of the three haptic conditions, one of the six virtual objects appeared in the center of the world. Participants were asked to touch a stationary red dot (Shown in Figure 14 (a)) on the virtual object with their fingers for 10 seconds. They then answered the question "How realistic was the experience?" displayed in the VR world on a Likert scale from 1 (not at all) to 7 (very much). Afterward, the next object was presented, and this process was repeated for all six objects with one trial for each object. The order of object presentation was randomized for each haptic condition. Next, participants performed the same procedure for the other two haptic conditions. The order of the haptic conditions was counterbalanced across participants. For the ground-truth and InflatableBots conditions, the real object or the InflatableBots’ cylinder top was placed exactly on the red dot in the virtual object. In contrast, no physical object was placed in the mid-air condition.

Figure 15 shows an overview of the results. The * mark indicates a significant difference as suggested by the Wilcoxon signed-rank test with Bonferroni correction. For all objects, the scores in both the InflatableBots and the ground-truth conditions were significantly higher than in the mid-air condition (p < 0.01). However, the ground-truth score was significantly higher than the InflatableBots score (p < 0.01). The average score of the mid-air condition was less than 2 for all objects, while that of the ground-truth condition exceeded 6. The InflatableBots condition had a different trend with a high variance for the different materials of the objects, as seen between cushion (5.2±1.1 SD) and plant (2.8±0.69 SD).

Looking at the mid-air vs. InflatableBots results, the presence of the tactile feedback from the physically inflated upper part successfully improved the reality of the haptics compared to the mid-air interaction. From the interview, they also felt an overall increased sense of physicality. However, the reality score of the InflatableBots was still significantly lower than the ground truth. These results suggest that the soft physical feedback of the InflatableBots offers some advantages for haptic representation but does not reach a realistic touch experience. Factors specifically mentioned in the interviews were surface hardness (softness), shape, and texture. InflatableBots are soft and almost flat on top, and their texture is uniform with thin vinyl. Therefore, objects with completely different materials, such as tables with a hard, flat surface, rough balls with many surface irregularities, or plants with complex leaf veins, cannot be fully rendered with InflatableBots. On the other hand, the cushion, which had similar hardness, shape, and surface texture with the virtual object, resulted in high reality. The animal had a relatively good result, we think the reason might be due to its softness. Participants mentioned that the virtual model did not visually render the animal’s fur fibers, so the reality of touch remained high even with InflatableBots’ vinyl material (P10, P11). This suggests that InflatableBots may be more effective for cartoon-like smooth models than rough textures.

When the VR system started, a red point immediately appeared in the air (Figure  16(c)). Participants were asked to keep touching the point with their fingers. The red point continuously moved 50 cm from the initial location in a given incline angle for 5 seconds to represent the virtual inclined board (Figure  16(c)). For the InflatableBots condition, its top’s height and position follow the red point’s motion to physically represent the inclined board (Figure 16(b)). For the ground-truth condition, we used the simplest way by setting up a mock-up with physically inclined boards (Figure 16(a)).

After finishing one trial, participants answered two questions regarding the realism of the rendered 3D surface in the virtual world. The first question concerning the surface’s continuous rendering ability“How much did you feel like you were touching a continuous surface?” and they responded on a Likert scale from 1 (did not feel) to 7 (felt). The second concerning the surface’s incline rendering ability “which angle did you perceive while touching the surface? ”. Several inclined board options were displayed in the VR (see Figure 16(d)), and they selected the one closest to what they experienced. After answering these questions, a subsequent trial was presented. Participants conducted one trial each for all five inclined angles. The inclination angles were given randomly, and the two haptic conditions were counterbalanced among participants.

Figure 17 shows an overview of the results. The * mark indicates a significant difference based on the Wilcoxon signed-rank test.

- Continuity: As shown in Figure 17(a), for all inclination angles, the continuity score under the ground-truth condition was significantly higher than the InflatableBots condition (p < 0.01). Furthermore, the average score under the ground-truth condition was above 6 for all inclination angles; the InflatableBots’ average score was above 4 for all angles.

- Angle estimation: Only for the 30-degree inclination angle, the error was significantly larger under the InflatableBots condition than the ground-truth condition (Figure 17(b)). No significant difference was observed between the InflatableBots and Ground Truth conditions for other inclination angles.

According to the interview, one reason that the InflatableBots condition scored lower than the ground-truth was that the real slope has an incline on the upper surface, but the top of the InflatableBots is almost flat. Another reason was no friction felt when moving the hand with the InflatableBots, and the force in the upward direction (e.g., pushing up the hand) from the inflated vinyl surface was weak, causing the hand to sink into the vinyl. Furthermore, the tilt estimation error becomes more negligible for extreme and clearly perceivable angle conditions (i.e., horizontal or near-vertical) which can be taken into account in designing applications.

In this task, we evaluated the user experience by obtaining various user feedback through representative application scenarios with InflatableBots. For this purpose, we implemented an application with four experiences, all themed around a farm (nature) (as shown in Figure 18). All experiences used different features of the InflatableBots.

Unlike Task 1 and 2, the InflatableBots maneuvered extensively around the room in this task. For safety considerations, their positions were visualized in the VR space. Furthermore, earplugs were used to reduce the noise from cooling fans and the robot’s mecanum drive. For hygiene reasons, disposable earplugs were adopted. The first experience involves touching plants. This is an action in which one can touch a stationary object, replicating Task 1. The second experience consists of repeatedly touching the entire horse’s body (e.g., from lower back to higher neck). As the participant’s hand moves, the InflatableBots moves to follow the hand, adjusting its height based on the height of the horse’s body parts that the user wants to touch. This experience is an extension of Task 2 and simulates changing height along an incline. The third experience is touching the necks of two dogs, where two InflatableBots were used, allowing two dogs to be touched simultaneously. This allows us to examine how multiple touch experience is perceived by users. The final experience involves using a controller instead of the highly sensitive hand to test how casual controller-based interaction is perceived. Here, when the controller in VR touches the pile, the pile is driven, lowering its height by 20 cm. The users can repeatedly touch the controller to the piles. They tried each experience once, and the order of the experiences was fixed. We did not prepare a mid-air or ground-truth condition, as such direct comparison was already tested with Task 1. Rather, we focused on testing users’ impressions when interacting with multiple InflatableBots around the users and also at real operating speeds.

Participants experienced the application in the order of plants, horse, dogs, and stakes.

After all of the experiences, participants answered survey questions regarding each experience in terms of "preference" and "how helpful InflatableBots is in understanding the content" on a 1 (did not like/not at all) to 7 (liked very much/very much) Likert scale, and "How loud was the sound" on a 1 (not at all) to 7 (very much) Likert scale, and gave their overall impressions of the application.

Figure 19(a) shows the scores for preference and Figure  19(b) shows how the InflatableBots aided the content understanding. The preference scores for plants, horses, dogs, and stakes were 3.7±1.1 SD, 5.3±1.1 SD, 5.5±1.0 SD, and 6.0±1.3 SD, respectively, with an average score exceeding 4 for all except plants. The scores for content understanding were 4.6±1.0 SD, 5.1±1.8 SD, 5.5±1.1 SD, and 5.8±1.3 SD for plants, horses, dogs, and stakes, respectively, with all experiences having an average score exceeding 4.6. The score for the sound being loud was 2.8±1.1 SD, with the average score being below 4.

Each experience: <Plant> The tactile presentation of the InflatableBots for stationary objects like plants deepens content understanding just by the sensation of touch. However, as mentioned in Task 1, unless there’s a match in hardness, shape, and texture, its effect is limited. In the interview, several participants mentioned that they felt a strong sense of unity with the InflatableBots, and they could not discern the independent sensations of leaves or branches. For a better experience, it is necessary to design applications considering these elements. <Horse> The interaction of touching the horse showed that participants could continuously touch the entire horse at varying heights. This indicates that the InflatableBots can adapt to body-scale objects. However, the InflatableBots moved to follow the participant’s hand, and some pointed out noticeable delay or waiting times for the robots to arrive at the hand position. Therefore, similar to the existing encounter-type haptic devices, using faster robots or employing multiple InflatableBots on the same continuous surface to reduce waiting time might be necessary. The following description, which is the result of the next experience, would support this consideration. <Dogs> The two InflatableBots were used, one was moving from the previous experience, and another was prepared around the second dog position. Overall, we received positive feedback that they could touch one dog and immediately touch the other without waiting time or noise, enhancing immersion. Some participants responded positively in touching both dogs simultaneously with both hands. This multi-robot approach can compensate for the disadvantages of single-robot operation and increase possible interactions. <Pile> Since the touch was through the controller, the influence of different textures was mitigated, and some suggested that it felt more realistic than direct touch. The fact that the InflatableBots are soft and slightly indented when hit with the controller made it feel like they were actually hammering. However, some were concerned about damaging the InflatableBots by hitting too hard.

Sound score: Thanks to earplugs, the score for the noise from the omni-wheel robot, fans, and motor drive was below 3. Participants could focus on the application without extra anxiety. The use of noise-canceling headphones could also be effective and might further reduce awareness of noise by playing appropriate sounds or background music during the application and interactions.

Safety: In Task 3, compared to the other tasks, the robot moved more rapidly around the room. However, interviews revealed that 10 out of 12 participants did not feel any fear of the moving robot. One reason was the in-VR visualization of InflatableBots’ positions, which allowed participants to adjust their hands or body postures to maintain sufficient distance from the robots. However, some felt that always-available visualization made them feel as if they were touching the visualized robot rather than the virtual object, which could potentially reduce immersion (P11). Additionally, some participants mentioned that the vinyl part of the InflatableBots was soft, safe to touch, and wouldn’t hurt. Others noted that they were familiar with household robots such as Roombas and trusted they wouldn’t collide with them.