Interactive restriction of a mobile robot’s workspace in a smart home environment

2.1.Mobile robot workspace restriction

As already stated in the introduction, current solutions to restrict the workspace of a mobile robot are based on sole HRI without support of a smart environment. Commercial solutions comprise magnetic stripes placed on the ground [29] and virtual wall systems based on beacon devices [8], that emit infrared light beams. Despite their effectiveness, they are intrusive, i.e. additional physical components on the ground are necessary to restrict the workspace, and inflexible, i.e. restriction areas are limited to certain sizes and shapes. Thus, they are not applicable for our problem because the considered scenarios require restriction areas of arbitrary sizes and shapes. An alternative is sketching restriction areas on a map of the environment containing occupancy probabilities [1]. These maps are created by today’s home robots and are used for navigation purposes. However, this interaction method is inaccurate, i.e. there are strong inaccuracies between the user-defined restriction areas as a result of an interaction process and the restriction areas to be intended for restriction by a human. This is caused by a correspondence problem between points on the map and in the environment [45]. In order to address this shortcoming, Sprute et al. introduced a framework for interactive teaching of virtual borders based on robot guidance [43]. This allows a user to guide a mobile robot, e.g. using a laser pointer as interaction device, and specify a virtual border as the robot’s trajectory [46]. Although this interaction is accurate due to the accurate localization of the mobile robot, the framework suffers from a linear interaction time regarding the border length and lacks an expressive feedback system. This is caused by the requirement concerning a direct line of sight between human and robot and limited on-board feedback capabilities. A comprehensive user study dealing with different HRI methods for teaching virtual borders revealed augmented reality as the most powerful interface for the given task [45]. However, this approach requires specialized hardware, i.e. a RGB-D tablet, which limits the potential number of users. Nonetheless, the user study showed that 36% of the participants, i.e. the second largest group, also preferred a laser pointer as interaction device due to its simplicity, comfort and intuitiveness during interaction. These advantages of a laser pointer are also revealed in other robot tasks, such as guiding a mobile robot to a 3D location [21], controlling a robot using stroke gestures [19] and teaching visual objects [34].

2.2.Network robot systems

In order to benefit from the advantages of laser pointers and reduce their limitations caused by the requirement concerning a direct line of sight for interaction and limited on-board feedback capabilities, we investigate the incorporation of a smart home environment with additional sensors and actuators into the interaction process. This integration of robots into smart environments is known as network robot systems (NRS) [38]. Such a system is characterized by five elements: physical embodiment, autonomous capabilities, network-based cooperation, environment sensors and actuators and human-robot interaction. Other related terms are ubiquitous robots [23], physically embedded intelligent systems (PEIS) [36], Kukanchi [30] and informationally structured environment [31]. This ubiquitous robotics paradigm is leveraged to provide more complex and more efficient robot services to humans. For example, Sandygulova et al. developed a portable ubiquitous robotics testbed consisting of a kitchen equipped with a wireless sensor network, a camera and mobile robot [37] and Djaid et al. integrate a robot wheelchair with a manipulatable arm into an intelligent environment [11]. Other applications comprise the improvement of a mobile robot’s object search [41], the development of a human-aware task planner based on observations of the smart environment [9] and a gesture-based object localization approach for robot applications in intelligent environments [42]. All these approaches show that the cooperation between robots and a smart environment can improve the quality of robot applications, e.g. through enhanced perception and interaction abilities [40].

2.3.Interaction in smart environments

However, only a single attempt was made to employ a smart environment to restrict the workspace of a mobile robot, i.e. Liu et al. use several bird’s eye view cameras mounted in the environment to visualize a top view of the environment on a tablet’s screen on which the user can draw arbitrary restriction areas [26]. But this approach relies on a full camera view coverage of the environment and does not deal with partial occlusions, e.g. by furniture. Therefore, it is not applicable to our problem because a home environment is typically not fully covered by cameras due to privacy concerns [3]. Nonetheless, there are already works dealing with the recognition of laser pointer gestures using cameras of a smart environment, e.g. Shibata et al. propose a laser pointer approach to allow humans to control a mobile robot [39]. Similarly, Ishii et al. use cameras integrated into the environment for laser spot tracking allowing a human to define stroke gestures with a laser pointer to control a network-connected mobile robot [19]. These approaches are no more limited to the mobile robot’s field of view and thus increase the interaction space. However, the gestures are exclusively recognized by cameras integrated in the smart environment. Hence, the interaction strongly depends on the number of cameras in the environment and their fields of view. There is no work considering a cooperative behavior employing cameras from the smart environment and the mobile robot for this task. This would be an opportunity to overcome issues concerning camera view coverage and occlusions. Moreover, smart environments provide additional visualization capabilities that could be used to provide feedback to the human, e.g. visual displays integrated into the environment [6,20].

2.4.Research gap

These findings lead us to the general question, we want to answer in this work: how can a smart home environment improve the interaction process of restricting a mobile robot’s workspace using a laser pointer with respect to the interaction time and user experience? This question involves research questions of (1) which sensors and actuators of a smart environment can be used to benefit the interaction process, (2) how to realize a cooperation of human, robot and smart environment in the interaction process and (3) how to cooperatively perceive and combine multiple camera observations of laser points to restrict the mobile robot’s workspace. These questions are the basis for the remainder of this work, which will be answered by prototypically implementing a solution and an empirical evaluation.

4.1.Smart environment design

As stated in Section 2, laser pointers are quite popular for human-robot interactions involving the transfer of spatial information due to their natural interaction mimicking human pointing gestures. For example, our baseline method for the restriction of a mobile robot’s workspace employs a laser pointer as interaction device [46]. Nonetheless, this kind of interaction features some drawbacks. The major drawback of the existing interaction method is the requirement concerning a direct line of sight between human and robot when specifying virtual borders. Thus, the mobile robot has to follow the laser spot, but it is restricted by velocity constraints, which lead to a relatively long interaction time. Moreover, only limited feedback about the current state of the interaction process can be conveyed using simple on-board LEDs and non-speech audio sound, e.g. the robot provides a beep sound whenever a laser spot is detected. However, no complex feedback concerning the spatial information of the specified virtual borders can be provided to the human using these communication channels. Another drawback of the baseline approach is the interaction to change between different states of the interaction process, e.g. specifying virtual border points or the seed point. For this purpose, visual codes generated by the laser pointer or push buttons on the mobile robot are provided. But visual codes can be error-prone due to changing light conditions and interaction using buttons requires the user to be in the vicinity of the robot.

Addressing these shortcomings of the baseline approach, we propose the incorporation of additional smart home components in the interaction process as the answer to our first research question. This deals with the question of which sensors and actuators of the smart environment can be used to benefit the interaction process. Our proposed smart home environment is shown in Fig. 1 and mainly consists of three components: (1) a camera network comprising stationary RGB cameras, (2) a smart display and (3) a smart speaker allowing the processing of voice commands. The first component, i.e. the camera network, is intended to increase the perceptual capabilities in combination with the mobile robot’s on-board camera. Stationary cameras are integrated (yellow and red fields of view) to cover certain areas of the environment, while a mobile camera mounted on a robot (blue field of view) can observe areas that are not covered by the stationary cameras due to their installation or occlusions, e.g. under the table. Hence, this combination allows perception even if the stationary cameras do not cover all areas of the environment, which is typically the case in smart home environments. The second component, i.e. the smart display, is intended to provide expressive feedback to the user by visualizing the progress and result of the interaction process. Finally, it is the idea to employ a smart speaker to facilitate the change of different states of the interaction process using voice commands.

Fig. 1.

A person defines a virtual border in the environment using a laser pointer. The spot is observed by stationary cameras in the environment (yellow and red) and a mobile camera on a robot (blue). A smart display (top right) provides visual feedback of the complex spatial information, and a smart speaker facilitates interaction.

4.2.Human-robot-environment interaction

In order to achieve our objective of a reduced interaction time and increased user experience compared to the baseline approach, we propose a new interaction method leveraging the combination of a mobile robot and the described smart home environment in the interaction process. We denote this combination as network robot system (NRS) in the following. This interaction method is intended to answer our second research question of how to realize a cooperation of human, robot and smart environment in the interaction process. As pointed out in the introduction, the interaction process between human and robot (1) has to allow a transfer of spatial information from human to robot and (2) has to provide feedback about the interaction process from robot to human. The first property is addressed by allowing a user to specify virtual borders by “drawing” directly in the environment using a common laser pointer. A laser spot is cooperatively perceived by the stationary camera network and the mobile robot’s on-board camera.

Fig. 2.

Architecture of the cooperative perception for specifying virtual borders based on multiple camera views.

The interaction method comprises several internal states, that reflect the three components of a virtual border as described in Section 3. The states are described as follows:

4.3.Cooperative perception

While the incorporation of the smart display and smart speaker in the interaction method is straightforward, the incorporation of the camera network, which is used to increase the interaction space, is more challenging. There are mainly two reasons: (1) multiple cameras, stationary and mobile, have to be integrated into an architecture that supports the interaction process and (2) a single virtual border has to be extracted from multiple camera observations including noisy data.

4.3.2.Virtual border extraction

Fig. 3.

Processing stages extracting a polygonal chain from a point set including noise. Each row corresponds to a user-defined point set, a polygon in the first and a separating curve in the second row. The first column visualizes the input point set containing virtual border points but also noise and other clusters. The points assigned to the virtual border cluster are colored green in the second column. Thinning the virtual border cluster yields the red point set in column three. This is used to generate a polygonal chain as shown in the last column.

While the extraction of the seed point s is not challenging, the extraction of the polygonal chain P includes several challenges that need to be adequately addressed. For example, the 2D point set acquired from multiple camera observations contains data points of a single user-defined polygonal chain P. However, this point set also contains (1) noisy data points and possibly (2) additional clusters. These can occur due to errors in the laser point detection algorithm or other areas in the environment, that have the same characteristics as a laser point. Moreover, the data points can be (3) spatially redundant since the points are obtained from different cameras that may have an overlap of their fields of view, e.g. two overlapping static cameras of the environment or an overlap between a static camera and the mobile robot’s camera. Moreover, calibration inaccuracies of the cameras and localization errors of the mobile robot can lead to (4) inaccurate user-defined points. Finally, the (5) generation of a polygonal chain from the point set is challenging because the polygonal chain can have an arbitrary shape. We address these challenges with a novel multi-stage virtual border extraction algorithm, which is the second part of our answer to the third research question. Figure 3 illustrates the stages of the algorithm with two exemplary polygonal chains in the first and second row. We reference this figure throughout this subsection to explain the multi-stage algorithm.

Algorithm 1

Clustering step

Clustering The first stage of the extraction algorithm is the clustering stage as denoted in Algorithm 1. The input is a 2D point set pointsetIn as shown in the first column of Fig. 3, and the data points belonging to the polygonal chain pointsetOut are the result. This stage is designed to addresses the first two challenges of the virtual border extraction step, i.e. extracting of the user-defined polygonal chain and discarding noisy data points and irrelevant clusters. A cluster is a group of spatially nearby data points, while noisy data points are single data points that do not belong to a cluster. Irrelevant clusters are characterized by certain expansion characteristics, that indicate the spatial expansion of the cluster measured as Euclidean distance between the diagonal points of the cluster’s minimum bounding box. Due to the flexibility of a virtual border, i.e. a user can define arbitrary shapes, we applied the DBSCAN [12] algorithm for clustering the data points (l. 3). This is a density-based clustering algorithm that can find clusters with different shapes and sizes. It is parameterized by eps to define the distance threshold for a neighboring point and minPts to define a core point. This is a point that has at least minPts points within its distance eps. The result of the DBSCAN algorithm is a set of clusters where each point of pointsetIn is assigned to a cluster. Noisy data points, that do not belong to a cluster, are discarded. Afterwards, the algorithm selects the largest cluster with certain expansion characteristics defined by minExp and maxExp. To this end, we order the clusters by their sizes (number of points) in descending order (l. 4) to iterate over the clusters beginning with the largest cluster (l. 5ff.). The additional parameter minSize is a lower threshold for the size of a cluster to exclude small clusters due to noise. In each iteration, it is checked if the expansion of the current cluster c lies within the expansion thresholds minExp and maxExp to ignore irrelevant clusters (l. 6). The first cluster, that fulfills this condition, is returned as cluster of the polygonal chain. The result is visualized in the second column of Fig. 3 as green points. The black points are either noise or irrelevant clusters.

Thinning The second stage of the algorithm is the thinning stage (Algorithm 2), that reduces the number of points in the largest cluster from the previous stage. It is designed to remove spatially redundant data points and to smooth data points due to localization errors and calibration inaccuracies. Thus, it addresses the third and fourth challenge of the virtual border extraction step. For this purpose, the thinning algorithm identifies spatially nearby data points and replaces them by their mean value. To this end, the point p with most neighbors within a distance maxNeighborDist is selected (l. 4) and its neighboring points n are determined (l. 5). If there is at least one neighboring point (l. 6), the mean point is calculated for these points p∪n (l. 7). Afterwards, these points are removed from the initial pointsetIn (l. 8) and the mean point is added to pointsetOut (l. 9). In case that no data point has at least one neighboring point (l. 10), the iterative procedure terminates. Finally, all remaining points contained in pointsetIn, i.e. points without neighbors, are added to pointsetOut (l. 12), i.e. the set containing the thinned points. Thus, the thinned cluster includes the initial points, that do not have neighboring points, and mean points representing subsets of the initial points. The result is shown in the third column of Fig. 3. Compared to the second column containing the cluster of the polygonal chain, there are fewer points due to the reduction of data points.

Algorithm 2

Thinning step

Algorithm 3

Polygon generation step

Polygon generation Finally, the thinned point set is the input pointsetIn for the last stage, in which the polygonal chain polygon is generated (Algorithm 3). This algorithm consists of two phases, i.e. forward and backward, and addresses the fifth challenge of the virtual border extraction step. Since a polygonal chain has a starting and ending point, we first select an arbitrary point of pointsetIn as starting point and collect neighboring points in one direction. If there is no more neighboring point available, we again select our starting point and collect neighboring points in the other direction. Afterwards, the selected points are concatenated. In this context, direction corresponds to the sequence of the points of the polygonal chain. For example, considering an arbitrary point of a polygonal chain, which is not the starting or ending point, there are two directions from this point, i.e. the directions to the starting and ending point. To realize this behavior, we first initialize two empty polygonal chains dir1 and dir2 for each direction (l. 2), and we set the variable forward, that indicates the phase of the algorithm (l. 3). We then select an arbitrary point p (here at index 0) of pointsetIn, mark it and append it to dir1 (l. 4ff.). Afterwards, the nearest neighboring point n within a distance maxNeighborDist, that it not already marked, is selected (l. 8). If there is a neighboring point n available (l. 9), we append n to one of the temporary polygonal chains depending on the variable forward, mark n and select n as the current point p (l. 10ff.). This procedure is repeated until there is no neighboring point n available for the current point p (l. 16), i.e. the neighboring points for the first direction are collected. In this case, we select our initial point again as current point p and switch the variable forward to collect neighboring points along the other direction (l. 17ff.). Subsequently, the same procedure is performed until there is again no neighboring point n available (l. 21) or all points of pointsetIn are marked (l. 7). As a last step, the order of the temporary polygonal chain dir1 is reversed (l. 22), and dir2 is appended resulting in the final polygonal chain (l. 23). This is necessary to create a single polygonal chain with a single direction. The result is visualized in the last column of Fig. 3.

We use the parameter values shown in Table 1 for the algorithms. These were determined experimentally.

Table 1

Parameter values for the algorithms

Stage	Parameter	Value
Clustering	eps	0.5 m
	minPts	1
	minExp	0.3 m
	maxExp	+∞ m
	minSize	10
Thinning	maxNeighborDist	0.1 m
Polygon generation	maxNeighborDist	0.5 m

5.1.Independent variables

In our experiment, we manipulated a single independent variable, i.e. the interaction method. This variable can have one of the two values:

5.2.Hypotheses

The objective of the experimental evaluation was the test of the following hypotheses:

5.3.Setup

Fig. 4.

Image and 3D sketch of the lab environment. The three evaluation scenarios are numbered, and the three cameras’ fields of view are visualized in red, green and blue. The position of the smart display for feedback is encircled in red, and the mobile robot’s initial pose is depicted in the bottom right of the sketch.

To test our hypotheses, we motivated our experimental setup by three scenarios for restriction areas as presented in the introduction, i.e. (1) privacy zones, (2) carpets and (3) dirty areas. For this purpose, we set up a home environment in our 8 m × 5 m lab environment comprising free space, walls, plants and furniture, such as tables and chairs. Additionally, we integrated adjustable walls with a height of 0.5 m to model two different rooms in the environment. Besides, we placed a carpet on the ground and some dirt in one area of the environment as basis for the evaluation scenarios. An image and 3D sketch of the environment is shown in Fig. 4.

As a mobile robotic platform, we employed a TurtleBot v2 equipped with a laser scanner for localization and a front-mounted RGB-D camera. This is a typical evaluation platform whose mobile basis is similar to a vacuum cleaning robot with differential-drive wheels. The camera’s color images were captured with a resolution of 640 × 480 pixels, and the depth sensor had a resolution of 160 × 120 pixels. Additionally, the robot had a colored on-board LED, three push buttons and a speaker. A prior OGM of the environment (2.5 cm per cell) was created beforehand using a Simultaneous Localization and Mapping (SLAM) algorithm running on the mobile robot [15], and the mobile robot was localized inside the environment using the adaptive Monte Carlo localization approach [13]. This allows the robot to determine its 3-DoF pose, i.e. position and orientation, with respect to the map coordinate frame.

In order to take advantage of a smart home environment as described in Section 4.1, we mounted three RGB cameras with an image resolution of 1920×1080 pixels on the ceiling (2.95 m height, pitch angle of 90∘). Thus, they provided top views of the environment. Their fields of view partly overlapped as illustrated in Fig. 4b, but they did not cover the entire environment. Hence, there was only a partial observation of the environment, which is typical for home environments. All RGB cameras were calibrated, i.e. their intrinsic camera parameters were known, and their relative transformations with respect to the map coordinate frame were determined in advance. We denote these stationary cameras as red, green and blue camera. The interaction using speech commands relied on a Wizard of Oz method in which a human operator reacted on the speech commands of participants, i.e. switching between system’s states per remote control. We did not use a voice-controlled intelligent personal assistant, such as Amazon’s Alexa or Google Assistant, due to network restrictions in the university’s network. However, this method was not recognized by the participants and did not change the way in which a participant interacted with the system. Moreover, we placed a 22-inch smart display on a table near the restriction areas to provide visual feedback to the participants. This display was network-connected to the system and showed the progress of the interaction process, i.e. the OGM of the environment, the mobile robot’s current pose and virtual borders if specified by the user.

5.4.Procedure

After setting up the experimental environment, each participant was introduced to the following three evaluation scenarios. These scenarios covered both types of virtual borders and are good representatives for restriction areas:

In this experiment, we applied a within-subjects design, i.e. each participant evaluated both interaction methods in all three evaluation scenarios. The order of the selected interaction method was randomized to avoid order effects. After selecting an interaction method, an experimenter explained a participant how to employ the selected interaction method, i.e. a short introduction into the different states of the system, how to switch between states and how to use the laser pointer. Afterwards, a participant had some time to get familiar with the interaction device, i.e. the participant could use the laser pointer and guide the mobile robot. This took approximately five minutes. Subsequently, the participant started to specify virtual borders for the three scenarios. The order of the different scenarios was randomized. At the beginning of each scenario, the mobile robot’s initial pose was set to a predefined pose to allow the comparison between the results. This initial pose, that was not covered by a stationary camera, is shown in the bottom right of Fig. 4b.

The shortest paths to the restriction areas in the three scenarios were between 2.50 m and 5.40 m (Scenario 1: 5.40 m, Scenario 2: 2.50 m and Scenario 3: 3.00 m). Since we applied a within-subjects design, this procedure resulted in six runs per participant (three scenarios and two interaction methods) and a participant could compare both interaction methods. After performing the practical part of the experiment, each participant was asked to answer a post-study questionnaire concerning his or her user experience with the interaction methods. An experiment with a single participant took approximately 20 minutes in total.

5.5.Measurement instruments

During the experiment, the time needed to specify a virtual border for each participant, interaction method and scenario was measured. Thus, we obtained six measurements for each participant. The time measurement started with the change from the Default state to one of the active states and ended with the integration of the virtual border into the prior map. Moreover, the time measurement was decomposed according to the three active states of the system, i.e. guiding the mobile robot (Guide), specifying virtual border points (Border) and the seed point (Seed), to identify reasons for potential time differences between the interaction methods.

In addition to the time, an experimenter documented if a participant successfully specified a virtual border for an evaluation scenario, i.e. a participant could specify the virtual border points P (independent of their accuracy) and a seed point s. The success values are used to assess the completeness of an interaction method. To this end, we calculate the ratio between the number of successful runs and the total number of runs for each interaction method and scenario.

To measure the user experience, participants were asked to answer a post-study questionnaire. In addition to general information, such as age, gender and experience with robots, the questionnaire contained five statements concerning the user experience, that could be rated on 5-point Likert items with numerical response format. This questionnaire was inspired by the questionnaire of Rouanet et al., who used a similar questionnaire to assess the usability and user experience of human-robot interfaces [34]. The questionnaire included the following statements about the experiment (translated from German):

The fourth evaluation criterion is the accuracy, that we assessed by calculating the Jaccard similarity index (JSI) between a user-defined UD and its associated ground truth GT virtual border:

5.6.Participants

The experiment was conducted with a total of 15 participants (11 male, 4 female) with a mean age of M=28.8 years and standard deviation of SD=11.4 years. The age group ranged from 17 to 55 years. Participants were recruited from the local environment by word of mouth and rated their experience with robots on a 5-point Likert item ranging from no experience (1) to highly experienced (5) with a mean of M=3.2 and standard deviation of SD=1.4. This corresponds to a moderate experience with robots and comprises users owning a mobile robot, such as a vacuum cleaning robot. However, they only deploy the mobile robots in their home environments according to the manual and do not know how they internally work.

5.7.Tools

We implemented all components of the system as ROS nodes [32]. ROS is a modular middleware architecture that allows communication between several components of a system, that are called nodes. ROS is the de facto standard for robot applications and provides a large set of tools to accelerate prototyping and error diagnosis. We organized all our nodes in packages to allow the easy distribution of our implementation. In order to save resulting maps of an interaction process for evaluation purposes, we implemented a node that directly stores the map on the hard disk whenever a new map is defined by a participant. Furthermore, we used integrated time functionality of ROS to perform time measurements for the evaluation. The visualization on the smart display was based on RVIZ, which is a 3D visualization tool for ROS.

5.8.Analysis & results

To test our hypotheses, we report the analysis and results of the experimental evaluation in this subsection. In case of statistical tests to identify differences between the interaction methods, we chose a significance level of α=0.05.

5.8.1.Interaction time

Fig. 5.

Interaction time for both interaction methods (blue and red contours around bars) based on the scenarios. Each bar is subdivided into the three active states of the system.

The interaction times of the experiment are summarized in Fig. 5. Each bar comprises the measurements of all 15 participants for an interaction method and scenario. Hypothesis 1 stated that the interaction time would be reduced when using the NRS compared to the Robot only interaction method. To test this hypothesis, we first ran Shapiro–Wilk tests to test for normality of the data (differences in the interaction times between the two interaction methods), which is an assumption of parametric statistical hypothesis tests, e.g. a paired t-test. These tests only became significant for the third scenario (p=0.016). Thus, we assume the data of the first two scenarios to be approximately normally distributed, while the third scenario is not normally distributed. Moreover, we interpreted the corresponding boxplots for outliers (1.5×interquartilerange). The boxplots revealed that there were some outliers in the data. Due to the presence of outliers and the violation of normality in the third scenario, we performed a non-parametric Wilcoxon signed-rank test to compare both interaction methods. The statistical results show that there is a significant difference between the interaction methods in all evaluation scenarios:

• – 1. Room exclusion: Z=−3.411, p<0.001

• – 2. Carpet exclusion: Z=−3.408, p<0.001

• – 3. Spot cleaning: Z=−3.409, p<0.001

Our proposed NRS approach is significantly faster compared to the baseline approach. This results in speedups of 2.8, 2.2 and 2.2 for Scenarios 1, 2 and 3.

Fig. 6.

User experience for both interaction methods showing mean and standard deviation depending on the statements.

The reason for this significant difference is revealed by the decomposition of the time measurements. While the time for the Robot only approach is composed of all active states of the system, our NRS approach does not include the Guide state. This is a consequence of the human-robot-environment interaction where the NRS automatically sends the mobile robot to the intended restriction area when a laser spot is detected by a stationary camera. Therefore, the user is not restricted to the mobile robot’s line of sight and does not have to manually guide the mobile robot to the intended restriction area. Thus, the NRS approach can avoid the time in the Guide state. This time primarily depends on the distance between the robot’s start pose and the restriction area. As described in Section 5.4, the distances in our scenarios ranged from 2.5 m to 5.4 m. If the distances would be smaller, the time in the Guide state would also decrease. However, we think that we chose the distances quite liberally since much larger distances would be even realistic, e.g. in typical home environments with a single charging station for the mobile robot.

Another reason for the time difference between the interaction methods is the time in the state Border, which is linear with respect to the border length [46]. Thus, if the user-defined virtual border is short, e.g. 0.70 m for Scenario 1, our NRS approach is only slightly faster in this state, i.e. 4 seconds difference. But if we consider a larger virtual border, e.g. the 6.50 m long border around the carpet (Scenario 2), our approach is even 26 seconds faster on average. The reason for this is the mobile robot’s velocity limitation (0.2 m/s) to ensure a safe and smooth movement of the robot. By using our NRS approach, this speed limitation can be reduced if the laser spot is in the field of view of one of the stationary cameras. Our interaction method is then only limited by the detection rate of the cameras (25 frames/s). Hence, it also features a linear interaction time, but with a smaller gradient.

Another speedup is achieved when specifying the seed point (Seed state) because a user can directly indicate the seed point with laser points, which are detected by a stationary camera. In case of the Robot only approach, a user additionally has to rotate the mobile robot around its vertical axis to adjust the camera’s field of view. This rotation takes additional interaction time. In summary, the results support Hypothesis 1.

5.8.2.User experience

In order to assess the user experience and test Hypothesis 2, i.e. the user experience is better when employing the NRS compared to the Robot only interaction method, we considered the participants’ answers of the questionnaire introduced in Section 5.5. The participants’ answers to the questionnaire are visualized in Fig. 6 with their mean (M) and standard deviation (SD) per statement and interaction method.

Since Likert-item data violate the assumption of normality, we ran a Wilcoxon signed-rank test on the 15 answers per statement to determine whether there are statistically significant differences between both interaction methods. We found statistically significant differences for all statements in the questionnaire:

• – Statement 1: Z=−3.051, p=0.002

• – Statement 2: Z=−2.830, p=0.004

• – Statement 3: Z=−3.126, p<0.001

• – Statement 4: Z=−2.373, p=0.023

• – Statement 5: Z=−2.745, p=0.005

Our proposed interaction method based on a NRS is better rated by the participants for all statements compared to the baseline approach. Participants had significantly less problems defining the virtual borders with a NRS (M=4.53) than without support of a smart environment (M=3.80).33 A reason could be that some participants had problems to rotate the mobile robot around its vertical axis, e.g. to specify the seed point. In this case, they moved the laser spot too fast so that the robot’s on-board camera could not follow the spot on the ground. In contrast to this, our approach avoids this problem by the additional cameras in the environment. Participants also found our approach more intuitive (M=4.60 and M=3.60) because they could more directly specify a virtual border without concentrating on the mobile robot. This was automatically sent to the restriction area when a stationary camera detected a laser spot. Furthermore, the speech commands provided a more intuitive interaction medium to change the system’s internal state than pushing on the robot’s buttons. The strongest effect was measured for Statement 3 that shows that our approach is less physically or mentally demanding (M=4.47) compared to the baseline approach (M=3.13). This coincides with the results of the interaction time as reported in Section 5.8.1. Another difference was observed for the learnability of the interaction methods (NRS: M=4.60 and Robot only: M=3.87). Since both interaction methods are based on a laser pointer as interaction device, the handling of a laser pointer does not influence the learnability. However, it could be easier to learn the speech commands than the assignments of the buttons to change between states of the system. Furthermore, the guiding of the mobile robot could affect the rating. Finally, there is a significant difference for the feedback of the interaction method (M=4.33 and M=3.07). Since the feedback system of the NRS is a superset of the baseline’s feedback system, a major reason for this difference is the additional smart display that visualizes the OGM with the user-defined virtual borders. This kind of feedback is missing for the Robot only approach that only features sound and colored LED feedback. The answers to the question if the smart environment supported the interaction process are summarized in Fig. 7. Most of the participants (14 out of 15) felt that the smart environment supported the interaction process. In summary, Hypothesis 2 is supported by the results.

Fig. 7.

Answers of the participants to the question if the smart environment supported the interaction process.

5.8.3.Completeness

Hypothesis 3 stated that the completeness does not significantly get worse when employing the NRS compared to the Robot only interaction method. The success rates are summarized in Fig. 8 as bars. Both interaction methods feature the same high success rate, i.e. 91.1% on average. Furthermore, the success rates for the three scenarios are the same (93.3% for Scenarios 1 and 2; 86.7% for Scenario 3). There were nine participants who performed all their runs successfully, four participants who failed for one of their six runs, and two participants incorrectly defined a virtual border in two of their six runs. The reason for these incorrect runs was the definition of the seed point s. While some participants were confused where to specify the seed point s, especially in Scenario 3, other participants were unfocused and noticed their mistake on their own after performing the experiments. There were no problems with the definition of the virtual border points P. Since both interaction methods feature an equally high completeness, we assume Hypothesis 3 to be supported by the results.

Fig. 8.

Completeness for both interaction methods depending on the scenarios.

5.8.4.Accuracy

Fig. 9.

Quantitative accuracy results for both interaction methods depending on the scenarios.

Fig. 10.

Visualization of representative accuracy results. The first row shows ground truth maps for all three scenarios containing occupied (black), free (white) and unknown (gray) areas. The yellow cells indicate the area to be specified by a participant during an interaction process (GT). The robot’s start pose is visualized as a red arrow. The following rows visualize the qualitative accuracy results for both interaction methods. A user-defined area as result of an interaction process is colored in green and red. Green pixels indicate the overlap of ground truth and user-defined areas (GT ∩ UD), while red pixels show areas defined by the user but not contained in the ground truth map (UD ∖ GT). A blue contour surrounds the union set of both areas (GT ∪ UD). Colors are only used for visualization.

Finally, Hypothesis 4 stated that the accuracy does not significantly get worse when employing the NRS compared to the Robot only interaction method. The JSI values resulting from the experimental evaluation are presented in Fig. 9. To test the hypothesis, we again explored the data for normality and outliers using Shapiro–Wilk test and boxplot interpretation. The Shapiro–Wilk tests did not become significant for any scenario indicating a normal distribution of the data. However, since the data for Scenarios 2 and 3 contain some outliers, we preferred a Wilcoxon signed-rank test to a paired t-test. The statistical results were different for the three scenarios:

• – 1. Room exclusion: Z=−0.594, p=0.588

• – 2. Carpet exclusion: Z=−3.110, p<0.001

• – 3. Spot cleaning: Z=−2.497, p=0.010

There is no significant difference for Scenario 1 (NRS: Mdn=0.971, Robot only: Mdn=0.974), the NRS (Mdn=0.856) performs significantly better than the Robot only (Mdn=0.779) in Scenario 2 and the Robot only (Mdn=0.906) has a significant higher accuracy than our NRS (Mdn=0.878) in Scenario 3. However, although the results are significant for Scenario 3, the difference between the medians is only 0.028, which is not notable in practice. This can also be qualitatively observed in Fig. 10, where the JSI can be visually interpreted as the fill degree of the green cells with respect to the cells encompassed by the blue contour. The results depicted in the figure are representative examples of different participants whose accuracy values coincide with the overall results per scenario and interaction method. The accuracy for Scenario 1 and 3 is similar, but there is a small difference for Scenario 2. The NRS interaction method features a higher fill degree of the green area with respect to the area surrounded by the blue contour compared to the Robot only approach. A reason could be that it is harder to specify the carpet’s corners when guiding the robot compared to our NRS approach. The highest accuracy value is achieved in Scenario 1 because the user excludes a relatively large area compared to the length of the virtual border. Thus, there is only minimal room for errors. In summary, the accuracy results support Hypothesis 4 because there is no significant degradation of the accuracy when employing the NRS approach.

5.9.Discussion

We designed our experimental evaluation according to the idea that the environment is partially observed by stationary cameras and that there are typical restriction areas, such as carpets or privacy zones. This is a typical setting for a smart home environment. In this case, our results suggest that a smart environment can significantly improve the interaction time (Hypothesis 1) and user experience (Hypothesis 2) while not negatively affecting the completeness (Hypothesis 3) and accuracy (Hypothesis 4).

However, there are two important aspects, that influence the speedup in the interaction time: (1) the stationary cameras’ coverage of the environment and (2) the distance between the mobile robot’s start pose and the restriction area. If we would decrease the number of cameras in the environment and thus the camera coverage, this would result in a smaller reduction of the interaction time. This would finally degenerate to the baseline approach if there is no camera coverage. Moreover, the speedup strongly depends on the distance between the mobile robot’s start pose and the restriction area, which influences the performance of the baseline approach. This is due to the fact that the baseline approach requires a direct line of sight between human and robot, and thus a human first has to guide the mobile robot to the restriction area. For these reasons, it is not possible to report a specific speedup value. Nonetheless, in the evaluation scenarios we chose a typical camera coverage and reasonable distances, which are typical for home environments with a single charging station for a mobile robot. Hence, we conclude that the interaction time improves with the support of a smart environment, but we cannot report a specific speedup value. Therefore, the reported speedups in Section 5.8.1 are intended to give an estimate and are only valid for this specific experimental evaluation.

In case of the user experience, participants appreciated aspects, such as a reduced mental or physical demand during interaction and an improved feedback of the system. These are effects of the incorporation of the smart environment’s components. However, a participant also wished an even stronger feedback system. The participant did not like the change of attention between the sketching of the virtual border on the ground and the view on the smart display, which was positioned aside on a table. Therefore, we conclude that our choice of smart home components improved the user experience significantly but the incorporation of more expressive feedback channels could improve the user experience even more.