Expressing Anger with Robot for Tackling the Onset of Robot Abuse

Robot abuse (also referred to as ‘robot bullying’ or ‘robot mistreatment’) has been reported to happen in public environments spontaneously [10, 24, 35, 37, 39]. The causes and mechanisms of robot abuse are not yet fully understood, but a number of findings exist. Curiosity, enjoyment or peer pressure are what children most commonly self-reported as reasons for abusing robots [28]. Yamada et al. [51] suggested that abusive behaviour follows a process of gradual escalation and identified specific events that lead to the escalation. Sanoubari et al. [38] compared human and robot bullying, proving that people perceived both as being wrong, although less strongly so for robots.

Various factors have been proposed as having an influence on the abuse, such as the perceived intelligence of the robot in [6] or the robot’s size in [25]. Keijsers et al. have shown that factors such as power, embodiment or mind attribution all play a role in robot abuse [22, 23].

Several specific methods to counter robot abuse have been discussed. Brščić et al. [10] proposed to let the robot escape from potential abuse by temporarily moving to a ‘safer place’, such as a more populated location. While they were able to reduce the number of abuse cases, it required the robot to interrupt its task so it was still not solving the disruption to the robot’s work. Tan et al. [47] have shown that it is possible to use a robot’s emotional reactions to induce human bystanders to intervene when robot abuse happens, whereas Connolly et al. [14] have proven that other robots in the group could use reactions to prompt people to help the abused robot. Both of these studies were run in a laboratory with only two people present and with small-sized robots. It is not known how well such these techniques would work in a real-world setting. The application of an educational technique called love-withdrawal was shown to have the potential to counter robot abuse in [18].

Finally, as one of the inspirations for this work, Scheeff et al. [39] anecdotally reported that boys have sometimes shown aggressive behaviour towards their social robot and that using angry expressions had a mitigating effect on the abuse, whereas other emotional expressions tended to have a detrimental effect.

Anger is common to all humans, and both expression and recognition of anger are universal across cultures [27]. People get angry for several reasons, and many researchers agree that people use it to prevent violations against them and to prevent being subordinated [32]. For example, according to the recalibration theory of anger [40], anger is an evolutionary adaptive mechanism that serves to communicate when one’s social value or status has been threatened, and one important purpose it has is to signal to others that we consider a situation to be unjust or disrespectful. Following that reasoning, seeing a robot express anger might also be a natural and easy-to-understand way of signalling that someone’s actions are not appropriate.

There is more than one way people can express anger. Speilberger et al. [44, 45] identified distinct factors of anger expressions: Anger/In—inward suppression of angry feelings, and Anger/Out—outward expression of anger as aggressive behaviour directed towards people or objects, to which they also add a third one: Anger/Control, which measures the degree of controlling the intensity and expression of anger.

Anger and other emotions in humans also manifest themselves through a number of modalities: facial expressions, body posture and movement, physiological characteristics (blood pressure, heartbeat, etc.), voice and speech content [12]. For example, the work by Ekman and Friesen [17] connects facial expressions for specific emotions to individual movements of the facial muscles.

Finally, human anger and anger expressions are also not constant and tend to change over time [31]. One of the distinct characteristics of the temporal dynamics of anger is that the intensity of anger tends to rise quickly but fall slowly over time [7, 33].

There is a large body of literature on the expression of emotions in robots. Similar to human emotional expressions, robots can also use various modalities for expressing anger and emotions.

Quite a few studies focused on robots’ facial emotional expressions, either through fully rendered faces or faces including mechanical parts [20, 34, 36]. The facial expressions for anger were studied also in virtual agents—e.g., de Melo et al. [16] show that in a negotiation task, people tend to concede more to an agent that shows anger than to one with a happy face. A large number of these works are based on Ekman’s model of facial expressions [17]. Most of them included anger as one of the emotions for which they implemented a facial expression.

Robot movement was also studied as a way to express emotions [21]. For example, Yagi et al. [50] used a humanoid robot’s upper body motion to express a range of emotions, including anger. Ajibo et al. [1] considered the use of gestures with an android robot specifically for expressing anger. Apart from movement, other expression modalities have been studied too [8], including ones that humans do not typically use, such as the use of colour or vibrations to express emotions in [43].

Only a few works reported purposefully using robot anger to affect people’s behaviour. For example, some of the early tour-guide robot implementations used angry facial expressions when people persistently blocked the robot’s path [42, 49]. Similarly, the already mentioned work by Scheeff et al. [39] employed anger to counter aggressive behaviour, with limited success. Reyes et al. [36] used facial expressions to give feedback during human–robot collaborative tasks and showed that negative expressions such as anger worked well for attracting the user’s attention and making them follow better the instructions. Despite the diversity of human anger expressions [44], previous studies have typically relied on only one way of expressing anger.

It is not obvious how should the expression of anger be implemented on a robot, especially when the robot has limited expression abilities. The few works in the literature on anger expressions in robots cited above provide hints, but there are no detailed guidelines on how to design anger for a specific robot. Therefore, to understand how to design anger expressions, we started by taking a look at human angry behaviour.

While there is no universal methodology for identifying human behaviours in specific situations and copying them to a robot, we used a two-step approach, with each step following typical practices from the literature. First, as described in this section, we identified the range of angry behaviours people may display in situations when they are being obstructed or abused. We used interviews and relied on the participants’ imagining and reflecting on a specific situation further enhanced by allowing them to enact it (this was inspired by methods such as [29]). The second step (in the following section) involved finding the specific expressive actions that humans do during each angry behaviour type, where we relied on the method of bodystorming [30] that has been frequently used for designing robot movements and expressions for interactions.

We collected examples of angry behaviour through interviews with participants—10 members of our laboratory. We first let the participants watch videos with examples of real-world robot abuse, which were collected during the study in [10]. After that, we posed them the question: ‘If you were in the place of the abused robot and angered, how would you behave?’ We let the participants freely talk about their suggested reactions, occasionally prompting them to think about other ideas by asking follow-up questions such as ‘What would you do if the children still did not move away?’ We also encouraged them to physically show the reactions and motions that they would use. The interview ended when the interviewee ran out of ideas.

A thematic analysis of the interviews conducted by one of the authors (K.N., following the 6-phase procedure from [9]) resulted in three main themes, indicating that the suggested reactions can be categorised into three main types: (1) expressing strong anger to startle the person (‘try to get them to move away by behaving more erratically than they do’, ‘using loud and violent, threatening behaviour’), (2) ignoring the person as much as possible (‘not speak or make eye contact to convey your intention not to be involved’) and (3) trying to persuade the abusing person by asking why they are doing this (‘ask them why they won’t move away and try to persuade them’). Interviewees mentioned other reactions, such as calling the police, but these were not considered to be expressions of anger so we excluded them.

Based on the results of the thematic analysis and through subsequent discussions between all three authors, we defined the following three distinct anger expression types:

It can be hard to imagine and design robot behaviours based solely on these definitions. Because of that, we additionally came up with ‘mental images’ describing how we envisioned each anger type should look like, i.e., the appearance that we wanted to achieve: furious anger as ‘someone almost mad with rage’; quiet anger as ‘trying to restrain, but actually very angry and about to explode any time’, and scolding anger as ‘adult, such as kindergarten teacher, who is very annoyed with a child, but holding back anger’.

While these were the three ways of expressing anger that came from the interviews, it is not guaranteed that they cover all possible types of anger expressions. Nonetheless, note that there are parallels between these three robot anger expression types and the three measurement factors of anger expression in humans as discussed in Spielberger et al. [44]. Furious anger could be seen as roughly corresponding to a high level of ‘Anger/Out’, quiet anger to a high ‘Anger/In’ and scolding anger to high ‘Anger/Control’.

To design the expressions for the three anger types, we further enlisted colleagues in our laboratory to put themselves in the robot’s shoes and express the defined three anger types in role-plays. The role-play consisted of the first author acting out an abusive person and the participants in the experiment showing their anger in response (such robot behaviour design through acting out is essentially the method of bodystorming described in Porfirio et al. [30]). Each role-play session lasted about 30 minutes.

Although not all of the observed expressions could be transferred well to the robot due to the limitation in its movements and expressive abilities, the data provided us with the starting examples of robot gestures and expressions. We then iteratively expanded and adjusted the expressions to match as closely as possible the mental images for each anger type described above. The basic details of the designed expressions are summarised in Table 1.

For the utterances of scolding anger, we purposefully used expressions that were adult-like and slightly too difficult for a child to understand. The reason was that we wanted to create an impression of intelligence, as this was suggested in previous work to be effective against robot abuse [6].

Photos of the robot during each of the three anger expressions can be seen in Figure 1(b)–(d). In addition to the three anger expressions, we also prepared a ‘normal’ expression, where the behaviour was designed such that it shows no emotional reaction, Figure 1(a): the gestures and movements are calm, there is no change in the voice and the utterances are limited to simple requests to stop the bothering behaviour. A video showing the expressions can be found in the supplementary materials.

From studies of human anger, it is known that anger intensity changes gradually [31]. This means that there are not just angry and not-angry states, but there are also states in between, such as mild irritation. Thus when a robot uses expressions of anger, direct switching between calm and full-blown anger states should be avoided, as it would feel unnatural to the observer. Also, due to its negative connotations, full-blown anger should probably be reserved only for most serious cases and otherwise avoided. However, implementing a full spectrum of intermediate anger states would be challenging, since it is not clear how to gradually change expressions such as gestures or utterances.

As a compromise, we used a three-level model of the robot’s anger intensity. We refer to these levels as ‘normal’, ‘mild anger’ and ‘strong anger’. The normal level corresponds to no anger, and in terms of behaviour, it is equivalent to the aforementioned ‘normal’ expression (Figure 1(a)). The strong anger corresponds to a high anger intensity, for which we used the anger expressions as described in Section 3.2. The mild anger level stands for a moderate intensity of anger. We designed the mild anger level for each of the three anger types by altering the expressions from Section 3.2 to make them appear less intense. For example, we used only half as much change in the voice loudness, rate and pitch, less forceful gestures, and we replaced some utterances with milder versions.

The gradual change of the robot’s anger intensity in response to the external stimuli was modelled mathematically as a change in an internal variable, which we denoted \(\eta\). The variable varied from 0 (no anger) to 100 (maximum anger intensity). When obstructed, the robot’s anger would increase, modelled as an increase in the value of \(\eta\). When undisturbed, the value would decrease. Since it is known from research on human anger that the rise of anger intensity is typically much faster than the fall [31], we used different factors for the increase and decrease of \(\eta\). The following expression shows the change of the value of \(\eta\) after time \(t\) (in seconds):

The \(\eta\) is then mapped to the three anger levels in the following way: \(\eta\in[0,65)\mapsto\) normal, \(\eta\in[65,95)\mapsto\) mild anger, \(\eta\in[95,100]\mapsto\) strong anger. The specific values used in this model were manually tuned such that they resemble the changes in anger intensity that a human would have in a similar situation.

For measuring the perception of the expression that the robot had in the video, we used a single-item questionnaire with the question ‘Did you feel that the robot was angry?’, with responses given on a 7-point Likert scale.

As previous research has shown that intelligent robots are less likely to be abused [6], we additionally also wanted to evaluate the perceived intelligence of the robot. As a measure, we used the Intelligence questionnaire from the Godspeed Questionnaire Series [5]. The responses were given on a 7-point scale.

Even though the intended target for the robot’s expressions was children, we hypothesised that there would be little difference between how children and adults perceive anger. We thus conducted the survey with adults, which was also easier to implement than with children.

We hired an online recruitment agency that recruited participants between 20 and 50 years of age, aiming to achieve a good balance of the participant’s age and gender. After excluding seven duplicate or invalid responses, we were left with 112 valid responses in total. The distribution of the participants was approximately balanced across both gender (M/F: 59/53) and age (20s/30s/40s: 37/35/40). All recruited participants were paid.

A repeated-measures ANOVA has shown a statistically significant effect of the conditions on the perceived anger, with a large effect size (\(F(3,333)=111.49\), \(\text{p} \lt .001\), \(\omega^{2}=.299\)). Post hoc analysis with Bonferroni correction revealed significant differences between the normal condition and each of the three anger conditions (for all \(\text{p} \lt .001\)), Figure 2(a). The average scores for all three anger conditions were higher than the score for the normal condition, showing that the participants perceived all three designed robot behaviours as expressions of anger.

Running a repeated-measures ANOVA on the intelligence questionnaire scores revealed a significant effect of anger conditions (\(F(3,333)=16.491\), \(\text{p} \lt .001\), \(\omega^{2}=.072\)). Post hoc comparisons with the Bonferroni method have shown significant differences between the pairs normal-furious and normal-quiet, as well as between the pairs scold-furious and scold-quiet (for all \(\text{p} \lt .001\)). The scold and normal conditions had a higher score than the other two conditions, which shows that between the three anger conditions, the robot expressing the scolding anger was perceived as more intelligent and on par with the non-emotional robot, whereas the other two anger types were perceived as less intelligent.

We used a within-subject experiment design. Consequently, we wanted each child to interact with the robot four times, in order to experience all anger expressions plus the robot not expressing anger. To achieve that, a key requirement was that children needed to do obstructive interactions with the robot in all conditions. However, designing an experiment in a laboratory setting in which children spontaneously initiate a bothering interaction with the robot and ensure that it occurs consistently across all four conditions can be very challenging.

To address this issue, we took inspiration from the findings in Yamada et al. [51] which have shown that the presence of other children and seeing robot abuse by other children is what typically encourages a child to start the abuse themselves. Therefore, we placed a confederate in the room with the robot and child who would guide the child to initiate the interaction with the robot. Obviously, having a child confederate would be difficult, so instead we enlisted a student colleague from our laboratory, who was not informed of the hypotheses of the experiment, to be the confederate.

The experimental space consisted of two rooms (A and B). At the beginning of the experiment and before interacting with the robot, we let the children solve simple arithmetic problems in room A for 6 minutes. After that, we led them to room B to ‘have a break’ for 6 minutes. In room B there was a robot, Robovie-II, driving back and forth. The confederate would briefly explain to the child that the robot is currently patrolling the room, and then stand in front of the robot to show the child how the robot will react.

The robot would detect when someone is obstructing its way and say something like ‘Please, move aside’ or ‘I am patrolling and you are in the way’. Depending on the experiment condition, if the obstruction persisted the robot would gradually get more angry, according to the anger intensity change model in Section 3.3.

After demonstrating the robot’s initial reaction, the confederate would ask the child to try obstructing the robot by standing in front of it. Once the child had seen both the mild and the strong anger reaction, the confederate would say that they had things to do so they would retreat to the corner of the room. They told the child that they could continue interacting with the robot if they wished and also reminded them that it was their break time so they were free to do what they wanted. After 3 minutes from the start of the interaction, and again after 5 minutes from the start, if the child was not interacting with the robot at that point, the confederate approached the child and tried to encourage them once to interact again with the robot. After that, he went back to sitting. At all other times, he sat quietly and avoided all interaction. If addressed, he would politely decline to interact saying that he cannot talk now. The whole interaction in room B was recorded using four cameras and microphones installed on the room ceiling.

After 6 minutes in room B, the child was led back to room A and asked to fill out a questionnaire about the robot. They then returned to solving arithmetic problems and the whole process was repeated for the remaining three conditions. The complete flow of the experiment is illustrated in Figure 3.

We used the same robot in the experiment in all conditions. To reinforce the impression that it is a different robot, we altered its appearance by putting a different hat on its head, and a name tag around its neck with a different name each time, Figure 4. The specific order of name tags and hats was randomised between participants and was independent of the order of conditions.

The study was approved by our institution’s Research Ethics Board.

Nomura et al. [28] reported that children who engaged in robot abuse in their study were between 4 and 9 years of age. Following that, and considering the difficulties in running experiments with very small children, we chose to have second- and third-grade elementary school children (7 to 9 years old) as participants.

Based on an a priori power analysis, we estimated the needed number of participants to be 24 (calculated using the G*Power software assuming a medium effect size \(f=.25\), with. 80 power and an alpha level of. 05).

The participants were recruited through a recruiting agency. We had to stop one experiment midway through due to a robot hardware failure, so the data from one of the participants could not be used. The distribution of the gender of the remaining 23 participants was M/F: 12/11. Each child participant and accompanying parent were compensated 14.000 Japanese yen for their participation.

We tested four conditions: Normal, Furious, Quiet and Scold, during which the robot was using different emotional expressions. In the Normal condition, the robot was not expressing anger. When persistently obstructed it would only repeatedly ask the child to move away from its path. In the other three conditions, the robot would react to continuous obstruction by a gradual increase in anger intensity, as described in Section 3.3. The difference between the conditions was only in the type of anger that the robot used: furious anger in the Furious condition, quiet anger in the Quiet condition and scolding anger in the Scold condition.

To avoid ordering effects as much as possible, the order of the conditions was counterbalanced. Given four conditions, there are 24 possible sequences, ensuring that each of the 24 participants experienced a different sequence of conditions. (This ordering was independent of the order of the robot’s hats and names described in Section 5.2.)

Our main prediction was that anger expression would have an effect on how much children obstruct the robot. Thus, we hypothesised the following.

In addition, following the result from Bartneck and Hu [6] that robots that appear to be more intelligent are less likely to be abused, we expected that the Scold condition would have a stronger effect on the children’s abusive behaviour than the other anger expressions. Hence, we made the following hypothesis.

During the experiment, most of the children participants obstructed the robot on their own through all four conditions, confirming that our experiment design worked well. Figure 5 shows an example scene at a moment when a child was left alone to interact with the robot.

There was a large variety in the ways how participants behaved during the experiment. Some children obstructed the robot over and over again on their own. In contrast, some children were reluctant to do any obstructive behaviour and did not even approach the robot unless invited to do so by the confederate.

In most cases, the children were obstructing the robot by simply standing in front of it. The participants mostly did not talk nor tried to address the confederate. In a few cases, we observed other actions such as putting the hand in front of the robot’s eyes, getting close to the robot and staring it in the face, or touching the robot’s hand, which could be seen as a slight escalation of abuse from just standing in front of the robot. One child who eagerly obstructed the robot at one point even laid down in front of the robot and pretended to sleep.

Another unexpected behaviour that we frequently observed was that the children would sometimes walk either directly behind or alongside the robot. As the robot only considered standing in front of it to be a hindrance, it did not show any reaction to the children in these situations. The children would continue to closely follow the robot for some time while it was patrolling and would not try to stand in front of it.

Based on videos collected during the experiment, the start and endpoints of the obstructing interactions were labelled, according to the criteria defined in Section 5.6.1. To ensure objectivity, the coding was done by a colleague from our laboratory who was not familiar with the purpose and settings of the experiment. Based on the labelled interaction start and endpoints, we calculated the scores for the number of interactions and the duration of interactions. A box-whisker plot of the results is shown in Figure 6(a).

We ran a repeated-measures ANOVA on both scores. For the measurement of the number of interactions, this revealed a statistically significant effect of anger expressions (\(F(3,66)=12.696\), \(\text{p}=.013\), \(\omega^{2}=.027\); this corresponds to Cohen’s \(f=.42\), which suggests that the study had enough power). Post hoc analysis using the Bonferroni method revealed a significant difference only between the Normal (\(M=4.65\), \(SD=3.74\)) and Furious (\(M=2.91\), \(SD=2.66\)) conditions, \(\text{p}=.011\). There was no significant effect between other pairs (e.g., for Normal-Quiet \(\text{p}=1\), for Normal-Scold \(\text{p}=.158\)).

For the duration of interactions, no significant effect was found (\(F(3,66)=1.683\), \(\text{p}=.179\), \(\omega^{2}=.01\)).

Based on these results our hypothesis H1 was partially supported. Namely, Furious had a significant effect on the number of interactions (but not the duration), whereas the other two anger types did not. Hypothesis H2 was not supported, since scolding anger did not lead to higher scores than the other two anger expressions.

We analysed the questionnaires filled out by the children participants to find out whether their perception of the robot’s anger and intelligence matched the results from the online video survey. We ran a repeated-measures ANOVA on both the perceived anger and perceived intelligence scores, followed by post hoc comparisons using \(t\)-tests with Bonferroni correction.

The result for anger revealed a significant effect of the conditions (\(F(3,66)=14.718\), \(\text{p} \lt .001\), \(\omega^{2}=.29\)), and the post hoc analysis revealed significant differences between the pairs Normal-Furious (\(\text{p} \lt .001\)), Normal-Quiet (\(\text{p} \lt .001\)) and Normal-Scold (\(\text{p} \lt .001\)). We also found a significant effect for the intelligence score (\(F(3,66)=7.987\), \(\text{p} \lt .001\), \(\omega^{2}=.068\)), with post hoc analysis showing differences between Normal-Furious (\(\text{p} \lt .001\)), Normal-Quiet (\(\text{p}=.032\)), Scold-Furious (\(\text{p}=.001\)) and Scold-Quiet (\(\text{p}=.010\)).

These results are very close to the results obtained in the online survey, suggesting that the children had essentially the same perception of the robot’s anger and intelligence as the adult survey participants.

The post-experiment interviews were transcribed after which one of the authors (D.B.) performed a deductive thematic analysis following the basic six-phase procedure described by Braun and Clarke [9]. As a result we identified seven main themes:

The results of the questionnaire filled out by the participants suggest that children had the same perception of the robot as the adults in the online survey and that they understood that the robot was expressing anger when they persistently stood in front of it. Nevertheless, we only found that this had a significant effect on their obstructing behaviour in the case of furious anger, which shows that not all anger expressions are equally effective for preventing hindering.

Many children mentioned curiosity as the reason for obstructing the robot. The eagerness to interact and actions such as lying down and pretending to sleep also suggest that some participants found the interaction to be enjoyable. This matches well with the results of the study of Nomura et al. [28] where children reported curiosity and enjoyment as the main reasons why they abused the robot.

Furious anger proved to be effective in influencing obstructive interactions; yet, we do not know a definite reason why. We suspect that the ‘scariness’ of the furious robot, which many of the children referred to in their interview responses, may have been one factor influencing the hindering behaviours. However, as we did not explicitly measure the perception of the robot’s scariness, we cannot quantitatively confirm this conjecture.

Meanwhile, we did not find an effect of the perceived intelligence on the children’s hindering. This seemingly contradicts the result from Bartneck and Hu [6], who reported a significant influence of the robot’s intelligence on robot abuse. That could be due to the difference between the behaviours that were studied, since in [6] the participants were asked to destroy a robot and not just hinder its movement. But it could also be due to differences between children and adults, as their study was done with 19- to 25-year-old participants.

We found a significant effect of anger expressions on the number of interactions, but not on the duration of interactions. We expected that these two measures would be correlated, and indeed the distributions of the scores (Figure 6) did show a similar trend—e.g., on average, in both cases, the Furious condition scores were lower than in the Normal condition. One possible explanation for why there was no significant effect on the interaction duration is that some of the children seemed not to be deterred by the robot’s anger at all so they engaged in very long interactions regardless of the condition, which can be seen as outliers in Figure 6(b). As a consequence of these outliers, the F-score was lower and the differences were not significant. But these data also provide a hint that a single solution like expressing anger may not be effective on all children.

In contrast to the children who eagerly interacted, a few were very reluctant to do it. Four of the children spent less than 10 seconds in total over all four conditions standing in front of the robots. This is somewhat similar to the rejection of interaction with robots by some children reported in Shiomi et al. [41], although they studied younger children. Arguably, such reluctant children would be unlikely to persistently bother the robot even in public, so they would not become the target of robot abuse prevention mechanisms studied in this work.

While earlier works reported that robot abuse was difficult to stop no matter what the robot did [10, 51], our study has shown that expressing anger is more effective as a signal of mistreatment and as a deterrent from further escalation of abuse. Nonetheless, anger is generally a negative emotion that may also cause negative reactions in the recipient of anger [32], so it is essential to consider how appropriate it is for robots to use expressions of anger towards people.

One argument for the use of anger is that it may be very important to prevent serious robot abuse. In particular, exhibiting or experiencing such negative behaviour towards robots could desensitise people to negative behaviours towards humans [15], which Axelsson et al. [2] refer to as ‘behaviour enforcement’. For example, if robots are employed in educational settings such as schools, where bullying tends to be a serious problem, letting children abuse a robot could have a detrimental effect and encourage children’s bullying behaviour. In such cases, a limited expression of anger by the robot solely for the purpose of stopping the abuse might be acceptable, provided that parents and teachers in that school agree to that.

If the use of anger is adopted, it would be important to carefully design it. In particular, unnecessary scaring of children should be avoided if possible. In our experiment, the robot was relatively quick to switch from normal behaviour to mild anger, and later to strong anger. This was done on purpose to keep the experiment from becoming excessively long. However, in an actual application, this change should happen slowly, such that the robot gets angry only at children who are persistently ignoring its polite requests. If other potentially successful strategies for stopping the obstruction are available, it might be better to try them first and only resort to anger if nothing else works.

Conversely, it could also be argued that expressing anger or other negative emotions with robots and potentially scaring or hurting people should simply never be allowed, even if that is the best way to stop robot abuse. Instead, people should be educated not to abuse robots, in a similar way to how we teach children not to mistreat animals or harass people at work (we could even consider legally banning violence against robots [26]).

The stance that robot anger should never be permissible may appeal to many people, and it is interesting to ponder why. We could compare it with the case of animals. If we were to continually poke and prod a dog or some other animal, we would think it natural that it would become angry with us. Anger has evolved in living things as a means of self-preservation [40], and we accept it as a matter of course. In the case of robots, the argument could be that as artificial objects they should not show anger. On the other hand, there is a multitude of examples of anthropomorphised robots, to which developers have given artificial emotions and other biological characteristics. One might wonder then why should anger be different.

The question could also be considered from the viewpoint of ethics and the debate on the moral standing of robots. First, there is the question of robots as ‘moral patients’, i.e., whether the fact that robots are being mistreated even merits our concern. While one possible position is that robots are just machines and can thus be treated like slaves [11], there are several ways how it can be argued that robots do have moral standing, if not in themselves, then at least indirectly [13]. For example, in the case of robot abuse, one could argue that abuse is wrong and the robot should have moral standing because the display of abusive behaviour in public can cause distress to bystanders and also negatively affect the person causing the abuse, or that interfering with the robot’s task can cause harm to society at large [4].

With a higher status of robots as moral patients, perhaps a robot that expresses anger would be naturally acceptable. Yet, expressing anger also entails trying to affect and change human moral behaviour on purpose, thus in effect robots become active ‘moral agents’. This in turn leads to the open questions of who should decide how and when a robot should exercise its moral actions and the responsibility for the consequences of such actions [13]. For example, who should be held responsible if a robot’s angry expression makes a child excessively frightened?

Considering all of the above, we believe careful debate is still needed on whether robots should express anger at all, and if yes, in which situations this may be adequate.

The study was done in a laboratory environment as this gave us better control of the conditions during the experiment than what can be achieved in a real-world study. In particular, we were able to study a single child interacting with the robot, whereas in the real world a lot of the time the interactions typically involve more children. However, an obvious limitation of in-lab experiments is that many of the experimental conditions are inevitably artificial. For example, children might not consider a robot patrolling a room to be doing an important task. While one participant said in the interview that the robot truly looked like it was patrolling, two participants said it did not (‘It said it was patrolling, but the room was small and it felt like it was playing around’.).

Repeating the interactions four times could also have felt awkward and caused the children to behave differently between conditions. However, we did not find any conspicuous behavioural changes in the repeated conditions, apart from the children requiring fewer explanations and being quicker to start interacting. We also believe that counterbalancing the order of conditions reduced the possible effects of ordering on the results.

The presence of a confederate in the room may have been felt as unnatural and potentially affected the interaction. In connection with that, one could question whether the experiments truly reproduced robot abuse at all. In particular, one worry is that the children doing the abuse were simply imitating the behaviour of the confederate. (The experiment setup reminds to some extent of the classical Bobo doll experiments by Albert Bandura, where children were shown to copy aggressive actions towards a doll after seeing adults doing them first [3].) While this is an important criticism, it should be noted that a similar social learning process was observed to be happening also during the real-world robot abuse, where children frequently started abuse only after seeing some of their peers doing it [51]. Moreover, analysis of the videos from our experiment also showed that once they were left alone most children paid almost no attention to the confederate (who pretended to ignore them). This could suggest that the children, on the whole, felt little pressure to do the obstructions just because they were asked to. Nevertheless, it would be important to confirm the obtained results in the case of naturally happening robot abuse in a real environment, which we will consider exploring in future work.

The work was done in Japan, with Japanese children and with one specific type of robot. It is possible that in a different culture and with a different setup the outcome would differ. It is also likely that parts of the robot’s utterances and behaviours would need to be adjusted. Moreover, it is not clear whether the same three anger types would be identified in other cultural settings. The age of the children in the experiment was also limited, and the results may not generalise to other age groups.