JP2007257088A - Robot device and its communication method - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To have a robot carry out human-like communication in a state of having a multiplicity of humans around the robot. <P>SOLUTION: When carrying out communication between the robot device, and humans around the robot device, the periphery of the robot device is photographed, detection of faces, and hands or arms of the humans is carried out from the photographed image, evaluation values of each state is obtained from states of the detected faces, and hands or arms, weighting coefficients predetermined per type of evaluation value are multiplied to the evaluation values of each state, a person with high priority is specified from a sum value of the evaluation values multiplied by respective coefficient values, and communication is carried out with the specified person. Priority is judged by obtaining distances to the detected people, and evaluation values in regard to detected sound source positions also. <P>COPYRIGHT: (C)2008,JPO&INPIT

Description

  The present invention relates to a robot apparatus and a communication method therefor, and more particularly to a technique for communicating with a person around the robot apparatus.

  Various studies have been conducted on communication between humans and robotic devices. However, most of the conventional research is related to one-to-one communication between one robot apparatus and one user. On the other hand, there is still little research on communication between robot devices and multiple users. For example, the distance between a robot device and a human can be measured with a sensor, and communication can be performed based on the measured distance. Are known. Specifically, the robot apparatus selects a user using the distance information and the touch sensor.

Patent Document 1 discloses a technique for performing one-to-one communication between one robot apparatus and one user.
JP 2005-199403 A

  However, when communicating, it is difficult to perform human-like communication only by selecting a partner using distance. For example, a nearby user only looks at the robot device, but a nearby user is selected even when a user in the distance is interested in the robot and shakes his hand. However, in such a situation, since a distant user is more interested in the robot apparatus, it is originally desirable to select a distant user, but it is difficult to deal with such a case with the conventional method. is there. In addition, a method for stably recognizing and extracting only a person (user) facing the camera from a moving image obtained by shooting a state in which a plurality of persons come and go as needed has been proposed. However, it is not enough to detect a person who is facing the camera.

  The present invention has been made in view of this point, and an object of the present invention is to allow a robot to perform human-like communication in a state where there are a large number of people around the robot.

  In the present invention, when communication is performed between a robot apparatus and a human being around the robot apparatus, the surroundings of the robot apparatus are photographed, and detection of a human face and detection of a hand or arm from the photographed image The evaluation value of each state is obtained from the detected face state and the detected hand or arm state, and the weighting coefficient predetermined for each type of evaluation value is obtained as the evaluation value of each state. And a person with high priority is identified from the total value of the evaluation values multiplied by the respective coefficient values, and communication is performed with the identified person.

  According to the present invention, a person with a high priority in communication is identified based on the detected state of the surrounding human face and the detected state of the surrounding human hand or arm. Thus, in a state where there are a plurality of persons around the robot apparatus, it is possible to satisfactorily identify a person who is most interested in the robot apparatus from among the plurality of persons.

  In this case, the evaluation value for the detected face orientation and the evaluation value for the detected hand or arm movement are calculated, and the priority is determined from each evaluation value, whereby the face orientation and the hand (or arm) are determined. The person who is interested in the robot apparatus can be well identified from the movement of the robot.

  In addition, the detection of the hand or arm is performed by detecting a part of a predetermined color state within the range where the hand or arm may exist, estimated from the position where the human face is detected, as a hand or arm. The hand or arm may be detected reliably and satisfactorily.

  Further, by measuring the distance to the person who recognized the face and calculating the measured distance as an evaluation value, it is possible to specify the distance in consideration.

  Further, the position of the sound source is identified from the sound data obtained by collecting the surrounding sounds, the correspondence between the identified sound source position and the person recognized by the image recognition means is obtained, and an evaluation value for the obtained person's sound data is obtained. By calculating, it is possible to specify the voice taking into account the voice.

  Hereinafter, an embodiment of the present invention will be described with reference to the accompanying drawings.

  In this example, humanoid communication is performed with the robot apparatus. FIG. 1 is a diagram showing a configuration of the robot apparatus of this example, and mainly shows a configuration for discriminating a user having a high priority from users in the vicinity of the robot apparatus.

  The robot apparatus 10 includes a camera 11 and takes a moving image in the robot apparatus 10. The camera 11 is a video camera configured to take a stereo image with two or more eyes. Image data photographed by the camera 11 is supplied to the image recognition unit 12, and a process of extracting a person image (particularly a face image and a hand or arm image) from the image data is performed. Details of the process of extracting an image of a person's face or hand by the image recognition unit 12 will be described later.

  Data recognized by the image recognition unit 12 is sent to the recognition data processing unit 13. The recognition data processing unit 13 functions as an evaluation value calculation unit that obtains an evaluation value of the state of the recognized image, and sends the obtained evaluation value to the priority determination unit 14. As the evaluation value of the state of the recognized image, the evaluation value of the detected state of the user's face, the evaluation value of the state of the user's hand (or arm), and between the detected user and the robot apparatus Get the distance evaluation value. As the evaluation value of the face state, for example, when the face faces the front of the robot apparatus, the highest evaluation value is set. As the evaluation value of the hand (or arm) state, for example, when the hand is waving, the highest evaluation value is set. As the evaluation value of the distance state, for example, when the distance between the user and the robot apparatus is short, a high evaluation value is set. The distance between the user and the robot apparatus can be detected from the captured image itself because the camera 11 is a video camera configured to capture a stereo image. However, other known distance measuring means such as an infrared sensor may be used in combination.

  In the robot apparatus 10 of this example, a plurality of microphones 21a to 21n are attached with their sound collecting directions changed, and sound around the robot apparatus 10 is collected. Audio signals collected by the microphones 21 a to 21 n are converted into digital audio data by individual analog / digital converters 22 a to 22 n, and the converted audio data is supplied to the digital audio processing unit 23. The digital audio processing unit 23 estimates the arrival time difference from the sound source to each microphone from a plurality of audio data, and creates a likelihood map for sound source position estimation. The created sound source likelihood map is sent to the sound source specifying unit 24. The sound source identification unit 24 combines a plurality of sound source likelihood maps, estimates the direction of the sound source viewed from the robot apparatus, and obtains an evaluation value that represents the rate at which the sound source matches the user's face area. The evaluation value obtained by the sound source identification unit 24 is sent to the priority determination unit 14.

  The priority determination unit 14 uses a supplied face evaluation value, a hand (or arm) evaluation value, a distance evaluation value, and a sound source evaluation value. The user who is most interested in the robot apparatus is identified from the users. In this case, a coefficient value determined in advance for each type of evaluation value is multiplied, and each type of evaluation value multiplied by the coefficient value is added to obtain an added evaluation value for each user. By comparing the value values, the user with the highest value is identified as the user who is most interested in the robot apparatus. A specific example of calculating the evaluation value will be described later. As for the evaluation value of the sound source, the matching process between the position of the sound source and the position where the face is detected from the photographed image is performed, and it is determined that the voice is the user's speech of the face where the match is detected in the process. The process of setting the evaluation value of the corresponding user is performed.

  When the priority determination unit 14 identifies a user who is most interested in the robot apparatus, the data indicating the direction and distance of the determined user is sent to the robot operation control unit 31. The robot operation control unit 31 sends a drive command to the rotation drive unit 32 so as to face the direction specified by the data, and communicates with the identified user. In addition, various other operations that need to be performed by specifying the user in the operation of the robot apparatus are also performed by referring to the data of the direction and distance in which the determined user is present, and communication with the specified user is performed.

  Next, an overall flow of processing for identifying a user performed by the robot apparatus 10 of this example will be described with reference to a flowchart of FIG. First, color moving image data photographed by the camera 11 is input to the image recognition unit 12 (step S11), and a skin color pixel region in the moving image data is extracted (step S12). A process of detecting a human face from the extracted skin color region is performed (step S13).

  Next, the user's upper body is detected based on the detected face area (step S14), and based on the skin color area data detected in step S12 and the user's upper body area data detected in step S14, The user's hand is extracted (step S15). Then, the extracted hand movement (that is, a change in the detected hand position with time) is detected, and gesture recognition is performed from the detected hand movement (step S16). Further, the distance between the face detected in step S13 and the robot apparatus 10 is detected by image processing (step S17).

  Further, the outputs of the microphones 21a to 21n are supplied (step S18), and the sound source position is estimated from the multi-channel audio signal (step S19). Then, evaluation values for each state detected so far are obtained, each evaluation value is multiplied by a coefficient for each type, and the degree of intimacy is calculated from the total value for each user of the coefficient-multiplied value. Is calculated (step S20).

Next, a detailed example of priority calculation processing for each user and extraction of each information will be described.
By integrating multimodal information such as the user's face orientation, gesture, and voice, and taking into account the distance to the user, the priority level representing the user's familiarity with the robot is calculated by the following equation (1). In Equation (1), U represents a priority, P _i represents intimacy determined from the appearance frequency and distance of each modal, and w _i represents a weight for intimacy P _i . The intimacy P _i corresponds to the evaluation value of each characteristic.

  Next, an example of processing for detecting a human face from an image taken by the camera 11 will be described. As processing for detecting a human face, first, a skin color region is extracted from an input image and a mask image is created to limit the processing range of face extraction. Then, face extraction is performed using known computer software for face image extraction. The software used for verifying the present invention is the trade name OKAO Vision (manufactured by OMRON), but is not limited thereto.

  Next, a process for performing skin color extraction from the face extracted image will be described. Regarding the skin color extraction, the skin color likelihood in the input image is obtained by using a skin color distribution GMM (Gaussian mixture model). Thereafter, a binary image is created by threshold processing, and noise removal processing is performed to create a mask image. The face extraction process is performed only in the area obtained by enlarging the mask area by a factor of 1.4. Thus, the face extraction accuracy is improved, and the processing speed can be improved.

  FIG. 3 shows an example of extracting a face from a photographed color image. FIG. 3B is a skin color likelihood image obtained with respect to the color image shown in FIG. 3A, and only the pixels of the skin color or the color approximated to the skin color are extracted. By performing binarization processing, noise removal processing, and the like from the skin color likelihood image, a mask image shown in FIG. 3C is obtained. An image obtained by enlarging the white area (mask) in the mask image is an image shown in FIG. In the image shown in FIG. 3D, a rectangular area including the user's face portion is set as a face area candidate.

  Face extraction is performed on the face area candidates obtained in this way using face image extraction software. In the case of the software used in this example, a face can be detected when the face is generally facing the front. Therefore, here, when the face is detected, it is determined that the face is facing the front, and in other cases, it is determined that the face is not facing the front. In the image shown in FIG. 3D, the square frame at the center is the face extraction result.

  Next, processing for detecting a hand or an arm and recognizing the detected hand gesture will be described. Here, gesture recognition is performed in six directions (up / down / left / right / front / back) indicated by hand movements. First, the hand is extracted by extracting the upper body. Next, gesture recognition is performed by examining hand movements.

  For the extraction of the upper body, the lower part of the face area is cut out from the face area obtained by the above-described process using the body structure of a person who has a torso under the face as shown in FIG. That is, based on the extracted face width W and height H, it is assumed that there is a torso of the upper body in the range of 2W × 1.5H below the face, and the hand or arm is on the left and right sides of the torso. Suppose there is.

  FIG. 5 is a diagram showing an example of extracting the upper body in this way. FIG. 5A shows an example in which the upper body range is determined from the actual color image by setting the range shown in FIG. 4. The upper frame represents the face area, and the lower frame represents the clipped portion. The HSV value of this part is obtained, and a histogram of hue (H) and saturation (S) is obtained. FIG. 5C shows a histogram of hue (H), and FIG. 5D shows a histogram of saturation (S). Here, the maximum value is obtained for each histogram. Then, a range of ± 10 is obtained centering on the obtained maximum value (portion surrounded by frames (c) and (d) in FIG. 5). The upper body of the user can be extracted by extracting pixels having this color information from the entire image and performing noise removal processing or the like. FIG. 5B shows the result of extracting the upper body region from the color image of FIG.

Next, processing for extracting a user's hand will be described.
Since the user's hand is connected to the arm, the user's upper body area should exist around the hand. Of the face candidate regions obtained by the above-described processing, a region other than the face region is set as a hand candidate region. The user's hand is extracted depending on whether or not the upper body area (chest, arm, etc.) of the user is within the candidate area of the hand. FIG. 6B shows the result of extracting faces and hands from the color image shown in FIG. In FIG. 6A, a white frame indicates a candidate region of the hand. In FIG. 6B, the top white area represents the user's face, the lower left area represents the user's right hand, and the lower right area represents the user's left hand.

  Next, a process for recognizing the detected hand gesture will be described. As for the gesture for indicating the direction by the movement of the hand, it was found that the movement of the hand in the indicated direction was faster than the movement of the returning hand as a result of actually capturing it as a moving image and examining the movement of the hand. Therefore, in this example, gesture recognition is performed using a hand movement vector. First, using the distance image, the three-dimensional coordinates of the hand obtained by hand extraction are obtained. The difference between the hand coordinates of the previous frame and the hand coordinates of the current frame is defined as a movement vector. The pointing motion is recognized by sequentially examining the movement vector during each gesture in each frame.

  FIG. 7 is a table showing movement vectors of two gestures (instructions on the right and left) with the central dotted line as a boundary. As can be seen from FIG. 7, the average value (68) of the right direction movement vector is larger than the average value (53) of the left direction movement vector in the case of the gesture indicating “right” (first gesture).

  Next, an example of processing for estimating the sound source position from the audio data captured by the microphone will be described. Here, the sound source position is estimated using the CSP method (Cross-power Spectrum Phase). A delay time map is created in advance for a set of two microphones, and a CSP coefficient is calculated to obtain a likelihood map of the sound source.

  Then, a sound source estimation range is obtained. FIG. 8 is a diagram illustrating a calculation example of the sound source estimation range. FIG. 8A is a side view of the photographing range of the camera. As shown in FIG. 8 (b), a space obtained by rotating the shooting range 360 degrees around the camera is set as a sound source position estimation range. FIG. 8C shows the expanded space, and the expanded range is used to calculate a likelihood map for sound source position estimation.

  A delay time map is created using the data of the sound source position estimation range. The delay time map is obtained by obtaining the maximum value and the minimum value of the delay sample number n (p) within the sound source position estimation range. The number of delayed samples n (p) is obtained using the following equation (2).

  Here, F, p, Ma, Mb, c are the sampling frequency of the sound signal, the three-dimensional coordinates of the point P in the real space, the three-dimensional coordinates of each of the two microphones, and the speed of sound.

Pixels of an image obtained by developing the sound source position estimation range (i, j) delay sample number n for each maximum value n _max of the (p) (p) and obtaining the minimum value n _min (p). The CSP coefficient is calculated, and the value of the CSP coefficient between the maximum value n _max (p) and the minimum value n _min (p) is set as the likelihood of the sound source position estimation of the pixel (i, j).

In the case of this example, five microphones are used, and ₅ C ₂ = 10 microphone pairs are obtained. The sound source position is estimated by synchronously adding the sound source likelihood maps in these 10 microphone pairs.

Next, a process for calculating the distance to the face detected from the image will be described. In this example, a stereo camera is used to acquire distance information. Since the distance to each pixel in the image can be calculated according to the stereo camera, the average distance between the pixels in the face area is defined as the distance between the robot and the user. When dealing with distances, intimate distance (about 50cm), personal distance (about 50-120cm), social distance (about 120-360cm), public distance (more than 360cm) Use classification. Specifically, it can be said that the closer the user is to the robot, the closer the user is to the robot, so the closer the distance between the user and the robot is, the larger the value of P _d representing the degree of intimacy by distance. For example, using the relationship between distance and intimacy shown in FIG. 9, the intimacy value (evaluation value) with the user according to the distance is calculated.

  Next, the priority of each user is calculated from the calculated intimacy values in each modal. When calculating the priority, it is necessary to observe the behavior of each user. Here, the number of observation frames nFrame = 40 (about 5 frames / second) is defined as one observation section. However, the number of observation frames is not limited to 40.

  First, the appearance frequency of the face front direction is determined. If the user's face is facing the robot, the face is extracted. The number of frames (nFrontFace) from which the face is extracted within the observation section is obtained, and the appearance frequency Pf of the face front direction in the observation section is calculated by Expression (3).

  Next, the appearance frequency of the gesture will be described. The expression frequency of the gesture is obtained using Expression (4). Here, within the observation section, the type of the instructed gesture and the number of gesture frames (nGesture) are obtained, and the appearance frequency of the gesture is obtained.

  Here, nGesture / nFrame indicates the ratio of the number of gestured frames. During communication, the hand movement in the front-rear direction indicates that the robot is most interested in the robot. Therefore, the coefficient (Kind) corresponding to the type of gesture is set to 0.7. The value of the coefficient (Kind) is, for example, 0.5 for reciprocating movements other than front and rear, and 0.0 for other movements.

  Next, the appearance frequency of conversation will be described. After the sound source position is estimated by the above-described processing, the number of frames (nVoice) in which the sound source position enters the user's face area in the observation section is obtained. Here, in consideration of changing the appearance frequency (Pv) of the conversation depending on the content of the speech by adding the function of speech recognition, the appearance frequency of the conversation is calculated using Expression (5). nVoice / nFrame indicates the ratio of the number of frames spoken.

  According to an experiment by the present inventor, when talking to the robot apparatus such as “Come here”, the number of frames being spoken is around 10 frames, and the ratio (nVoice / nFrame) in the observation section is small. So Ia is set to 0.5 and Pv is increased.

  Using each of these values, the priority for each user is calculated. That is, the priority is calculated by the above-described equation (1) using Pf, Pg, Pv and Pd for each observation section. Here, considering the importance of the orientation of the face at the time of communication, it is necessary to increase the weight Wf of Pf. Since the information to be emphasized differs depending on the person, the robot apparatus can behave with individuality by changing the value of the weight Wi. Here, assuming that sound information and gesture information are equally emphasized, the values of the weights Wf, Wg, Wv, and Wd are set to 0.5, 0.2, 0.2, and 0.1, respectively. By calculating in this way, a user who is most interested in the robot apparatus 10 can be selected.

  Next, FIG. 10 shows an example of a change in priority when an experiment is performed in a state where there are actually three users around the robot apparatus. The vertical axis represents the priority, the horizontal axis represents the frame number of the image, and shows an example of changes in the state of the three users U1, U2, U3. In this example, since the user U1 is merely looking at the robot, the priority change of the user U1 is gradual and the value is low (section I in FIG. 10). Since the user U2 is gesturing while talking to the robot, the priority of the user U2 changes rapidly and the value is large (section II in FIG. 10). Furthermore, since the user U3 is only talking while looking at the robot apparatus, the change in the priority of the user U3 is not so steep and the value is not so large (section III in FIG. 10). After that, since the priority of the user U2 is the highest, the robot turns to the user U2 and looks at the user U2, but the user U2 is talking while looking at the user U1 (not facing the direction of the robot). ), The priority is reduced (section IV in FIG. 10). Further, the robot apparatus changes the direction to the user U3 having the highest priority, and calculates the priority of the user U3. Since the user U3 is still talking with interest in the robot, the priority is high (section V in FIG. 10). At this time, the priority of the user U3 is large, and the robot communicates with the user U3.

  As described above, according to the present embodiment, it is possible to define the priority indicating the friendliness between the user and the robot apparatus by integrating the multimodal information such as the user's face direction, gesture, voice, and distance. it can. Then, by calculating the priority for each user, it becomes possible to give priority to communication among the users who are most interested in the robot.

  In addition, when communicating, the information emphasized differs depending on the person. In this embodiment, the weights of gesture information and voice information are the same. However, if a certain user places importance on gesture information, the weight of the gesture information is increased, etc. It is also possible to have

It is a block diagram which shows the structural example of the robot apparatus by one embodiment of this invention. It is a flowchart which shows the flow of the whole process by one embodiment of this invention. It is explanatory drawing which shows the example of an extraction process of the face image by one embodiment of this invention. It is explanatory drawing which shows the example of the detection range of the hand or arm by one embodiment of this invention. It is explanatory drawing which shows the example of an extraction process of the upper body by one embodiment of this invention. It is explanatory drawing which shows the example of extraction of the image of the hand by one embodiment of this invention. It is a wave form diagram which shows the example of a time change of the movement vector by one embodiment of this invention. It is explanatory drawing which shows the example of a sound source position estimation by one embodiment of this invention. It is a characteristic view which shows the example of the value of intimacy according to the distance by one embodiment of this invention. It is explanatory drawing which shows the example of a time change of the priority by one embodiment of this invention.

Explanation of symbols

  DESCRIPTION OF SYMBOLS 10 ... Robot apparatus, 11 ... Camera, 12 ... Image recognition part, 13 ... Recognition data processing part, 14 ... Priority determination part, 21a-21n ... Microphone, 22a-22n ... Analog / digital converter, 23 ... Digital audio processing , 24 ... Sound source identification unit, 31 ... Robot operation control unit, 32 ... Rotation drive unit

Claims (6)

Photographing means for photographing the surroundings;
An image recognition means for detecting a human face and a hand or arm from the image photographed by the photographing means;
Evaluation value calculation means for obtaining an evaluation value of each state from the detected face state and the detected hand or arm state;
Multiplying the evaluation value of each state by a weighting factor determined in advance for each type of evaluation value, and specifying a person with high priority from the total value of the evaluation values multiplied by each coefficient value Means and
A robot apparatus that communicates with a person specified by the specifying means. The robot apparatus according to claim 1, wherein
The robot apparatus, wherein the evaluation value calculation means calculates an evaluation value for the face orientation detected by the image recognition means and an evaluation value for the hand or arm movement detected by the image recognition means. The robot apparatus according to claim 2, wherein
The detection of the hand or arm by the image recognition means is performed by detecting a portion of a predetermined color state within a range where the hand or arm may exist, estimated from the position where the human face is detected. A robot device characterized by detecting. The robot apparatus according to claim 1, wherein
Furthermore, the image recognition means comprises a measurement means for measuring the distance from the person who recognized the face,
The evaluation value calculation means calculates an evaluation value from the distance measured by the measurement means. The robot apparatus according to claim 1,
Furthermore, sound collecting means for collecting surrounding sounds,
A sound source identifying means for identifying a position of a sound source from the sound data collected by the sound collecting means, and obtaining a correspondence relationship between the identified sound source position and the person recognized by the image recognition means;
The robot apparatus according to claim 1, wherein the evaluation value calculation means calculates an evaluation value related to human voice data obtained by the sound source identification means. In a communication method of a robot apparatus that performs communication between the robot apparatus and a person around the robot apparatus,
Photograph the surroundings of the robotic device,
From the captured image, human face detection and hand or arm detection,
From the detected face state and the detected hand or arm state, obtain an evaluation value of each state,
Multiply the evaluation value of each state by a weighting factor determined in advance for each type of evaluation value, and identify a person with high priority from the total value of the evaluation values multiplied by each coefficient value,
A communication method of a robot apparatus, characterized by communicating with a specified person.