JP7026066B2 - Voice guidance system and voice guidance method - Google Patents

Description

The present invention relates to a voice guidance system and a voice guidance method.

In recent years, as a voice guidance system, there is known a system that applies voice recognition technology or conversation technology to a user's voice input and returns an appropriate answer by voice. An example of such a voice guidance system is a conversation robot. For example, Patent Document 1 discloses a technique for naturally performing a conversation with a plurality of speakers in a conversation robot.

Japanese Unexamined Patent Publication No. 2012-76162

The voice guidance system that has been developed in the past has a problem that it is not possible to talk with a plurality of guide targets at the same time.
For example, if a question is asked by another guide target person B during a conversation with one guide target person A, the conversation robot answers to the guide target person B while maintaining the conversation status with the guide target person A. Therefore, it is not possible to give an appropriate answer.

Further, in a usage scene where the conversation between the guide target person and the non-guidance target person (guidance executor) is assisted, the conversation robot may respond to the utterance (question) of the guide target person, but the guidance execution If you respond to a person's utterance (answer), the conversation will not be established.
Furthermore, it is convenient for the conversation robot to answer instead when the non-guidance target person cannot answer, but at present, it is difficult to determine whether or not the conversation robot (voice guidance system) should answer.
As a situation in which the conversation robot assists the conversation between the guidance target person and the non-guidance target person (guidance executor), for example, the conversation robot translates the conversation between the guidance target person and the non-guidance target person. However, at the time of such translation, it is necessary to make a more complicated judgment as to whether or not to answer.

It is an object of the present invention to provide a voice guidance system and a voice guidance method capable of appropriately performing a response when a plurality of speakers are present.

In order to solve the above problems, for example, the configuration described in the claims is adopted.
The present application includes a plurality of means for solving the above problems. For example, a camera, a microphone, and a speaker are provided, and the speaker provides guidance voice based on the voice input to the microphone. It is a voice guidance system that outputs.
Then, a question reception unit that accepts a question by voice input to the microphone and a voice recognition unit that calculates the characteristics of the user's voice corresponding to the question received by the question reception unit and recognizes the user based on the calculated voice characteristics. When the question reception unit detects the voice, the image recognition unit that calculates the characteristics of the user's image taken by the camera and recognizes the user based on the calculated image or the characteristics of the image, and the voice recognition unit calculates . A guidance unit that selects a user to provide guidance using the characteristics of the user's voice and the characteristics of the user's image calculated by the image recognition unit, and outputs the guidance voice to the selected user from the speaker , and the output voice. It is equipped with a translation unit that uses voice translated into a predetermined language, and based on the reception status of the voice input to the microphone in the question reception unit, the guidance unit uses the question reception unit instead of outputting the guidance voice. The voice translated from the received voice into a predetermined language is output from the speaker.

According to the present invention, even in a situation where a plurality of users are nearby, each user can be individually recognized, and an appropriate answer can be given to each user. It also enables guidance to support conversations between multiple users.
Issues, configurations and effects other than those described above will be clarified by the following description of the embodiments.

It is a block diagram which shows the configuration example of the whole voice guidance system in 1st Embodiment of this invention. It is a block diagram which shows the structural example of the robot in 1st Embodiment of this invention. It is a block diagram which shows the structural example of the robot control apparatus in 1st Embodiment of this invention. It is a block diagram which shows the configuration example of the robot management server in the 1st Embodiment example of this invention. It is a figure which shows the appearance example of the robot in 1st Embodiment of this invention. It is a figure which shows the use example in 1st Embodiment of this invention. It is a figure which shows the selection screen of the language used in 1st Embodiment of this invention. It is a flowchart which shows the flow of the whole service in 1st Embodiment of this invention. It is a flowchart which shows the processing example of the translation service in 1st Embodiment of this invention. It is a sequence diagram (first half) of the robot answer processing based on the conversation failure in the 1st Embodiment example of this invention. It is a sequence diagram of the robot response process based on the conversation failure in the first embodiment of the present invention (second half: continuation of FIG. 10). It is a sequence diagram at the time of robot response based on the expiration of the response timer in the first embodiment of the present invention. It is a flowchart which shows the flow of the whole service in 2nd Embodiment of this invention. It is a flowchart which shows the processing example of the conversation assisting service in 2nd Embodiment of this invention. It is a flowchart which shows the processing example of the translation service in the 3rd Embodiment example of this invention. It is a sequence diagram (first half) at the time of a robot response at the time of a conversation failure in the third embodiment of the present invention. It is a sequence diagram (second half: continuation of FIG. 16) at the time of a robot response at the time of conversation failure in the 3rd Embodiment example of this invention.

<1. Example of First Embodiment>
Hereinafter, an example of the first embodiment of the present invention will be described with reference to FIGS. 1 to 12.
The voice guidance system of the first embodiment of the present invention translates a conversation between a guidance target person (facility user) and a guidance performer (facility staff). Here, the voice guidance system of the first embodiment can answer the question of the guidance target person on behalf of the guidance executor when the guidance executor cannot answer the question.

[System configuration]
FIG. 1 shows the overall configuration of a voice guidance system according to an example of the first embodiment of the present invention.
The voice guidance system 1 includes a robot 100, a robot control device 200, and a robot management server 300. The robot 100 and the robot control device 200 are installed at the site 2 where the robot 100 is operated. Site 2 is a facility such as a shopping center.
The robot 100 executes guidance by conversation.
The robot control device 200 controls the robot 100.
The robot management server 300 monitors the operation status of the robot 100, and is operated by, for example, a business operator that provides the system. The robot management server 300 is connected to the robot control device 200 installed at the site 2 via a network.

The robot 100 of the voice guidance system 1 assists the guidance by translating the conversation between the guidance executor and the guidance target person who speaks a different language from the guidance executor at the site 2 such as a commercial facility. If the guidance executor cannot answer, the robot 100 responds to the guidance target person on behalf of the guidance executor.

FIG. 2 shows a configuration example of the robot 100.
The robot 100 communicates with a CPU (Central Processing Unit) 110 that controls the processing operation of guidance by voice, a storage device 120 that stores each software and data, an input / output device 130, and an external device. It is composed of a communication interface 140.

The storage device 120 includes an input / output unit 121, a scenario execution unit 122, and a screen processing unit 123.
The input / output unit 121 holds the data and instructions received from the robot control device 200 and sends them to each processing unit.
The scenario execution unit 122 outputs voice and gives an instruction to display a screen according to a scenario instruction received from the robot control device 200. Further, the scenario execution unit 122 acquires data from the input / output device 130 and the like.
The screen processing unit 123 accesses the robot control device 200 and displays the screen according to the screen display instruction received from the scenario execution unit 122. Further, the screen processing unit 123 accepts touch input from the input / output device 130.

The input / output device 130 includes a microphone array 131 composed of a plurality of microphones, a camera 132, a speaker 133, and a touch panel 134.
The microphone array 131 acquires audio data acquired by a plurality of microphones as multi-channel data. The plurality of microphones included in the microphone array 131 acquire voices in different directions of arrival. The camera 132 acquires visual data such as a video or an image. The speaker 133 outputs sound. The touch panel 134 acquires screen output and touch input data.

The robot 100 always transmits the data acquired by the microphone array 131 and the camera 132 to the robot control device 200 at regular intervals through the communication interface 140, and receives a scenario execution instruction from the robot control device 200 through the communication interface 140.

FIG. 3 is a diagram showing a configuration example of the robot control device 200, in which the robot control device 200 communicates with a CPU 210 that processes each part, a storage device 220 that stores software and data, and an external device. It is composed of a communication interface 230 to perform.
The main storage device 220 includes an input / output unit 221, a question reception unit 222, a voice recognition unit 223, an image recognition unit 224, a guidance unit 225, and a translation unit 226.
The input / output unit 221 processes input data and output data from the robot 100 and the robot control server 300, holds the data, and sends the data to each unit in the robot control device 200.

The question reception unit 222 performs a question reception process for receiving the voice input of the microphone (channel) having the loudest voice of the user stored from the multi-channel voice data received from the robot 100. At this time, the question receiving unit 222 estimates the arrival direction of the voice data based on the installation direction of the microphone.

The voice recognition unit 223 performs voice recognition processing. As the voice recognition process in the voice recognition unit 223, the feature amount of the voice received by the question reception unit 222 is calculated, and the user is identified based on the voice. The calculation of the feature amount of the voice here is performed in order to distinguish the voices of a plurality of people around the robot 100.
Further, the voice recognition unit 223 converts the voice received by the question reception unit 222 into text.
The image recognition unit 224 performs image processing on the image data received from the robot 100, calculates the feature amount of the user's image, and identifies the user based on the image. When multiple people appear in the image, each user is recognized individually.

When the voice recognition unit 223 or the image recognition unit 224 recognizes the user, the user registered in advance can be recognized as a specific user. That is, the voice feature and the image feature of the guidance executor 4 are registered in the voice recognition unit 223 and the image recognition unit 224 in advance. By registering in this way, the user who has detected the pre-registered voice features and / or image features is designated as the non-guidance target person (guidance executor 4), and the pre-registered voice features or features. The user who has detected the feature of the image can be the guide target person 3.

The guidance unit 225 is composed of a conversation continuation determination unit 225a, an answer availability determination unit 225b, a conversation failure determination unit 225c, and an answer output unit 225d, and is used as a user identification result in the voice processing unit 223 and the image recognition unit 224. Based on this, a user is selected and a conversation with the selected user is executed.
The conversation continuation determination unit 225a determines whether or not the conversation between the guidance target person and the guidance executor is continuing.
The answer availability determination unit 225b determines whether or not it is possible to answer the question of the guide target person.
The conversation failure determination unit 225c determines whether or not the conversation between the guidance target person and the guidance executor is broken.
The answer output unit 225d searches for and outputs an answer to the question of the guide target person.
The translation unit 226 translates the utterances converted into text by the voice recognition unit 223. The translation source language and translation destination language are determined based on the user's standing position and the input of the touch panel 134.

FIG. 4 shows a configuration example of the robot management server 300.
The robot management server 300 includes a CPU 310 that controls the processing of each unit, a storage device 320 that stores software and data, and a communication interface 330 that communicates with an external device.
The storage device 320 has a robot management unit 321, and the robot management unit 321 manages the state of each robot and notifies maintenance plans and repairs.

[Examples of robot shapes and usage patterns]
FIG. 5 shows an example of the appearance of the robot 100. The robot 100 of the present embodiment is a signage type robot, and is composed of a tablet 101 including a camera 132, a speaker 133, and a touch panel 134, and a credor 102 including a microphone array 131. The claim doll 102 holds the tablet 101 and is connected to the held tablet 101.

A plurality of microphones are arranged in the horizontal direction of the array microphone 131, and each microphone sets the direction of collecting sound to a different direction (horizontal angle). The appearance of the robot 100 shown in FIG. 5 is an example, and other appearance shapes may be used. For example, the camera 132 and the speaker 133 may be provided by the credor 102.
The camera 132 is arranged in front of the tablet 101 and photographs a user in front of the tablet 101. The speaker 133 is also arranged in front of the tablet 101.

The touch panel 134 includes a text display unit 134a and an avatar display unit 134b. The text of the translation result or the text of the answer by the voice guidance system is displayed on the text display unit 134a. The avatar display unit 134b displays an image used for animation and guidance of the avatar that operates according to the text displayed on the text display unit 134a. FIG. 5 shows an example in which a robot is displayed as an avatar, and a conversation sentence (here, “hello”) is displayed on the text display unit 134a, and the robot (avatar) is displayed as if it had a conversation.

FIG. 6 shows an example of using translation by a voice guidance system. The guidance target person 3 and the guidance executor 4 stand in front of the robot 100 while facing the robot 100, and the robot 100 assists and translates the guidance based on the voice and the text translated by the voice guidance system 1. , The guidance target person 3 and the guidance executor 4 have a conversation with each other. The guidance target person 3 here is a user of the site 2, and the guidance executor 4 is a person (such as an employee of the site 2) who is waiting for guidance at the site 2.

FIG. 7 shows a screen for selecting the language used by the guidance target person 3 and the guidance executor 4.
The button 134c indicates a language selection button of the guide target person 3. Button 134d indicates a language selection button of the guidance executor 4. The button 134e indicates a button for switching the standing positions of the guidance target person 3 and the guidance executor 4.

By exchanging the guidance target person 3 and the guidance executor 4 with the standing position exchange button 134e, the positions of the guidance target person language selection button 134c and the guidance executor language selection button 134d on the touch panel 134 are exchanged.
The guidance target person 3 and the guidance performer 4 each determine the language to be used for speech recognition and translation by selecting the language selection buttons 134c and 134d, respectively.

[Execution example of guidance service]
FIG. 8 is a flowchart showing an example of an execution procedure of the guidance service by the voice guidance system 1. Here, as shown in FIG. 6, there is a guidance target person 3 and a guidance executor 4 in close proximity to the robot 100.

First, in the robot control device 200, a user determination (step S100) is performed. In this user determination, it is determined whether the plurality of users standing in front of the robot 100 are the guidance target person 3 or the guidance executor 4, respectively, based on the image, the voice, and the touch input.
The determination of whether the guidance target person 3 or the guidance executor 4 is performed by, for example, the following processing.
First, the image recognition process in the image recognition unit 224 identifies whether the guidance executor is standing on the right or the left from the face image of the registered guidance executor.
Next, the voice recognition unit 223 detects whether the speaker is on the left or right by detecting the direction of arrival of the voice, and estimates whether the speaker is the guide performer or not (guidance target person).

Returning to the description of the flowchart of FIG. 8, the robot control device 200 performs the language determination (step S200). Here, the utterance language (first language) of the guidance target person 3 and the utterance language (second language) of the guidance executor 4 are determined.
This language determination is executed, for example, by language identification based on the voice acquired by the voice recognition unit 223. Alternatively, the setting is performed by the touch input shown in FIG. 7.

Subsequently, the robot control device 200 provides a translation service (step S300). Here, the translation from the first language to the second language and the translation from the second language to the first language are performed. Here, an example in which the first language is English and the second language is Japanese will be described.
By setting the language in this way, when the guidance executor 4 cannot give an appropriate answer, the robot 100 uses the first language (English) to give an answer on behalf of the guidance executor 4.

FIG. 9 is a flowchart showing the flow of the translation service in step S300.
Further, FIGS. 10 and 11 (both figures are continuous views) are sequence diagrams in which the robot responds based on the conversation failure by executing the flowchart of FIG. FIG. 12 is a sequence diagram when the flowchart of FIG. 9 is executed and the robot responds based on the expiration of the response timer. In these sequence diagrams, the same step numbers are assigned to the parts corresponding to each step in the flowchart of FIG.

The details of the translation service S300 shown in FIG. 9 will be described below.
First, the question receiving unit 222 of the robot control device 200 checks the interrupt processing (step S1) and determines the presence / absence of the interrupt processing (step S2). The interrupt processing determined by the question receiving unit 222 here includes an interrupt due to voice input and an interrupt due to the expiration of the answer timer.
When there is no interrupt processing in step S2 (“no interrupt” in step S2), the question receiving unit 222 returns to the check in step S1.

Further, when it is determined in step S2 that there is an interrupt process (“interrupted” in step S2), the question receiving unit 222 determines whether the interrupt content is an interrupt due to voice input or an interrupt due to the expiration of the answer timer. (Step S3).
When it is determined in step S3 that the interrupt is due to voice input (“voice” in step S3), the question receiving unit 222 detects the direction of arrival of voice from the voice data of a plurality of channels, and the speaker is the guide target person 3. It is determined whether the user is the guide executor 4 (step S4).

Then, when the speaker determined from the voice direction in step S4 is the guidance target person 3 (“customer” in step S4), the result of voice recognition in English by the voice recognition unit 223 is translated into Japanese by the translation unit 226. It is translated and output by voice from the speaker 133 of the robot 100 (step S5). At this time, the translated text on the touch panel 134 may be output at the same time as or instead of the audio output.

When the translation of the utterance of the guidance target person 3 is completed, the customer conversation failure detection unit 225c increments the number of questions (step S6), and the conversation continuation determination unit 225a starts the answer timer (step S7).

Further, when the speaker determined from the voice direction in step S4 is the guidance executor 4 (“site administrator” in step S4), the result of voice recognition in Japanese by the voice recognition unit 223 is translated into English by the translation unit 226. Translate to. Then, the speaker 133 and the touch panel 134 of the robot 100 output the translated voice or sentence (step S9).

When the translation process of the utterance of the guidance executor 4 in step S9 is completed, the conversation failure detection unit 225c acquires an image from the robot 100, and the image recognition unit 224 recognizes the face to determine the emotion, and the user's reaction. Is positive or negative (step S10).
Here, when the user's reaction is positive (“positive” in step S10), the conversation failure detection unit 225c clears the number of customer questions (step S11), and the conversation continuation determination unit 225a clears the answer timer (step). S12).

Further, when the user's reaction is negative in the determination in step S10 (“negative” in step S10), the conversation failure detection unit 225c determines whether or not the number of customer questions is equal to or greater than the threshold value (step S13). Here, if it is equal to or less than the threshold value (“below the threshold value” in step S13), the process proceeds to step S12, and the conversation determination unit 225a clears the answer timer.

Further, if the number of customer questions in step S13 is equal to or greater than the threshold value (“greater than or equal to the threshold value” in step S13), the conversation failure detection unit 225c considers that the conversation has failed, and the answer possibility determination unit 225b determines whether or not the answer is possible. (Step S14).

If no answer is possible in the determination of step S14 (“No” in step S14), the conversation failure detection unit 225c clears the number of customer questions (step S11), and the conversation continuation determination unit 225a clears the answer timer (step S12). ..
If the answer can be answered by the determination in step S14 (“OK” in step S14), the answer output unit 225d searches for the answer (step S15), and the search result is output by the speaker 133 and / or the touch panel 134 of the robot 100. (Step S16).

After the robot answers, the conversation failure detection unit 225c clears the number of customer questions (step S11), and the conversation continuation determination unit 225a clears the answer timer (step S12).

If the answer timer expires during the interrupt check (steps S1 and S2), the conversation continuation determination unit 225a interrupts the timer expiration, and the question receiving unit 222 detects the timer expiration interrupt (“step S3”. Answer timer has expired "). In this case, the answerability determination unit 225b determines whether or not the question of the immediately preceding guide target person can be answered (step S14).
Here, when the answer is possible (“OK” in step S14), the answer output unit 225d searches for the answer (step S15), and outputs the search result to the speaker 133 or the touch panel 134 of the robot 100 (step S16).

Then, after the answer timer start (step S7) and after the answer timer clear (step S12), the conversation continuation determination unit 225a determines whether or not the user is in front of the robot 100 based on the face recognition of the image recognition unit 224. Check the usage status to determine (step S8).
Here, when the conversation continuation determination unit 225a determines that there is a user (“in use” in step S8), the conversation continuation determination unit 225a returns to the interrupt check in step S1 again. Further, when it is determined that there is no user (“end of use” in step S8), the translation service is terminated.

10 and 11 (both figures are continuous sequence diagrams) are sequence diagrams when the robot responds based on the conversation failure.
In this example, the guidance target person 3 first speaks in English, and the robot 100 transmits the voice input to the microphone array 131 to the question reception unit 222 of the robot control device 200. This voice interruption is detected by the question receiving unit 222, the voice direction is detected by the voice recognition unit 223, text conversion is performed, and the translation unit 226 translates from English to Japanese. At this time, the number of questions is incremented by the conversation failure detection unit 225c.
Then, the translation result in the translation unit 226 is output by the robot 100, and the answer timer is started in the conversation continuation determination unit 225a. The above is the translation process of the utterance of the guidance target person 3 shown in the upper half of FIG.

Subsequently, when the guidance executor 4 utters an answer, the robot 100 transmits the voice input to the microphone array 131 to the question reception unit 222 of the robot control device 200. This voice interruption is detected by the question receiving unit 222, the voice direction is detected by the voice recognition unit 223, text conversion is performed, and the translation unit 226 translates from Japanese to English.
Then, the translation result in the translation unit 226 is output by the robot 100. The above is the translation process of the utterance of the guidance executor 4 shown in the lower half of FIG.

In the case of the present embodiment, the robot 100 further gives an answer based on the facial expression of the guide target person 3.
That is, after the voice output of the answer shown in FIG. 10, as shown in FIG. 11, face recognition and emotion determination of the recognized face are performed from the image taken by the camera 132 of the robot 100, and the conversation failure detection unit 225c performs the face recognition. , Determine if the conversation is broken. When it is detected that the conversation is broken by this judgment, the answer possibility determination unit 225b determines whether the question of the guidance target person 3 can be answered, and if possible, the answer output unit 225d searches for the answer and searches. The answer is output from the robot 100.

Further, the conversation failure detection unit 225c clears the number of questions, and the conversation continuation determination unit 225a clears the answer timer, and the process ends after the usage status is confirmed. Alternatively, when the conversation continues, return to the beginning of FIG.

FIG. 12 is a sequence diagram at the time of robot response based on the expiration of the response timer.
In this example, as in the case of FIG. 10, the guidance target person 3 first speaks in English, and the robot 100 transmits the voice input to the microphone array 131 to the question reception unit 222 of the robot control device 200. This voice interruption is detected by the question receiving unit 222, the voice direction is detected by the voice recognition unit 223, text conversion is performed, and the translation unit 226 translates from English to Japanese. At this time, the number of questions is incremented by the conversation failure detection unit 225c.
Then, the translation result in the translation unit 226 is output by the robot 100, and the answer timer is started in the conversation continuation determination unit 225a. Up to this point, the process is the same as the translation process of the utterance of the guidance target person 3 shown in the upper half of FIG.

After that, the conversation continuation determination unit 225a expires the answer timer, and the question reception unit 222 detects an interrupt of the timer expiration. At this time, the answer possibility determination unit 225b determines whether or not the answer is possible, and when the answer is possible, the answer output unit 225d searches for the answer, and the robot 100 outputs the answer by voice and / or image.
Further, after the answer search by the answer output unit 225d, the number of questions is cleared by the conversation failure detection unit 225c, and the number of answers is also cleared by the conversation continuation determination unit 225a. End the process.

Here, a specific example of a conversation and an answer by the guidance target person 3, the guidance executor 4, and the robot 100 in the embodiment of the present embodiment is shown below.
Utterance of Guidance Target 3: "Hello"
Translation of Robot 100: "Hello"
Guidance executor 4 utterance: "Are you in trouble?"
Robot 100 translation: "Can you help you with something?"
Utterance of Guidance Target 3: "I'm looking for coin locker"
Robot 100 Translation: "Looking for a coin locker"
Utterance of Guidance Executer 4: "Hmm ..."
Translation of Robot 100: "Umm ..."
Utterance of Guidance Target 3: "Don't you know?"
Utterance of Guidance Executer 4: "Um ..."
Translation of Robot 100: "Umm ..." (conversation failure detection)
Robot 100's answer: "I'm answer behalf of him.
There are coinlockers at ~~~ "
Utterance of Guidance Target 3: "Oh! Thank you !!"
Robot 100 translation: "Oh! Thank you!"
Judgment of the end of conversation of the robot 100 leaving the guidance target person 3

As described above, according to the voice guidance system 1 of the embodiment of the present embodiment, the question of the guidance target person 3 and the answer of the guidance executor 4 can be appropriately performed while translating.
In particular, by distinguishing the guidance target person 3 and the guidance executor 4 by voice recognition and image recognition, it becomes possible to appropriately execute the acceptance of the question and the output of the answer to the question.
Further, when the conversation between the guidance target person 3 and the guidance executor 4 is broken, that is, when the answer is not appropriate, the voice guidance system 1 can answer on behalf of the person, and the guidance by the conversation can be established.

<2. Example of the second embodiment>
Next, an example of the second embodiment of the present invention will be described with reference to FIGS. 13 to 14.
The configuration of the voice guidance system 1 of the second embodiment of the present invention is the same as the configuration described with reference to FIGS. 1 to 7 in the first embodiment, and duplicate description will be omitted.
In the second embodiment, the voice guidance system 1 provides a conversation assisting service that assists the conversation between the guidance target person 3 and the guidance executor 4.

[Execution example of conversation assistance service]
FIG. 13 is a flowchart showing the flow of the conversation assisting service by the voice guidance system 1.
First, the robot control device 200 determines the spoken language (step S400).
Subsequently, in the robot control device 200, the conversation assist service (step S500) is performed based on the utterance language determined in step S400.

FIG. 14 is a flowchart showing the details of the conversation assisting service in step S500.
First, the question receiving unit 222 of the robot control device 200 executes a check for the presence / absence of an interrupt (step S21), and determines the presence / absence of an interrupt (step S22). Here, when there is no interrupt (“no interruption” in step S22), the question receiving unit 222 returns to the check for the presence or absence of an interrupt in step S21.

If there is a voice interrupt in the judgment of step S22 (“interruption available” in step S22), the question receiving unit 222 determines whether the interrupt content is an interrupt due to voice input or an interrupt due to the expiration of the answer timer. (Step S23).
When it is determined in step S23 that the interrupt is due to the input of voice (“voice” in step S23), the response availability determination unit 225b determines whether or not the recognized voice can be answered (step S24).

If the answer can be answered here (“OK” in step S24), the answer availability determination unit 225b searches for the answer (step S25) and outputs an image of the answer (step S26). Here, the answer is output only as an image, and no sound is output.

After outputting the image of the answer in step S26, the conversation failure determination unit 225c determines the reaction of the guidance target person 3 (step S27). Here, if the reaction is positive (“positive” in step S27), the conversation failure determination unit 225c clears the number of customer questions (step S28) and clears the conversation timer (step S29).

If the reaction determined in step S27 is negative (“negative” in step S27), the conversation failure determination unit 225c determines whether or not the number of customer questions is equal to or greater than the threshold value (step S31). Here, if it is equal to or less than the threshold value (“below the threshold value” in step S31), the process proceeds to step S29, and the conversation determination unit 225a clears the answer timer.

Further, if the number of customer questions in step S31 is equal to or greater than the threshold value (“greater than or equal to the threshold value” in step S31), the conversation failure detection unit 225c considers that the conversation has failed, and the answer output unit 225d responds immediately before (answer by image). ) Is output by voice (step S32). After that, the process proceeds to step S28, and the conversation failure determination unit 225c clears the number of customer questions.

After clearing the answer timer (step S29), the conversation continuation determination unit 225a confirms the usage state for determining whether or not the user is in front of the robot 100 based on the face recognition of the image recognition unit 224 (step). S30).
Here, when the conversation continuation determination unit 225a determines that there is a user (“in use” in step S30), the conversation continuation determination unit 225a returns to the interrupt check in step S21 again. Further, when it is determined that there is no user (“end of use” in step S30), the translation service is terminated.

As described above, in the embodiment of the present embodiment, the robot 100 assists the conversation by displaying images related to the conversation of a plurality of people, and when the conversation breaks down, the conversation is continued by answering by voice. be able to.

<3. Example of Third Embodiment>
Next, an example of the third embodiment of the present invention will be described with reference to FIGS. 15 to 17.
The configuration of the voice guidance system 1 of the third embodiment of the present invention is the same as the configuration described with reference to FIGS. 1 to 7 in the first embodiment, and duplicate description will be omitted.
In the third embodiment, in the voice guidance system 1, when the robot 100 answers an utterance (question) from the guidance target person 3 and the conversation between the guidance target person 3 and the robot 100 breaks down. In addition, the guidance of the guidance executor 4 is translated and transmitted to the guidance target person 3.
As shown in FIG. 8, the overall flow of the service is performed in the order of user determination (step S100), language determination (step S200), and translation service (step S300), and is described by the flowchart of FIG. 15 at the time of translation service. It is executed by the procedure to be performed.

[Execution example of guidance service]
FIG. 15 is a flowchart showing the flow of the translation service in the example of the present embodiment.
Further, FIGS. 16 and 17 (both figures are continuous views) are sequence diagrams in which the guidance executor 4 answers based on the conversation failure by executing the flowchart of FIG. In these sequence diagrams, the same step numbers are assigned to the parts corresponding to each step in the flowchart of FIG. Also in the case of the example of FIG. 15, the guidance target person 3 talks in English, and the guidance executor 4 talks in Japanese.

The details of the translation service shown in FIG. 15 will be described below.
First, the question receiving unit 222 of the robot control device 200 checks the interrupt processing (step S41) and determines the presence / absence of the interrupt processing (step S42). The interrupt processing determined by the question receiving unit 222 here includes an interrupt due to voice input and an interrupt due to the expiration of the answer timer.
When there is no interrupt processing in step S42 (“no interrupt” in step S42), the question receiving unit 222 returns to the check in step S41.

Further, when it is determined in step S42 that there is interrupt processing (“interruption available” in step S42), the question receiving unit 222 detects the arrival direction of the voice from the voice data of the plurality of channels (step S43). Then, the voice recognition unit 223 recognizes the input voice (step S44), the image recognition unit 224 recognizes the input image (step S45), and performs face recognition processing and identification processing (step S46).

Here, the question reception unit 222 determines whether the speaker is the guidance target person 3 or the guidance executor 4 (step S47). In step S47, when the determined speaker is the guidance target person 3 (“customer (English speaker)” in step S47), the answerability determination unit 225b determines whether or not the answer is possible (step S48).

When the answer can be answered by the determination in step S48 (“OK” in step S48), the answer output unit 225d searches for the answer to the question (step S49), and outputs the searched answer by voice and / or image from the robot 100. (Step S50).

Then, the conversation failure detection unit 225c recognizes the face by the image recognition unit 224 and determines the emotion, and determines whether the user's reaction is positive or negative (step S51).
Here, when the user's reaction is positive (“positive” in step S51), the conversation continuation determination unit 225a determines whether or not the user is in front of the robot 100 based on the face recognition of the image recognition unit 224. Check the usage status (step S52).
Here, when the conversation continuation determination unit 225a determines that there is a user (“in use” in step S52), the conversation continuation determination unit 225a returns to the interrupt check in step S41 again. Further, when it is determined that there is no user (“end of use” in step S41), the translation service is terminated.

The flow up to this point is the case where the robot 100 answers the question from the guidance target person 3 and the guidance target person 3 gives a positive reaction to the conversation, and as long as such a positive reaction continues, as long as such a positive reaction continues. The conversation between the guide target person 3 and the robot 100 continues.
However, if the reaction of the user determined in step S51 is negative, the conversation by the robot 100 will break down.

That is, when the reaction of the user determined in step S51 is negative (“negative” in step S51), the image recognition unit 224 determines the presence / absence of the guidance executor 4 (step S53). Even if it is determined in step S48 that the answer cannot be answered, the process proceeds to step S53 to determine the presence or absence of the guidance executor 4.

Then, when it is determined that the guidance executor 4 is present (“Yes” in step S53), the translation unit 226 translates the question (English) from the guidance target person 3 into Japanese (step S55). The translation result is output from the robot 100 as audio and / or an image (step S56). Further, when it is determined in step S53 that the guidance executor 4 does not exist (“none” in step S53), the process of calling the guidance executor 4 is performed (step S54), and then the process proceeds to step S55. After outputting the translation result, the process proceeds to the determination of the usage state in step S52.

If the determined speaker in step S47 is the guidance executor 4 (“site administrator (Japanese speaker)” in step S47), the translation unit 226 responds from the guidance executor 4 (Japan). The word) is translated into English (step S57), and the translation result is output from the robot 100 as voice and / or image (step S58). After outputting the translation result, the process proceeds to the determination of the usage state in step S52.

16 and 17 (both figures are continuous sequence diagrams) are sequence diagrams when the guidance executor 4 answers based on the conversation failure in the robot.
In this example, the guidance target person 3 first speaks in English, and the robot 100 transmits the voice input to the microphone array 131 to the question reception unit 222 of the robot control device 200. This voice interrupt is detected by the question reception unit 222. At this time, the voice direction is detected by the voice recognition unit 223, face recognition is performed by the image recognition unit 224 based on the image acquired by the camera 132 of the robot 100, and the speaker is guided by the question reception unit 222. It is identified as 3.

When it is identified as the guidance target person 3, the answer possibility determination unit 225b determines whether the answer is possible, and when the answer is possible, the answer output unit 225d searches for the answer conversation, and the answer as the search result is the robot. It is output from 100 as an English voice and / or an image of an English sentence.
Up to this point, the process of answering by the robot 100 shown in the upper half of FIG. 16 is performed.

Then, at the time of outputting this answer, the image recognition unit 224 recognizes the face of the guidance target person 3 from the image taken by the camera 132 of the robot 100, and the conversation failure detection unit 225c is based on the emotion determination of the guidance target person 3. Detects conversation breakdown.
Here, when the conversation failure detection unit 225c detects the conversation failure and confirms that the guidance executor 4 is present, a process of translating the question sentence spoken in English by the guidance target person 3 into Japanese is performed. The translation result is output.
Up to this point, the processing at the time of conversation failure detection by the robot 100 shown in the lower half of FIG. 16 is performed.

After that, in the case of the present embodiment, the process of translating the answer of the guidance executor 4 is performed.
That is, as shown in FIG. 17, the answer (Japanese utterance) of the guidance executor 4 is transmitted from the robot 100 to the question receiving unit 222 of the robot control device 200, and the voice interruption is detected by the question receiving unit 222. .. At this time, the voice recognition unit 223 detects the voice direction, the voice recognition is performed, and the image recognition unit 224 performs image recognition to identify the speaker (guidance executor 4).

When the guidance executor 4 is identified, the answer of the guidance executor 4 is translated into English, and the answer as the translation result is output from the robot 100 as an English voice and / or an image of an English sentence.

As described above, in the present embodiment, when the robot 100 answers the question of the guidance target person 3 and the conversation between the guidance target person 3 and the robot 100 breaks down, the answer of the guidance executor 4 is translated. Processing is performed. Therefore, it becomes possible to appropriately continue the conversation between the guidance target person 3 and the robot 100 while assisting the conversation by the guidance executor 4.

<4. Modification example>
The present invention is not limited to the above-described embodiments, but includes various modifications.
For example, in the above-described embodiment, the tablet terminal that displays the avatar is applied as the robot 100, but a robot having another shape may be used. Further, in the above-described embodiment, the robot 100 performs input processing by a microphone or a camera and output processing by a speaker, and data processing such as user identification, conversation processing, and translation processing is performed by the robot control device 200. I did it. On the other hand, some data processing or all data processing may be performed in the robot 100.

Further, in each of the embodiments up to this point, the guidance target person 3 and the guidance executor 4 have been shown as one person each, but the guidance target person 3 and the guidance executor 4 may be a plurality of persons. .. For example, when there are a plurality of guidance target persons 3, each guidance target person 3 can be identified by voice and an image, and the question of each guidance target person 3 can be answered.

Further, the above-described embodiment is described in detail in order to explain the present invention in an easy-to-understand manner, and is not necessarily limited to the one including all the described configurations. Further, in the configuration diagrams such as FIGS. 1 to 4, only the control lines and information lines considered to be necessary for explanation are shown, and not all the control lines and information lines are shown in the product. .. In practice, it can be considered that almost all configurations are interconnected. Further, in the flowcharts shown in FIGS. 8, 9, 13, 14, and 15, and the sequence diagrams of FIGS. 10, 11, 12, 16, and 17, the processing results of the embodiment are not affected. Within the range, the execution order of some processing steps may be changed, or some processing steps may be executed at the same time.

Further, the configuration described in the above-described embodiment may be realized by software by the processor interpreting and executing a program that realizes each function. Information such as a program that realizes each function can be placed in a memory, a hard disk, a recording device such as an SSD (Solid State Drive), or a recording medium such as an IC card, an SD card, or an optical disk.

1 ... Voice guidance system, 2 ... Site, 3 ... Guidance target person, 4 ... Guidance executor, 100 ... Robot, 110 ... CPU, 120 ... Storage device, 121 ... Input / output unit, 122 ... Scenario execution unit, 123 ... Screen Processing unit, 130 ... input / output device, 131 ... microphone array, 132 ... camera, 133 ... speaker, 134 ... touch panel, 134a ... text display unit, 134b ... avatar display unit, 134c ... guidance target language selection button, 134d ... guidance Executer language selection button, 134e ... Standing position switching button, 140 ... Communication interface, 200 ... Robot control device, 210 ... CPU, 220 ... Storage device, 221 ... Input / output device, 222 ... Question reception unit, 223 ... Voice recognition unit , 224 ... Image recognition unit, 225 ... Guidance unit, 225a ... Conversation continuation determination unit, 225b ... Answer availability determination unit, 225c ... Conversation failure determination unit, 225d ... Answer output unit, 226 ... Translation unit, 230 ... Robot control device 200 Communication interface, 300 ... Robot management server, 310 ... CPU, 320 ... Storage device, 321 ... Robot management unit, 330 ... Communication interface

Claims (5)

A voice guidance system including a camera, a microphone, and a speaker, which outputs a guidance voice based on the voice input to the microphone from the speaker.
A question reception unit that accepts questions by voice input to the microphone,
A voice recognition unit that calculates the characteristics of the user's voice corresponding to the question received by the question reception unit and recognizes the user based on the calculated voice characteristics.
An image recognition unit that calculates the characteristics of the user's image taken by the camera when the question reception unit detects voice, and recognizes the user based on the calculated characteristics of the image .
A user to be guided is selected using the characteristics of the user's voice calculated by the voice recognition unit and the characteristics of the user's image calculated by the image recognition unit, and the guidance voice to the selected user is transmitted from the speaker. Equipped with a guide to output
The guide unit includes a translation unit that translates the output voice into a predetermined language.
Based on the reception status of the voice input to the microphone in the question reception unit, the guidance unit outputs the voice received by the question reception unit in a predetermined language instead of outputting the guidance voice. The voice translated into the above is output from the speaker.
Voice guidance system. The guide unit selects a user who matches a user recognized by the voice recognition unit and a user recognized by the image recognition unit, and executes a conversation with the selected user using the microphone and the speaker. The voice guidance system according to 1. The question reception unit targets users who have detected pre-registered voice features or image features as non-guidance targets, and guides users who have detected pre-registered voice features or image features. And
The voice guidance system according to claim 1, wherein the guidance unit answers only the questions of the guidance target person. The image recognition unit performs a process of detecting the reaction of the guide target person photographed by the camera, and performs a process.
Further, a conversation failure determination unit that detects that the utterance of the non-guidance target person is not the utterance expected by the guidance target person based on the number of utterances of the non-guidance target person and the reaction of the guidance target person.
It is provided with an answerability determination unit that determines whether or not the question can be answered by voice received by the question reception unit.
The voice guidance system according to claim 3 , wherein when the conversation failure determination unit determines the conversation failure and the answer availability determination unit determines that the answer is possible, the guidance unit outputs an answer to the question of the guidance target person. .. It is a voice guidance method that provides guidance by voice by outputting a guidance voice based on the voice input to the microphone by the speaker.
The question reception process, in which the question reception department accepts questions by voice input to the microphone,
The voice recognition unit calculates the characteristics of the user's voice corresponding to the question received by the question reception process in the question reception unit, and recognizes the user based on the calculated voice characteristics.
When the voice is detected by the question reception process in the question reception unit, the image recognition unit calculates the characteristics of the user's image taken by the camera and recognizes the user based on the calculated characteristics of the image. Recognition processing and
The guidance unit selects a user who provides guidance using the characteristics of the user's voice calculated by the voice recognition processing in the image recognition unit and the characteristics of the user's image calculated by the image recognition processing, and for the selected user. Guidance processing that outputs guidance voice from the speaker,
Includes a translation process in which the output voice from the speaker is translated into a predetermined language.
Based on the reception status of the voice input to the microphone in the question reception process, in the guidance process, instead of outputting the guidance voice, the voice received in the question reception process is translated into a predetermined language. The voice translated into the above is output from the speaker.
Voice guidance method.

Images