JPWO2018230345A1 - Dialogue robot, dialogue system, and dialogue program - Google Patents

Abstract

A dialogue robot, a dialogue system, and a dialogue program capable of realizing a natural dialogue with a user are provided. In the interactive robot according to the present invention, when a user specifies that the user is unregistered on the server based on the user's face information, the interactive robot generates a name question for asking the name of the user generated by the server Ii) when the server specifies that the user is registered based on the face information, receives response sentence information including the name of the user generated by the server, and speaks The unit makes a start utterance to the user including the name question or the name of the user, depending on whether or not the user has been registered. The interactive robot of the present invention can arbitrarily change the level of conversation with the user, and may perform pre-processing such as various questions before changing the set level. [Selection] Figure 1

Description

  The present invention relates to a technique using an interactive robot, and more particularly to an interactive robot, an interactive system, and an interactive program capable of actively interacting with various patterns of voice while listening to a user's facial expressions.

  Advances in information means have increased opportunities to communicate with foreign people in foreign languages. In addition, with the progress of transportation means, it has become an era where people can easily travel abroad, and there are increasing opportunities to understand and speak the language of the destination country. Therefore, acquisition of foreign languages including English, especially acquisition of conversation skills, is important regardless of age.

In recent years, with the development of communication technologies such as the Internet, various technologies for performing pseudo-conversations via a network have been proposed.
For example, Patent Document 1 proposes an interactive robot in which abundant response sentences are accumulated, and an interactive system that can interact with a character in addition to the robot by connecting the robot to a computer.

  On the other hand, in Patent Document 2, for example, in an English conversation learning support system using a computer, a metaagent (11) that manages the information of the user (10) and each person who gives guidance to the user (10) A learning support system characterized by having a teacher agent (13) is proposed.

Japanese Patent No. 3958253 JP 2002-304114 A

However, it is not limited to the above-described Patent Document 1 and Patent Document 2, and it cannot be said that the current technology appropriately satisfies market needs, and there are various problems as described below.
In other words, when realizing a pseudo-conversation system via a network, since the person to talk with is a virtual reality person, an unusual atmosphere is inevitably produced unlike a real person.

For example, Patent Document 2 has a problem that since it only interacts with a character on a computer screen, it is not realistic, and it is difficult for a user to show interest, particularly.
On the other hand, in Patent Document 1, although conversational with robots and other characters is possible, it is not easy to get tired of, but since conversations with users are mainly performed only by voice, I cannot deny the feeling of becoming.

Originally, the act of conversation between humans can be changed by the natural progress of dialogue and changes in emotions between speakers. In this regard, for example, if the interactive robot proposed in Patent Document 2 is improved in performance and has a large amount of information, although a more human conversation can be realized, the processing operation may be delayed. The price itself becomes very expensive.
As described above, in the prior art including the above-described patent documents, there is no level that can realize a natural conversation (lively as if humans are speaking) with a user at a low cost. There is much room for improvement.
In addition to the above-mentioned language learning such as English conversation, for example, it is expected that the need for a system for interacting with a user for nursing care or entertainment will increase more in the future.

  The present invention has been made in view of the above problem as an example, and an object thereof is to provide a dialog robot, a dialog system, and a dialog program that can realize a natural dialog with a user.

  In order to solve the above problems, a dialog robot according to an embodiment of the present invention is (1) a dialog robot that is connected to a server on a network via a communication line and performs an English conversation with a user. An imaging unit that images the face of the user, a speech recognition unit that recognizes speech spoken by the user, a speech unit that speaks to the user, facial information regarding the face via the communication line, and the A user information transmission unit that transmits voice information recognized by the voice recognition unit to the server; and a response statement information reception unit that receives response statement information from the server, wherein the user information transmission unit includes the imaging unit. Transmits the face information captured by the server to the server, and i) generates the server when the server identifies the user as unregistered based on the face information. When the response text information receiving unit receives the response text information including a name question for asking the name of the user, and ii) when the server specifies that the user is registered based on the face information, The response text information receiving unit receives the response text information including the name of the user generated by the server, and the utterance unit sends the name to the user according to whether or not the user has been registered. A start utterance including a question or the name of the user is made to the user, and the user information transmission unit transmits the voice information uttered by the user with respect to the start utterance recognized by the voice recognition unit to the server. It is characterized by.

  Furthermore, in order to solve the above-described problem, an interactive system according to an embodiment of the present invention is (2) an interactive system including the interactive robot according to (1) and the server, wherein the server is And a database individually managed for each user, and information relating to a dialogue with the user is recorded in the database.

  In the dialogue system according to (2) above, (3) the server converts the user's voice information received from the user information transmission unit into text, and the voiced voice information and the response text in the database. Matching with information is performed, and the response sentence information including an appropriate response sentence based on the matching is transmitted to the response sentence information receiving unit, and the dialogue robot is based on the received response sentence information including the response sentence. It is preferable to speak to the user via the utterance unit.

  In the dialogue system according to (3), (4) the imaging unit images the user who is responding to the response sentence, and the user information transmission unit utters the response sentence. Along with the user's voice information, the image information of the user being answered, which is captured by the imaging unit, is transmitted to the server, and the server is based on the image information of the user received from the user information transmission unit. It is preferable to determine the facial expression of the user and transmit the response text information including the next response text whose contents are changed based on the determination result of the facial expression to the response text information receiving unit.

  In the dialogue system according to (3) or (4), (5) the server transmits response sentence information including a level determination question for determining a conversation level to the response sentence information receiving unit. Then, the dialogue robot utters the received response sentence including the level determination question to the user via the utterance unit, and based on the response to the level determination question issued by the user It is preferable that a conversation level is determined, and a response sentence in the database is selected by the server based on the determined conversation level and transmitted to the response sentence information receiving unit.

  Furthermore, in order to solve the above-described problem, a dialogue program according to an embodiment of the present invention causes a dialogue robot or a server to capture an image of a user's face using an imaging unit, and information about the captured user's face is stored in the server. Response text information including a name question that asks the user's name when the user is unregistered, and i) specifies whether the user is unregistered based on the information about the face And ii) when registered, the response sentence information including the user's name is generated, and the user is uttered as a start utterance based on the response sentence information using the utterance unit, and the voice Using the recognition unit, the user's utterance with respect to the start utterance is voice-recognized, and the user's utterance that is voice-recognized Characterized in that to transmit via the serial communication line.

  According to the present invention, it is possible to have a dialogue with a dialogue robot at a relatively low cost, and a user who has a dialogue with the dialogue robot for the first time can start a dialogue in a natural manner, so that it can be used for a wide range of ages. It is possible to provide an opportunity to talk with interest without getting bored.

It is a mimetic diagram showing the whole dialogue system 100 composition of a 1st embodiment. It is a functional block diagram which shows the structure of the dialogue robot 10 of 1st Embodiment. It is a functional block diagram which shows the structure of the server 20 of 1st Embodiment. It is a flowchart explaining the initial operation | movement in the English conversation of 1st Embodiment. It is a flowchart which determines the content of conversation in 1st Embodiment. It is a schematic diagram which shows the whole structure of dialogue system 100 'of 2nd Embodiment. It is a flowchart explaining the operation | movement flow of the level setting in a modification. It is a flowchart explaining the operation | movement flow of the level setting in a modification.

  In the following, an example in which a user performs an English conversation with a server and a dialogue robot via the network N will be described with respect to an embodiment for carrying out the present invention. However, the present invention is not limited to the following. For example, not only English conversation but also conversation in another language may be used, and an example using a plurality of interactive robots may be used. Further, the present embodiment is not limited to language learning as described below, and can be applied as an interactive robot in the nursing and entertainment fields.

<< First Embodiment >>
<Dialogue system 100>
FIG. 1 is a schematic diagram illustrating a configuration of the entire dialogue system 100 according to the embodiment.
The interactive system 100 according to the present embodiment includes an interactive robot 10 and a server 20. The interactive robot 10 and the server 20 are connected so that they can communicate with each other via a network N such as the Internet. Then, the user U performs an English conversation with respect to the interactive robot 10 that is communicably connected to the server 20 for the purpose of learning words.
As described above, the dialogue system 100 of the present embodiment functions as an English conversation system for carrying out an English conversation of the user U.

  Although details will be described later, in this interactive system 100, information processing is performed as follows. That is, first, the speech of the user U is sent to the server 20 via the Internet via the voice recognition unit of the dialogue robot 10. In the server 20, the speech is converted into text by the voice conversion engine. Thereafter, matching between the uttered voice converted to the text and the conversation sentence database is performed, and an optimal response sentence for the uttered voice is selected, and the selected sentence is voiced by the voice engine and is sent to the dialogue robot 10. The dialogue robot 10 is uttered by being sent.

  As will be described later, the interactive system 100 may further include an information processing terminal 50. In this case, for example, the user U can execute a conversation with a plurality of persons including not only the interactive robot 10 but also a virtual person (character on the PC) displayed on the screen of the information processing terminal 50. About the character on this PC, you may incorporate suitably in the format along the meaning of this embodiment with reference to above-mentioned patent 3958253.

Hereinafter, each component of the interactive system 100 of this embodiment will be described in detail.
<Dialogue robot 10>
First, the configuration and functions of the interactive robot 10 of this embodiment will be described in detail with reference to FIG.
The interactive robot 10 according to the present embodiment is a robot that is connected to a server 20 on a network N (for example, the Internet) via a communication line and performs an English conversation with a user U. The interactive robot 10 of the present embodiment is a robot having a form of a small bird that can interact with the user U in English, for example.

As shown in the figure, the bird-type dialogue robot 10 includes an imaging unit 11, a voice recognition unit 12, a utterance unit 13, a communication interface 14 (user information transmission unit 14a, response sentence information reception unit 14b), and a ROM / RAM unit. 15, and a power source 16.
The interactive robot 10 may include a part of the function of the overall control unit 38 in the server 20 described later as the control unit 17. A specific example of the control unit 17 in this case is a known processor such as a CPU.

  The imaging unit 11 has a function of imaging the face of the user U and the like. As a specific example of the imaging unit 11, a camera on which an image sensor such as a CCD or a CMOS is mounted can be exemplified. In the present embodiment, a camera is mounted around the forehead of the small bird-type interactive robot 10, but the eye of the small bird may be the imaging unit 11, for example. When the eye of the bird becomes the imaging unit 11, the lens surface may be covered with an arbitrary timing (for example, as a trigger for imaging the user U), for example, with a movable piglet.

  The voice recognition unit 12 extracts a speech voice (spoken language) uttered by the user U via a sound collector (such as a microphone) (not shown), and converts the extracted speech voice into a character string that can be recognized by the computer. have. The speech recognition algorithm performed by the speech recognition unit 12 is not particularly limited. For example, a statistical method using an acoustic model or a known algorithm based on a hidden Markov model can be applied. For such voice recognition, various known software can be applied. Furthermore, as another configuration example in speech recognition, a known speech recognition technique such as Japanese Patent No. 3968133 and Japanese Patent No. 3361732 may be applied.

The generation unit 13 has a function of speaking to the user U. The generating unit 13 in the present embodiment is a speaker and is mounted, for example, in the vicinity of the abdomen of the small bird type interactive robot 10.
As will be described later, an utterance based on response sentence information generated in the server 20 is performed via the generation unit 13.

  The communication interface 14 has a function of transmitting and receiving various data to and from other terminals via a communication line such as a network N (for example, the Internet). In this embodiment, the user information transmission unit 14a and the response text information reception unit 14b is included.

Among these, the user information transmitting unit 14a has a function of transmitting information about the user U through the communication line. As the “information about the user U”, for example, information for identifying the user U acquired by the imaging unit 11 (image information about the face (expression), image information such as gestures by the body, etc.), the voice recognition unit 12, etc. Includes the utterance (speech) information of the user U acquired by.
On the other hand, the response text information receiving unit 14b has a function of receiving response text information generated by the server 20 described later via the communication line.

  The ROM / RAM unit 15 has a function of recording and holding various types of information. The “various information” includes, for example, information acquired by the imaging unit 11 and the voice recognition unit 12, information received via the communication interface 14, and the like. Specific examples of the ROM / RAM unit 15 of the present embodiment include a nonvolatile memory such as an EEPROM and a flash memory, and a volatile memory such as an SRAM and a DRAM.

  The power supply 16 has a function of supplying power necessary for the operation of the interactive robot 10 including the above-described configuration. The power source 16 may be an outlet plug for obtaining power from a known commercial power source, or may be a battery such as a primary battery or a secondary battery.

<Server 20>
Next, the detailed configuration of the server 20 of the present embodiment will be described with reference to FIG.
The server 20 of this embodiment functions as a computer that performs information communication with the interactive robot 10 via a network N such as the Internet.
More specifically, the server 20 includes an application server 30 and a storage server 40. When such a server 20 is functionally classified, the communication interface 31, the response sentence matching unit 32a, the response sentence voice converting unit 32b, the response sentence information transmitting unit 32c, the utterance voice recording unit 33, the information analysis recording unit 34, the face The information matching unit 35, the conversation level setting unit 36, the ROM / RAM unit 37, the overall control unit 38 and the database 40 are configured.

  The communication interface 31 has a function of connecting to a network N such as the Internet. The server 20 can send and receive various types of information via the communication interface 31.

  The response sentence matching unit 32a has a function of performing content analysis including text conversion of the uttered voice, and selects a response sentence corresponding to the semantic content of the analyzed uttered voice from the response sentence database 41 described later. When the server 20 includes the conversation level setting unit 36 as a constituent element, a response sentence corresponding to the conversation level of the user U is further displayed based on the conversation level set by the conversation level setting unit 36. You may have the function to choose from.

  If the response sentence matching unit 32a analyzes the content of the speech of the user U and determines that the semantic content of the speech is not a semantic content that allows the response sentence to be selected, the response sentence matching unit 32a prompts the user U. On the other hand, a response sentence that prompts the user to speak again may be selected from the response sentence database 41 and transmitted to the dialogue robot 10.

  The response sentence voice conversion unit 32b has a function of converting the response sentence selected by the response sentence matching unit 32a into voice data. The file format of the audio data is not particularly limited, and may be an uncompressed format such as WAV or a compressed format such as MP3 or AAC. Various known software can also be applied to such voice conversion. Further, the response sentence voice conversion unit 32b converts the voice data into voice data uttered by a native speaker, for example.

  The response sentence information transmitting unit 32c has a function of transmitting the voice data converted by the response sentence voice converting unit 32b as response sentence information to the interactive robot 10 via the network N.

  The utterance voice recording unit 33 has a function of recording the utterance voice after the content analysis is performed by the response sentence matching unit 32 a in the database 40. More specifically, the utterance voice recording unit 33 records the utterance voice of the user U received from the interactive robot 10 via the network N in the learning history DB 43 as needed. As will be described later, the learning history DB 43 is managed for each user, and is individually recorded in the learning history DB 43 together with, for example, a date, a conversation time, conversation contents (used theme, etc.) or a correct answer rate to a question. As described above, in the present embodiment, the server 20 includes the database 40 managed individually for each user U, and information related to English conversation with the user U is recorded in the database 40.

  The information analysis recording unit 34 mainly functions to analyze the image information acquired by the imaging unit 11 transmitted from the user information transmission unit 14a of the interactive robot 10, and to record the image information after the analysis in the database 40. have. More specifically, the information analysis recording unit 34 analyzes whether or not a person is included in the image information received from the user information transmission unit 14a, and if the person is included, an image of the head or upper body, etc. Is recorded in the ID information DB 42. At this time, when the person is registered in the ID information DB 42 for the first time, it is preferable to record the person together with the name information of the person as described later.

  Note that when analyzing whether the acquired image information includes a person, a known person extraction algorithm such as morphological processing can be applied. Further, in this person extraction algorithm, the extracted person's face can be easily specified, so the information analysis recording unit 34 may record the specified face information in the ID information DB 42.

  The face information matching unit 35 has a function of matching the face information of the user U received from the user information transmitting unit 14a of the interactive robot 10 with the face information already recorded in the ID information DB 42. In other words, it can be said that the face information matching unit 35 has a function of determining whether the face information of the user U received from the user information transmitting unit 14a is registered or not registered.

  The conversation level setting unit 36 has a function of setting the level of the response text that the response text matching unit 32a selects from the response text database 41 in accordance with the content of the answer to the question of the user U. The conversation level setting unit 36 of the present embodiment can set a plurality of levels (in this embodiment, 12 levels) in descending order of difficulty. As a specific level setting method, for example, the conversation level setting unit 36 may set an intermediate level (in the case of 12 levels, the middle level 6) as an initial value. After the setting, the response sentence from the dialogue robot 10 is controlled to respond with the determined level of English sentences.

As another specific level setting method, for example, an intermediate (level 6) response sentence is used. If the user U gives an appropriate response to this response sentence, the level is increased by one, and an incorrect answer is given. For example, it may be lowered by one. As for the response text and the content of the response, a plurality of English examples are stored in the response text DB 41 corresponding to the above-described levels. As examples of specific question and answer content sentences, for example, a well-known English question collection or a public English test question may be applied.
Accordingly, the conversation level setting unit 36 controls the response sentence matching unit 32a so as to extract a response sentence corresponding to the set level from the response sentence DB 41, while the response of the user U received from the voice recognition unit 12 is the response sentence. What is necessary is just to determine whether it corresponds with the example of a response stored in DB41.
In addition, since the dialogue system 100 of the present embodiment is an English conversation system as language learning, it is possible to set a level based on the correctness of the utterance content from the user U. For example, when the present system is applied to the care field, etc. The conversation level setting unit 36 is not necessarily required.

The ROM / RAM unit 37 has a function of storing software operating in the server 20 and software operating in the interactive robot 10. Specific examples of the ROM / RAM unit 37 include a hard disk drive and a large capacity memory.
Note that a part or all of the software operating on the interactive robot 10 may be stored by the interactive robot 10 itself. In this case, the ROM / RAM unit 37 stores various software operating on the server 20. The

  The overall control unit 38 is a known processor such as a CPU, and has a function of controlling the operation of each component of the server 20 in an integrated manner.

The database 40 has a function of recording and holding various data used in the interactive system 100. The database 40 of this embodiment can be exemplified by a large capacity hard disk drive, for example. As described above, the database 40 of the present embodiment includes the response sentence DB 41, the ID information DB 42, and the learning history DB 43 described above. Among these, at least the learning history DB 43 is configured such that a registered user's learning history is individually managed for each user.
Among these, examples of the response text stored in the response text DB 41 include “Good morning” and “Hi”. As other examples of the response sentence to be stored, various known sentence examples and known data structures disclosed in, for example, International Publication No. WO2005 / 076258 and Japanese Patent Application Laid-Open No. 2012-215645 may be used as appropriate. .
Since the present embodiment is an English conversation system, the above-mentioned example sentences are accumulated in the response sentence DB 41. For example, when applied to the nursing field, examples of daily conversation sentences and hobbies / entertainment topics are examples of response sentences. Accumulated as. That is, the response sentence DB 41 of the dialogue system 100 according to the present embodiment stores necessary example sentences according to the field to which it is applied (such as business assistance such as language learning, nursing care, entertainment, reception reception, etc.).

<Dialogue program>
The program applied to the dialogue system 100 described above causes either the dialogue robot 10 or the server 20 to cause the dialogue robot or the server to image the user's face using the imaging unit, and the imaged information on the user's face is captured. To the server via a communication line, to identify whether the user is unregistered based on the information about the face, and i) a response including a name question that asks the user's name if unregistered And ii) generating response sentence information including the user's name when registered, and using the utterance unit to utter the user as a start utterance based on the response sentence information The user who recognized the speech of the user with respect to the start utterance using the speech recognition unit and recognized the speech The speech is transmitted via the communication line to the server.

<Startup (user specific) operation flow>
Next, with reference to FIG. 4, the user specifying operation in the initial stage of activation of the interactive system 100 in the present embodiment will be described in detail. 4 to 6 are executed under the control of the overall control unit 38 of the server 20 and the control unit 17 of the interactive robot 10 provided as necessary.
First, in step S10, when a user U who desires a conversation with the interactive robot 10 (English conversation in this example) turns on the power switch (not shown) of the interactive robot 10, the system of the interactive robot 10 is activated and the network N (hereinafter referred to as “network N”). The connection request is made with the server 20 via the Internet).

  Next, in step S <b> 11, the server 20 transmits an ID information request to the interactive robot 10 via the Internet based on the connection request from the interactive robot 10. The ID information is not particularly limited as long as it is information that can be identified by the user U, and is not particularly limited, such as facial information and fingerprint information. Therefore, below, it demonstrates using the face information regarding a user's face as an example of such ID information.

  When the interactive robot 10 receives this ID information request, in step 12, the imaging unit 11 operates to acquire the user U's ID information (hereinafter referred to as face information). More specifically, the dialogue robot 10 adjusts the image pickup unit 11 (camera) so that the face of the user U facing the user U enters the object scene, and picks up an image.

  In step S <b> 13, the user information transmission unit 14 a of the interactive robot 10 transmits the face information of the user U captured by the imaging unit 11 to the server 20.

  Next, in step S14, the server 20 determines whether the received face information is already registered or unregistered. More specifically, the face information matching unit 35 of the server 20 searches whether the face information received this time matches the face information registered in the ID information DB 42. At this time, facial information matching is identified by extracting features that stand out from the facial image. For example, eigenfaces using principal component analysis, linear discriminant analysis, elastic bunch graph matching, dynamic link matching, etc. An algorithm may be applied.

  In step S15, i) when the user U is identified as unregistered based on the received face information, response sentence information including a name question for asking the name of the user U is generated (S15-1). For example, an example sentence such as “What is your name?” Is stored in the response sentence DB 41 as the “name question”, and the overall control unit 38 accesses the response sentence DB 41 and generates response sentence information including the name question. To do. Thereafter, the generated response text information is received by the response text information receiving unit 14b of the interactive robot 10 via the Internet.

  On the other hand, ii) when the user U is specified to be registered based on the received face information, response sentence information including the name of the user U is generated (S15-2). For example, an example sentence such as “Hello, (name).” Is stored in the response sentence DB 41 as the “response sentence including the name”, and the overall control unit 38 accesses the response sentence DB 41 and registers the registered user. Response text information combined with the name of U is generated. Thereafter, the generated response text information is received by the response text information receiving unit 14b of the interactive robot 10 via the Internet.

In step S16, the interactive robot 10 that has received the response text information by the response text information reception unit 14b utters the received response text as a start utterance via the generation unit 13 (speaker).
As described above, in the present embodiment, the utterance unit 13 gives the user U a name question that asks the user U for a name or a start utterance that includes the name of the user U, depending on whether or not the user U has been registered. Do it.

  Next, in step S17, since an answer corresponding to the start utterance from the utterance unit 13 is made by the user U, the utterance content (answer) of the user U is recognized by voice. More specifically, the voice recognition unit 12 of the interactive robot 10 recognizes the utterance content that is the answer of the user U by voice.

  Then, the speech content of the user U that has been voice-recognized is transmitted to the server 20 via the Internet in the subsequent step S18. More specifically, the user information transmission unit 14 a of the interactive robot 10 transmits the voice information uttered by the user to the start utterance recognized by the voice recognition unit 12 to the server 20.

  When the current user U is registered for the first time, in subsequent step S19, the information analysis recording unit 34 of the server 20 records the received name information of the user U together with the ID information used in step S14 in the database 40. More specifically, in the present embodiment, name information is recorded and stored in the ID information DB 42 together with the facial information of the user U. The ID information recorded together with the name information is face information in this example, but may be other information when the information used in step S14 is other identification information such as fingerprint information.

When the above startup initial (user identification) operation flow is completed, a conversation between the user U and the interactive robot 10 is started.
In step S10, the interactive robot 10 may be equipped with a display means such as an LED that indicates that the system has been activated or that the online operation has been successful. In this case, for example, it is preferable to change the display color of the display means in the order of system startup and online.

  As described above, when the power switch of the interactive robot 10 is turned on and connected to the network N, the imaging unit 11 of the interactive robot 10 operates to capture the face of the user U in front of the interactive robot 10. The captured user image information is transmitted to the information analysis recording unit 34 and the face information matching unit 35 of the server 20 via the network N. Then, the face information matching unit 35 performs matching with the registered face image, and specifies who the user is already registered.

  If the face information matching unit 35 matches the registered face image, the person is identified and the utterance “Hello, (name)” is sent to the dialogue robot. On the other hand, if there is no facial image that can be determined as the same person in the registered image, the user U saves the image as a new registrant and sends a name question “What is your name?” To the interactive robot 10. When the user U answers the question and speaks his / her name, the user U registers the name together with the previous image.

<Determination flow of conversation content>
Next, the operation in which the dialogue system 100 determines the content of the conversation with the user U will be described in detail with reference to FIG.

  First, in step S30, a theme request for requesting what should be the subject of English conversation is transmitted from the server 20 to the interactive robot 10 via the Internet. Specifically, the overall control unit 38 of the server 20 extracts a response sentence related to the theme request from the response sentence DB 41 and transmits voice information based on the response sentence to the interactive robot 10 via the communication interface 31. In addition, as the contents of this theme request, for example, you may ask "What do you want the theme?" In the user U's native language, or "Sport, weather, meal, XX" You may ask a question to present as an option. The theme request itself may be made in English as a learning language instead of the user U's native language.

Next, in step S31, the dialogue robot 10 utters voice information related to the received theme request from the utterance unit 13, and executes an English conversation theme inquiry to the user U this time.
In step S32, the voice recognition unit 12 recognizes the speech about the theme answer from the user U, and the user information transmission unit 14a transmits the voice information about the theme answer recognized by the voice recognition unit 12 to the server 20. .

Next, in step S33, the information analysis recording unit 34 of the server 20 analyzes the voice information related to the theme answer, and determines the theme to be learned this time.
After the learning theme is determined in step S33, it is determined in step S34 whether a learning history exists. More specifically, the overall control unit 38 of the server 20 refers to the learning history DB 43 and searches whether there is a history learned about the user U in the past. If it is determined that there is a past learning history, in step S35, the overall control unit 38 reflects the content of the extracted learning history in the current English conversation. Specific reflection contents include, for example, continuing a conversation level that has been performed in the past and the presence or absence of an already learned theme.

On the other hand, when the current user U is new, the history in the learning history DB 43 is still zero, so that the learning histories after this time are sequentially accumulated.
Thus, after passing through this step S34, the conversation between the user U and the dialogue robot 10 is recorded and managed by the utterance voice recording unit 33 and the like under the control of the overall control unit 38. Further, since subsequent conversations between the user U and the dialogue robot 10 are recorded in the learning history DB, the proficiency level of the user U, the conversation level, and the like are updated in the learning history DB as needed.

  Next, after the confirmation of the presence / absence of the learning history and its reflection are completed, a specific English conversation related to the current theme is started in step S36. That is, the overall control unit 38 controls the response sentence matching unit 32a to select a response sentence suitable for the current theme from the response sentence DB 41. Then, after the selected response sentence is converted into voice information by the response sentence voice converting unit 32b, the response sentence information transmitting unit 32c transmits the response sentence information as response sentence information to the interactive robot 10 via the communication interface 31.

In step S37, the interactive robot 10 that has received the response message information generated in step S36 operates the imaging unit 11 and images the user U who is responding to the response message. At this time, the imaging unit 11 may use the ROM / RAM unit 15 as appropriate to capture one or a plurality of still images, or capture a moving image when replying for a predetermined time (for example, within a few seconds to a few minutes). Also good.
This step S37 may be always performed when the user U replies to the response sentence, or may be executed at an arbitrary timing such as every 5 minutes or every 10 minutes. In that case, steps S40 to S42 described later may be executed at the timing when step S37 is activated.

In Step S38, a response sentence is uttered by the utterance unit 13 based on the response sentence information generated in Step 36. Note that the order of step S37 and step S38 described above may be reversed.
Next, in step S39, the voice recognition unit 12 recognizes the utterance to the response sentence from the user U, and the user information transmission unit 14a utters the response sentence recognized by the voice recognition unit 12 (an answer to the response sentence). The voice information regarding the response sentence is transmitted to the server 20.

At substantially the same time as this step S39, in step S40, the image information of the user U acquired in step S37 (the still image or moving image of the user U at the time of answering) is transmitted to the server 20 via the Internet. Note that step S39 and step S40 described above may be executed substantially simultaneously, or step S39 may be performed earlier or later.
As described above, in this embodiment, the user information transmission unit 14a of the interactive robot 10 includes the voice information of the user U who utters (responds) to the response sentence and the image information of the user U who is responding captured by the imaging unit 11. Is transmitted to the server 20.

  In step S41, the server 20 determines the facial expression of the user U based on the image information of the user U received from the user information transmission unit 14a. More specifically, for example, when a numerical value of “anger” or “sadness” greater than or equal to a predetermined value is detected among emotion elements that can be read from the facial expression of the user U, the server 20 is confused by the facial expression of the user U. (In other words, the user U feels boring or tired).

As the quantification of emotion elements, for example, “Emotion API” using a face recognition technology published by Microsoft Corporation can be applied. This technology recognizes a face included in an image, and is an eight major emotion elements. "Anger", "contempt", "fear", "hate", "happiness", "intermediate", "sadness", and "surprise" are quantified. In addition, it means that the feeling is so strong that this figure is high.
As other determination methods described above, for example, JP-A-3-252775, JP-A-8-249447, JP-A-8-249453, JP-A-10-255043, JP-A-2001-051338. A publicly known technique for recognizing facial expressions, such as a gazette, may be applied, and emotions related to “confused” (“angry”, “anxiety”, “hate”, etc.) may be extracted and determined from the facial expressions. .

  If it is determined in step S41 that the facial expression of the user U is confused, in step S42, the server 20 controls the conversation level setting unit 36 to change the setting level. More specifically, the conversation level setting unit 36 executes processing for lowering one or more levels from the currently set conversation level. Thereafter, the response sentence matching unit 32a is controlled to select an appropriate response sentence candidate from the response sentence DB 41 based on the changed conversation level.

Note that when it is determined in step S41 that the expression of the user U is not confused, in this embodiment, the setting level is unchanged, but this is not a limitation. For example, if a numerical value of “joy” or “fun” greater than or equal to a predetermined value is detected among emotion elements that can be read from the facial expression of the user U in step S41, the conversation level setting unit 36 may be controlled to increase the setting level. Good. In this case, the overall control unit 38 may transmit a response sentence that suggests raising the setting level to the interactive robot 10 before raising the setting level. Various response sentences that suggest increasing the setting level can be stored in the response sentence DB 41 in advance.
Thus, in this embodiment, it can be said that the system side, not the user U, actively changes the content (topic, etc.) and level of the conversation according to the facial expression determination result of the user U.

Next, in step S43, response sentence information including the next response sentence whose content has been changed based on the determination result of the facial expression of the user U is transmitted to the interactive robot 10 via the Internet. More specifically, the response sentence matching unit 32a selects an appropriate response sentence candidate from the response sentence DB 41 as the next response sentence based on the set conversation level. Then, after the response sentence voice converting unit 32b converts the selected next response sentence into the voice information, the response sentence information including the next response sentence is received by the response sentence information transmitting unit 32c via the communication interface 31. It is transmitted to the information receiving unit 14b.
In this step S43, a response sentence that prompts the user to make a grammatical error indication or correction or pronunciation correction and training for the utterance made by the user U in step S39 may be transmitted to the dialogue robot 10.

  In step S43, the following processing is executed in the server 20. That is, first in step S39, the speech information related to the user's utterance (response) received from the user information transmitting unit 14a is converted into text by the utterance speech recording unit 33. Then, the response text matching unit 32a performs matching between the voiced text information and the response text information of the response text DB 41. Further, response text information including an appropriate response text based on the matching result is generated via the response text voice conversion unit 32b and the response text information transmission unit 32c and transmitted to the response text information reception unit 14b of the interactive robot 10. The

In step S44, the interactive robot 10 utters the user U via the utterance unit 13 based on the received response text information including the next response text.
Next, in step S45, the speech recognition unit 12 recognizes the speech of the user U in response to the next response sentence uttered, and the speech information about the speech content of the user U that has been speech-recognized is also transmitted to the user information transmission unit 14a. Is transmitted to the server 20 via.

  In step S46, after the server 20 receives the audio information transmitted in step S45, it is determined whether or not the current English conversation theme learning has been completed. More specifically, the overall control unit 38 refers to the response sentence DB 41 or the like to search for the next response sentence, and completes the learning when it is determined that the theme learning has ended. At this time, a notification that learning is completed may be transmitted to the interactive robot 10 via the Internet and the utterance unit 13 may utter. Further, the overall control unit 38 may perform control to update the learning history in the learning history DB 43 such as updating the conversation level when the theme learning is completed.

On the other hand, when the overall control unit 38 determines in step S46 that the theme learning has not been completed yet, the process returns to step S36, and the response sentence matching unit 32a refers to the response sentence DB 41 to appropriately respond to the user U's response. Repeat control to select response text.
As described above, in this embodiment, when the interactive robot 10 throws a conversation to the user U, a face image during the response of the user U is captured. Then, the facial expression being answered by the user U is received by the server 20 via the Internet together with the user's speech, and the server 20 determines the emotion of the user U based on the image. Then, as a result of the determination, there is a main feature in that the response sentence is changed to a level adapted to each according to the facial expression of the user U.

<< Second Embodiment >>
The dialogue system 100 ′ according to the second embodiment of the present invention will be described below using FIG. In the first embodiment, the interactive robot 10 is connected to the physical server 20 via the network N so as to be capable of information communication. However, the present embodiment is different in that cloud computing is used. Therefore, below, a different point from 1st Embodiment is demonstrated, the thing which show | plays the same function and effect | action as the above-mentioned structure attaches | subjects the same reference number, and abbreviate | omits the description suitably.

As shown in FIG. 6, the dialogue system 100 ′ according to the present embodiment is a system using so-called cloud computing, and includes a dialogue robot 10, a logical virtual server 20 ′, and a physical data center 60. It consists of The interactive robot 10 and the virtual server 20 ′ are connected so as to be able to communicate with each other via the Internet. Then, the user U performs an English conversation with the interactive robot 10 that is communicably connected to the virtual server 20 ′ for the purpose of learning words.
The virtual server 20 ′ is connected to a data center 60 in which various physical types are arranged via a network N.

In the virtual server 20 ′, a logical virtual application server 30 ′ and a virtual storage server 40 ′ are configured. Note that the specific function of the virtual server 20 ′ is the same as that of the first embodiment, and thus the description thereof is omitted.
That is, in the present embodiment, the interactive robot 10 is to perform information communication with the virtual server 20 ′ in the cloud via the network N (for example, the Internet).
As described above, the present invention is not limited to the case of directly exchanging information with the physical server 20, but can also be implemented using cloud computing.

  Each of the above-described embodiments can be variously modified without departing from the spirit of the present invention. Hereinafter, modified examples applicable to each embodiment will be described.

<Conversation level decision operation flow>
Hereinafter, the conversation level determination operation flow according to the modification will be described with reference to FIG.
In FIG. 7, steps S30 to S46 other than steps R1 to R7 that are level determination routines are the processes having the same contents as those in FIG. 5 of the first embodiment described above, and thus description thereof will be omitted as appropriate.

  That is, after the reflection of the learning history is appropriately executed in steps S34 and S35, the overall control unit 38 of the server 20 determines whether the level determination routine is activated in step R1. In this level determination routine, the interactive robot 10 asks the user U several questions, and the user U's conversation level is determined in multiple stages depending on how the user U answers the question. A response sentence from the dialogue robot 10 after the conversation level is determined by the level determination routine is appropriately selected from the response sentence DB 41 based on the determined conversation level.

In the present modification, processing by the level determination routine is executed in correspondence with the conversation level (12 levels) set by the conversation level setting unit 36 described above.
The activation frequency of the level determination routine is not particularly limited and may be activated at an arbitrary timing. For example, the user U operates by operating a switch (not shown) of the interactive robot 10 or every predetermined time such as 10 minutes. It may be activated so that the level is determined at the beginning, or may be activated only once in the initial stage when the interactive robot 10 is activated.

If the level determination routine is activated in step R1, the conversation level determination unit 36 of the server 20 controls the response sentence matching unit 32a to select and extract a level determination question from the response sentence DB 41. Next, after the response sentence voice conversion unit 32b converts the response sentence including the level determination question into voice information, the converted voice information (response sentence information) is communicated by the response sentence information transmission unit 32c via the communication interface 31. It is transmitted to the robot 10.
As a level determination question for determining the conversation level, a plurality of English examples based on the above-mentioned well-known English question collections and public English test questions can be applied. Such level determination questions Is stored in the response sentence DB 41.

Next, in step R3, the dialogue robot 10 utters a question sentence by the utterance unit 13 based on the received response sentence information including the level determination question.
In the subsequent step R4, the speech recognition unit 12 recognizes the speech (response) to the question sentence from the user U, and the speech related to the utterance (response that becomes an answer to the question sentence) recognized by the speech recognition unit 12 The user information transmission unit 14 a transmits information to the server 20.

In step R <b> 5, the server 20 that has received the voice information from the user U collates correct / incorrect regarding the content of the reply made by the user U. More specifically, as described above, it is determined whether the response of the user U received from the interactive robot 10 matches the response example stored in the response text DB 41.
Next, at step R6, it is determined whether or not the predetermined number of questions has been reached by the server 20, and if it has not reached yet, control is returned to step R2 to repeat the question. On the other hand, if the predetermined number of questions has been reached, an appropriate conversation level is set in step R7 based on the correctness / incorrectness of the questions made so far.

The predetermined number of questions is not particularly limited. For example, the number of questions corresponding to the number of hierarchical levels of the conversation level (for example, 12 if the level is 12) is asked to the user U and the conversation level is determined according to the correct answer rate. May be determined.
As described above, in the first embodiment, the facial expression at the time of reply of the user U is analyzed and the conversation level is changed, but in this modified example, the conversation is performed according to the actual answer content of the user U in addition to the facial expression. It is going to set the level.

The content described above is an example for carrying out the present invention, and the elements of the embodiments and modifications described above may be appropriately combined without departing from the spirit of the present invention.
For example, in the above-described embodiment and modification, information processing for selecting an appropriate response sentence or the like is performed on the server side via the network N without being performed on the interactive robot 10 side.

  In other words, the server side may manage only the database 40 and perform the above-described various information processing by the control unit 17 of the interactive robot 10. In this case, the dialogue robot 10 includes the response sentence matching unit 32a, the response sentence voice conversion unit 32b, the response sentence information transmission unit 32c, the utterance voice recording unit 33, the information analysis recording unit 34, the face information matching unit 35, And the function of the conversation level setting unit 36 is provided. Alternatively, the dialogue robot 10 includes the response sentence matching unit 32a, the response sentence voice converting unit 32b, the response sentence information transmitting unit 32c, the utterance voice recording unit 33, the information analysis recording unit 34, the face information matching unit 35, and the conversation level setting unit 36. It is good also as a form which shares at least 1 function of these, and processes the remainder by the server side.

  In the above embodiment, an example in which the user U and the interactive robot 10 have a conversation with each other has been described. However, as described above, the present invention is not limited to this form, and may have a form in which three or more parties have a conversation. In this case, for example, the following conversation modes can be considered.

(A) In an application example in the nursing care field, a mode in which the user U, the dialogue robot 10, and the caregiver interact with each other:
In this case, two people are added to the dialogue robot 10. The voice information converted by the voice recognition unit 12 of the interactive robot 10 is managed individually (differentiated between the user U and the caregiver) by the server 20. Then, the response sentence matching unit 32a selects an appropriate response sentence for each speaker from the response sentence DB 41 and transmits it to the conversation robot 10 via the network N.

(B) In an application example in the entertainment field or the like, a mode in which the user U, the dialogue robot 10 and the character in the information processing terminal 50 (which may be a portable information device such as a smartphone) interact with each other:
In this case, the information processing terminal 50 is connected to the server 20 via the network N, and a plurality of information devices participate in the dialogue for one user U. Then, the response sentence matching unit 32a receives the response sentence information from the information processing terminal 50, selects an appropriate response sentence from the response sentence DB 41 based on the response sentence information from the information processing terminal 50, and selects a conversation robot. 10 to the network N.

  For example, when the conversation content is English conversation learning, when the user U is interacting with the digital character displayed on the information processing terminal 50 (English conversation), the conversation robot 10 gives a hint of the response content to the user U, Alternatively, the dialogue robot 10 may respond to the digital character on behalf of the user U. In other words, in such a case, the dialogue robot 10 has a function as an assistant of the user U.

In the case of (a), three or more people may be added. In this case, the conversation content may be identified by the server 20 for each speaker. In the case of (b), two or more information terminals may be added in addition to the interactive robot 10.
Further, in the above cases (a) and (b), the server 20 may set different conversation contents (themes) for each speaker when the voice information can be identified for each speaker. . Moreover, you may make it present the information relevant to a theme positively from the server 20, the information processing terminal 50, or the said information apparatus.
In addition, the dialogue system, the dialogue robot, and the dialogue program of the present invention are not limited to the field of language learning including the above-mentioned English conversation performed at home or in a classroom, for example, some kind of dialogue with the user such as the nursing care field and the entertainment field. It can be widely applied to various necessary fields.

<Pre-processing before changing setting level>
In the first embodiment described above, when it is determined in step S41 that the facial expression of the user U is confused, in the subsequent step S42, the conversation level setting unit 36 is immediately controlled to change the setting level.
However, the present invention is not limited to this mode, and pre-processing before changing the setting level may be executed as follows.

  That is, in the following modified example, some patterns (for example, “confused”, “smile”, “no expression”, etc.) are set in advance for the recognition of the facial expression of the user U, and the user U determined by the server 20 The response content from the server 20 is changed according to the facial expression. As an example, for example, after determining the facial expression of the user U, various prior notices such as an answer prompt and a prior notice are performed before changing the conversation setting level.

More specifically, for example, as shown in steps S41a and S41b of FIG. 8, the user U is prompted for an answer and confirms the intention to change the setting level based on the recognition result of the user U's facial expression.
First, in step S41, the facial expression of the user U is determined in the same manner as in the first embodiment. If the user U's facial expression is confused and the voice information related to the response from the user U is not received, the server 20 transmits a message prompting the user U to answer in step S41a. Examples of such messages include “What did you do?” And “Wait a little more?”.

  When receiving a response from the user U after sending a message in step S41a, for example, an expected answer pattern from the user U (such as “Yes”, “No”, or “OK”) is stored in advance. What is necessary is just to collate with the response from the actual user U. Of course, even if a significant response cannot be obtained from the user U, the processing may proceed to the subsequent step S41b when a predetermined time has elapsed.

Then, in the subsequent step S41b, a message for prompting the user U to change the setting level from the current conversation level is transmitted. As an example of such a message, for example, a prior notice such as “Let's talk more clearly” or “Let's change the topic” can be exemplified.
After such prompting and advance notice up to step S41b, processing for changing the setting level of the conversation is performed in step S42 as in the first embodiment. Note that step S41a and step S41b need not be executed together, and at least one of them may be executed.

  On the other hand, if the facial expression of the user U is not confused, the server 20 determines whether the facial expression of the user U is a smile in step S41c. Then, if the user U encounters a smile with the facial expression of the user U, the server 20 transmits a message asking the reason for the smile to the user U in step S41d. Examples of such messages include “Is it a good thing?” Or “Happy?”. Note that this step may be executed even when voice information related to a response sentence from the user U is not received.

  In addition, when the past learning history is reflected in step S35, the past response content is compared with the response content from the past user U (the utterance and the facial expression to the response sentence performed in the past). If the contents of the change this time (the utterance or its expression with respect to the response sentence in step S38) are different, a background inquiry such as “what happened?” May be made.

<Utilization of big data>
Further, in the present invention, the server 20 holds the response sentence example acquired by the exchange with another user as big data, and the message transmitted from the server 20 to the user U is changed using this big data. May be.
More specifically, in FIG. 8, the next response sentence to the user U is selected with reference to big data according to the content of the response from the user U (the utterance to the response sentence in step S38). Also has features.

That is, in step P1, it is determined whether the content of the response from the user U (the utterance for the response sentence in step S38) is a correct answer.
If the response from the user U is not a correct answer, such as when the response content is different from the assumed response content stored in advance, or when there is a grammatical error, the big response is given at step P2. Browse the data.

Here, the server 20 shares and learns past interactions with other users and the user U. For example, an example of a response sentence having a high appearance frequency for each response sentence can be made into a database.
Therefore, for example, when there is a frequent grammatical error in a response from the user U, the server 20 selects a response sentence having contents related to the grammar and transmits it to the user U in the subsequent step S43. It may be. Further, for example, if the response content from the user U is not included in the assumed response content, the server 20 selects a response sentence whose setting level has been lowered in the subsequent step S43 and transmits it to the user U. Also good.

Note that “steps S41a to S41d” and “steps P1 and P2” described with reference to FIG.
Also in FIG. 8, the level determination routine (steps R1 to R7) described using FIG. 7 may be executed.

   As described above, the interactive robot, the interactive system, and the interactive program of the present invention are suitable for constructing a natural conversation with a user.

U user 10 interactive robot 11 imaging unit 12 voice recognition unit 13 utterance unit 14 communication interface 15 ROM / RAM unit 16 power supply 17 control unit 20 server 20 ′ virtual server
30 'virtual application server 31 communication interface 32a response sentence matching unit 33 utterance voice analysis recording unit 34 information analysis recording unit 35 face information matching unit 36 conversation level setting unit 37 ROM / RAM unit 38 overall control unit 40 database 41 response sentence DB
42 ID information DB
43 Learning history DB
100 Dialogue system

Claims (6)

A dialogue robot connected to a server on a network via a communication line and carrying out a dialogue with a user,
An imaging unit for imaging the user's face;
A speech recognition unit for recognizing speech uttered by the user;
An utterance unit that utters to the user;
A user information transmission unit that transmits the facial information on the face and the voice information recognized by the voice recognition unit to the server via the communication line;
A response sentence information receiving unit for receiving response sentence information from the server,
The user information transmission unit transmits the facial information captured by the imaging unit to the server,
i) When the server specifies that the user is unregistered based on the face information, the response sentence information including a name question for asking a name of the user generated by the server is received as the response sentence information Department receives,
ii) When the server specifies that the user is registered based on the face information, the response sentence information receiving unit receives the response sentence information including the name of the user generated by the server,
The utterance unit makes a start utterance to the user including the name question or the user's name for the user according to whether or not the user has been registered,
The interactive robot characterized in that the user information transmission unit transmits voice information uttered by the user with respect to the start utterance recognized by the voice recognition unit to the server. A dialogue robot according to claim 1;
An interactive system including the server,
The dialogue system according to claim 1, wherein the server includes a database managed individually for each user, and information relating to a dialogue with the user is recorded in the database. The server converts the speech information related to the user's utterance received from the user information transmission unit into text, performs matching between the textualized speech information and response sentence information in the database, and an appropriate response sentence based on the matching Sending the response sentence information including the response sentence information receiving unit,
The dialogue system according to claim 2, wherein the dialogue robot utters to the user via the utterance unit based on response sentence information including the received response sentence. The imaging unit images the user in response to the response sentence,
The user information transmission unit transmits the user's image information in the response captured by the imaging unit to the server together with the user's voice information spoken to the response sentence,
The server determines the facial expression of the user based on the image information of the user received from the user information transmission unit, and includes the next response sentence that has been changed based on the determination result of the facial expression. The dialogue system according to claim 3, wherein information is transmitted to the response text information receiving unit. The server transmits response sentence information including a level determination question for determining a conversation level to the response sentence information receiving unit,
The interactive robot utters a response sentence including the received level determination question to the user via the utterance unit,
The conversation level is determined based on a response to the level determination question issued by the user,
5. The dialogue system according to claim 3 or 4, wherein a response sentence in the database is selected by the server based on the determined conversation level and transmitted to the response sentence information receiving unit. In a dialogue robot or server,
Use the imaging unit to capture the user's face,
Send the imaged information about the user's face to the server via a communication line,
Determine whether the user is unregistered based on information about the face, i) generate response text information including a name question that asks the user's name if unregistered, and ii) registered If so, response sentence information including the user's name is generated,
Let the user utter as a start utterance based on the response sentence information using the utterance unit,
Using the voice recognition unit to recognize the user's utterance with respect to the start utterance,
An interactive program that causes the server to transmit the speech of the user that has been voice-recognized to the server via the communication line.