JP4735965B2 - Remote communication system - Google Patents

Description

  The present invention relates to a remote place dialogue system, and more particularly, to a remote place dialogue system that communicates voices of talkers away from a remote place via a network.

  When a dialogue is performed between remote locations, the delay time of the dialogue becomes long due to the delay, and it is difficult to take the timing of the utterance. Conventionally, there has been no technique for preventing such duplication of utterances and generating a smooth dialogue between remote locations.

  For example, Patent Document 1 discloses an example of a technique for detecting a superimposition of a voice output from a voice input / output device and a voice input by a user's utterance.

Patent Document 2 and Patent Document 3 disclose an example of a technique for performing a gesture by installing a robot in front of a conversation person as well as simply outputting a conversation voice. In the technology of Patent Document 2, the speaker's robot performs gestures based on the speaker's voice, while the listener's robot performs gestures based on the voice received by the listener. The feeling is enhanced. In the technique of Patent Document 3, the gesture is reproduced by the partner robot in response to transmission of gesture information on the speaker side.
JP 7-264103 A JP 2000-349920 A JP 2001-156930 A

  In the technique of Patent Document 1, the superimposition of the output voice and the input voice is detected and the operation of the echo canceller is changed, but the speaker's speech timing cannot be controlled. Further, in the techniques of Patent Documents 2 and 3, a smooth conversation is realized by gesturing a robot installed in front of a conversation person, but the speaker's utterance timing cannot be controlled. As described above, in the prior art, since the timing of the utterance could not be controlled, it was not possible to cope with an abnormal blank time in the dialog due to the delay. Therefore, it was not possible to prevent the utterances of both interlocutors from overlapping, and a smooth conversation could not be realized.

  SUMMARY OF THE INVENTION Therefore, a main object of the present invention is to provide a remote place dialogue system that can provide an appropriate space for a remote place dialogue and can realize a smooth dialogue.

The invention of claim 1 is a system for carrying out a dialogue between remote locations including two dialogue devices connected via a network. Each dialog device includes an acquisition unit that acquires voice, a transmission unit that transmits the voice acquired by the acquisition unit to the other party's dialog device, a reception unit that receives the voice transmitted from the other party's dialog device, and a reception unit. Output means for outputting the received voice is included. The system includes a pattern storage means for storing a plurality of inter-patterns including information relating to at least blank time and features of speech voice, a history for recording a dialog state history including at least a voice acquisition state and a voice output state in each dialog device When it is determined that there is no utterance state in both interactive devices based on the history recorded by the recording means and the history recording means, a situation between at least a blank time and a feature of the speech voice before the blank and a plurality of matching means for matching and between patterns, when there is between pattern results match verification by collation means, dialogue predetermined sound corresponding to the inter-pattern, present to the other party of a speaker of the predetermined sound while the control means for outputting from the output means of the apparatus, and heavy utterance in both interactive device based on the history recorded by the history recording means When it is determined the a, by recording one voice, a delay reproduction means then speech outputting the recorded speech at the completion of the output means of the other interactive device.

In the invention according to claim 1, the dialog system between remote locations includes two dialog devices, and the dialogue between the remote sites can be performed between the remote locations by communicating the voices acquired by each dialog device and outputting them on the other side. Done. A plurality of inter-patterns are stored in the inter-pattern storage means. The interval pattern indicates an appropriate interval in the dialogue, and includes at least information regarding the blank time and the characteristics of the speech. For example, the features of spoken speech may include fundamental frequency (pitch), amplitude, syllable average duration, and the like. The history recording means records a dialog state history including at least a voice acquisition state and a voice output state in each dialog device. The dialogue state history is an utterance flag table in the embodiment described later, and records the utterance presence / absence state (SPEAKING flag, SILENT flag), processing state (RECORDING flag, INTERPOLATING flag), etc. at each time. When it is determined that both of the interactive devices are in the no-speech state, the collating unit collates the situation between at least the blank time and the feature of the speech voice before the blank and a plurality of patterns. That is, when the dialogue is silent, it is confirmed whether the current situation matches any of a plurality of patterns. When there is a matching pattern as a result of the collation, the inter-control unit outputs a predetermined voice corresponding to the inter-pattern from the output unit of the interactive apparatus existing on the other party of the speaker of the predetermined voice. However, when the utterances are duplicated, one voice is recorded by the delay reproduction means, and when the utterance is thereafter ended, the recorded voice is output from the other interactive device.
Na us, the inter-pattern storage means, history recording unit, matching hand stage, while the control unit and the delay reproduction means, to one of the two interactive device, or, in another apparatus (the examples included in this system (Speech timing control server). Or these means may be distributed and provided in two interactive apparatuses.

According to the first aspect of the present invention, when a silent state in a conversation is detected, it is possible to output a sound corresponding to a pattern that gives an appropriate interval. Therefore, it is possible to prevent utterances from overlapping, and a smooth conversation can be established. Moreover, even if utterances overlap, it can be avoided that both utterances are output at the other party at the same time.

The invention of claim 2 is dependent on the invention of claim 1, and when at least one of the interactive devices is a robot capable of performing gestures, the inter-control means further outputs a predetermined output corresponding to the inter-pattern along with the output of the voice. Let the dialogue device perform gestures.

In the invention of claim 2, at least one of the interactive devices may be a robot capable of performing gestures. The inter-control unit causes the robot to output a sound corresponding to the inter-pattern and execute gesture corresponding to the inter-pattern. Therefore, since a suitable space can be given to the dialogue using voice and gestures, a smoother dialogue between remote locations can be established.

  According to the present invention, since a word is inserted so as to take an appropriate interval when a silent state is detected in the dialog, the blank time of the dialog can be set to an appropriate length. For this reason, it can be avoided that the blank time becomes longer due to the delay and the conversation person feels uncomfortable. Therefore, since it becomes easy for the interlocutor to measure the timing of the utterance, duplication of both utterances can be prevented and a smooth conversation can be established.

  The above object, other objects, features and advantages of the present invention will become more apparent from the following detailed description of embodiments with reference to the drawings.

  Referring to FIG. 1, a remote place dialogue system (hereinafter also simply referred to as “system”) 10 of this embodiment is for a dialogue between remote talkers who are remote. The system 10 includes at least two interactive devices 12 (12a, 12b). The two interactive devices 12 are connected via a network, for example, a public Internet network, and communicate voice data of speech uttered by the interactive person A on the interactive device 12a side and the interactive person B on the interactive device 12b side. The system 10 of this embodiment includes an utterance timing control server (hereinafter also simply referred to as “server”) 14, and the server 14 is communicably connected to at least two interactive devices 12 via a network.

  In this embodiment, a case where a computer 12a is applied as one interactive device 12 and a communication robot (hereinafter simply referred to as “robot”) 12b is applied as the other interactive device 12 will be described.

  The interactive device 12 a includes a microphone 16 and a speaker 18. The interactive device 12a is a personal computer, for example, and includes a CPU, a main memory, a communication device, an input device, and the like. The main memory stores programs and data necessary for functioning as the interactive apparatus 12 of the present invention. The program and data may be fixedly stored in the main memory in advance, or may be acquired from an information storage medium or a network. In accordance with the program, the CPU executes processing for dialogue while generating or acquiring temporary data in the working memory of the main memory.

  The microphone 16 is for acquiring the voice uttered by the conversation person, and the voice is converted into data by the voice input / output board and stored in the main memory as voice data. The speaker 18 is for outputting the voice of the conversation partner and the voice of the system 10. The CPU gives the received audio data to the audio input / output board and outputs the audio from the speaker 18. The communication device transmits / receives data to / from the other interactive device 12 or server 14 via the network. The input device is a keyboard or a pointing device.

  In this embodiment, since the robot 12b is used as the interactive apparatus 12 on the other side, the user can instruct the gesture of the robot 12b using the input device. The gesture of the robot 12b may be selected from a list displayed on the screen of the display device, or may be assigned to each key of the input device. By instructing the operation using the input device while speaking, the conversation person A can output his / her voice from the partner robot 12 and can cause the robot 12 to perform desired gestures.

  The interactive device 12a may be any computer that can input and output voice and can communicate, and is not limited to a PC, and may be another computer such as a game machine, a mobile phone, or a portable game machine.

  The other interaction device 12b is a robot having a body part like a human, and can present a predetermined gesture to the conversation person B by moving the body part. The robot 12 b includes a microphone 20 and a speaker 22. Specifically, FIG. 2 shows an example of the appearance of the robot 12b, and FIG. 3 shows an example of the electrical configuration of the robot 12.

  Referring to FIG. 2, the robot 12 b includes a carriage 24, and wheels 26 that autonomously move the robot 12 b are provided on the lower surface of the carriage 24. The wheel 26 is driven by a wheel motor (indicated by reference numeral “28” in FIG. 3), and the carriage 24, that is, the robot 12b can be moved in any direction. Although not shown in FIG. 2, a collision sensor (indicated by reference numeral “30” in FIG. 3) is attached to the front surface of the carriage 24. Detects contact with obstacles. If contact is detected during the movement of the robot 12b, the driving of the wheels 26 can be stopped immediately.

  A polygonal column sensor mounting panel 32 is provided on the carriage 24, and an ultrasonic distance sensor 34 is mounted on each surface of the sensor mounting panel 32. The ultrasonic distance sensor 34 is for measuring the distance between the mounting panel 32, that is, the robot 12b and mainly the person.

  Further, the body of the robot 12b is mounted on the carriage 24 so that the lower part of the body is surrounded by the mounting panel 32 described above. The body includes a lower body 36 and an upper body 38, and the lower body 36 and the upper body 38 are connected by a connecting portion 40. Although not shown, the connecting portion 40 has a built-in lifting mechanism. By using this lifting mechanism, the height of the upper body 38, that is, the height of the robot 12b can be changed. The lifting mechanism is driven by a waist motor (indicated by reference numeral “42” in FIG. 3).

  One omnidirectional camera 44 and one microphone 20 are provided in the approximate center of the upper body 38. The omnidirectional camera 44 photographs the surroundings of the robot 12b and is distinguished from an eye camera 46 described later. As described above, the microphone 20 captures ambient sounds, particularly human voices.

  Upper arms 50R and 50L are attached to both shoulders of upper body 38 by shoulder joints 48R and 48L, respectively. The shoulder joints 48R and 48L each have three axes of freedom. That is, the right shoulder joint 48R can control the angle of the upper arm 50R around each of the X, Y, and Z axes. The Y axis is an axis parallel to the longitudinal direction (or axis) of the upper arm 50R, and the X axis and the Z axis are axes orthogonal to the Y axis from different directions. The left shoulder joint 48L can control the angle of the upper arm 50L around each of the A, B, and C axes. The B axis is an axis parallel to the longitudinal direction (or axis) of the upper arm 50L, and the A axis and the C axis are axes orthogonal to the B axis from different directions.

  Forearms 54R and 54L are attached to the tips of upper arms 50R and 50L via elbow joints 52R and 52L, respectively. The elbow joints 52R and 52L can control the angles of the forearms 54R and 54L around the W axis and the D axis, respectively.

  In the X, Y, Z, W axes and A, B, C, and D axes that control the displacement of the upper arms 50R and 50L and the forearms 54R and 54L, "0 degrees" is the home position. The upper arms 50R and 50L and the forearms 54R and 54L are directed downward.

  Although not shown in FIG. 2, a touch sensor (FIG. 3) is provided on the shoulder portion including the shoulder joints 48R and 48L of the upper body 38 and the arm portion including the upper arms 50R and 50L and the forearms 54R and 54L. The touch sensor 56 detects whether or not a person has touched these parts of the robot 12b.

  Spheres 58R and 58L corresponding to hands are fixedly attached to the tips of the forearms 54R and 54L, respectively. Instead of the spheres 58R and 58L, a “hand” in the shape of a human hand can be used when a finger function is required unlike the robot 12b of this embodiment.

  A head portion 62 is attached to the upper center of the upper body 38 via a neck joint 60. The neck joint 60 has a degree of freedom of three axes, and the angle can be controlled around each of the S, T, and U axes. The S-axis is an axis that goes directly from the neck, and the T-axis and the U-axis are axes that are orthogonal to the S-axis in different directions. The head 62 is provided with the speaker 22 described above at a position corresponding to a human mouth. The speaker 22 may be used for the robot 12b to communicate with a person around it by voice or voice. Further, the speaker 22 may be provided in another part of the robot 12, for example, the trunk.

  The head 62 is provided with eyeball portions 64R and 64L at positions corresponding to the eyes. Eyeball portions 64R and 64L include eye cameras 46R and 46L, respectively. Note that the left and right eyeball portions 64R and 64L may be collectively indicated by reference numeral “64”, and the left and right eye cameras 46R and 46L may be collectively indicated by reference numeral “46”. The eye camera 46 captures the video signal by photographing the face of the person approaching the robot 12b and other parts or objects.

  Note that each of the omnidirectional camera 44 and the eye camera 46 described above may be a camera using a solid-state imaging device such as a CCD or a CMOS.

  For example, the eye camera 46 is fixed in the eyeball unit 64, and the eyeball unit 64 is attached to a predetermined position in the head 62 via an eyeball support unit (not shown). The eyeball support unit has two degrees of freedom and can be controlled in angle around each of the α axis and the β axis. The α-axis and the β-axis are axes set with respect to the head 62, the α-axis is an axis in a direction toward the top of the head 62, the β-axis is orthogonal to the α-axis, and the front side of the head 62 It is an axis in a direction orthogonal to the direction in which (face) faces. In this embodiment, when the head 62 is at the home position, the α axis is set to be parallel to the S axis and the β axis is set to be parallel to the U axis. In such a head 62, by rotating the eyeball support portion around each of the α axis and the β axis, the tip (front) side of the eyeball portion 64 or the eye camera 46 is displaced, and the camera axis, that is, the line-of-sight direction is changed. Moved.

  In the α axis and β axis that control the displacement of the eye camera 46, “0 degree” is the home position. At this home position, the camera axis of the eye camera 46 is the head 62 as shown in FIG. The direction of the front side (face) is directed, and the line of sight is in the normal viewing state.

  Referring to FIG. 3, the robot 12b includes a microcomputer or a CPU 66 for overall control. The CPU 66 is connected to a memory 70, a motor control board 72, a sensor input / output board 74, and a voice through a bus 68. An input / output board 76 is connected.

  Although not shown, the memory 70 includes a ROM, an HDD, a RAM, and the like, and a program and data for causing the robot 12b to function as the interactive device 12 of the present invention are stored in advance in the ROM or the HDD. The CPU 66 executes processing according to this program. The RAM is used as a buffer memory or a working memory.

  The motor control board 72 is composed of, for example, a DSP (Digital Signal Processor) and controls a motor for driving body parts such as the right arm, the left arm, the head, and the eyes. That is, the motor control board 72 receives the control data from the CPU 66 and controls the angles of the three motors for controlling the X, Y and Z axes of the right shoulder joint 48R and the axis W of the right elbow 52R. The rotation angles of a total of four motors including one motor (collectively shown as “right arm motor” in FIG. 3) 78 are adjusted. The motor control board 72 includes a total of four motors including three motors for controlling the angles of the A, B, and C axes of the left shoulder joint 48L and one motor for controlling the angle of the D axis of the left elbow joint 52L. The rotation angle of the motor 80 (collectively shown as “left arm motor” in FIG. 3) 80 is adjusted. The motor control board 72 also adjusts the rotation angle of three motors 82 (collectively shown as “head motors” in FIG. 3) that control the angles of the S, T, and U axes of the neck joint 60. . The motor control board 72 also controls two motors 28 (shown collectively as “wheel motors” in FIG. 3) that drive the waist motor 42 and the wheels 26. Further, the motor control board 72 adjusts the rotation angle of two motors 84 (collectively shown as “right eyeball motor” in FIG. 3) that control the angles of the α axis and β axis of the right eyeball portion 64R. Further, the rotation angle of two motors 86 (collectively shown as “left eyeball motor” in FIG. 3) for controlling the angles of the α axis and the β axis of the left eyeball portion 64L is adjusted.

  The above-described motors of this embodiment are stepping motors or pulse motors for simplifying the control, except for the wheel motors 28, but may be direct-current motors as with the wheel motors 28. In this embodiment, a motor using electric power as a drive source is used as an actuator for driving a body part such as an arm, a head, or an eye of the robot 12b. However, as this robot 12b, a robot that drives a body part by other actuators such as air pressure (or negative pressure), oil pressure, piezoelectric element, shape memory alloy, or the like may be applied.

  Similarly, the sensor input / output board 74 is also constituted by a DSP, and takes in signals from each sensor and camera and gives them to the CPU 66. That is, data relating to the reflection time from each of the ultrasonic distance sensors 34 is input to the CPU 66 through the sensor input / output board 74. The video signal from the omnidirectional camera 44 is input to the CPU 66 after being subjected to predetermined processing by the sensor input / output board 74 as necessary. Similarly, the video signal from the eye camera 46 is also supplied to the CPU 66. Further, a signal from the touch sensor 56 is given to the CPU 66 via the sensor input / output board 74.

  Audio data is given to the speaker 22 from the CPU 66 through the audio input / output board 76, and accordingly, audio or voice according to the data is output from the speaker 22. Also, the voice input from the microphone 20 is taken into the CPU 66 as voice data via the voice input / output board 76.

  Similarly, the communication LAN board 88 is configured by a DSP, and sends the transmission data given from the CPU 66 to the wireless communication device 90 so that the data is transmitted from the wireless communication device 90. The communication LAN board 88 receives data via the wireless communication device 90 and provides the received data to the CPU 66.

  Returning to FIG. 1, the server 14 is provided to control the timing of the utterances of the two interactors. The server 14 includes a CPU, a main memory, a communication device, and the like. A program and data for controlling the server 14 are stored in the main memory. The CPU executes processing according to the program.

  The server 14 also includes a voice analysis history database (DB) 92 and an inter-pattern DB 94. The speech analysis history DB 92 stores a history of speech analysis data obtained by the dialog device 12.

  As will be described later, the inter-pattern DB 94 stores inter-pattern data (see FIG. 4) for giving an appropriate interval to the dialogue. The inter-pattern data is extracted by pattern recognition from the utterance data obtained by measuring the utterance in advance. When the measurement is performed on an actual user, the personal characteristics of how to make an interval can be extracted. However, since it is considered that there is a standard or general way of taking an interval, the utterance may be measured with an arbitrary person as a subject and the inter-pattern data may be extracted.

  In this system 10, each interactive device 12 detects a sound with the microphone 16 or 20 every predetermined time ΔT. ΔT may be, for example, one frame or a predetermined number of frames. One frame is, for example, 1/30 second. The dialogue apparatus 12 performs processing according to the detected result, that is, the presence or absence of an utterance. The dialogue device 12 transmits information related to the dialogue state in the device 12 such as the utterance state (voice acquisition state) and the executed processing (voice output state) at the detection time to the server 14. The server 14 receives information related to the dialog state and stores the state in the dialog device 12 one by one. Such a dialog state history is stored in the memory as an utterance flag table (see FIG. 5). As will be described later, the utterance flag table stores a flag indicating a dialogue state including at least a voice acquisition state and a voice output state in the dialogue device 12 for each detection time.

  For convenience, the operation seen from the interactive device 12a side will be described here. However, since the difference between the interactive device 12a and the interactive device 12b is mainly only the function related to the presentation of the body movement, the operation of the interactive device 12b is the same as that of the interactive device 12a.

  When the dialogue apparatus 12a detects voice from the microphone 16, the dialogue apparatus 12a refers to the utterance flag table of the server 14. Then, the interactive device 12a executes processing according to the state of the other party at the previous detection time (that is, before ΔT). Specifically, when the partner state flag at the previous detection time is not the SPEAKING flag, that is, when the partner is not transmitting voice at the previous detection time, the interactive device 12a While recording the audio data detected in 16 as a local file in the memory, the audio data is transmitted to the other-side interactive device 12b. In response to this, the dialogue apparatus 12b receives the audio data and outputs the audio from the speaker 22. As described above, when one conversation person A speaks and the voice of the other conversation person B is not transmitted at the previous detection time, the voice data of the conversation person A is immediately transmitted and the voice is transmitted to the other party. Is output by the interactive device 12b on the side and is sent to the partner B.

  In this case, the dialogue apparatus 12a transmits the SPEAKING flag to the server 14, and in response thereto, the server 14 stores the SPEAKING flag in the utterance flag table as the state of the dialogue apparatus 12a at the detection time t. The SPEAKING flag means a state in which one of the interlocutors is speaking, that is, a state in which the spoken voice is directly transmitted to the other party and reproduced. Furthermore, the dialogue apparatus 12a turns on the SEND flag in the memory and stores that the voice is being transmitted to the other party as its processing state.

  Note that, for the local file in which the voice is recorded, voice analysis is performed when the utterance is finished, and the feature or state of the uttered voice is stored in the voice analysis history DB 92 of the server 14.

  On the other hand, when the dialogue apparatus 12a does not detect the voice from the microphone 16, it transmits a SILENT flag to the server 14, and the server 14 stores the SILENT flag in the utterance flag table accordingly. The SILENT flag means a state in which the conversation person is not speaking at the detection time t. As described above, when no voice is detected, the SILENT flag is recorded in the utterance flag table.

  In the server 14, when a state in which neither conversation person is speaking (SILENT flag) is detected, the inter-pattern is compared with the current speech state (inter-state) in the conversation. The situation in between includes at least the blank time (silence time) of the dialogue, and the features (voice analysis results) of the speech voice in the dialogue before the blank.

  An example of the inter-pattern data stored in the inter-pattern DB 94 is shown in FIG. The inter-pattern DB 94 stores a plurality of inter-pattern data that can give an appropriate interval to the conversation. The inter-pattern includes at least information related to the blank time and the characteristics of the previous speech. In this embodiment, the inter-pattern data includes information such as a conversation blank time, the last speaker, condition (I), inter-function language, speaker, and operation command. The blank time (t) is a condition of time during which both silent states are continued. The last speaker is the condition of the last speaker A or B in the dialogue before the blank. The condition (I) is a condition of the speech analysis result of the dialogue before the blank, and includes elements such as a fundamental frequency (pitch), an amplitude, and an average duration of a syllable.

  When such a blank time and a situation corresponding to an inter-pattern defined by the feature of the previous speech is detected, words and gestures based on the inter-pattern are inserted into the dialogue. Specifically, in this embodiment, when a blank time after dialogue that meets the last speaker, the condition (I) and the blank time condition (t) occurs, the inter-function from the speaker side specified by the pattern is performed. A word is emitted. Further, when the interactive device 12 that utters a function word during the time instead of the speaker is the robot 12b, the gesture corresponding to the output word is also reproduced according to the operation command.

  The inter-function words are words that are inserted during the silence time to function as silence between them and give an appropriate interval to the conversation. For example, it may be a word such as a response, a joint, or an interjection. In FIG. 4, “Yun”, “Um”, “Yes yes”, “Ut”, etc. are shown as inter-function words. Each inter-function function word is associated with an action command for executing a body action that is presented together with the word. In FIG. 4, “Noun” is a “Unazuku” command, “Yun” is a “Tilt” command, “Yes” is a “Unazuku” command, and “Uto” is “Gaze up”. ”Commands are associated with each other.

  In FIG. 4, the speaker means a person who is a human being, and the interactive device 12 that outputs a function word during this period is the interactive device 12 that exists at the location of the other party of the speaker. For example, in the case of the uppermost pattern in FIG. 4, the last speaker is the conversation person A, and the conversation apparatus that outputs the interactivity function is the conversation apparatus 12 a on the conversation person A side who is the partner of the speaker B. Become. The uppermost pattern is associated with the word “yeah” and “nodding” behavior, and the robot 12b existing on the side of the final speaker responds to the action of the speaker who is the partner of the final speaker. Expressing it gives you a gap. On the other hand, the second pattern from the top is associated with the word “hmm” and the “tilt” action, and the robot 12b on the other side of the last speaker continues further speaking and gesturing. Is given.

  As a result of collation using the inter-pattern, if inter-pattern data exists in the inter-pattern DB 94 that matches the current inter-state situation, that is, it is determined that an appropriate interval needs to be taken according to the inter-pattern. The server 14 instructs the necessary dialogue device 12 to reproduce the words for the time being. In this embodiment, the voice data of the word and the reproduction instruction are transmitted. In response to this, the voice is output on the interactive device 12 side. Further, when the interactive device 12 is an interactive device 12b capable of expressing the body, the server 14 instructs the execution of the gesture corresponding to the words for taking the interval. In this embodiment, an operation command and a reproduction instruction are transmitted to the gesture. In response to this, in the dialogue apparatus 12b, the corresponding body part is moved and the gesture is executed.

  In this way, when a silent state (silent state) is detected in the dialogue, the current situation and the inter-pattern are collated. And if necessary, words and gestures are inserted into the blank time so as to take an appropriate time, so that the blank time in the dialogue can be maintained at an appropriate time. For this reason, it is possible to avoid a situation in which the blank time in the dialogue becomes long due to the delay and gives a stranger to the dialogue person. Therefore, it becomes easy for the interlocutor to measure the timing of the utterance, and the situation where the utterances of both interlocutors overlap can be prevented. In this way, the speech timing of the interlocutor can be controlled by inserting words and gestures, so that the conversation can be continued and the speech can be promoted, and a natural conversation flow can be created. Therefore, a smooth dialogue can be established.

  In addition, in this system 10, in the unlikely event that the utterances of both dialoguers overlap, the system 10 is provided with a function of preventing the utterances from completely overlapping by delaying the output of one utterance. .

  Specifically, when voice is detected by the dialog device 12a, when the partner status flag at the previous detection time is the SPEAKING flag, that is, when both utterances overlap, the dialog device 12a Recording of the voice detected by the microphone 16 is started, and the voice data is stored in the memory as a voice file. Further, the dialogue apparatus 12a transmits a RECORDING flag to the server 14, and in response thereto, the server 14 stores the RECORDING flag as a state at the detection time t of the dialogue apparatus 12a in the utterance flag table. The REDORDING flag means that voice data is being recorded and the voice is not transmitted to the other party.

  Further, the dialogue apparatus 12a turns on the RECORD flag in the memory, and stores that the voice is being recorded as its processing state.

  In the utterance flag table of the server 14, a PLAY flag is stored as information for controlling the reproduction of the recorded audio file. In this embodiment, the interactive device 12a instructs the server 14 to add 1 to the value of the PLAY flag when recording. The initial value of the PLAY flag is 0. When recording is performed, 1 is added to the value of the previous detection time at every detection time, and when the recording is not performed, the value of the previous detection time is maintained. Is done.

  Thereafter, when no voice is detected by the interactive apparatus 12a, the recorded voice file is transmitted to the server 14. In response to this, the server 14 stores the received audio file in the memory in association with the detection time t at which the recording was performed. In the utterance flag table, the storage position where the audio file is stored is stored. The voice file is subjected to voice analysis processing before being sent to the server 14, and the analysis data is sent to the server 14 and stored in the voice analysis history DB 92.

  If the server 14 detects that neither of the dialog devices 12 is outputting sound, that is, if it is detected that the SPEAKING flag is not stored in any of the states of both dialog parties, It is determined whether or not the dialogue file is stored without being reproduced. If an unreproduced recording file remains, that is, if the value of the PLAY flag is 1 or more, the audio file with the earliest recording start time is reproduced. Specifically, the server 14 transmits the audio data and the reproduction instruction to the partner interactive device 12 until the reproduction of the file is completed. In response to this, the dialogue apparatus 12 outputs the voice based on the received voice data.

  In this way, if the utterances of both parties interact, record the voice of the side that started speaking later, and then output the recorded voice on the other side when both utterances are finished. Can do. In addition, when both utterances start at the same time, the sound can be delayed and reproduced according to the priority order. Therefore, since the output of the utterance when it overlaps can be delayed, a smooth remote conversation can be established.

  FIG. 5 shows an example of the utterance flag table stored in the server 14. In the utterance flag table, information such as a user, a status flag, a target, a storage position of a saved voice file, a storage position of a saved command file, and a PLAY flag are stored for each detection time t. The user information is the subject of the data. For example, A means that the data is in the state of the interactive device 12a, and B means that the data is in the state of the interactive device 12b. The target indicates a user's utterance target.

  The status flag indicates a voice acquisition state and a voice output state in the interactive device 12, and the SPEAKING flag, the SILENT flag, and the RECORDING flag are stored as described above. At time t = T + 2ΔT in FIG. 5, the status flag is an INTERPOLATING flag. As described above, in the case where the state flag of both the interlocutors is the SILENT flag, when the inter-function language is inserted according to the inter-pattern, the INTERPOLATING flag is overwritten as the state flag at the time. Yes. As a result, the detection time is not measured as a blank time in the dialogue.

  The stored audio file indicates the storage location of the recorded audio file. For example, in FIG. 5, at time T + 4ΔT and time T + 5ΔT, the RECORDING flag is stored as the status flag of user A, and the storage location of the audio file corresponding to the recording at that time is shown. Also, the PLAY flag at time T + 4T is 1, which means that recording has started, and the PLAY flag at time T + 5T, which is 2, means that recording has been continued. The PLAY flag at time T + 6T remains at 2, which means that recording has ended.

  The saved command file indicates the storage location of a file in which an operation command input by the user with the interactive device 12a is recorded during recording. This command file is transmitted together with the voice file to the other-side dialog device 12b. Accordingly, the dialog device 12b performs the gesture of input instruction together with the recorded voice.

  FIGS. 6 to 9 show an example of the operation in the input process of the CPU of the dialogue apparatus 12. When the input process is started, initialization is performed in the first step S1 of FIG. For example, the SEND flag is turned off, the RECORD flag is turned off, and the current time T (or initial value T) is substituted for time (or frame number) t. Subsequent processing from step S3 to step S69 in FIG. 9 is repeatedly executed at regular time intervals ΔT, for example, every frame.

  In step S3, the input of the microphone 16 or 20 is checked. In step S5, it is determined whether there is a voice input based on the input data. If “YES” in the step S5, that is, if the conversation person is speaking, the utterance flag table of the server 14 is referred to in a step S7. For example, the dialogue apparatus 12 transmits a request for the utterance flag to the server 14. In response to this, the server 14 transmits the data of the utterance flag table to the dialogue apparatus 12. The dialogue device 12 receives the utterance flag table data and stores it in the memory. Since the state where there is no voice input continues after the start, the SILENT flag is stored in the utterance flag table as the states of the two talkers before the first utterance.

  Subsequently, an inter-process measurement process is executed in step S9. An example of the operation of the measurement process during this time is shown in detail in FIG. When the interval measurement process is started, in the first step S81 of FIG. 10, it is determined whether or not the state flag at the time before ΔT of the current time t is the SILENT flag based on the utterance flag table. In step S81, it is determined whether there is a voice input at the current detection time and no voice input at the previous detection time. That is, it is determined whether or not it is the timing when the user on the interactive device 12 side starts speaking.

  If “YES” in the step S81, the two types until the start of the current conversation are measured based on the speech flag table. Specifically, in step S83, the blank time from the end of speaking a word before the start of speaking is measured. In step S85, a blank time from when the partner finishes speaking a word until when the partner starts speaking is measured. In step S87, the measurement data is transmitted to the server 14. In response to this, the server 14 stores inter-measurement data. If step S87 ends or if “NO” in the step S81, the process returns to the step S11 in FIG.

  In this way, by storing the measurement data history in the server 14, it is possible to record in the server 14 how long the interlocutor is interacting. A pattern between the history data and the voice analysis history data can be extracted.

  Subsequently, in step S11 of FIG. 6, it is determined based on the utterance flag table whether or not the flag of the conversation partner at the time before ΔT is the SPEAKING flag. In this step, it is determined whether or not the conversation person on the dialog device 12 side is speaking even though the other party is speaking. If “YES” in the step S11, that is, if the utterances of both the interlocutors overlap, it is determined whether or not the RECORD flag is turned on in a step S13.

  If “NO” in the step S13, that is, if the utterance duplication has just begun, the voice recording is started in a step S15, and the obtained voice data is converted into a voice file and stored in the memory. For example, the audio data may be PCM format data, and the audio file may be in WAVE format. The audio data may be compressed by an appropriate method before transmission and decoded before reproduction. In step S17, the RECORD flag in the memory is turned on to store that recording is in progress.

  When the other party starts speaking first, the dialogue apparatus 12 receives the voice from the other party and outputs it from the speaker. Therefore, the dialogue person on the dialogue apparatus 12 side usually keeps speaking forcibly. Without thinking, you can stop your utterance and listen to the other party's voice. For this reason, it is considered that the voice to be recorded is very short, and in this embodiment, the voice is transmitted to the server 14 at once after the recording is completed. However, in other embodiments, sound may be transmitted to the server 14 each time.

  On the other hand, if “YES” in the step S13, that is, if the recording has already been started, the recording started in the step S15 is continued in the step S19, and the acquired sound data is stored in the sound file. .

  When step S17 or S19 is completed, the RECORDING flag is recorded in the utterance flag table of the server 14 in step S21. Specifically, the dialogue device 12 includes information indicating that recording is being performed, together with information such as the time t, the speaker (identification information of the dialogue device 12), and the target (identification information of the partner dialogue device 12). RECORDING flag) is transmitted to the server 14. In response to this, the server 14 stores the time, speaker, subject, and RECORDING flag in the utterance flag table based on the received information.

  When the system 10 includes three or more interactive devices 12, an utterance target (identification information of the partner interactive device 12) may be selected by operating an input device or the like.

  Further, in step S23, a value obtained by adding 1 to the value before ΔT is recorded in the PLAY flag of the utterance flag table of the server 14. Specifically, the dialogue apparatus 12 transmits an instruction to increase the PLAY flag to the server 14 together with information such as the time t, the speaker, and the subject. In response to this, the server 14 reads the value immediately before the time t of the PLAY flag in the utterance flag table, adds 1 to this value, and stores the calculated value as the value of the PLAY flag at the time t. . When recording is started with no unreproduced audio file remaining, 1 is stored in the PLAY flag, and the value of the PLAY flag increases by 1 as time t progresses as long as recording continues. Is done. When step S23 ends, the process proceeds to step S59 in FIG.

  On the other hand, if “NO” in the step S11, the process proceeds to a step S25 in FIG. That is, when the conversation person on the dialog apparatus 12 side is speaking, if the other party is not speaking or recording at the previous time, the voice of the conversation person on the conversation apparatus 12 side is Tell the other party.

  If “NO” in the step S5, that is, if the dialog person does not speak on the dialog device 12, the process proceeds to a step S35 in FIG.

  In step S25 of FIG. 7, the SPEAKING flag is recorded in the utterance flag table of the server 14. Specifically, the dialogue apparatus 12 transmits information indicating that the user is speaking (SPEAKING flag) to the server 14 together with information such as the time t, the speaker, and the subject. In response to this, the server 14 stores the time, speaker, target, and SPEAKING flag in the utterance flag table based on the received information.

  In the subsequent step S27, the acquired audio data is converted into an audio file and stored as a local file in the memory. In step S27, the acquired voice data and the reproduction instruction thereof are transmitted directly (that is, not via the server 14) to the interactive apparatus 12 on the other side. Upon receiving the voice data and the playback instruction, the other party's dialogue apparatus 12 executes a playback process of the voice data and outputs the voice from the speaker 18 or 22. In this way, when the utterance is being performed only on the dialog device 12 side, or when the other party is recording, the voice on the dialog device 12 side is recorded in the local file and directly transmitted to the other side. Then, the voice is immediately reproduced and output by the other-side dialog device 12.

  In step S31, the SEND flag of the memory is turned on to store that transmission is in progress. In step S33, the value before ΔT is recorded as it is in the PLAY flag of the utterance flag table of the server 14. Specifically, the dialogue apparatus 12 transmits a PLAY flag maintenance instruction to the server 14 together with information such as the time t, the speaker, and the subject. In response to this, the server 14 reads the value immediately before the time t of the PLAY flag in the utterance flag table, and stores this value as the value of the PLAY flag at the time t. Thus, when recording is not in progress, the previous value is maintained as the value of the PLAY flag. When step S33 ends, the process proceeds to step S59 in FIG.

  If no voice input is performed on the interactive apparatus 12, it is determined in step S35 in FIG. 8 whether or not the RECORD flag is on. If “YES” in the step S35, that is, if the recording has been performed up to the previous time, the recording of the sound to the sound file is ended in a step S37. In step S39, the recording of the operation command input using the input device during the recording of the voice to the command file is ended. In step S41, the RECORD flag of the memory is turned off.

  In step S43, voice analysis processing is performed on the recorded voice file. An example of this voice analysis processing operation is shown in detail in FIG. The operation of the voice analysis process executed in step S53 in FIG. 8 is the same.

  When the voice analysis process is started, the voice file recorded in the memory is read in step S91 in FIG. Note that when executed in step S53 of FIG. 8, the audio data of the local file is read in step S91.

  Next, in step S93, the fundamental frequency (pitch) and amplitude of the read sound source are calculated. In step S95, an attempt is made to divide the audio data into syllables. In step S97, it is determined whether there is a divided syllable. If “YES” in the step S97, the duration of the syllable is calculated in a succeeding step S99. Further, the average duration of the syllable is calculated. When step S99 ends, the process returns to step S97, and if the divided syllables remain, the process of step S99 is repeated for the syllable. If “NO” in the step S97, the voice analysis data is transmitted to the server 14 in a step S101. Therefore, the voice analysis data includes information such as fundamental frequency, amplitude, and average duration of syllables. This voice analysis data is transmitted to the server 14 in association with information such as the time, the speaker, and the object, for example. In response to this, the server 14 stores the received voice analysis data in the voice analysis history DB 92. In this way, the features of the uttered voice are extracted and the history is recorded. When step S101 ends, the voice analysis process ends, and the process returns to step S45 (in the case of step S43) or step S55 (in the case of step S53) in FIG.

  When step S43 ends, the recorded voice file and command file are transmitted to the server 14 in step S45. The voice file and the command file are transmitted to the server 14 in association with information such as the time, the speaker, and the object, for example. In response to this, the server 14 stores the received audio file and command file in a predetermined area of the memory. In the utterance flag table, the storage location of the voice file is registered as the saved voice file information at the recorded time, and the storage location of the command file is registered as the saved command file information at the same time (see FIG. 5). . When step S45 ends, the process proceeds to step S55.

  On the other hand, if “NO” in the step S35, it is determined whether or not the SEND flag is turned on in a step S47. If “YES” in the step S47, that is, if the voice has been transmitted to the interactive apparatus 12 on the other side until the previous time, the recording of the voice in the local file is ended in a step S49. In step S51, the SEND flag of the memory is turned off. In step S53, the above-described voice analysis processing of FIG. 11 is executed on the voice data of the local file. When step S53 ends or if “NO” in the step S47, the process proceeds to a step S55.

  In step S55, the SILENT flag is recorded in the utterance flag table of the server 14. Specifically, the dialogue apparatus 12 transmits information indicating that there is no voice input (SILENT flag) to the server 14 together with information such as the time t, the speaker, and the target. In response to this, the server 14 stores the time, speaker, object, and SILENT flag in the utterance flag table based on the received information.

  In step S57, the value before ΔT is recorded as it is in the PLAY flag of the utterance flag table of the server in the same manner as in step S33 of FIG. When step S57 ends, the process proceeds to step S59 in FIG.

  Steps S59 to S67 in FIG. 9 are processing when the partner interactive device 12 is the robot 12b. Therefore, when the other party is not the robot 12b, these processes need not be performed.

  In step S59 of FIG. 9, input of an operation command is checked. Specifically, input data from the input device is acquired, and it is determined whether or not an operation command for gesture of the robot 12b has been selected. For example, the operation commands may be displayed as a selectable list on the display. If the interactive device 12 is a robot 12b, it is necessary to provide an input device and a display.

  In step S61, it is determined whether there is a selected operation command. If “YES”, it is determined in step S63 whether the RECORD flag of the memory is on. If “YES” in the step S63, that is, if voice recording is being performed, the operation command is stored in a command file in the memory in a step S65. As described above, when the voice is recorded, the input of the operation command is also recorded at the same time and transmitted to the server 14 at the above-described step S45 after the recording is completed. In this case, it is possible to play back on the other party side after giving a delay to speech and gesture at the same time.

  On the other hand, if “NO” in the step S63, the operation command and the reproduction instruction are directly transmitted to the partner interactive apparatus 12b in a step S67. Upon receiving the operation command and the reproduction instruction, the other-side dialog device 12b operates according to the program and data corresponding to the operation command and performs the gesture.

  When step S65 or S67 is completed, or if “NO” in the step S61, the time (or frame number) t is updated by adding a predetermined time ΔT (for example, one frame) in a step S69. Then, the process returns to step S3 in FIG. 6 to repeat the process at the next time t. In this way, in the dialog device 12, processing according to the state of the utterance of the conversation person on the dialog device 12 side and the state of the utterance on the other party side is executed.

  FIG. 12 shows an example of the operation of the server 14 in the continuation promotion process. FIG. 13 shows an example of the operation in the delayed reproduction process of the server 14.

  It should be noted that other processes of the server 14 such as a reception process, an utterance flag table creation process, and a transmission process are omitted. The server 14 executes each process as described above in parallel. As described above, when the server 14 receives data from the interactive apparatus 12, the server 14 stores the data in a memory and executes predetermined processing corresponding to the data as necessary. For example, the server 14 creates an utterance flag table when it receives data related to the state of utterance or processing from the dialogue device 12. When an audio file, an operation command file, or the like is received, these files are stored and the storage location is written in the utterance flag table. When the voice analysis data is received, the data is stored in the voice analysis history DB 92. Further, when there is a request for the utterance flag table from the dialogue apparatus 12, the utterance flag table is transmitted to the dialogue apparatus 12.

  In the continuation promotion process shown in FIG. 12, the CPU of the server 14 executes initialization in step S111, and sets an initial value T for a variable t, for example. This initial value T is the first value T at time t in the utterance flag table, that is, the initial value T at time t in the interactive device 12. Therefore, the continuation promotion process is executed after the utterance flag table is created. The CPU of the server 14 repeatedly executes the processing of subsequent steps S113 to S135 every predetermined time ΔT, for example, every frame.

  In step S113, the utterance flag table in the memory is referred to. For example, data at the current time t is read. Then, in step S115, it is determined whether or not the SILENT flag is set between the interlocutors. For example, it is determined that there are both interacting parties in which the user and the utterance target are paired at the current time t, and the status flag of both interacting parties is the SILENT flag. For example, in FIG. 5, the time T + ΔT corresponds to this state.

  If “YES” in the step S115, that is, if there is a silent state in the dialogue, a blank time is calculated in a step S117. For example, the data in the utterance flag table before the current time t is read, and the time (or the number of frames) taken until either state flag of the two talkers is no longer the SILENT flag is calculated from the current time t.

  Subsequently, in step S119, the latest data of the interlocutors is extracted from the voice analysis history DB 92. Specifically, the fundamental frequency, the amplitude, the average duration of the syllable, and the like are read from the speech analysis data of the speaker closest to the current time t. As described above, in steps S117 and S119, a situation between at least the blank time and the feature of the speech before the blank is detected.

  In step S121, the current situation and the inter-pattern are collated, and in step S123, it is determined whether or not there is an inter-pattern that matches the situation during the current dialogue. As shown in FIG. 4 above, since the blank time (t) and the condition (I) are set in the inter-pattern data, the blank time and speech characteristics (basic frequency, amplitude) that match the inter-pattern are set. , The average duration of syllables, etc.) (ie, silence after the last speaker's utterance) is determined. If there is a matching pattern, a function word corresponding to the pattern is selected. In addition, based on the relationship between the last speaker and the speaker (partner or me) set in the inter-pattern data, the dialogue device 12 that utters the inter-function language is specified.

  If “YES” in the step S123, that is, if it is determined that the situation in the current dialogue is a situation to be inserted based on the inter-pattern, the selected interval in the step S125. The function language speech file is read into the work area of the memory, the pitch and the inflection pattern are adjusted, and the function language speech file is generated during the adjustment. This makes it possible to output function words while matching the characteristics of the speaker's utterance (for example, an elevated tone, a light utterance, etc.). Therefore, even if the synthesized speech is inserted into the conversation, it is possible to prevent the conversation person from feeling a sense of incongruity, and it is possible to continue the tone and flow of the conversation so far.

  In step S127, an operation command suitable for the selected function word is selected. In this embodiment, since the operation command corresponding to the inter-function word is registered in the inter-pattern data, the operation command is selected.

  In step S129, the voice file and the operation command file are transmitted to the dialogue apparatus 12 that speaks. After the file transmission, in step S131, the voice and operation playback instructions are transmitted to the same interactive device 12. As a result, words and gestures can be inserted into the silent area in the dialogue. The dialog device 12 executes reproduction of the audio file and outputs the audio. When the interactive device 12 is the robot 12b, the operation command is further reproduced, and the gesture corresponding to the operation command is performed.

  In step S133, the state flag at the current time t is overwritten with the INTERPOLATING flag in the utterance flag table (see time T + 2ΔT in FIG. 5). Thereby, in the subsequent step S117, the time t can be prevented from being regarded as silent.

  On the other hand, if “NO” in the step S123, that is, if it is not yet necessary to give a space according to the interval pattern, the process proceeds to a step S135 as it is. In step S135, the time (or frame number) t is updated by adding a predetermined time ΔT (for example, one frame). Note that ΔT in the server 14 is the same as ΔT in the interactive device 12. And it returns to step S113 and repeats the process in the following time t. In this way, when silence is detected in the dialogue, the silence time can be changed appropriately by inserting words or gestures as necessary.

  In the delayed playback process shown in FIG. 13, the CPU of the server 14 performs initialization in step S151. For example, the PLAYING flag is turned off. The PLAYING flag indicates whether or not the recorded audio file and operation command file are being reproduced. Further, as in the continuation promotion process of FIG. 12, an initial value T is set to the variable t. This initial value T is the first value T at time t in the utterance flag table, that is, the initial value T at time t in the interactive device 12. Therefore, this delayed reproduction process is also executed after the utterance flag table is created. The CPU of the server 14 repeatedly executes the processing of subsequent steps S153 to S179 every predetermined time ΔT, for example, every frame.

  In step S153, the utterance flag table in the memory is referred to. In step S155, it is determined whether or not the PLAYING flag of the memory is on. If “NO” in the step S115, that is, if reproduction is not being performed, it is determined whether or not there is a SPEAKING flag as a state flag of either of the two interactors at the current time t in a step S157. If “YES” in the step S157, one of the voices is speaking and the voice should be output from the other interactive device 12. Accordingly, the delayed reproduction is not performed, and the process proceeds to step S179.

  On the other hand, if “NO” in the step S157, that is, if no sound is output from both the interactive devices 12, it is determined whether or not the PLAY flag of either of the interactive parties is 1 or more in a step S159. Judging. If “NO” in the step S159, since there is no audio file that has been recorded but not reproduced, the process proceeds to a step S179 as it is.

  However, if “YES” in the step S159, that is, if a recorded but unreproduced audio file remains, a user whose earliest time t at which the PLAY flag is 1 is indicated in the utterance flag table in a step S161. Reference from That is, the user whose recording start time is early is specified. When the start of recording is the same time for both conversation parties, the user is specified based on a preset priority (for example, B> A).

  Subsequently, in step S164, settings for reproduction are executed, 1 is set in the variable F, and the specified user is set in the variable U. A variable F is a frame counter for audio reproduction. In step S165, the PLAYING flag in the memory is turned on to store that playback is in progress. In step S167, the voice and operation in which the PLAY flag of the variable U is the value of the variable F are reproduced. Specifically, the audio file is read out, and the audio file and a reproduction instruction are transmitted to the dialog device 12 on the other side of the user. If the operation command file is also stored, the operation command file is also read out and transmitted to the partner interactive device 12 together with the audio file. In response to this, the dialogue apparatus 12 stores the audio file and the operation command file, and executes the reproduction thereof. As a result, when a voice is output from the speaker 18 or 22 and an operation command is received, the gesture is also executed. In this way, playback of the recorded voice and the stored operation is started.

  When step S167 ends, the process proceeds to step S179. In step S179, the predetermined time ΔT is added to the time t, and the time t is updated. When step S179 ends, the process returns to step S153, and the process at the next time t is repeated.

  When reproduction is started, “YES” is determined in the step S155, and it is determined in a subsequent step S169 whether or not the PLAY flag of the variable U at the time t is equal to the value of the variable F. As described above, when the recording is completed, the value of the PLAY flag is maintained at the previous time, so in this step S169, it is determined whether or not the reproduction of the audio file being reproduced has been completed. .

  If “NO” in the step S169, that is, if the reproduction of the audio file is not yet completed, the variable F is incremented in a step S171. After that, in step S173, the voice and operation in which the PLAY flag of the variable U is the value of the variable F are reproduced. As a result, data is transmitted in the same manner as in step S167 described above, and the voice and operation of the next frame are reproduced by the dialogue apparatus 12. When step S173 ends, the process proceeds to step S179.

  On the other hand, if “YES” in the step S169, that is, if the reproduction of the audio file is completed, the PLAYING flag of the memory is turned off in a step S175. In step S177, all the values of the PLAY flag of the variable U are subtracted by the value of the variable F. When the value becomes negative as a result of the subtraction, the value of the PLAY flag is set to 0. As a result, the reproduced variable U and the value of the PLAY flag at time t are all zero. In addition, when there is an unreproduced audio file of the user of the variable U, the value of the PLAY flag at the earliest time among the oldest recorded audio files of the user is 1. Therefore, next time, the unreproduced sound can be reproduced. When step S177 ends, the process proceeds to step S179.

  In this way, it is possible to reproduce the voice recorded by the overlap of the utterances of the two interactors and the recorded operation command later.

  FIG. 14 shows an example of the output processing operation of the CPU of the interactive apparatus 12. This output process is executed in parallel with the input processes shown in FIGS. Further, this output process is repeatedly executed at regular intervals, for example, every frame.

  In step S191, it is determined whether or not voice is received. If “YES”, the received voice file or voice data is stored in the memory in step S193.

  Subsequently, in step S195, it is determined whether or not an operation command has been received. If “YES”, the received operation command file is stored in the memory in step S197.

  Subsequently, in step S199, it is determined whether or not a reproduction instruction is received. If “YES”, the audio is reproduced in step S201. Specifically, the CPU of the dialogue apparatus 12 starts playing the received audio file, gives the audio data to the audio input / output board, and outputs the audio from the speaker. If the interactive device 12 is an interactive device 12b having a physical motion function, the motion is reproduced in step S203. Specifically, the corresponding gesture is executed according to the operation command. A program and control data for executing gestures corresponding to the operation commands are stored in advance in the memory 70 of the interactive device 12b. The CPU 66 gives control data to the motor control board 72 according to a program corresponding to the operation command to control the corresponding motor. As a result, the corresponding body part is moved to express a predetermined gesture.

  In the above-described embodiment, when the utterances of both of the interlocutors overlap, the voice of the person who was uttered later is recorded, and after both ceases to speak, the recorded voice is output on the other party's side. I was doing. However, in another embodiment, when the utterances of the two interlocutors overlap, the voice of the one uttered later may be canceled.

  Further, in each of the above-described embodiments, the server 14 stores the voice data of the inter-function language and transmits it from the server 14 to the dialogue device 12. However, in another embodiment, voice data of inter-function words may be stored in advance in each interactive device 12 and information specifying the inter-function words to be reproduced may be transmitted from the server 14.

  Further, in each of the above-described embodiments, the system 10 includes the interactive device 12a that does not have a physical motion function and the interactive device 12b that has a physical motion function. However, in other embodiments, only the interactive device 12a that does not have a body motion function may be used, and in this case, processing related to motion commands is unnecessary. Conversely, only the interactive device 12b having a body movement function may be used.

  Further, in each of the above-described embodiments, the system 10 includes the server 14 that manages information (that is, the utterance flag table) indicating the voice acquisition state and the voice output state of each interactive device 12 separately from the interactive device 12. However, in another embodiment, the server 14 may be provided with the functions of the server 14 (speech flag table management, continuation promotion processing, delayed reproduction processing, etc.) without providing the server 14 separately. Alternatively, the two interactive devices 12 may be distributed and provided.

It is an illustration figure which shows the structure of the remote place dialogue system of one Example of this invention. It is an illustration figure which shows an example of the external appearance of the dialogue apparatus which has a body movement function. FIG. 3 is a block diagram illustrating an example of an electrical configuration of the interactive apparatus in FIG. 2. It is an illustration figure which shows an example of the interval pattern data memorize | stored in interval pattern DB. It is an illustration figure which shows an example of the speech flag table memorize | stored in a server. It is a flowchart which shows a part of example of operation | movement of the input process of a dialogue apparatus. It is a flowchart which shows a part of continuation of FIG. It is a flowchart which shows a part of continuation of FIG. FIG. 9 is a flowchart showing a continuation of FIGS. 6, 7, and 8. It is a flowchart which shows an example of the operation | movement of a measurement process between FIG. It is a flowchart which shows an example of the operation | movement of the audio | voice analysis process of FIG. It is a flowchart which shows an example of the operation | movement of a continuation promotion process of a server. It is a flowchart which shows an example of the operation | movement of a delayed reproduction process of a server. It is a flowchart which shows an example of operation | movement of the output process of a dialogue apparatus.

Explanation of symbols

DESCRIPTION OF SYMBOLS 10 ... Remote place interaction system 12, 12a, 12b ... Dialogue device 14 ... Speech timing control server 16, 20 ... Microphone 18, 22 ... Speaker 92 ... Voice analysis history database 94 ... Inter-pattern database

Claims (2)

A system for conducting a dialogue between remote locations including two dialogue devices connected via a network,
Each of the interactive devices
Acquisition means for acquiring audio;
Transmitting means for transmitting the voice acquired by the acquiring means to the interactive apparatus on the other side;
Receiving means for receiving the voice transmitted from the interactive apparatus on the other side, and output means for outputting the voice received by the receiving means,
An inter-pattern storage means for storing a plurality of inter-patterns including at least information regarding the blank time and the features of the speech;
A history recording means for recording a history of dialogue states including at least a voice acquisition state and a voice output state in each of the dialogue devices;
When it is determined that both of the interactive devices are in a no-speech state based on the history recorded by the history recording means, the situation between at least the blank time and the features of the uttered voice before the blank and the plural A matching means for matching between the patterns,
When there is an inter-pattern that matches as a result of the collation by the collating unit, a predetermined voice corresponding to the inter-pattern is output from the output unit of the dialogue apparatus existing on the other party of the speaker of the predetermined voice And when it is determined that the utterances are duplicated in both of the interactive devices based on the history recorded by the history control unit, and when the utterance is finished, A remote interaction system comprising delay reproduction means for outputting a recorded voice from the output means of the other interactive device.   The at least one of the interactive devices is a robot capable of performing gestures, and the interval control unit causes the interactive device to execute a predetermined gesture corresponding to the interval pattern together with the output of the voice. Remote site dialogue system.

Images