JP2019060921A - Information processor and program - Google Patents

Abstract

To express data corresponding to utterance of a speaker in a mode along feeling of the speaker when the data corresponding to the utterance of the speaker is generated from lip movement of the speaker and is expressed.SOLUTION: A first acquisition section 117 acquires images obtained by allowing an imaging section 18 to take a face of a speaker for a plurality of times. A specification section 111 extracts a change of a shape of speaker's lips from the images and specifies utterance of the speaker. A second acquisition section 118 acquires biological information of the speaker mounting a measuring device 2 from the measuring device. An estimation section 112 extracts expression of the speaker from the image of the speaker, which the first acquisition section 117 acquires, collates the expression and the biological information which the second acquisition section 118 acquires with an expression condition determined in feeling DB 121 and a biological condition, and estimates the feeling of the speaker. A selection section 113 selects phoneme data corresponding to utterance information indicating the specified utterance of the speaker. A generation section 114 generates voice data corresponding to the specified utterance of the speaker and the estimated feeling of the speaker.SELECTED DRAWING: Figure 5

Description

  The present invention relates to an information processing apparatus and a program.

Patent Document 1 discloses a portable terminal device which extracts a character corresponding to the movement of the mouth based on the acquired image of the movement of the mouth, generates a voice using the character, and transmits it to a communication destination. .
According to Patent Document 2, when the loudness of the sound input to the microphone is equal to or greater than a predetermined threshold, a screen for selecting a lip reading call mode is displayed on the display unit when a call is received, and a lip reading call mode is selected. If the call mode is switched to the lip reading call mode and switched to the lip reading call mode, the shape of the lip of the speaker is detected from the image obtained by imaging with the imaging device, and at least one of speech data of speech and text data Is disclosed and transmitted to the outside.

JP, 2015-115926, A JP, 2015-220684, A

  An object of the present invention is, when generating and expressing data corresponding to a speaker's utterance from the movement of the speaker's mouth, to express the data in a mode in accordance with the speaker's emotion.

  The information processing apparatus according to claim 1 of the present invention extracts a change in the shape of the speaker's mouth from the image, a first acquisition unit that acquires an image of the speaker's face taken a plurality of times, It is an information processing device which has a generation part which generates data corresponding to a talk of the speaker specified, and an emotion of the speaker presumed.

  An information processing apparatus according to a second aspect of the present invention, in the aspect according to the first aspect, includes an identifying unit that identifies the utterance from the extracted change, and an estimating unit that estimates the emotion. The generation unit is an information processing apparatus characterized by generating the data corresponding to the utterance specified by the specification unit and the emotion estimated by the estimation unit.

  An information processing apparatus according to a third aspect of the present invention, in the aspect according to the second aspect, further includes a second acquisition unit that acquires biological information of the speaker, and the estimation unit acquires the biological information acquired. The information processing apparatus is characterized in that the emotion is estimated according to.

  In the information processing apparatus according to a fourth aspect of the present invention, in the aspect according to the second or third aspect, the estimation unit extracts the expression of the speaker from the image, and the emotion is calculated according to the expression. It is an information processing apparatus characterized by estimating.

  In the information processing apparatus according to a fifth aspect of the present invention, in the aspect according to any one of the first to fourth aspects, the generation unit selects a phoneme from the utterance, and corresponds to the emotion from the phoneme. It is an information processor characterized by generating the data which show the voice of expression.

  In the information processing apparatus according to a sixth aspect of the present invention, in the aspect according to any one of the second to fourth aspects, the identification unit identifies at least one of the speed and volume of the speaker's speech; The generation unit is configured to select a phoneme from the utterance, and to generate the data of the velocity or volume specified by the specification unit, which is an expression corresponding to the emotion from the phoneme. It is an apparatus.

  An information processing apparatus according to a seventh aspect of the present invention is the information processing apparatus according to any one of the first to sixth aspects, wherein the generation unit uses the speaker data indicating the voice of the speaker. An information processing apparatus characterized by generating.

  In the information processing apparatus according to an eighth aspect of the present invention, in the aspect according to the seventh aspect, the generation unit transmits the speaker data indicating a voice corresponding to the estimated emotion among the voices of the speaker An information processing apparatus characterized by generating the data using

  An information processing apparatus according to a ninth aspect of the present invention is the information processing apparatus according to the third aspect, wherein the generation unit is speaker data indicating a voice corresponding to biological information of the speaker among voices of the speaker An information processing apparatus characterized by generating the data using

  An information processing apparatus according to a tenth aspect of the present invention is the information processing apparatus according to any one of the first to ninth aspects, wherein the generation unit receives the operation determined from the speaker. An information processing apparatus characterized by generating

  An information processing apparatus according to an eleventh aspect of the present invention, in the aspect according to any one of the first to tenth aspects, further comprising: a measurement unit that measures the volume of the voice uttered by the speaker; The information processing apparatus is characterized in that the unit generates the data when the volume is less than a predetermined threshold.

  An information processing apparatus according to a twelfth aspect of the present invention, in the aspect according to any one of the first to tenth aspects, further comprising: a measurement unit that measures a light amount around the speaker; The information processing apparatus is characterized in that generation of the data is stopped when the light amount is less than a determined threshold.

  The information processing apparatus according to claim 13 of the present invention is characterized in that, in the aspect according to any one of claims 1 to 12, the generation unit notifies whether or not the data is generated. It is an information processing apparatus.

  A program according to a fourteenth aspect of the present invention is a computer comprising: a first acquisition unit for acquiring an image obtained by imaging a speaker's face a plurality of times; and a change in the shape of the speaker's mouth from the image. It is a program for functioning as a generation part which generates data corresponding to a talk of the speaker specified, and an emotion of the speaker presumed.

According to the inventions of claims 1, 2 and 14, when generating and expressing data corresponding to the speech of the speaker from the movement of the mouth of the speaker, the data is in line with the emotion of the speaker It can be expressed in an aspect.
According to the third aspect of the present invention, the emotion of the speaker used for expression is estimated according to the biological information of the speaker.
According to the invention of claim 4, the emotion of the speaker is estimated from the expression of the speaker.
According to the fifth aspect of the present invention, data indicative of speech is synthesized by synthesizing phonemes.
According to the invention of claim 6, at least one of the speech speed and the volume is reflected in the data to be synthesized.
According to the seventh aspect of the invention, data corresponding to the voice of the speaker is synthesized.
According to the invention of claim 8, the data is synthesized using the voice corresponding to the emotion of the speaker.
According to the invention of claim 9, the data is synthesized by using the voice corresponding to the biometric information of the speaker.
According to the invention of claim 10, the data is synthesized by the speaker performing the determined operation.
According to the invention of claim 11, the data is synthesized when the volume of the voice uttered by the speaker is less than the threshold.
According to the invention of claim 12, the data synthesis is stopped when the amount of light around the speaker is less than the threshold.
According to the invention as set forth in claim 13, it can be known whether or not data according to the speech and emotion of the speaker are synthesized.

FIG. 1 is a view showing an example of the configuration of a call system 9 according to the present embodiment. FIG. 2 is a view showing an example of the configuration of a measurement apparatus 2; The figure which shows the example of a structure of the terminal 1. FIG. FIG. 2 is a view showing various databases stored in a storage unit 12; FIG. 2 is a diagram showing a functional configuration of a terminal 1; FIG. 7 is a flow diagram showing a flow of an operation of extracting a representative color and a color arrangement candidate of the terminal 1; The figure for demonstrating the expression of the audio | voice which changes for every speaker's emotion. The figure which shows the structure of the server apparatus 4 shown with the broken line in FIG. FIG. 2 is a diagram showing a functional configuration of a server device 4; The figure which shows the example which the terminal 1a and the terminal 1b share a function. The figure which shows the example which the terminal 1a and the terminal 1b share a function. The figure which shows an example of phoneme DB124 in a modification.

1. Embodiment 1-1. Overall Configuration of Call System FIG. 1 is a diagram showing an example of a configuration of a call system 9 according to the present embodiment. The call system 9 includes a terminal 1a used by the speaker, a terminal 1b used by the other party of the talk of the speaker (hereinafter simply referred to as "terminal 1" when the terminals 1a and 1b are not distinguished), and The communication line 3 which connects the terminals 1 of Further, the communication system 9 shown in FIG. 1 includes the measuring device 2 which is attached to any part of the body, such as the speaker's wrist, and measures information on the speaker's living body (hereinafter referred to as "biological information"). doing. The biological information is, for example, information represented by blood pressure, body temperature, sweat rate, pulse rate, heart rate pattern and the like.

1-2. Configuration of Measuring Device FIG. 2 is a view showing an example of the configuration of the measuring device 2. The measuring device 2 includes a control unit 21, a storage unit 22, a communication unit 23, and a measuring unit 28.

  The control unit 21 includes a central processing unit (CPU), a read only memory (ROM), and a random access memory (RAM), and a computer program (hereinafter, simply referred to as a program) stored in the ROM and the storage unit 22 by the CPU. Are read out and executed to control each part of the measuring apparatus 2.

  The storage unit 22 is storage means such as a solid state drive, and stores various programs read by the CPU of the control unit 21.

  The measurement unit 28 is a device that measures biological information of a speaker who uses the terminal 1. The measuring unit 28 irradiates light of a determined wavelength to the tissue of the speaker's wrist by an irradiation device such as a light emitting diode (LED), for example, and is not absorbed by the hemoglobin contained in the blood flowing in the blood vessel. It is a pulsimeter that receives the reflected light reflected by the light receiving element and measures the pulse rate from the period of time-dependent change of the reflected light. In addition, the measurement unit 28 may measure the blood pressure, the temperature, the amount of sweating, the heartbeat pattern, and the like of the speaker.

  The communication unit 23 is a communication circuit connected to the terminal 1 wirelessly or by wire. The measuring device 2 supplies biological information indicating the result of measurement by the measuring unit 28 to the terminal 1 through the communication unit 23.

1-3. Terminal Configuration FIG. 3 is a diagram showing an example of the configuration of the terminal 1. The terminal 1 includes a control unit 11, a storage unit 12, a communication unit 13, a display unit 14, an operation unit 15, a sound collection unit 16, a sound emission unit 17, and an imaging unit 18.

  The control unit 11 includes a CPU, a ROM, and a RAM, and controls the units of the terminal 1 by the CPU reading and executing programs stored in the ROM and the storage unit 12.

  The operation unit 15 includes operation buttons for giving various instructions and operation elements such as a keyboard, receives an operation by the user, and supplies a signal corresponding to the content of the operation to the control unit 11. The operation unit 15 may have a touch panel that detects an operation object such as a user's finger or a stylus pen.

  The display unit 14 has a display screen such as a liquid crystal display, and displays an image under the control of the control unit 11. The transparent touch panel of the operation unit 15 may be overlapped on the display screen.

The sound collection unit 16 is a microphone or the like that records the voice of the speaker and converts it into a voice signal.
The sound emitting unit 17 is a speaker or the like that converts an audio signal supplied from the control unit 11 into sound and emits the sound.
The imaging unit 18 includes an imaging element such as a complementary metal oxide semiconductor (CMOS) or a charge coupled device (CCD), and an optical system that forms an image on the imaging element. The imaging unit 18 is a so-called in-camera, and generates image data indicating an image obtained by the imaging device. When the speaker uses the terminal 1, the imaging unit 18 images the face of the speaker a plurality of times.

  In addition, this imaging unit 18 obtains image data showing a plurality of still images by photographing a moving image showing the face of the speaker once instead of imaging the face of the speaker a plurality of times. May be

  The communication unit 13 has a first communication unit 131 and a second communication unit 132. The first communication unit 131 is a communication circuit connected to the communication line 3 wirelessly or by wire. The second communication unit 132 is a communication circuit connected to the measurement device 2 wirelessly or by wire. For communication by the second communication unit 132, for example, a close proximity wireless communication method or the like conforming to IEEE 802.15 is applied.

  The storage unit 12 is a large-capacity storage unit such as a solid state drive or a hard disk drive, and stores various programs read by the CPU of the control unit 11. In addition, the storage unit 12 stores an emotion DB 121, an utterance DB 122, an expression DB 123, and a phoneme DB 124.

  FIG. 4 is a diagram showing various databases stored in the storage unit 12. The emotion DB 121 shown in FIG. 4A is a condition related to the expression of the speaker (referred to as “expression condition”), a condition related to the biological information of the speaker (referred to as “biological condition”), and at least the expression condition and the biological condition. It is stored in association with identification information (referred to as "emotional ID") for identifying the speaker's emotion when one is satisfied. The emotions are, for example, "joy", "anger", "sadness", "surprise", "excitation", "puzzle", "question", "irritability", and the like.

  The expression condition is, for example, a condition in which the position and the range of movement of each part constituting the expression, such as the mouth angle, eyebrow and pupil of the speaker, are defined from the image obtained by imaging the speaker's face. The living condition is a condition that defines the range of the speaker's blood pressure, body temperature, amount of sweating, pulse rate, heart rate pattern, etc. from the speaker's biological information acquired from the measuring device 2 via the second communication unit 132. is there.

  By referring to the emotion DB 121, for example, the control unit 11 determines that the expression condition that the speaker's mouth angle rises above the determined reference condition is satisfied, and the pulse rate is determined. When it is determined that the living condition of being in the range is satisfied, it is specified that this speaker has the emotion of “joy” associated with the expression condition and the living condition. When it is determined that one of the expression condition and the biological condition is satisfied, the control unit 11 may specify that the emotion associated with the condition is included.

  The speech DB 122 shown in FIG. 4 (b) is a condition related to the shape of the speaker's mouth or lip (referred to as "lip condition"), and information indicating the speaker's speech when the lip condition is satisfied (referred to as "message information" And are stored in association with each other. By referring to the speech DB 122, the control unit 11 specifies the speech of the speaker from the shape of the speaker's mouth or lip.

  The utterance indicated by the utterance information may be a speech segment (referred to as a "phoneme") which is a unit of sound distinguished in the language used by the speaker, but is a word composed of a combination of several phonemes. It may be.

  The expression DB 123 illustrated in FIG. 4C associates and stores the emotion ID and information (referred to as “expression information”) indicating an aspect of expression of speech. By referring to the expression DB 123, the control unit 11 specifies an aspect in which the voice of the speaker is expressed when the speaker has the emotion indicated by the designated emotion ID.

  Specifically, the aspect in which the speech is expressed is, for example, an aspect indicated by prosody such as stress, intonation, pitch, or rhythm of speech. In the expression DB 123, for example, “the tonality in which the beginning and the end become pitches higher than the threshold at a high pitch overall,” for the emotion of “joy”, for the emotion of “rage”, Each of “the intonation in which the change of the pitch is less than the threshold at a lower pitch overall” is associated with each as the expression information.

  The correspondence between the emotion and the manner in which the voice is expressed may be determined by statistically processing a plurality of human samples, or by statistically processing a sample of a speaker using the terminal 1 It may be determined. Note that processing statistically includes, for example, removal of an abnormal value by comparison with a threshold value, calculation of arithmetic mean and geometric mean, and identification of representative values such as mode value and median value.

  The phoneme DB 124 illustrated in FIG. 4D stores speech information and phoneme data indicating a phoneme when a person speaks a speech indicated by the speech information, in association with each other.

  The phoneme data may be a voice signal generated by statistically processing waveform data when a plurality of persons utter a phoneme, or indicates a voice recorded by a speaker using the terminal 1 in the past It may be an audio signal generated by statistically processing data (referred to as "speaker data").

1-4. Functional Configuration of Terminal FIG. 5 is a diagram showing a functional configuration of the terminal 1. The control unit 11 of the terminal 1 shown in FIG. 5 executes the program stored in the storage unit 12 shown in FIG. 1 to obtain the identification unit 111, the estimation unit 112, the selection unit 113, the generation unit 114, and the first acquisition unit. The function 117 functions as the second acquisition unit 118. The control unit 11 illustrated in FIG. 5 functions as the measurement unit 119 and the reception unit 110 by executing the above-described program.

  The first acquisition unit 117 acquires an image obtained by imaging the face of the speaker a plurality of times by the imaging unit 18. The identifying unit 111 extracts a change in the shape of the speaker's mouth from the image to specify the speaker's utterance.

  In addition, the identification unit 111 uses the lip information of the speaker extracted from the acquired image, for example, the size and shape of the open mouth, and the speed of the change of the opening speed of the speaker's speech. And / or the volume may be specified.

  The second acquisition unit 118 communicates with the measurement apparatus 2 via the second communication unit 132, and acquires biological information of the speaker wearing the measurement apparatus 2. The estimation unit 112 extracts the expression of the speaker from the image of the speaker acquired by the first acquisition unit 117. Then, the estimation unit 112 collates the extracted expression with the biological information acquired by the second acquisition unit 118 with the expression condition and the living condition defined in the emotion DB 121 to estimate the emotion of the speaker.

  The selection unit 113 searches the phoneme DB 124 based on the speech information indicating the speech of the specified speaker, and selects phoneme data corresponding to the speech information. The phoneme data selected by the selection unit 113 may be one or more for one piece of speech information.

  The generation unit 114 generates voice data according to the identified speaker's utterance and the estimated speaker's emotion. For example, the generation unit 114 generates voice data by deforming the selected phoneme data according to the estimated emotion, arranging the selected phoneme data in an arrangement along the utterance, and connecting them to each other. That is, when the functions of the selection unit 113 and the generation unit 114 are combined, a phoneme is selected from the utterance of the speaker identified by the identification unit 111, and a voice of an expression corresponding to the speaker's emotion is indicated from the selected phoneme. It becomes a function to generate voice data.

  After generating the voice data, the generation unit 114 transmits the voice data to the terminal 1 used by the other party of the call via the first communication unit 131 of the communication unit 13. Note that the destination terminal 1b does not have to have all the functions of the terminal 1a that has generated the audio data, and has a function of outputting the audio data received from the terminal 1a from the sound emitting unit 17, for example. Just do it.

  By the control unit 11 realizing the above-described function, the terminal 1 extracts a change in the shape of the speaker's mouth from the image, and a first acquisition unit that acquires an image of the speaker's face taken a plurality of times. The information processing apparatus functions as an information processing apparatus including a specification unit that specifies a speaker's utterance, an estimation unit that estimates a speaker's emotion, and a generator that generates data according to the utterance and the emotion.

  When the phoneme data stored in the phoneme DB 124 is a voice signal generated based on the speaker data, the generation unit 114 generates the above-described voice using the voice signal generated based on the speaker data. Generate data. In this case, the generated voice data is likely to include features specific to the voice of the speaker using the terminal 1.

  The measuring unit 119 measures the volume of the voice of the speaker. The generation unit 114 generates audio data when the measured volume is less than the determined threshold. According to this configuration, when the volume of the voice of the speaker is less than the determined level, the terminal 1 identifies the content of the speech (referred to as the "content of speech") from the shape and movement of the lip of the speaker Then, voice data corresponding to the content of the message is generated and transmitted to the terminal 1 of the other party. Therefore, when the voice is small and can not be heard, the terminal 1 generates voice data according to the content of the statement and transmits it to the other party.

  The measuring unit 119 also measures the amount of light around the speaker. For example, the measurement unit 119 may measure the above-described light amount from the image acquired by the first acquisition unit. Further, the terminal 1 may have an illuminance meter or the like that measures the amount of light around the speaker. In this case, the measuring unit 119 may obtain the information of the light amount described above from the luminometer.

  The generation unit 114 stops the generation of audio data when the measured light amount is less than the determined threshold. According to this configuration, when the brightness around the speaker is not determined, voice data according to the content of the utterance specified from the shape and movement of the lip of the speaker is not generated. Therefore, the terminal 1 does not generate voice data when there is a possibility that the captured image is too dark and it is difficult to identify the movement of the speaker's lips, and the identified utterance content may be inaccurate. , Do not send to the other party.

  The reception unit 110 receives the operation of the speaker via the operation unit 15. When the receiving unit 110 receives an operation determined by the speaker, the generating unit 114 generates the above-described voice data. For example, the generation unit 114 generates the above-described voice data during a period from when the reception unit 110 presses the determined button of the operation unit 15 until the button is released, that is, during a period when the button is kept pressed.

1-5. Operation of Terminal FIG. 6 is a flowchart showing a flow of an operation of extracting a representative color and color arrangement candidate of the terminal 1. According to the flowchart shown in FIG. 6A, the control unit 11 of the terminal 1 monitors the first communication unit 131 and determines whether there is an incoming call from another terminal 1 (step S101). While determining that there is no incoming call (step S101; NO), the control unit 11 continues this determination. When it is determined that there is an incoming call (step S101; YES), the control unit 11 receives an operation for selecting a mode for receiving an incoming call, and determines whether or not the speaker has selected a mode ( Step S102).

  If it is determined that the speaker has not made a selection (step S102; NO), the control unit 11 continues this determination. If it is determined that the speaker has made a selection (step S102; YES), the control unit 11 determines whether the selected mode is a mode for making a call without voice (referred to as "non-voice mode"). (Step S103).

  When it is determined that the silent mode is not selected (step S103; NO), the control unit 11 performs a normal call via voice (step S104). When it is determined that the silent mode is selected (step S103; YES), the control unit 11 drives the imaging unit 18 to capture the face of the speaker (step S105), and acquires those images from the imaging unit 18 Also, biological information is acquired from the measuring device 2 (step S106).

  The control unit 11 determines whether or not there has been an operation (referred to as a "speech operation") for the speaker to make a speech (step S107). The speech operation is, for example, an operation of continuing to press the determined button of the operation unit 15. While judging that there is no speech operation (step S107; NO), the control unit 11 continues this judgment.

  If it is determined that there is a speaking operation (step S107; YES), the control unit 11 extracts lip information of the speaker from the acquired image and specifies the speaker's speech (step S108).

  In addition, the control unit 11 may specify the speed of the speaking manner of the speaker, the size of the voice that should have been emitted, etc., using the lip information of the speaker extracted from the acquired image (step S109).

  The control unit 11 estimates the emotion of the speaker from the acquired biological information or the image of the speaker's face (step S110). The control unit 11 selects phoneme data indicating one or more phonemes associated with the identified speaker's utterance (step S111), and transforms and connects those phoneme data along the estimated emotion. , And generates voice data of an expression according to the emotion (step S112). When the speech speed and volume of the speaker are specified in step S109, such information may be reflected in the generation of the voice data. The control unit 11 transmits the generated voice to the other party's terminal 1 (step S113).

  By the above-described operation, the terminal 1 acquires an image obtained by imaging the face of the speaker a plurality of times, extracts the change in the shape of the speaker's mouth from the image, and specifies the utterance. Then, the terminal 1 estimates the speaker's emotion from the above-described image or biological information, and generates voice data according to the identified utterance and the estimated emotion.

  FIG. 7 is a diagram for explaining the expression of voice that changes for each speaker's emotion. In the table shown in FIG. 7, the phonemes pronounced in order from left to right are arranged, and the emotions of the speaker such as "joy", "anger", "sadness" are arranged vertically. And in FIG. 7, the pitch when each phoneme is pronounced is represented by the height of the bar graph of the corresponding position for each emotion.

  As shown in FIG. 7, for example, the statement "thank you" is represented by a sequence of 11 phonemes. When the speaker has a feeling of "joy", this statement has an overall higher pitch compared to when the speaker has another feeling, and the beginning and end are above the threshold It becomes a high pitch.

  On the other hand, when the speaker is feeling anger, this statement has an overall lower pitch than when feeling the joy, and there is relatively little intonation. In addition, when the speaker is feeling "sorrow", this statement has a pitch that is generally lower compared to when feeling the other feeling, and the pitch drops from the third phoneme. to continue.

  According to the above-described operation of the terminal 1, the utterance is specified from the change in the shape of the speaker's mouth, and the speaker's emotion is compared to the terminal transmitting the voice data generated based only on the specified utterance. The voice data is transmitted to the other party's terminal in a corresponding expression.

2. Although the embodiment has been described above, the contents of this embodiment can be modified as follows. Also, the following modifications may be combined.

2-1. Modification 1
In the embodiment described above, the estimation unit 112 estimates the emotion of the speaker using the facial expression of the speaker extracted from the image acquired by the first acquisition unit 117 and the biological information acquired by the second acquisition unit 118. However, it may be estimated based on either expression or biological information.

  For example, the estimation unit 112 may estimate the emotion of the speaker from the expression of the speaker extracted from the image acquired by the first acquisition unit 117. In this case, the field of the living condition may not be included in the emotion DB 121 shown in FIG. 4A. Further, in this case, the speaker may not wear the measurement device 2 to the region of his or her body, and the control unit 11 may not function as the second acquisition unit 118.

2-2. Modification 2
Further, the estimation unit 112 may estimate the emotion of the speaker from the biological information acquired by the second acquisition unit 118. In this case, the field of the expression condition may not be included in the emotion DB 121 shown in FIG. Also, in this case, the estimation unit 112 may not extract the expression of the speaker from the image acquired by the first acquisition unit 117.

In addition, the estimation unit 112 may estimate the emotion of the speaker when any one of the conditions of the expression and the biological information is satisfied.
Further, the estimation unit 112 may estimate the speaker's emotion based on conditions different from the expression and the biological information. For example, the estimation unit 112 may specify the features and prosody of the speaker's voice recorded by the sound collection unit 16 and estimate the speaker's emotion based on these.

2-3. Modification 3
In the embodiment described above, the generation unit 114 and the selection unit 113 select a phoneme from the utterance of the speaker identified by the identification unit 111, and indicate the speech of the expression corresponding to the speaker's emotion from the selected phoneme. Although the voice data is generated, the generation unit 114 may generate data indicating the content of the speech of the speaker.

  The generation unit 114 may generate, for example, text data representing the content of the speaker's speech in characters. In this case, the generation unit 114 uses the text or the display mode of the text, such as an annotation, character color, font modification, character modification such as underlining, font selection, etc., of the speaker's emotion estimated by the estimation unit 112. It may be expressed. In short, the generation unit 114 may generate data according to the identified utterance and the estimated emotion.

2-4. Modification 4
In the embodiment described above, the identification unit 111 identifies the speed or volume of the speaker's speech using lip information, but may not identify them. Also in this case, the identification unit 111 identifies the speech of the speaker from the lip information, and the generation unit 114 generates data corresponding to the speech.

2-5. Modification 5
In the embodiment described above, the phoneme DB 124 stores speech information and phoneme data indicating a phoneme when a person speaks a speech indicated by the speech information in association with each other. A set of phoneme data indicating phonemes generated may be stored for each speaker's emotions.

  For example, even when the speaker utters the same utterance "A", when the speaker has an emotion of "joy" and an emotion of "anger", The prosody may be different. In this modification, the phoneme DB 124 stores, for each feeling of the speaker, a set of speech information and phoneme data corresponding thereto.

  In this configuration, the selection unit 113 searches the phoneme DB 124 based on, for example, utterance information indicating the identified speaker's utterance and the emotion ID indicating the estimated speaker emotion, and the corresponding phoneme Select data The generation unit 114 connects the selected phoneme data to generate the above-described voice data.

2-6. Modification 6
Further, the phoneme DB 124 may store a set of speech information and phoneme data of the speaker for each of the speaker's biometric information. For example, even when the speaker utters the same utterance "A", the condition of the living body such as the speaker's blood pressure, body temperature, amount of sweating, pulse rate, heart rate pattern, etc. affects the speaker's voice There is. In this modification, the phoneme DB 124 stores, for each living body information of the speaker, a set of speech information and phoneme data corresponding to the speech information.

  In this configuration, the selection unit 113 searches the phoneme DB 124 based on, for example, speech information indicating the speech of the specified speaker and the biological information of the speaker measured in the measuring device 2, and the speech information and the speech information The phoneme data corresponding to each of the biological information is selected. The generation unit 114 connects the selected phoneme data to generate the above-described voice data.

2-7. Modification 7
In the embodiment described above, the reception unit 110 receives the operation of the speaker via the operation unit 15, and the generation unit 114 receives the voice data described above when the reception unit 110 receives the operation determined from the speaker. The generation unit 114 may generate voice data without depending on the operation from the speaker.

2-8. Modification 8
In the embodiment described above, the generation unit 114 generates voice data when the measured volume is less than the determined threshold, but when the volume satisfies the determined condition, the generation unit 114 generates the voice data. The generation may be started or stopped. In addition, the generation unit 114 may start or stop generation of voice data, for example, when receiving a determined operation, regardless of the volume of the voice of the speaker. In this case, the measuring unit 119 may not measure the volume of the voice of the speaker and may not be provided in the terminal 1.

  Also, the terminal 1 is provided with a human sensor such as an infrared sensor, for example, and it is detected by this human sensor whether a speaker is present within a determined distance from a determined part of the terminal 1 Good.

  For example, when the voiceless mode is selected, a part of the speaker's face or the like is present within a determined distance that indicates the range in which the human sensor can not perform imaging by the imaging unit 18 or the like. May be stopped, the generation of the audio data by the generation unit 114 may be stopped. Also, at this time, the control unit 11 uses the sound emitting unit 17 to “generate voice data in the voiceless mode for the speaker, so keep the terminal away from the face and talk while taking a face to the in-camera. You may make an announcement.

2-9. Modification 9
In addition, although the generation unit 114 stops the generation of audio data when the measured light amount is less than the determined threshold, the generation unit 114 generates the audio data when the light amount satisfies the determined condition. It may be started or stopped. In addition, the generation unit 114 may start or stop generation of voice data, for example, when receiving a determined operation, regardless of the amount of light around the speaker. In this case, the measuring unit 119 may not measure the amount of light around the speaker, and may not be provided in the terminal 1.

  The measuring unit 119 may measure a physical quantity in addition to the volume of the voice of the speaker and the light intensity around the speaker. The generation unit 114 may start or stop the generation of audio data when the measured physical quantity satisfies the determined condition.

2-10. Modification 10
When communicating with the terminal 1 of the other party, the terminal 1 may notify whether the generation unit 114 generates the above-described voice data. Thereby, the user of the terminal 1b of the communication destination grasps whether or not the transmitted voice data is generated in the silent mode.

2-11. Modification 11
The control unit 11 of the terminal 1 described above functions as the specification unit 111, the estimation unit 112, the selection unit 113, the generation unit 114, the first acquisition unit 117, and the second acquisition unit 118. The server apparatus 4 connected to 3 may perform.

  FIG. 8 is a diagram showing the configuration of the server device 4 indicated by a broken line in FIG. The server device 4 is connected to the communication line 3 and mediates the calls of the terminal 1a and the terminal 1b.

  As illustrated in FIG. 8, the server device 4 includes a control unit 41, a storage unit 42, and a communication unit 43.

The control unit 41 includes a CPU, a ROM, and a RAM, and controls the units of the server device 4 by the CPU reading and executing programs stored in the ROM and the storage unit 42.
The communication unit 43 is a communication circuit connected to the communication line 3 wirelessly or by wire.

  The storage unit 42 is a large-capacity storage unit such as a solid state drive or a hard disk drive, and stores various programs read by the CPU of the control unit 41. In addition, the storage unit 42 stores an emotion DB 421, a statement DB 422, an expression DB 423, and a phoneme DB 424. These databases correspond to the emotion DB 121, the speech DB 122, the expression DB 123, and the phoneme DB 124 stored in the storage unit 12.

  FIG. 9 is a diagram showing a functional configuration of the server device 4. The control unit 41 of the terminal 1 illustrated in FIG. 9 functions as an identification unit 411, an estimation unit 412, a selection unit 413, a generation unit 414, and an acquisition unit 417 by executing the program stored in the storage unit 42.

  The control unit 11 of the terminal 1a transmits the image to the server device 4 via the communication unit 13 and the communication line 3 when the imaging unit 18 acquires an image obtained by imaging the face of the speaker a plurality of times. Further, when the control unit 11 of the terminal 1 a acquires the biological information of the speaker from the measurement device 2, the control unit 11 transmits the biological information to the server device 4 via the communication unit 13 and the communication line 3.

  The acquisition unit 417 acquires an image and biological information from the terminal 1a. The identifying unit 411 extracts a change in the shape of the speaker's mouth from the image to specify the speaker's utterance. The estimation unit 412 estimates the emotion of the speaker using the facial expression and the biological information extracted from the image.

  The selection unit 413 selects phoneme data corresponding to the speech information indicating the speech of the specified speaker. The generation unit 414 generates voice data in accordance with the identified speaker's utterance and the estimated speaker's emotion. Then, when the voice data is generated, the generation unit 414 transmits the voice data to the terminal 1 b used by the other party of the call via the communication unit 43.

  When the control unit 41 realizes the above-described function, the server device 4 extracts a change in the shape of the speaker's mouth from the image, an acquisition unit that acquires an image of the speaker's face taken a plurality of times, and an image. The information processing apparatus functions as an information processing apparatus including: a specification unit that specifies a speaker's utterance; an estimation unit that estimates a speaker's emotion; and a generator that generates data according to the utterance and the emotion.

2-12. Modification 12
The control unit 11 of the terminal 1 described above functions as the identification unit 111, the estimation unit 112, the selection unit 113, the generation unit 114, the first acquisition unit 117, and the second acquisition unit 118, but is a transmission source of audio data. These functions may be shared between the terminal 1a and the terminal 1b which is the transmission destination.

  FIG. 10 is a diagram showing an example in which the terminal 1a and the terminal 1b share functions. The control unit 11 of the terminal 1a illustrated in FIG. 10 functions as a specification unit 111, an estimation unit 112, a first acquisition unit 117, and a second acquisition unit 118. The functions of the identifying unit 111, the estimating unit 112, the first acquiring unit 117, and the second acquiring unit 118 illustrated in FIG. 10 are the same as the functions illustrated in FIG.

  The control unit 11 of the terminal 1 a includes the first communication unit 131 of the communication unit 13 and the communication line 3 of the utterance information of the speaker identified by the identification unit 111 and the information indicating the speaker's emotion estimated by the estimation unit 112. To the terminal 1b.

  The control unit 11 of the terminal 1 b illustrated in FIG. 10 functions as a selection unit 113, a generation unit 114, and a sound emission control unit 115. Also, the control unit 11 of the terminal 1 b shown in FIG. 10 may function as the display control unit 116.

  The selection unit 113 selects phoneme data corresponding to each of the speech information indicating the speech of the speaker based on the speech information transmitted from the terminal 1a. The generation unit 114 receives the information indicating the emotion of the speaker transmitted from the terminal 1a, and transforms / connects the phoneme data selected by the selection unit 113 according to the emotion of the speaker to generate voice data.

  The sound emission control unit 115 causes the sound emission unit 17 to output (sound emission) the audio data generated by the generation unit 114. The display control unit 116 causes the display unit 14 to display the generated text data when the generation unit 114 generates the text data described in the third modification, for example.

  In the configuration shown in FIG. 10, since the speech information of the speaker and the information indicating the emotion of the speaker are exchanged via the communication line 3, the communication load can be reduced as compared with the case of exchanging voice data. There is sex.

FIG. 11 is a diagram showing an example in which the terminal 1a and the terminal 1b share functions. The control unit 11 of the terminal 1a illustrated in FIG. 11 functions as a first acquisition unit 117 and a second acquisition unit 118.
The control unit 11 of the terminal 1a transmits the image acquired by the first acquisition unit 117 and the biological information acquired by the second acquisition unit 118 via the first communication unit 131 of the communication unit 13 and the communication line 3, respectively. , To the terminal 1b.

  The control unit 11 of the terminal 1 b illustrated in FIG. 11 functions as a specification unit 111, an estimation unit 112, a selection unit 113, a generation unit 114, and a sound emission control unit 115. Further, the control unit 11 of the terminal 1 b shown in FIG. 11 may function as the display control unit 116. The identifying unit 111 analyzes the image transmitted from the terminal 1a, extracts a change in the shape of the speaker's mouth, and specifies the speaker's utterance. The estimation unit 112 analyzes the image sent from the terminal 1a and compares the expression of the speaker extracted and the biological information sent from the terminal 1a with the expression condition and the living condition defined in the emotion DB 121. Estimate the emotion of this speaker.

  The selection unit 113 searches the phoneme DB 124 based on the speech information indicating the speech of the specified speaker, and selects phoneme data corresponding to the speech information. The generation unit 114 generates voice data according to the identified speaker's utterance and the estimated speaker's emotion. The sound emission control unit 115 causes the sound emission unit 17 to output the generated sound data, and the display control unit 116 causes the display unit 14 to display the generated text data.

  In the configuration shown in FIG. 11, the image acquired by the first acquisition unit 117 and the biological information acquired by the second acquisition unit 118 are exchanged via the communication line 3, so this is compared to the case of exchanging audio data. Communication load may be reduced.

  The terminal 1a and the terminal 1b may share the functions as described above, but the control unit 11 of the terminal 1 consigns the functions of the identification unit 111 and the estimation unit 112 to other devices and executes them. It is also good. In short, the control unit 11 of the terminal 1 extracts a change in the shape of the speaker's mouth from the acquired first image acquisition unit 117 which acquires an image of the speaker's face taken a plurality of times, It suffices to function as the generation unit 114 that generates data corresponding to the utterance of the speaker identified to be entrusted to such a device and the emotion of the speaker estimated to be entrusted to any of the devices.

2-13. Modification 13
Although the phoneme DB 124 described above stores speech information and phoneme data indicating a phoneme when a person speaks a speech indicated by the speech information in association with each other, one phoneme data is stored for each phoneme. It is not necessary to memorize. For example, the phoneme DB 124 may store, for a group of phonemes (phoneme group), a data string (referred to as a phoneme data string) in which phoneme data indicating phonemes that constitute the phoneme group are arranged in order.

  FIG. 12 is a diagram showing an example of the phoneme DB 124 in the modification. The phoneme DB 124 shown in FIG. 12 has a phoneme group table 1241 in which phoneme groups in which a plurality of phonemes are grouped together are associated in advance with a group of speech information, and a phoneme data string 1242 respectively indicating the phoneme groups. For example, in the phoneme group table 1241 shown in FIG. 12, a phoneme group "thank you", a phoneme group "there was", etc. are associated with the speech information. A phoneme data string 1242 is associated with each phoneme group.

  In this case, the selection unit 113 realized by the control unit 11 may search the phoneme DB 124 based on the speech information indicating the speech of the specified speaker, and select the phoneme data string 1242 corresponding to the speech information. .

  For example, when the utterance information indicating the utterance identified by the identifying unit 111 matches any of the utterance information described in the phoneme group table 1241 of the phoneme DB 124 within the determined error range, the selecting unit 113 In the phoneme group table 1241, the phoneme group associated with the utterance information is specified, and the phoneme data string 1242 associated with the specified phoneme group is selected. At this time, the selection unit 113 does not have to select phoneme data for each phoneme because it selects the phoneme data string 1242 according to the speech information.

  Then, in this case, the generation unit 114 realized by the control unit 11 may generate voice data by deforming the selected phoneme data sequence 1242 according to the estimated emotion. In the generation unit 114, since the selection unit 113 selects the phoneme data string 1242, it is not necessary to connect a plurality of phoneme data respectively indicating one phoneme to generate voice data.

  That is, when the functions of the selection unit 113 and the generation unit 114 shown in this modification are combined, a phoneme group is selected from the speech of the speaker identified by the identification unit 111, and from the selected phoneme group to the speaker's emotion. The function is to generate voice data indicating the voice of the corresponding expression.

2-14. Modification 14
The program executed by the control unit 11 of the terminal 1 is a recording medium readable by a computer device such as a magnetic recording medium such as a magnetic tape and a magnetic disk, an optical recording medium such as an optical disc, a magnetooptical recording medium, and a semiconductor memory It can be provided in a stored state. It is also possible to download this program via a communication line such as the Internet. In addition, as a control means illustrated by the control part 11 mentioned above, various apparatuses other than CPU may be applied, for example, a processor for exclusive use etc. are used.

1, 1a, 1b: terminal, 11: control unit, 110: reception unit, 111: identification unit, 112: estimation unit, 113: selection unit, 114: generation unit, 117: first acquisition unit, 118: second acquisition Unit 119 Measurement unit 12 Storage unit 121 Emotion DB 122 Speech DB 123 Expression DB 124 Phoneme DB 13 Communication unit 131 First communication unit 132 Second communication unit , 14: display unit, 15: operation unit, 16: sound collection unit, 17: sound emission unit, 18: imaging unit, 2: measuring device, 21: control unit, 22: storage unit, 23: communication unit, 28: ... Measurement unit, 3 ... communication line, 9 ... call system.

Claims (14)

A first acquisition unit that acquires an image obtained by imaging the face of the speaker a plurality of times;
A generation unit that extracts a change in the shape of the speaker's mouth from the image and generates data corresponding to the identified speaker's utterance and the speaker's estimated emotion;
An information processing apparatus having An identifying unit that identifies the utterance from the extracted change;
And an estimation unit for estimating the emotion.
The information processing apparatus according to claim 1, wherein the generation unit generates the data corresponding to the utterance specified by the identification unit and the emotion estimated by the estimation unit. A second acquisition unit configured to acquire biological information of the speaker;
The information processing apparatus according to claim 2, wherein the estimation unit estimates the emotion according to the acquired biological information. The information processing apparatus according to claim 2, wherein the estimation unit extracts an expression of the speaker from the image, and estimates the emotion according to the expression. The information according to any one of claims 1 to 4, wherein the generation unit selects a phoneme from the utterance, and generates the data indicating a voice of an expression corresponding to the emotion from the phoneme. Processing unit. The identification unit identifies at least one of the speaking speed and volume of the speaker,
The generation unit is configured to select a phoneme from the speech, and to generate the data of the velocity or volume specified by the specification unit, which is an expression corresponding to the emotion from the phoneme. The information processing apparatus according to any one of 2 to 4. The information processing apparatus according to any one of claims 1 to 6, wherein the generation unit generates the data using speaker data indicating a voice of the speaker. The information processing apparatus according to claim 7, wherein the generation unit generates the data using the speaker data indicating a voice corresponding to the estimated emotion among the voices of the speaker. . The information processing apparatus according to claim 3, wherein the generation unit generates the data using speaker data indicating a voice corresponding to biological information of the speaker among voices of the speaker. . The information processing apparatus according to any one of claims 1 to 9, wherein the generation unit generates the data when receiving a predetermined operation from the speaker. A measurement unit that measures the volume of the voice of the speaker;
The information processing apparatus according to any one of claims 1 to 10, wherein the generation unit generates the data when the volume is less than a determined threshold. A measurement unit that measures the amount of light around the speaker;
The information processing apparatus according to any one of claims 1 to 10, wherein the generation unit stops generation of the data when the light amount is less than a determined threshold. Notifying whether the generation unit generates the data;
The information processing apparatus according to any one of claims 1 to 12, characterized in that: Computer,
A first acquisition unit that acquires an image obtained by imaging the face of the speaker a plurality of times;
A generation unit that extracts a change in the shape of the speaker's mouth from the image, and generates data corresponding to the identified speaker's utterance and the speaker's estimated emotion;
Program to function as.

Images