JP2003271532A - Communication system, data transfer method of the system, server of the system, processing program for the system and record medium - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To enable a hearer to visually and intuitively sense the emotion of a speaker inputting a text sentence. <P>SOLUTION: On receiving a text sentence from a user terminal 10-1 of the speaker connected through a network 30, a server 20 deforms a face image corresponding to the user according to the value of the emotion expressed in words of the text sentence to create a deformed face image, relates the deformed image with the words of the text sentence, transmits the related deformed image to a user terminal 10-2 as a hearer, and displays the face image related with the words where a cursor showing the word position is located together with the cursor on a display part 114 of the user terminal 10-2. <P>COPYRIGHT: (C)2003,JPO

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a communication system (chat system) for carrying out a conversation (chat) via a network, for example, and a data transfer method of the system.

[0002]

2. Description of the Related Art Generally, a chat system allows users (users) to have a pseudo conversation by transmitting and receiving texts such as letters, symbols, and numbers in real time between terminals (computers) via a network. It is a possible system. And this chat system allows multiple users to talk at the same time,
One utterance is monitored by all other users.

[0003]

However, since the data transmitted and received is only text, there is naturally a limit in transmitting the emotion of the user (speaker) who inputs the text sentence to the listener. Therefore, the conventional chat system has a drawback in that the listener's feelings are difficult to understand intuitively for the listener, and above all, the list of words is visually insufficient. The present invention has been made in view of the above circumstances, and an object thereof is to provide a communication system that visually expresses a feeling of a speaker and makes a listener intuitive. Regarding the method.

[0004]

In order to achieve the above object, in a data transfer method of a communication system according to the present invention, a server receives a text sentence from one user terminal connected via a network. Then, the first process of transforming the face image corresponding to the user of the one user terminal according to the emotional value indicated by the word of the text sentence, and the face corresponding to the user by the server. A second step of associating an image and a deformed image obtained by deforming the face image with a word string of the text sentence, and another use of the face image and the deformed image associated with the word string by the server together with the text sentence. The third step of transmitting to the user terminal and the other user terminal, together with the cursor indicating the text sentence and the word position thereof, the face associated with the word where the cursor is located. And having a fourth step of displaying the image or modified image.

According to this method, in other user terminals,
A cursor is displayed together with a text sentence transmitted from one user terminal, and an image obtained by transforming a face image associated with the word at the cursor position according to a value indicating the emotion of the word is displayed. Therefore, since the face image of the one user, that is, the speaker changes in accordance with the emotion that the word in the text sentence means as the cursor moves,
The listener can visually and intuitively understand the emotion of the speaker. Further, since the image deformation is performed by the server that receives the text sentence transmitted by one user terminal, the user of the speaker and the listener is not burdened.

[0006] Here, in the first step, the server decomposes the received text sentence into words, obtains a vector having an emotional value given by the decomposed words as a component, and asks the one user. Correspondingly, it is preferable to deform the face image by moving the feature points of the face image stored in advance by the component of the vector of the decomposed word. Further, in the fourth step, it is preferable that the other user terminal voice-synthesize and output the text sentence with the cursor position as a reading point.

In addition to the above-mentioned data transfer method, the present invention may be applied to the communication system itself, the server thereof, or a processing program executed by the server. It is possible to visually express the feelings of the speaker to the listener in order to make them intuitive.

[0008]

BEST MODE FOR CARRYING OUT THE INVENTION A communication system according to an embodiment of the present invention will be described below with reference to the drawings.

<Overall Configuration> FIG. 1 is a diagram showing the configuration of this communication system. As shown in this figure, the communication system includes a user terminal 1
0-1, 10-2, 10-3, ... And the server 20 are connected to each other via the network 30.

Here, the user terminals 10-1, 10-2,
The configuration of 10-3, ... Will be described. Since these configurations are substantially the same as each other, the reference numeral will be described as 10 unless it is necessary to specify the user terminals of the speaker and the listener as described later. Figure 2
FIG. 3 is a block diagram showing a hardware configuration of this user terminal 10.

As shown in this figure, the user terminal 10
Is a CPU 102 that executes various calculations and controls, a ROM 104 that stores basic input / output programs and the like, a RAM 106 that is used as a main memory for storing information such as programs and data, a voice database and various types used in voice synthesis. Storage unit 10 used as an external storage device for storing the application program of
8, an input unit 112 for inputting various operations of the user such as a keyboard and a mouse, a display unit 114 for providing various display screens to the user, and a unit for voice-synthesizing and outputting a received text sentence. It has a speaker 116 and a network connection unit 118 for connecting to the network 30 and transmitting / receiving data, and these units are connected to each other via a system bus B. The server 2
The hardware configuration of 0 is basically the same as that of the user terminal 10 except that the speaker 116 is unnecessary. However, since the user terminal 10 and the server 20 have different roles in the communication system, their functional configurations are also different as described below.

<Functional Structure> Next, functional blocks in the communication system will be described with reference to FIG. This functional block is constructed in software by the user terminal 10 and the server 20 executing the application program for chat, and does not necessarily exist in hardware. In addition, here, for convenience, while the user terminal 10-1 is operated by the user of the speaker,
It is assumed that the user terminal 10-2 is operated by the listener user.

First, a user terminal 10-corresponding to a speaker.
In the input unit 112 of No. 1, the user inputs a text sentence to be a chat sentence. The entered text sentence is
While being displayed on the display unit 114 for confirmation, the network connection unit 11 together with the information for identifying the sender.
8 through the transmission unit 118a corresponding to the network 30
Sent to.

On the other hand, in the server 20, the receiving section 21
8b receives the text sentence from the speaker user terminal 10-1 and the information for identifying the speaker who is the sender. Next, the syntactic analysis unit 222 morphologically analyzes the received text sentence to decompose it into words. Word extraction unit 22
In step 4, the words decomposed by the syntax analysis unit 222 are extracted one by one in order from the beginning and are processed.

Next, in order to quantitatively handle the emotional expression indicated by a word, a vector having a plurality of emotions as elements is used for each word. The emotion dictionary 228 is a kind of database that stores a vector of words for each element component, and in the present embodiment, for example, as shown in FIG. 4, “joy”, “sadness”, and “surprise” for each word. , “Fear”, “anger”, and “dislike” are assigned in advance as numerical data. For example, for the word "don't know", the component values of "sadness" and "surprise" are assigned as "0.5", and the other component values are assigned as "0". The emotion extraction unit 226 determines whether or not the word extracted by the word extraction unit 224 is registered in the emotion dictionary 228, and if registered, sequentially extracts the vector component values.

A face image database (DB) 232 stores (for each user) a face image (data) indicating the user of this communication system (or sufficient to identify the user). The emotion dictionary 228 and the face image DB 232 correspond to a storage unit in the server 20.

The image transforming unit 230, whose detailed operation will be described later, displays a face image corresponding to the sender of the text sentence.
The face image is read from the face image DB 232, and the face image is associated with the extracted word and
It is transformed according to the vector component value extracted in step 6 and temporarily stored. The image creation unit 234, the face image read from the face image DB 232 corresponding to the extracted word,
Alternatively, an image file is created from the face image transformed by the image transforming unit 230. Table creation unit 236
Is a correspondence table that shows the correspondence relationship between the words extracted by the word extracting unit 224 and the image files created by the image creating unit 234. Specifically, at least the words in the text sentence are associated with which image file. Create a correspondence table that shows whether or not it exists. A specific example of the image file created by the image creating unit 234 and the correspondence table created by the table creating unit 236 will be described later.

The transmitting unit 218a uses the text sentence received by the receiving unit 218b, the correspondence table created by the table creating unit 236, and the image file created by the image creating unit 234 as a listener of the user terminal. Send to 10-2.

The text sentence, the table and the image file are received by the receiving unit 118b of the user terminal 10-2 corresponding to the listener. The receiving unit 118b
Corresponds to the network connection unit 118 described above. The cursor position control unit 122 outputs the coordinate information of the position to be indicated by the cursor in the text sentence received by the receiving unit 118b. The voice synthesis unit 124 performs voice synthesis on the text sentence received by the reception unit 118 b according to the coordinate information indicated by the cursor position control unit 122, and outputs it to the speaker 116. here,
The voice synthesis unit 124 uses the voice data stored in the voice DB 128 corresponding to the storage unit 108 at the time of voice synthesis. Note that voice data may be registered in advance in the voice DB 128 for each user, and voice data corresponding to the sender of the text sentence may be used when performing voice synthesis.

The image instructing unit 126 refers to the correspondence table received by the receiving unit 118b and points out the image file corresponding to the coordinate information of the cursor. The image reproduction unit 127 reproduces the image file pointed out by the image instruction unit 126 among the image files received by the reception unit 118b. Then, the display unit 114 of the user terminal 10-2 displays the text sentence received by the reception unit 118b and the cursor at the coordinates indicated by the coordinate information of the cursor, and further in the vicinity thereof, the image reproduction unit. 1
The image reproduced by 27 is also displayed.

<System Operation> Next, the operation of the communication system will be described. In this communication system, data such as a text sentence is transmitted from the user terminal 10-1 corresponding to the speaker as described above.
Once received by the server 20, it is transmitted to the listener user terminal 10-2.

<Processing Operation of Server> First, the operation of the server 20 will be described. 5 and 6
Is a flowchart showing the operation of the server 20. In the server 20, first, in the receiving unit 218b, the text sentence and the information for specifying the sender are
It is determined whether or not it has been received (step S11). If it is determined that it has not been received, it is not necessary to execute the following processing, and thus the standby state is entered. When the text sentence and the information for identifying the sender are received, the image transformation unit 2
30 reads the face image of the sender specified by the information from the face image DB 232 and prepares for image transformation (step S12).

Subsequently, the syntactic analysis unit 222 morphologically analyzes the received text sentence to decompose it into words (step S13). In order to process the words decomposed by the syntax analysis unit 222 in order, the word extraction unit 224 first sets the word located at the beginning in the received text sentence as the focused word indicating the processing target (step S1).
4).

Next, the emotion extraction unit 226 determines whether or not the word of interest at the present time is a word registered in the emotion dictionary 228 (step S15). If the result of this determination is affirmative, the emotion extraction unit 226 sequentially sets the six vector components of "joy", "sadness", "surprise", "fear", "anger", and "disgust" indicating emotions. In order to process, “sadness” is first set as the vector component of the processing target.
Attention is paid to (step S16). And the emotion extraction unit 2
26 accesses the emotion dictionary 228 to acquire and output the value of the currently focused component of the vector component of the focused word (step S17).

The image transformation unit 230 determines whether or not the value of the component output from the emotion extraction unit 226 is other than zero (step S18). When the value of this component is other than zero, the image transformation unit 230 moves the characteristic portion of the face image read in the previous step S12 in the direction corresponding to the component by the distance according to the component value. , The face image is transformed and temporarily stored (step S19). on the other hand,
If the value of the component output from the emotion extraction unit 226 is zero, the face deforming process is skipped.

Next, the emotion extraction unit 226 determines whether or not the component of interest at the present time is "dislike" (step S20). If the determination result is negative, the emotion extraction unit 226 shifts the focused component by 1 in the above order (step S21), and returns the processing procedure to step S17. As a result, for example, if the focused component is “joy” that is focused first, the process shifts to the next component, “sadness”, and steps S17 and S1 are performed for this “sadness”.
8, the process of S19 is executed again. That is, the processing from steps S17 to S21 is repeatedly executed corresponding to each of the six components of "joy", "sadness", "surprise", "fear", "anger", and "disgust". Therefore, if the determination result in step S20 is affirmative, that is, if the focused component is "dislike", it means that all the component values of the above six vectors have been acquired for the focused word. . Among these, when the component value is other than zero, it also means that the face image deformed according to the component is temporarily stored.

Then, the image creating section 234 determines whether or not there are two or more component values having a value other than zero among the six vector components extracted for the target word (step S22). If the determination result is affirmative, the image creating unit 234 creates a moving image file in which these deformed images are smoothly changed by using the temporarily stored two or more deformed face images (step S23). If the determination result is negative, a still image file showing the modified face image itself is created from the temporarily stored one modified face image (step S24). On the other hand, when it is determined in step S15 that the word of interest is not registered in the emotion dictionary, the image creating unit 234 creates a still image file showing the face image itself of the user who sent the text sentence (step S25). . Next, the table creation unit 236
The moving image file or the still image file created by the image creating unit 234 is stored in association with the word of interest (step S26).

Then, the word extracting unit 224 determines whether or not another word exists next to the word of interest (step S).
27). If the determination result is negative, the processing step is performed in step S15 in order to shift the focused word to the next position (step S28) and similarly create a moving image file or a still image file for the shifted focused word. Return to. On the other hand, when the result of this determination is positive, it means that a moving image file or a still image file has been created for the last word located at the end of the sentence as a result of processing the decomposed words in order, so it was received The text sentence, the correspondence table created by the table creation unit 236, and all the image files created by the image creation unit 234 are transmitted via the network 30 to the user terminal 10-2 as the listener.

<Processing Operation of Receiving Side User Terminal>
The operation of the user terminal 10-2 that receives the text sentence, the correspondence table, and the image file transmitted from the server 20 will be described. FIG. 7 is a flowchart showing the operation of this user terminal 10-2.

First, in the user terminal 10-2, the receiving unit 1
In 18b, it is determined whether or not information such as a text sentence has been received (step S51). When the result of this determination is negative, it is not necessary to execute the following processing, so the system is in a standby state. On the other hand, when the determination result is affirmative, the text of the received information is displayed on the display unit 1.
14 is displayed (step S52). Further, the received text sentence is displayed by the cursor position control unit 122.
It is used for determining the movable range of the cursor, preparing for voice synthesis by the voice synthesizer 124, and the like. here,
Preparations for voice synthesis include, for example, a process of reading voice data from the voice DB 128 in advance for voice generation, parsing a received text sentence, and intonation at the cursor position, voice quality, and pitch. And the like.

Here, the cursor position control unit 122 outputs the coordinate information of the cursor so that the cursor in the displayed text sentence gradually moves from the beginning to the end of the text sentence (step S53). The coordinate information of this cursor is the user terminal 10- that receives the text sentence.
2 is used as follows (step S5)
4). That is, firstly, in the display unit 114, the cursor is displayed at the position indicated by the coordinate information so as to overlap with the text sentence, and secondly, in the voice synthesis unit 124, the coordinate information of the received text sentence is displayed. The voice is synthesized so that the word at the position indicated by is read aloud and is sounded through the speaker 116. Thirdly, the image instruction unit 126 refers to the received correspondence table to indicate the coordinate information. The image file corresponding to the section including the position is pointed out. Further, in the image reproduction unit 127, of the received image files, the image file pointed out by the image instruction unit 126 is expanded, and the moving image or the still image according to the image file is displayed on the display unit 114 ( Step S55).

Therefore, at the user terminal 10-2 that has received the information such as the text sentence, the moving image or the still image deformed according to the emotion of the word indicated by the cursor is displayed together with the cursor indicating the text sentence and its reference position. The image is displayed on the display unit 114, and the word indicated by the cursor is voice-synthesized. That is, in synchronization with the movement of the cursor, the moving image or the still image is displayed, and the synthesized voice for reading the text sentence is output.

Then, the cursor position controller 122 determines whether or not the cursor has reached the end of the text sentence (step S56). If this determination is negative,
The processing procedure is returned to step S53. Therefore, in the processing of the above steps S53, S54, and S55, that is, the display of the moving image or the still image synchronized with the movement of the cursor and the speech synthesis of reading the text sentence are performed from the beginning to the end of the text sentence. It will be implemented for the period up to the point. On the other hand, when the cursor reaches the end of the text sentence, the process of moving the cursor, displaying the moving image or the still image, and synthesizing the voice ends.
The text sentence may be displayed continuously.

<Specific Operation> Next, the operation of such a communication system, in particular, the operation of the server 20 which is a feature of the present invention, and the user terminal 10-2 which receives a text sentence and becomes a listener. The operation will be described with a specific example. Here, as a specific example, the following text is transmitted from the user terminal 10-1 that is the speaker, and the content of the emotion dictionary is shown in FIG.
8 and the case where the face image of the user is as shown in FIG. 8A is assumed. That is, it is assumed that a text sentence "Thank you for telling me what I didn't understand yesterday. It was saved." Is transmitted from the user terminal 10-1.

In this case, the text sentence is the server 20.
For example, “yesterday / ha /
I don't understand / Place / Teach / Teach / Give / Te / Thank you /. / It was saved / Yo /. It is decomposed to be separated by "/". Of these, the words registered in the emotion dictionary are "I don't know" and "Thank you."
There are two.

When the word of interest is "I don't know",
The six component values of the vector are sequentially output by the emotion extraction unit 226, and the non-zero components are “sadness” and “surprise”. Therefore, the image transforming unit 230 sets the feature points corresponding to “sadness” among the feature points indicated by the circles in FIG. 8A as component values (= 0.
5) is deformed by a distance corresponding to 5) to create the image shown in FIG. 8B, and among the feature points in FIG.
The feature point corresponding to "surprise" is transformed by the distance corresponding to the component value (= 0.5) to create the image shown in FIG. Then, the image creating unit 234 creates a moving image (moving image file) for these words “I don't know” using these two modified images. For example, a moving image that gradually changes from one side of the deformed image to the other may be created, or an image in which the display frames of the deformed image are distributed according to the component values is created as the moving image here. Is also good.

When the word of interest is "thank you", the six component values of the vector are output in order, but the only non-zero component is "joy". Therefore, the image transforming unit 230 determines the feature value corresponding to "joy" among the feature points indicated by a circle in FIG.
The image shown in FIG. 8D is created by deforming by a distance corresponding to 1.0). Then, the image creation unit 234
The word "thank you" is created as a still image file showing this one modified image.

In the above sentence example, words other than "I don't know" and "Thank you" are not registered in the emotion dictionary 228, so the image creating section 234 makes the words of the face image shown in FIG. That is, a still image file of a face image corresponding to the sender of the above sentence example is created.

Therefore, in the case of the above sentence example, the correspondence table created by the table creation unit 236 is
For example, it becomes as shown in FIG. That is, in the section a corresponding to "I don't know" in the above sentence example,
A moving image file created from a transformed image of two components (its name is "moving image 1.mpg" for convenience)
In the section b corresponding to “Thank you”, a still image file created from a deformed image by one component (the name is referred to as “still image 2.jpg” for convenience) corresponds to In section c, a still image file of a face image corresponding to the sender (its name is referred to as “still image 1.
jpg ”) is corresponding.

Therefore, such a correspondence table and image files (moving image 1.mpg, still image 1.jpg,
Still image 2. In the user terminal 10-2 that has received the jpg) together with the text sentence, as shown in FIG. 10, the image files are reproduced and displayed in time series with the movement of the cursor.

As described above, according to this embodiment, when a text sentence is input to the speaker user terminal 10-1, the listener user terminal 10-2 synchronizes with the cursor position of the text sentence. Moreover, since the image of the face is displayed according to the emotional expression of the text sentence, it is possible to visually intuitively understand the emotion of the speaker. Furthermore, since the face image is created and transformed by the server 20, the speaker need only enter a text sentence, which avoids complication of operability and configuration.

In the embodiment, the voices are simply synthesized in accordance with the cursor position, but the voices to be synthesized, the voice quality, and the volume are changed in accordance with the extracted vector. Also good. Further, it may be configured such that symbols such as “!” And “?” Are additionally displayed on the face image depending on the emotional expression.
In the embodiment, the vector indicating the emotion of the word is
The six elements of "joy", "sadness", "surprise", "fear", "anger", and "disgust" were used as components, but other elements may be used or the number of elements other than 6 may be used. good. Further, as for the face image, of course, FIG.
The facial image is not limited to the one shown in FIG. 2, and a portrait, a photographic image, or the like may be used, and so-called emoticons may be used if there is no corresponding one in the face image DB 232.

[0043]

As described above, according to the present invention, it becomes possible for the listener to visually feel the emotion of the speaker who inputs the text sentence.

[Brief description of drawings]

FIG. 1 is a diagram showing a configuration of a communication system according to an embodiment of the present invention.

FIG. 2 is a block diagram showing a configuration of a user terminal (server) of the communication system.

FIG. 3 is a block diagram showing a functional configuration of the communication system.

FIG. 4 is a diagram showing a structure of an emotion dictionary stored in a server in the communication system.

FIG. 5 is a flowchart for explaining the operation of the server in the communication system.

FIG. 6 is a flowchart for explaining the operation of the server in the communication system.

FIG. 7 is a flowchart for explaining an operation of a listener user terminal in the communication system.

FIG. 8A to FIG. 8D are diagrams for explaining the deformation operation of the face image.

FIG. 9 is a diagram showing a correspondence table of cursor positions and images created in the server.

FIG. 10 is a diagram for explaining a relationship between an image and a text.

[Explanation of symbols]

10-1, 10-2, 10-3 ... User terminal 20 ... server 30 ... Network 114 ... Display unit 122 ... Cursor position control unit 124 ... Voice synthesizer 127 ... Image playback unit 224 ... Word extraction unit 226 ... Emotion extraction unit 228 ... emotion dictionary 230 ... Image transformation unit 234 ... Image creation unit 236 ... Table creation section

Claims (7)

[Claims] 1. When a server receives a text sentence from one user terminal connected via a network,
A first process of transforming a face image corresponding to the user of the one user terminal according to an emotion value indicated by a word of the text sentence; and a face image corresponding to the user by the server, The second step of associating the deformed image obtained by deforming the face image with the word string of the text sentence, and the server, together with the text sentence, the face image and the deformed image associated with the word string to another user terminal. A third step of transmitting, and a fourth step in which the other user terminal displays, together with the text sentence and a cursor indicating the word position, a face image or a modified image associated with the word at which the cursor is located.
A data transfer method for a communication system, comprising: 2. In the first step, the server decomposes the received text sentence into words, obtains a vector having an emotional value given by the decomposed words as a component, and responds to the one user. The data transfer method of the communication system according to claim 1, wherein the feature points of the face image stored in advance are moved by the components of the vector of the decomposed word to deform the face image. 3. The communication system according to claim 2, wherein, in the fourth step, the other user terminal synthesizes and outputs the text sentence with the cursor position as a reading point. Data transfer method. 4. A communication system having a plurality of user terminals and a server connected to each other via a network, wherein the server is operated by one user of the plurality of user terminals. When the text sentence is received from the user terminal of, the image transforming unit that transforms the face image corresponding to the one user according to the emotion value indicated by the word of the text sentence, and the corresponding image transforming unit. The face image and the deformed image obtained by transforming the face image are associated with a word string of the text sentence, and a table creating unit that creates a table showing the association, and the text sentence are associated with the word string. And a transmission means for transmitting the face image and the deformed image to another user terminal, wherein the other user terminal displays the text sentence and its word position. A communication system characterized by having a display means for displaying a face image or a deformed image associated with the word in which the cursor is located together with the cursor. 5. A server connected to a network, wherein a text sentence is received from a user terminal operated by one of a plurality of user terminals connected via the network. , An image transforming unit that transforms the face image corresponding to the one user according to the emotion value indicated by the word of the text sentence, and a face image corresponding to the user and a transform that transforms the face image. And associate the image with the word string of the text sentence,
Table creating means for creating a table indicating the relation, and transmitting means for transmitting the face image and the modified image associated with the word string to another user terminal together with the text sentence. Communication system server. 6. When a server connected to a network receives a text sentence from one user terminal operated by one of a plurality of user terminals connected via the network, An image transforming unit that transforms the face image corresponding to one user according to the emotional value indicated by the word of the text sentence, a face image corresponding to the user, and a transformed image obtained by transforming the face image. , Associated with the word string of the text,
Table creating means for creating a table showing the relationship,
And a processing program for a communication system that causes the face image and the deformed image associated with the word string to function as a transmission unit that transmits the text sentence to another user terminal. 7. When a server connected to a network receives a text sentence from a user terminal operated by one of a plurality of user terminals connected via the network, An image transforming unit that transforms the face image corresponding to one user according to the emotional value indicated by the word of the text sentence, a face image corresponding to the user, and a transformed image obtained by transforming the face image. , Associated with the word string of the text,
Table creating means for creating a table showing the relationship,
And a computer-readable recording medium recording a processing program for a communication system for causing the text image and the face image and the modified image associated with the word string to function as transmitting means for transmitting to another user terminal. .