JP2024108744A - Embedded Expression Generation System - Google Patents

Abstract

To obtain an embedded expression which properly expresses a relation between entities.SOLUTION: An embedded expression generating system 1 includes: a language understanding unit for acquiring a decoded text by inputting, to a decoding portion, a synthesized embedded expression obtained by synthesizing a user embedded expression and a user speech embedded expression obtained by inputting a first user speech text into an embedded portion, and for carrying out a machine learning of a language model having the embedded portion and the decoded portion and of the user embedded expression based on an error between a second user speech text and the decoded text; an expression acquiring unit for acquiring a topic embedded expression by inputting a topic word to the embedded portion; and a relation learning unit for, with respect to a relation graph set with an edge based on a user speech result by using a user and a topic as nodes, acquiring an embedded expression of each node by learning through a graph neural network using the user embedded expression and the topic embedded expression as a feature amount of each node.SELECTED DRAWING: Figure 1

Description

The present invention relates to an embedded expression generation system.

A system for summarizing abstract text is known. For example, the system described in Patent Document 1 uses an encoder-decoder model implemented by a recurrent neural network, in which the encoder processes input token embedding in a document, and the decoder's output tokens are processed to obtain summary token output.

JP 2021-22404 A

By using a properly constructed encoder-decoder model, it is possible to obtain embeddings of words that represent topics, places, etc. Similarly, it is also possible to obtain embeddings of people, which are a different type of entity from words. Since the embeddings are represented as real vectors, it was possible to calculate the distance between entities of the same type, such as between words and between people. However, if the relationship between people and words has not been learned in generating the embeddings of words and people, those relationships are not reflected in the embeddings of people and words, and it was not possible to calculate the distance between different types of entities, such as between people and words.

The present invention has been made in consideration of the above problems, and aims to obtain an embedding representation of an entity in which the relationships between different entities are appropriately represented.

In order to solve the above problem, an embedded expression generation system according to one aspect of the present disclosure is an embedded expression generation system that generates embedded expressions of at least a user and a topic, and a language understanding unit that learns a language model composed of an encoder-decoder model including an embedding unit and a decoding unit, in which the embedding unit outputs embedded expressions that represent the features of the input text, the decoding unit decodes the embedded expressions that include at least the output from the embedding unit, obtains a user utterance embedded expression output from the embedding unit by inputting a first user utterance text that represents the utterance content of one user out of the utterance text that represents the content of the user's utterance to the embedding unit, obtains a decoded text output from the decoding unit by inputting a composite embedded expression that combines the user utterance embedded expression and the user embedded expression that is the embedded expression of the one user to the decoding unit, and learns the language model so that an error between the second user utterance text that follows the first user utterance text in the utterance text and the decoded text is reduced. The system includes a language understanding unit that performs machine learning to adjust the model and user embedded expressions, where the user embedded expressions are initial user embedded expressions before learning or user embedded expressions in the learning process, a topic extraction unit that extracts topic words, which are phrases that represent topics in the user's utterance, from the spoken text, an embedded expression acquisition unit that inputs the topic words to a learned embedding unit and acquires topic embedded expressions output from the embedding unit, a relationship extraction unit that generates a relationship graph based on the user's utterance history and behavior history, in which at least users and topics are nodes, conversation records between users are edges that connect users, and user utterance records of topic words are edges that connect users and topics, a relationship learning unit that obtains learned embedded expressions for each node by learning a graph neural network in which each of the learned user embedded expressions and topic embedded expressions is a feature of the user and topic nodes in the relationship graph, and an embedded expression output unit that outputs the embedded expression for each node.

According to the above aspect, a language model composed of an encoder-decoder model uses a pair of a first user utterance text and a second user utterance text as teacher data, inputs the first user utterance text to an embedding unit, and inputs a composite embedding representation obtained by combining the user utterance embedding representation and the user embedding representation obtained by inputting the first user utterance text to the embedding unit into the decoding unit. The language model and the user embedding representation are machine-learned so that the error between the decoded text output from the decoding unit and the second user utterance text is reduced, thereby obtaining an embedding unit (encoder) that outputs a suitable topic embedded representation in response to the input of a topic word, and obtaining a user embedded representation that suitably reflects the user's characteristics. Then, a relationship graph is generated in which the user and the topic are nodes and edges are drawn between the nodes based on the user's utterance and behavior history, and a graph neural network is trained in which the topic embedded representation obtained by inputting the topic word into the embedding unit and the learned user embedded representation are each used as the feature of the topic word and the user, thereby obtaining a learned topic embedded representation and a user embedded representation that suitably reflect the topic word and the user's characteristics. The resulting topic embeddings and user embeddings reflect the relationships between those entities, making it possible to calculate the distance between the user and the topic.

It is possible to obtain embedding representations of entities that properly represent the relationships between different entities.

1 is a block diagram showing a functional configuration of an embedded expression generation device according to an embodiment of the present invention; FIG. 2 is a hardware block diagram of the embedded expression generating device. FIG. 2 is a diagram for explaining an outline of a process for acquiring spoken text. FIG. 2 is a diagram illustrating an example of a language model configuration and machine learning processing of the language model. FIG. 13 is a diagram illustrating an example of an embedded expression acquisition process using an embedding unit of a trained language model. FIG. 13 is a diagram illustrating an example of edge acquisition for generating a relationship graph. 1A and 1B are diagrams illustrating an example of a relationship graph and an example of extraction of positive examples and negative examples from the relationship graph. FIG. 1 is a diagram showing an example of an embedded representation of each entity obtained by learning a graph neural network that configures a relationship graph. 10 is a flowchart showing the process of an embedded-expression generating method in the embedded-expression generating device. 11 is a flowchart showing the processing contents of machine learning of a language model. FIG. 13 is a diagram showing a configuration of an embedded expression generation program.

An embodiment of an embedded expression generation system according to the present invention will be described with reference to the drawings. Where possible, identical parts will be given the same reference numerals and duplicated explanations will be omitted.

Figure 1 is a diagram showing the functional configuration of an embedded expression generation system according to this embodiment. The embedded expression generation system 1 of this embodiment is a system that generates embedded expressions for at least users and topics, and is, as an example, configured by an embedded expression generation device 10.

As shown in FIG. 1, the embedded expression generation device 10 functionally comprises a speech log acquisition unit 11, a voice recognition unit 12, a text acquisition unit 13, an emotion acquisition unit 14, a language understanding unit 15, a topic extraction unit 16, an embedded expression acquisition unit 17, a relationship extraction unit 18, a relationship learning unit 19, an embedded expression output unit 20, and a link prediction unit 21. Each of these functional units 11 to 21 may be configured in a single device as illustrated in FIG. 1, or may be distributed across multiple devices.

The block diagram shown in FIG. 1 shows functional blocks. These functional blocks (components) are realized by any combination of at least one of hardware and software. Furthermore, the method of realizing each functional block is not particularly limited. That is, each functional block may be realized using one device that is physically or logically coupled, or may be realized using two or more devices that are physically or logically separated and connected directly or indirectly (e.g., using wires, wirelessly, etc.). The functional blocks may be realized by combining the one device or the multiple devices with software.

Functions include, but are not limited to, judgement, determination, judgment, calculation, computation, processing, derivation, investigation, search, confirmation, reception, transmission, output, access, resolution, selection, selection, establishment, comparison, assumption, expectation, regard, broadcasting, notifying, communicating, forwarding, configuring, reconfiguring, allocating, mapping, and assignment. For example, a functional block (component) that performs the transmission function is called a transmitting unit or transmitter. As mentioned above, there are no particular limitations on the method of realization for either of these.

For example, the embedded expression generation device 10 in one embodiment of the present invention may function as a computer. FIG. 2 is a diagram showing an example of the hardware configuration of the embedded expression generation device 10 according to this embodiment. The embedded expression generation device 10 may be physically configured as a computer device including a processor 1001, a memory 1002, a storage 1003, a communication device 1004, an input device 1005, an output device 1006, a bus 1007, etc.

In the following description, the term "apparatus" may be interpreted as a circuit, device, unit, etc. The hardware configuration of the embedded expression generation device 10 may be configured to include one or more of the devices shown in the figure, or may be configured to exclude some of the devices.

Each function of the embedded expression generation device 10 is realized by loading a specific software (program) onto hardware such as the processor 1001 and memory 1002, causing the processor 1001 to perform calculations and control communication via the communication device 1004 and the reading and/or writing of data in the memory 1002 and storage 1003.

The processor 1001, for example, runs an operating system to control the entire computer. The processor 1001 may be configured as a central processing unit (CPU) that includes an interface with peripheral devices, a control device, an arithmetic unit, a register, etc. For example, each of the functional units 11 to 21 shown in FIG. 1 may be realized by the processor 1001.

The processor 1001 also reads out programs (program codes), software modules, and data from the storage 1003 and/or the communication device 1004 into the memory 1002, and executes various processes according to these. The programs used are those that cause a computer to execute at least some of the operations described in the above-mentioned embodiments. For example, the functional units 11 to 21 of the embedded expression generation device 10 may be realized by a control program stored in the memory 1002 and operated by the processor 1001. Although the above-mentioned various processes have been described as being executed by one processor 1001, they may be executed simultaneously or sequentially by two or more processors 1001. The processor 1001 may be implemented in one or more chips. The programs may be transmitted from a network via a telecommunications line.

The memory 1002 is a computer-readable recording medium, and may be composed of at least one of, for example, a read only memory (ROM), an erasable programmable ROM (EPROM), an electrically erasable programmable ROM (EEPROM), a random access memory (RAM), etc. The memory 1002 may also be called a register, a cache, a main memory (primary storage device), etc. The memory 1002 can store executable programs (program codes), software modules, etc. for implementing the embedded representation generation method according to one embodiment of the present invention.

Storage 1003 is a computer-readable recording medium, and may be, for example, at least one of an optical disk such as a CD-ROM (Compact Disc ROM), a hard disk drive, a flexible disk, a magneto-optical disk (e.g., a compact disk, a digital versatile disk, a Blu-ray (registered trademark) disk), a smart card, a flash memory (e.g., a card, a stick, a key drive), a floppy (registered trademark) disk, a magnetic strip, and the like. Storage 1003 may also be referred to as an auxiliary storage device. The above-mentioned storage medium may be, for example, a database, a server, or other suitable medium including memory 1002 and/or storage 1003.

The communication device 1004 is hardware (transmitting/receiving device) for communicating between computers via a wired and/or wireless network, and is also referred to as, for example, a network device, a network controller, a network card, a communication module, etc.

The input device 1005 is an input device (e.g., a keyboard, a mouse, a microphone, a switch, a button, a sensor, etc.) that accepts input from the outside. The output device 1006 is an output device (e.g., a display, a speaker, an LED lamp, etc.) that performs output to the outside. Note that the input device 1005 and the output device 1006 may be integrated into one configuration (e.g., a touch panel).

In addition, each device, such as the processor 1001 and memory 1002, is connected by a bus 1007 for communicating information. The bus 1007 may be configured as a single bus, or may be configured as different buses between the devices.

The embedded expression generation device 10 may also be configured to include hardware such as a microprocessor, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a programmable logic device (PLD), or a field programmable gate array (FPGA), and some or all of the functional blocks may be realized by the hardware. For example, the processor 1001 may be implemented by at least one of these pieces of hardware.

Next, each functional unit of the embedded expression generation device 10 will be described. The speech log acquisition unit 11 acquires a speech log that represents the content of the user's utterance. The voice recognition unit 12 converts the speech log into text when the speech log is voice. The text acquisition unit 13 acquires a speech text, which is text that represents the content of the user's utterance, based on the speech log. The emotion acquisition unit 14 acquires emotion information that represents the emotion of the user at the time the user uttered the utterance based on the voice of the utterance or the facial expression of the user, and associates the acquired emotion information with the speech text that represents the content of the utterance.

The processing operations of the speech log acquisition unit 11, the voice recognition unit 12, the text acquisition unit 13, and the emotion acquisition unit 14 will be specifically described with reference to FIG. 3. FIG. 3 is a diagram that outlines the process of acquiring speech text.

The speech log acquisition unit 11 may acquire a speech log representing the contents of the user's speech in the form of text, based on input via an input device 41, for example, a keyboard, a touch panel, or the like. The speech log acquisition unit 11 may also acquire a speech log representing the contents of the user's speech in the form of audio data, based on audio input via a microphone 42, for example.

The speech log acquired by the speech log acquisition unit 11 may be voice or text (chat) representing the contents of the user's speech in a specified virtual space. The specified virtual space may be, for example, a virtual space known as the metaverse. The user's speech may be speech by an avatar in a virtual space such as the metaverse, and the speech log acquisition unit 11 may acquire the speech log representing the speech by the avatar in the form of voice or text.

The speech recognition unit 12 converts speech into text when a speech log of speech form is acquired by the speech log acquisition unit 11. The speech recognition unit 12 may convert the speech log consisting of speech into text by any method, and may convert speech into text by, for example, well-known speech recognition technology.

The text acquisition unit 13 acquires speech text, which is text that represents the content of the user's utterance, based on the speech log. When the speech log acquisition unit 11 acquires the speech log in the form of text, the text acquisition unit 13 acquires the text that represents the speech log as the speech text. When the speech log acquisition unit 11 acquires the speech log in the form of speech, the text acquisition unit 13 acquires the speech log converted into text by the speech recognition unit 12 as the speech text. Then, the text acquisition unit 13 sends the acquired speech text t1 to the language understanding unit 15.

The emotion acquisition unit 14 acquires emotion information that represents the emotion of the user when the user speaks, for example, based on the user's speech acquired via the microphone 42 or an image representing the user's facial expression acquired via the camera 43.

The emotion acquisition unit 14 may acquire the user's emotion information from the spoken voice by any method, for example, may acquire the emotion information from the spoken voice by well-known emotion recognition technology. The emotion acquisition unit 14 may also acquire the user's emotion information from an image showing the user's facial expression by any method, for example, may acquire the emotion information from an image showing the user's facial expression by well-known facial expression recognition technology.

In addition, the source of emotion information is not limited to the user's facial expressions and spoken voice, and the emotion acquisition unit 14 may acquire the emotion information from the state of the avatar when the user speaks in the virtual space.

Emotion information includes categories such as "joy," "anger," "sadness," and "surprise," and some predetermined emotion categories such as "fun" and "calm" can be classified as positive emotions.

The emotion acquisition unit 14 associates emotion information acquired from the user's facial expression and voice when speaking with the spoken text t1 that represents the content of the utterance. Therefore, the language understanding unit 15 can acquire the spoken text t1 associated with the emotion information.

The language understanding unit 15 performs machine learning of a language model configured by an encoder-decoder model. FIG. 4 is a diagram showing an example of the configuration of a language model and a machine learning process of the language model. The language model md is an encoder-decoder model configured including a neural network, and includes an embedding unit en (encoder) and a decoding unit de (decoder).

The configuration of the language model md is not limited, but may be, for example, an encoder-decoder model composed of a pair of recurrent neural networks such as seq2seq, or may be composed of a transformer such as T5 (Text-to-Text Transfer Transformer).

The embedding unit en encodes the input text and outputs an embedded representation that represents the characteristics of the text. The decoding unit decodes the embedded representation that includes at least the output from the embedding unit en, and outputs the decoded text dt. Note that in the explanation of the input and output of the language model, what is referred to as "text" may be vector data in which text has been converted using a specified method, or may be output as vector data representing text.

The language understanding unit 15 inputs a first user utterance text representing the content of a user's utterance from among the utterance texts representing the content of the user's utterance to the embedding unit en, thereby acquiring the user utterance embedded expression output from the embedding unit en.

In the example shown in FIG. 4, the language understanding unit 15 inputs the first user utterance text ut1 (What's for dinner tonight?) from the utterance text ut ("What's for dinner tonight?", "Curry") representing the content of the utterance of user A, which is teacher data for learning the language model md, to the embedding unit en. Then, the language understanding unit 15 acquires the user utterance embedded expression ebs encoded and output by the embedding unit en.

Here, the language understanding unit 15 acquires user embedded expressions, which are embedded expressions of the user. For example, the embedded expression generation system 1 may further include a user embedded expression management unit 22. The user embedded expression management unit 22 may generate and manage initial user embedded expressions before learning. The user embedded expression management unit 22 may also manage user embedded expressions during the learning process. The user embedded expression management unit 22 may be configured as a functional unit of the embedded expression generation device 10 shown in FIG. 1, or may be configured in a separate device.

The user embedded representation is represented by a real vector. The initial user embedded representation may be a random real vector, or may be a real vector consisting of feature quantities that reflect some characteristic of the user. In the embedded representation generation system 1 of this embodiment, the method of obtaining the initial user embedded representation is not limited, and any well-known method may be used.

The language understanding unit 15 generates a composite embedded expression by combining the user utterance embedded expression and the user embedded expression that is the embedded expression of the one user. The language understanding unit 15 may generate the composite embedded expression by linking the user utterance embedded expression and the user embedded expression. In the example shown in FIG. 4, the language understanding unit 15 acquires the user embedded expression ebu of the user A from the user embedded expression management unit 22, and links the user utterance embedded expression ebs that is the embedded expression of the first user utterance text ut1 with the user embedded expression ebu of the user A to generate the composite embedded expression ebl. Then, the language understanding unit 15 inputs the composite embedded expression ebl to the decoding unit de to acquire the decoded text dt that has been decoded by the decoding unit de.

The language understanding unit 15 performs machine learning to adjust the language model and the user-embedded expression ebu so that the error between the second user utterance text following the first user utterance text in the utterance text and the decoded text is reduced. In the example shown in FIG. 4, the language understanding unit 15 adjusts the language model md and the user-embedded expression ebu so that the error between the second user utterance text ut2 (curry) following the first user utterance text ut1 in the utterance text ut ("What's for dinner tonight?" "Curry") and the decoded text dt is reduced.

The language understanding unit 15 may perform machine learning to adjust the language model md and the user-embedded expression using spoken text associated with emotional information expressing a predetermined positive emotion. As described above, the spoken text ut may be accompanied by emotional information expressing the user's emotion at the time the utterance related to the spoken text was uttered. In such a case, the language understanding unit 15 may perform machine learning to adjust the language model md and the user-embedded expression using spoken text ut associated with emotional information expressing positive emotions such as "fun" and "calm" as training data.

In this way, by using speech text associated with emotional information expressing positive emotions in machine learning, a combination of the first and second user speech texts that is likely to be expressed when the user is feeling positive emotions can be used as training data. By performing machine learning using such training data, an embedding unit and user embedding expressions that can generate topic embedding expressions that reflect the user's preferred relationship with topic words, etc. can be obtained.

The language model md, which is a model that includes a trained neural network, can be considered as a program that is loaded or referenced by a computer and causes the computer to execute specified processes and realize specified functions.

That is, the trained language model md of this embodiment is used in a computer equipped with a CPU and memory. Specifically, the computer's CPU operates in accordance with instructions from the trained language model md stored in the memory to perform calculations on input data input to the input layer of the neural network based on, for example, trained weighting coefficients (parameters) and response functions corresponding to each layer, and to output results (probabilities) from the output layer.

Referring again to FIG. 1, the topic extraction unit 16 extracts topic words, which are words that express topics in the user's utterance, from the speech text. There are no limitations on the method applied to extract topic words, and the topic extraction unit 16 can extract topic words by using well-known methods such as morphological analysis and text mining, for example.

The embedded expression acquisition unit 17 inputs topic words to the trained embedding unit and acquires topic embedded expressions output from the embedding unit. FIG. 5 is a diagram showing an example of an embedded expression acquisition process using an embedding unit of a trained language model. As shown in FIG. 5, the embedded expression acquisition unit 17 inputs topic words tp extracted by the topic extraction unit 16 to the trained embedding unit en to acquire topic embedded expressions ebt. The trained embedding unit en can output suitable topic embedded expressions that appropriately reflect the characteristics of the topic in response to the input of topic words.

The embedded expression acquisition unit 17 may further acquire a location embedded expression output from the embedding unit en by inputting a location text representing a location to the learned embedding unit en. The location text may be, for example, the name of the location and an explanatory text explaining the location. This makes it possible to obtain a location embedded expression that appropriately reflects the characteristics of the location.

The relationship extraction unit 18 generates a relationship graph with at least users and topics as nodes based on the user's speech history (speech log) and behavior history. The relationship extraction unit 18 may also generate a relationship graph that further includes locations as nodes.

The relationship extraction unit 18 extracts relationships between nodes based on the user's speech, actions, and other records, and creates edges based on the extracted relationships. In this embodiment, the relationship extraction unit 18 generates a relationship graph based on the user's speech history and action history in a specified virtual space.

Figure 6 is a diagram showing an example of edge acquisition for generating a relationship graph. As shown in Figure 6, the relationship extraction unit 18 acquires a user's speech history hs (speech log, speech text, etc.) in a virtual space such as the metaverse. The relationship extraction unit 18 extracts the results r1 of dialogue between users from the user's speech history hs, and assigns it as an edge ed1 between the nodes of the users in the relationship graph.

The relationship extraction unit 18 also extracts the user's utterance record r2 of the topic word from the user's utterance history hs, and assigns it as an edge ed2 connecting the user's node and the topic word node.

Furthermore, the relationship extraction unit 18 acquires the user's behavior history ha in the virtual space. Then, the relationship extraction unit 18 extracts the user's visit record r3 to a location from the user's behavior history ha, and assigns it as an edge ed3 connecting the node of the user and the node of the location.

The relationship learning unit 19 obtains the learned embedded representations of each node by learning a graph neural network that treats each of the learned user embedded representations and topic embedded representations as features of the user and topic nodes in the relationship graph.

In addition, for a relationship graph that further includes a location node, the relationship learning unit 19 may obtain a learned embedding representation for each node by learning the graph neural network of the relationship graph, using the location embedding representation as a feature of the location node.

Specifically, the relationship learning unit 19 associates the learned user embedded expressions ebu obtained by machine learning by the language understanding unit 15 and the topic embedded expressions ebt acquired by the embedded expression acquisition unit 17 as features with each user and topic node in the relationship graph. In addition, the relationship learning unit 19 associates the location embedded expressions acquired by the embedded expression acquisition unit 17 as features with the location nodes in the relationship graph.

Then, the relationship learning unit 19 learns the graph neural network of the relationship graph in which the embedded representation is the feature of each node, thereby changing the feature and weight of each node and obtaining the learned embedded representation of each node.

The relationship learning unit 19 can learn the relationship graph using a well-known graph neural network learning method. The learning of the relationship graph will be generally described with reference to FIG. 7. FIG. 7 is a diagram showing an example of a relationship graph and an example of extracting positive examples and negative examples from the relationship graph.

The relationship graph gn illustrated in FIG. 7 includes nodes n1 to n5 that correspond to users, topics, and locations. The relationship learning unit 19 randomly samples a node of interest. In the example illustrated in FIG. 7, it is assumed that node n2 is sampled as the node of interest.

The relationship learning unit 19 extracts a positive example graph g1 and a negative example graph g2 from the relationship graph gn. The positive example graph g1 includes node n2, which is the node of interest, and nodes n1 and n5 that are connected to node n2 by edges. The negative example graph g2 includes node n2, which is the node of interest, and nodes n3 and n4 that are not connected to node n2 by edges. Note that the negative example graph g2 does not need to include all nodes that are not connected to the node of interest by edges.

Below, we will explain an example of learning the relationship graph gn, but since the learning process for graph neural networks is a well-known technique, we will explain it simply.

First, learning in the positive example graph g1 will be described. Based on the positive example graph g1, the relationship learning unit 19 extracts an adjacency matrix A in which the nodes included in the graph are represented as rows and columns, and the connection relationships via edges with the node of interest, node n2, are represented as elements.

Furthermore, the relationship learning unit 19 extracts a diagonal matrix I in which the nodes included in the graph are represented as rows and columns, and the self-loops of the nodes are represented as elements. If a real vector representing the feature of a node is represented as node feature X, the feature of each node is represented by the following formula as the sum (convolution) of the feature of the connected node represented by the adjacency matrix A and the feature of the node itself represented by the diagonal matrix I.
(A+I)・X

The relation learning unit 19 multiplies the feature amount of each convoluted node by a weight W, as expressed by the following equation, and further inputs the result to an activation function f to obtain an output H.
H (positive example) = f ((A+I)・X・W)
Then, the relationship learning unit 19 learns the weights and features so that the output H (positive example) obtained based on the positive example graph g1 becomes 1.

The relationship learning unit 19 similarly obtains an output H (negative example) based on the negative example graph g2. Then, the relationship learning unit 19 learns the weights and features so that the output H (negative example) obtained based on the negative example graph g2 becomes 0.

Referring again to FIG. 1, the embedded representation output unit 20 outputs the embedded representation of each node that has been learned by the relationship learning unit 19. FIG. 8 is a diagram showing an example of an embedded representation of each entity obtained by learning the graph neural network that configures the relationship graph. As shown in FIG. 8, the embedded representation output unit 20 outputs embedded representations EB of entities 1, 2, 3, 4, 5, ... corresponding to each node of the relationship graph gn, based on the learning gm of the graph neural network that targets the relationship graph gn by the relationship learning unit 19.

The embedded representation of each node obtained in this way appropriately reflects the characteristics of each entity corresponding to each node, and is a real vector that reflects the relationships between the entities, making it possible to calculate the distance between the entities.
Thus, where each node in the relationship graph corresponds to a different type of entity, such as a user, topic, and place, it becomes possible to calculate the distance between entities of different types.

The manner in which the embedded expression is output by the embedded expression output unit 20 is not limited, and may be storage in a specified storage means, transmission to a specified device, display on a specified display device, etc.

Referring again to FIG. 1, the link prediction unit 21 calculates the distance between nodes based on the learned embedded representation of each node, and calculates link prediction information indicating the possibility of an edge being established between each node based on the calculated distance between the nodes.

Specifically, the link prediction unit 21 determines whether the distance between nodes, calculated as the distance between real vectors, is equal to or less than a given threshold. If the link prediction unit 21 determines that the distance between nodes is equal to or less than the threshold, it outputs link prediction information indicating that an edge is predicted to exist between the nodes.

In this way, by learning gm of the graph neural network on the relationship graph gn, an embedded representation expressed by a real vector is obtained that allows the distance between different types of entities to be calculated, and link prediction information is calculated that allows the evaluation of the possibility that an edge will be established between each node of the graph. Therefore, it becomes possible to predict that there is a certain degree of relationship between the entities corresponding to each node.

In addition, based on a given threshold value for the distance between nodes, the link prediction unit 21 outputs information indicating each node whose distance between nodes is equal to or less than the threshold value as link prediction information.

Specifically, the link prediction unit 21 determines whether the distance between nodes, calculated as the distance between real vectors, is equal to or less than a given threshold, and outputs information indicating an entity corresponding to a node whose distance is determined to be equal to or less than the threshold as link prediction information. If at least one of the entities corresponding to a node whose distance is determined to be equal to or less than the threshold is a user, information indicating the other entity may be provided to the user as recommendation information.

Figure 9 is a flowchart showing the processing steps of the embedded expression generation method in the embedded expression generation device 10.

In step S1, the text acquisition unit 13 acquires speech text, which is text that represents the content of the user's utterance, based on the speech log.

In step S2, the language understanding unit 15 performs machine learning of a language model configured by an encoder-decoder model. The processing content of step S2 will be described with reference to FIG. 10.

Figure 10 is a flowchart showing the process of machine learning for a language model. In step S21, the language understanding unit 15 inputs a first user utterance text representing the utterance content of one user from among the utterance texts to the embedding unit en.

In step S22, the language understanding unit 15 acquires the user utterance embedded expression ebs encoded and output by the embedding unit en.

In step S23, the language understanding unit 15 generates a composite embedded representation ebl by combining the user utterance embedded representation and the user embedded representation, which is the embedded representation of the particular user. Then, the language understanding unit 15 inputs the composite embedded representation ebl to the decoding unit de.

In step S24, the language understanding unit 15 obtains the decoded text dt decoded by the decoding unit de.

In step S25, the language understanding unit 15 performs machine learning to adjust the language model and the user-embedded expressions so as to reduce the error between the second user-uttered text that follows the first user-uttered text in the utterance text and the decoded text.

In step S26, the language understanding unit 15 determines whether or not to end machine learning of the language model. If it is determined that machine learning of the language model is to end, the process proceeds to step S27. On the other hand, if it is not determined that machine learning of the language model is to end, the processes of steps S21 to S25 are repeated using the spoken text (the first and second user spoken text) as training data.

In step S27, the language understanding unit 15 outputs the trained language model and the user-embedded expressions. For example, the language understanding unit 15 may store the trained language model in a predetermined storage means. The language understanding unit 15 may also store the trained user-embedded expressions in a predetermined storage means, or may have the user-embedded expressions managed by the user-embedded expression management unit 22.

Referring again to FIG. 9, in step S3, the topic extraction unit 16 extracts topic words, which are words that represent topics in the user's utterance, from the speech text.

In step S4, the embedded expression acquisition unit 17 inputs the topic word to the learned embedding unit en, and acquires the topic embedded expression output from the embedding unit en. Here, the embedded expression acquisition unit 17 may further acquire the location embedded expression output from the embedding unit en by inputting a location text representing a location to the learned embedding unit en.

In step S5, the relationship extraction unit 18 generates a relationship graph in which at least users and topics are nodes based on the user's speech history (speech log) and behavior history. The relationship extraction unit 18 may also generate a relationship graph that further includes locations as nodes.

In step S6, the relationship learning unit 19 performs learning of a graph neural network in which each of the learned user-embedded expressions and topic-embedded expressions is treated as a feature of the user and topic nodes in the relationship graph. The relationship graph used for learning may further include locations as nodes, and the location-embedded expressions may be treated as a feature of the location nodes.

In step S7, the relationship learning unit 19 learns the graph neural network of the relationship graph in which the embedded representation is the feature of each node, thereby changing the feature and weight of each node and obtaining the learned embedded representation of each node.

In step S8, the embedded representation output unit 20 outputs the embedded representation of each node that has been learned by the relation learning unit 19.

Next, referring to FIG. 11, an embedded expression generation program for causing a computer to function as the embedded expression generation device 10 of this embodiment will be described. FIG. 11 is a diagram showing the configuration of the embedded expression generation program. The embedded expression generation program P1 is configured to include a main module m10 that controls the embedded expression generation process in the embedded expression generation device 10 in an overall manner, an utterance log acquisition module m11, a voice recognition module m12, a text acquisition module m13, an emotion acquisition module m14, a language understanding module m15, a topic extraction module m16, an embedded expression acquisition module m17, a relationship extraction module m18, a relationship learning module m19, an embedded expression output module m20, and a link prediction module m21. Each of the modules m11 to m21 realizes a function for each of the functional units 11 to 21.

The embedded expression generation program P1 may be transmitted via a transmission medium such as a communication line, or may be stored in a recording medium M1 as shown in FIG. 11.

According to the above-described embodiment of the embedded expression generation device 10, the embedded expression generation method, and the embedded expression generation program P1, the language model configured by the encoder-decoder model uses a pair of a first user utterance text and a second user utterance text as teacher data, inputs the first user utterance text to the embedding unit, and inputs a composite embedded expression obtained by combining the user utterance embedded expression and the user embedded expression into the decoding unit. The language model and the user embedded expression are machine-learned so that the error between the decoded text output from the decoding unit and the second user utterance text is reduced, thereby obtaining an embedding unit (encoder) that outputs a suitable topic embedded expression in response to the input of a topic word, and obtaining a user embedded expression that suitably reflects the user's characteristics. Then, a relationship graph is generated in which the user and topic are nodes and edges are drawn between the nodes based on the user's utterance and behavior history, and a graph neural network is trained in which the topic embedded expression obtained by inputting the topic word into the embedding unit and the learned user embedded expression are each used as the feature of the topic word and the user, thereby obtaining a learned topic embedded expression and a user embedded expression that suitably reflect the topic word and the user's characteristics. The resulting topic embeddings and user embeddings reflect the relationships between those entities, making it possible to calculate the distance between the user and the topic.

The invention disclosed herein can be understood, for example, as follows:

An embedded expression generation system according to a first aspect of the present disclosure is an embedded expression generation system that generates embedded expressions of at least a user and a topic, and a language understanding unit that learns a language model composed of an encoder-decoder model including an embedding unit and a decoding unit, in which the embedding unit outputs embedded expressions that represent the characteristics of the input text, the decoding unit decodes the embedded expressions that include at least the output from the embedding unit, obtains a user utterance embedded expression output from the embedding unit by inputting a first user utterance text that represents the utterance content of one user out of the utterance text that represents the content of the user's utterance to the embedding unit, obtains a decoded text output from the decoding unit by inputting a composite embedded expression that combines the user utterance embedded expression and the user embedded expression that is the embedded expression of the one user to the decoding unit, and adjusts the language model and the user utterance so that an error between the second user utterance text that follows the first user utterance text in the utterance text and the decoded text is reduced. The system includes a language understanding unit that performs machine learning to adjust user embedded expressions, the user embedded expressions being initial user embedded expressions before learning or user embedded expressions in the learning process, a topic extraction unit that extracts topic words, which are phrases that express topics in the user's utterance, from the spoken text, an embedded expression acquisition unit that inputs the topic words to a learned embedding unit and acquires topic embedded expressions output from the embedding unit, a relationship extraction unit that generates a relationship graph based on the user's utterance history and behavior history, in which at least users and topics are nodes, conversation records between users are edges connecting users, and user utterance records of topic words are edges connecting users and topics, a relationship learning unit that obtains learned embedded expressions for each node by learning a graph neural network in which each of the learned user embedded expressions and topic embedded expressions is a feature of the user and topic nodes in the relationship graph, and an embedded expression output unit that outputs the embedded expression for each node.

According to the above aspect, a language model composed of an encoder-decoder model uses a pair of a first user utterance text and a second user utterance text as teacher data, inputs the first user utterance text to an embedding unit, and inputs a composite embedding representation obtained by combining the user utterance embedding representation and the user embedding representation obtained by inputting the first user utterance text to the embedding unit into the decoding unit. The language model and the user embedding representation are machine-learned so that the error between the decoded text output from the decoding unit and the second user utterance text is reduced, thereby obtaining an embedding unit (encoder) that outputs a suitable topic embedded representation in response to the input of a topic word, and obtaining a user embedded representation that suitably reflects the user's characteristics. Then, a relationship graph is generated in which the user and the topic are nodes and edges are drawn between the nodes based on the user's utterance and behavior history, and a graph neural network is trained in which the topic embedded representation obtained by inputting the topic word into the embedding unit and the learned user embedded representation are each used as the feature of the topic word and the user, thereby obtaining a learned topic embedded representation and a user embedded representation that suitably reflect the topic word and the user's characteristics. The resulting topic embeddings and user embeddings reflect the relationships between those entities, making it possible to calculate the distance between the user and the topic.

The embedded expression generation system according to the second aspect may further include an emotion acquisition unit that acquires emotion information representing the emotion of the user when the user uttered an utterance based on the voice of the utterance or the facial expression of the user, and associates the acquired emotion information with a speech text representing the content of the utterance, in the embedded expression generation system according to the first aspect, and the language understanding unit may perform machine learning to adjust the language model and the user embedded expression using the speech text associated with emotion information representing a predetermined positive emotion.

According to the above aspect, speech texts that represent utterances made by a user when the user is likely to have positive emotions are used for machine learning. Therefore, the combination of the first and second user speech texts that constitute the training data is a combination that is likely to occur when the user is having positive emotions. By performing machine learning using such training data, an embedding unit and user embedding expressions that can generate topic embedding expressions that reflect the user's preferred relationship with topic words are obtained.

In the embedded expression generation system according to the third aspect, in the embedded expression generation system according to the first or second aspect, the embedded expression acquisition unit further acquires a location embedded expression output from the embedding unit by inputting a location text representing a location to the learned embedding unit, and the relationship extraction unit generates a relationship graph based on the user's speech history and behavior history, in which at least users, topics, and locations are nodes, records of conversations between users are edges connecting users, records of utterances of topic words by users are edges connecting the users and topics, and records of visits to locations by users are edges connecting the users and locations, and the relationship learning unit may obtain a learned embedded expression for each node by learning a graph neural network in which each of the learned user embedded expressions, topic embedded expressions, and location embedded expressions is a feature of the user, topic, and location nodes in the relationship graph.

According to the above aspect, by inputting location text into the embedding section of the trained language model, a location embedding representation that appropriately reflects the features of the location is obtained. Then, a relationship graph is generated in which the nodes are users, topics, and locations, and edges are drawn between the nodes based on the user's utterance and behavior history. A graph neural network is trained in which the topic embedding representation, the location embedding representation, and the trained user embedding representation are the features of the topic word, the location, and the user, respectively, to obtain trained topic embedding representations, location embedding representations, and user embedding representations that appropriately reflect the features of the topic word, the location, and the user. The obtained topic embedding representations, location embedding representations, and user embedding representations reflect the relationships between those entities, so it is possible to calculate the distance between the user and the topic and the location.

In the embedded expression generation system according to the fourth aspect, the embedded expression generation system according to any one of the first to third aspects may further include a link prediction unit that calculates the distance between nodes based on the learned embedded expression of each node, and calculates link prediction information indicating the possibility of an edge being established between each node based on the calculated distance between the nodes.

According to the above aspect, by learning a graph neural network about a relationship graph, an embedded representation expressed by a real number vector is obtained, which allows the distance between different types of entities to be calculated, and link prediction information is calculated that allows the evaluation of the possibility that an edge will be established between each node of the graph. Therefore, it becomes possible to predict that there is a certain degree of relationship between the entities corresponding to each node.

In the embedded expression generation system according to the fifth aspect, the link prediction unit in the embedded expression generation system according to the fourth aspect may output, as link prediction information, information indicating each node whose inter-node distance is equal to or less than a threshold based on a given threshold for the inter-node distance.

According to the above aspect, it is possible to obtain information about entities that have a relationship with a predetermined degree or more based on information indicating nodes whose internode distance is equal to or less than a given threshold.

In the embedded expression generation system according to the sixth aspect, in the embedded expression generation system according to any one of the first to fifth aspects, the spoken text may be obtained based on a speech log of audio or text representing the content of a user's utterance in a specified virtual space.

According to the above aspect, voice or text representing a user's speech can be easily acquired in a virtual space, making it easy to acquire speech text.

In the embedded expression generation system according to the seventh aspect, which is an embedded expression generation system according to any one of the first to sixth aspects, the relationship extraction unit may generate a relationship graph based on a user's speech history and behavior history in a specified virtual space.

According to the above aspect, since a user's speech history and behavior history can be easily acquired in a virtual space, a relationship graph can be easily generated.

Although the present disclosure has been described in detail above, it is clear to those skilled in the art that the present disclosure is not limited to the embodiments described herein. The present disclosure can be implemented in modified and altered forms without departing from the spirit and scope of the present disclosure as defined by the claims. Therefore, the description of the present disclosure is intended as an illustrative example and does not have any limiting meaning on the present disclosure.

The notification of information is not limited to the aspects/embodiments described in the present disclosure, and may be performed using other methods. For example, the notification of information may be performed by physical layer signaling (e.g., Downlink Control Information (DCI), Uplink Control Information (UCI)), higher layer signaling (e.g., Radio Resource Control (RRC) signaling, Medium Access Control (MAC) signaling, broadcast information (Master Information Block (MIB), System Information Block (SIB))), other signals, or a combination of these. In addition, the RRC signaling may be called an RRC message, and may be, for example, an RRC Connection Setup message, an RRC Connection Reconfiguration message, etc.

Each aspect/embodiment described herein may be applied to systems using LTE (Long Term Evolution), LTE-Advanced (LTE-A), SUPER 3G, IMT-Advanced, 4G, 5G, FRA (Future Radio Access), W-CDMA (registered trademark), GSM (registered trademark), CDMA2000, UMB (Ultra Mobile Broadband), IEEE 802.11 (Wi-Fi), IEEE 802.16 (WiMAX), IEEE 802.20, UWB (Ultra-WideBand), Bluetooth (registered trademark), or other suitable systems and/or next generation systems enhanced based on these. In addition, multiple systems may be combined (e.g., a combination of at least one of LTE and LTE-A with 5G, etc.).

The steps, sequences, flow charts, etc. of each aspect/embodiment described herein may be reordered unless inconsistent. For example, the methods described herein present elements of various steps in an example order and are not limited to the particular order presented.

Specific operations that are described as being performed by a base station in this disclosure may also be performed by its upper node in some cases. In a network consisting of one or more network nodes having a base station, it is clear that various operations performed for communication with a terminal may be performed by at least one of the base station and other network nodes other than the base station (e.g., MME or S-GW, etc., but are not limited to these). Although the above example shows a case where there is one other network node other than the base station, it may also be a combination of multiple other network nodes (e.g., MME and S-GW).

Information, etc. (see the "Information, Signals" section) can be output from a higher layer (or a lower layer) to a lower layer (or a higher layer). It may also be input and output via multiple network nodes.

The input and output information may be stored in a specific location (e.g., memory) or may be managed in a management table. The input and output information may be overwritten, updated, or added to. The output information may be deleted. The input information may be sent to another device.

The determination may be based on a value represented by one bit (0 or 1), a Boolean (true or false) value, or a numerical comparison (e.g., with a predetermined value).

Each aspect/embodiment described in this disclosure may be used alone, in combination, or switched depending on the execution. In addition, notification of specific information (e.g., notification that "X is the case") is not limited to being done explicitly, but may be done implicitly (e.g., not notifying the specific information).

Software shall be construed broadly to mean instructions, instruction sets, code, code segments, program code, programs, subprograms, software modules, applications, software applications, software packages, routines, subroutines, objects, executable files, threads of execution, procedures, functions, etc., whether referred to as software, firmware, middleware, microcode, hardware description language, or otherwise.

Software, instructions, etc. may also be transmitted and received over a transmission medium. For example, if the software is transmitted from a website, server, or other remote source using wired technologies, such as coaxial cable, fiber optic cable, twisted pair, and digital subscriber line (DSL), and/or wireless technologies, such as infrared, radio, and microwave, these wired and/or wireless technologies are included within the definition of a transmission medium.

The information, signals, etc. described in this disclosure may be represented using any of a variety of different technologies. For example, data, instructions, commands, information, signals, bits, symbols, chips, etc. that may be referred to throughout the above description may be represented by voltages, currents, electromagnetic waves, magnetic fields or magnetic particles, optical fields or photons, or any combination thereof.

In addition, terms explained in this disclosure and/or terms necessary for understanding this specification may be replaced with terms having the same or similar meanings.

As used herein, the terms "system" and "network" are used interchangeably.

In addition, the information, parameters, etc. described in this specification may be expressed as absolute values, as relative values from a predetermined value, or as corresponding other information. For example, radio resources may be indicated by an index.

The names used for the parameters described above are not intended to be limiting in any way. Furthermore, the formulas etc. using these parameters may differ from those explicitly disclosed in this disclosure. The various channels (e.g., PUCCH, PDCCH, etc.) and information elements may be identified by any suitable names, and therefore the various names assigned to these various channels and information elements are not intended to be limiting in any way.

As used in this disclosure, the terms "determining" and "determining" may encompass a wide variety of actions. "Determining" and "determining" may include, for example, judging, calculating, computing, processing, deriving, investigating, looking up, searching, inquiring (e.g., searching in a table, database, or other data structure), ascertaining, and the like. "Determining" and "determining" may also include receiving (e.g., receiving information), transmitting (e.g., sending information), input, output, accessing (e.g., accessing data in memory), and the like. Additionally, "judgment" and "decision" can include considering resolving, selecting, choosing, establishing, comparing, etc., to have been "judged" or "decided." In other words, "judgment" and "decision" can include considering some action to have been "judged" or "decided." Additionally, "judgment (decision)" can be interpreted as "assuming," "expecting," "considering," etc.

As used in this disclosure, the phrase "based on" does not mean "based only on," unless expressly stated otherwise. In other words, the phrase "based on" means both "based only on" and "based at least on."

When designations such as "first," "second," and the like are used herein, any reference to that element is not intended to generally limit the quantity or order of those elements. These designations may be used herein as a convenient way to distinguish between two or more elements. Thus, a reference to a first and a second element does not imply that only two elements may be employed therein or that the first element must precede the second element in some way.

To the extent that the terms "include," "including," and variations thereof are used herein in the specification or claims, these terms are intended to be inclusive, similar to the term "comprising." Further, the term "or" as used herein is not intended to be an exclusive or.

In this disclosure, where articles have been added through translation, such as a, an, and the in English, this disclosure may include that the nouns following these articles are in the plural form.

In this disclosure, the term "A and B are different" may mean "A and B are different from each other." The term may also mean "A and B are each different from C." Terms such as "separate" and "combined" may also be interpreted in the same way as "different."

1...embedded expression generation system, 10...embedded expression generation device, 11...speech log acquisition unit, 12...speech recognition unit, 13...text acquisition unit, 14...emotion acquisition unit, 15...language understanding unit, 16...topic extraction unit, 17...embedded expression acquisition unit, 18...relation extraction unit, 19...relation learning unit, 20...embedded expression output unit, 21...link prediction unit, 22...expression management unit, de...decoding unit, en...embedding unit, gn...relation graph, M1...recording medium, m10...main module, m11...speech log acquisition module, m12...speech recognition module, m13...text acquisition module, m14...emotion acquisition module, m15...language understanding module, m16...topic extraction module, m17...embedded expression acquisition module, m18...relation extraction module, m19...relation learning module, m20...embedded expression output module, m21...link prediction module, md...language model, P1...embedded expression generation program.

Claims (7)

An embedded expression generation system for generating embedded expressions of at least a user and a topic, comprising:
A language understanding unit that learns a language model configured by an encoder-decoder model including an embedding unit and a decoding unit,
The embedding unit outputs an embedding expression representing a feature of the input text;
The decoding unit decodes an embedded representation including at least an output from the embedding unit;
a first user utterance text representing the content of a user's utterance among the utterance texts representing the content of the utterances of the users is input to the embedding unit to obtain a user utterance embedded expression output from the embedding unit; a composite embedded expression obtained by combining the user utterance embedded expression and a user embedded expression that is an embedded expression of the one user is input to the decoding unit to obtain a decoded text output from the decoding unit; and machine learning is performed to adjust the language model and the user embedded expression so that an error between a second user utterance text following the first user utterance text in the utterance text and the decoded text is reduced;
A language understanding unit, wherein the user-embedded representation is an initial user-embedded representation before learning or a user-embedded representation in a learning process;
a topic extraction unit that extracts topic words, which are words expressing topics in the user's utterance, from the utterance text;
an embedding expression acquisition unit that inputs the topic word to the learned embedding unit and acquires the topic embedding expression output from the embedding unit;
a relationship extraction unit that generates a relationship graph based on the user's speech history and behavior history, the relationship graph being a graph in which at least users and topics are nodes, conversation records between users are edges connecting users, and the user's speech records of the topic words are edges connecting the user and the topics; and
a relationship learning unit that obtains a learned embedding representation of each node by learning a graph neural network in which the learned user embedding representation and the topic embedding representation are each set as features of the user and topic nodes in the relationship graph;
an embedding representation output unit that outputs the learned embedding representation of each node;
An embedded representation generation system comprising: an emotion acquisition unit that acquires emotion information representing an emotion of the user when the user uttered an utterance based on a voice of the utterance or a facial expression of the user, and associates the acquired emotion information with the utterance text representing the content of the utterance,
The language understanding unit performs machine learning to adjust the language model and the user-embedded expressions using the spoken text associated with emotion information expressing a predetermined positive emotion.
The embedded representation generation system of claim 1 . The embedded expression acquisition unit further acquires a location embedded expression output from the embedding unit by inputting a location text representing a location to the learned embedding unit;
the relationship extraction unit generates the relationship graph based on the user's speech history and action history, the relationship graph being a graph in which at least users, topics, and places are nodes, conversation records between users are edges connecting users, the user's speech records of the topic words are edges connecting the user and topics, and the user's visit records to places are edges connecting the user and places;
the relationship learning unit obtains a learned embedding representation for each node by learning a graph neural network in which the learned user embedding representation, the topic embedding representation, and the place embedding representation are set as features of the user, topic, and place nodes in the relationship graph;
The embedded expression generation system according to claim 1 or 2. The method further includes a link prediction unit that calculates a distance between the nodes based on the learned embedding representation of each node, and calculates link prediction information indicating a possibility that an edge will be established between each node based on the calculated distance between the nodes.
The embedded representation generation system of claim 1 . The link prediction unit outputs, based on a given threshold value regarding a distance between nodes, information indicating each node whose inter-node distance is equal to or less than the threshold value as the link prediction information.
The system of claim 4. The speech text is obtained based on a speech log of voice or text representing the content of the user's utterance in a predetermined virtual space.
The embedded representation generation system of claim 1 . The relationship extraction unit generates the relationship graph based on a speech history and a behavior history of the user in a predetermined virtual space.
The embedded representation generation system of claim 1 .

Images