JP2024112283A - Dialogue system, dialogue control method, and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a dialogue system for conducting a dialogue with a user using a dialogue agent, configured to generate a response of the dialogue agent on the basis of verbal information of the user and non-verbal information of the user.

SOLUTION: A dialogue system for conducting a dialogue with a user using a dialogue agent includes: a first acquisition unit which acquires verbal information of the user from the dialogue; a second acquisition unit which acquires non-verbal information of the user from the dialogue; a generation unit which generates a response containing a verbal response and a non-verbal response of the dialogue agent, on the basis of the verbal information of the user and the non-verbal information of the user; and a control unit which controls the dialogue agent on the basis of the response generated by the generation unit.

SELECTED DRAWING: Figure 7

COPYRIGHT: (C)2024,JPO&INPIT

Description

The present invention relates to a dialogue system, a dialogue control method, and a program.

There are dialogue systems in which a dialogue agent automatically responds to messages from a user. There are also known agent systems that learn from dialogues with users and change attributes of the dialogue agent, such as its appearance or personality (see, for example, Patent Document 1).

Conventional technology has the problem that dialogue agents cannot generate responses based on both the user's linguistic and non-linguistic information.

One embodiment of the present invention has been made in consideration of the above problems, and in a dialogue system that uses a dialogue agent to engage in dialogue with a user, it is possible to generate the dialogue agent's response content based on the user's linguistic information and non-verbal information.

In order to solve the above problems, a dialogue system according to one embodiment is a dialogue system that uses a dialogue agent to engage in a dialogue with a user, and includes a first acquisition unit that acquires linguistic information of the user from the dialogue, a second acquisition unit that acquires non-verbal information of the user from the dialogue, a generation unit that generates response content including a verbal response and a non-verbal response of the dialogue agent based on the linguistic information and non-verbal information of the user, and a control unit that controls the dialogue agent based on the response content generated by the generation unit.

According to one embodiment of the present invention, in a dialogue system that uses a dialogue agent to dialogue with a user, it becomes possible to generate the response content of the dialogue agent based on the user's linguistic information and the user's non-verbal information.

FIG. 1 is a diagram illustrating an example of a system configuration of a dialogue system according to an embodiment. FIG. 2 illustrates an example of a dialogue agent according to an embodiment. FIG. 2 illustrates another example of a dialogue agent according to an embodiment. FIG. 1 is a diagram for explaining an overview of dialogue processing according to an embodiment. FIG. 2 is a diagram illustrating an example of a hardware configuration of a computer according to an embodiment. FIG. 2 is a diagram illustrating an example of a hardware configuration of a terminal device according to an embodiment. FIG. 2 is a diagram illustrating an example of a functional configuration of a dialogue system according to an embodiment. 1 is a flowchart illustrating an overview of an interaction process according to an embodiment. FIG. 2 is a diagram illustrating an example of a functional configuration of a generating unit according to the first embodiment. 1 is a flowchart (1) illustrating an example of an interaction process according to the first embodiment. 11 is a flowchart (2) illustrating an example of the dialogue process according to the first embodiment. 1 is a diagram for explaining an example of use of non-language information according to the first embodiment; FIG. 11 is a diagram showing an example of a transition of a dialogue scenario according to the second embodiment; FIG. 13 is a diagram showing an example of a transition of a dialogue scenario according to the second embodiment; FIG. 13 is a diagram showing an example of a dialogue screen according to the third embodiment. FIG. 13 is a diagram illustrating an example of a functional configuration of a dialogue system according to a third embodiment. 13 is a flowchart illustrating an example of an interaction process according to the third embodiment. FIG. 13 is a diagram illustrating an example of a functional configuration of a dialogue system according to a fourth embodiment. FIG. 13 is a diagram illustrating an example of a dialogue log according to the fourth embodiment. FIG. 13 is a diagram illustrating an example of a functional configuration of a dialogue system according to a fifth embodiment. 13 is a flowchart illustrating an example of a catchphrase presentation process according to the fifth embodiment. FIG. 23 is a diagram illustrating an example of a functional configuration of a dialogue system according to a sixth embodiment. FIG. 23 is a diagram illustrating an example of input/output information according to the sixth embodiment. 23 is a flowchart illustrating an example of an interaction process according to the sixth embodiment. FIG. 1 is a diagram illustrating an example of a system configuration of a usage scene 1 according to an embodiment. 11 is a flowchart illustrating an example of a dialogue start process in a usage scene 2 according to an embodiment. FIG. 1 is a diagram illustrating an example of a system configuration of a usage scene 2 according to an embodiment. 11 is a flowchart illustrating an example of a dialogue start process in a usage scene 2 according to an embodiment. FIG. 11 is a diagram illustrating an example of a system configuration of a usage scene 3 according to an embodiment. 11 is a flowchart illustrating an example of a dialogue start process in a usage scene 2 according to an embodiment.

Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.
<System Configuration>
Fig. 1 is a diagram showing an example of a system configuration of a dialogue system according to an embodiment. In the example of Fig. 1, the dialogue system 1 includes a server device 100 and a terminal device 10 connected to a communication network N, such as the Internet and a LAN (Local Area Network).

The server device 100 is, for example, an information processing device having a computer configuration, or a system consisting of multiple computers. The server device 100 provides a dialogue service in which a dialogue agent automatically responds to messages from a user 11 using a terminal device 10 by causing the computer included in the server device 100 to execute a specific program.

The terminal device 10 is an information terminal used by a user 11, such as a PC (Personal Computer), a tablet terminal, or a smartphone. The terminal device 10 is capable of communicating with the server device 100 via a communication network N. The user 11 can use the interactive service provided by the server device 100 by using the terminal device 10.

Preferably, the dialogue system 1 assists in the performance of a specified task, such as business negotiations or caregiving, through a dialogue in which the dialogue agent automatically responds to messages from the user.

The system configuration of the dialogue system 1 shown in FIG. 1 is an example. The terminal device 10 is not limited to a general-purpose information terminal, and may be, for example, a dedicated terminal device or various electronic devices. The dialogue system 1 may be realized, for example, by a single information processing device having a computer configuration. Here, the following description will be given assuming that the dialogue system 1 has the system configuration shown in FIG. 1.

(Image of a conversational agent)
A dialogue agent is a system that automatically responds to questions from a user or a customer using knowledge including registered information and knowledge, or AI (Artificial Intelligence), etc.

Examples of use cases for the dialogue agent include web conferences, web sites, smartphone apps, or unmanned AI avatars in the metaverse.

Figure 2 shows an example of an image of a dialogue agent according to an embodiment. This figure shows an example of a dialogue screen 200 for business negotiations that the server device 100 causes the terminal device 10 to display. In the example of Figure 2, a virtual human 201 generated by 3D (three-dimensional) modeling is displayed on the dialogue screen 200. Note that the virtual human 201 is an example of a dialogue agent. For example, the server device 100 controls the virtual human 201 to advance business negotiations while having a dialogue with the user 11 on this dialogue screen 200.

As a suitable example, the dialogue screen 200 for business negotiations displays a large display 202. The server device 100 can display, for example, a product proposed by the user on this display 202, and can also control the virtual human 201 to explain the product.

Figure 3 shows another example of an image of a dialogue agent according to an embodiment. This figure shows an example of a dialogue screen 300 for nursing care purposes that the server device 100 causes the terminal device 10 to display. In the example of Figure 3, the dialogue screen 300 displays another virtual human 301 generated by 3D modeling, similar to Figure 2. Note that the virtual human 301 is another example of a dialogue agent. The server device 100 controls the virtual human 301 in this dialogue screen 300 so that it communicates with, for example, elderly people living alone, in order to prevent dementia.

As a preferred example, the interaction between the user 11 and the virtual human 301 can be through text-based interaction 302 in addition to (or instead of) voice, as shown in FIG. 3.

In this way, the dialogue system 1 can change the dialogue scenario to change the dialogue content to suit various purposes, such as business negotiations, nursing care, classes, or counseling.

(Overview of interactive processing)
4 is a diagram for explaining an overview of a dialogue process according to an embodiment. This diagram, with the horizontal axis representing time, shows an example of the relationship between the linguistic information and non-linguistic information of the user 11 and the linguistic responses and non-linguistic responses of the dialogue agent in a dialogue between the user 11 and the dialogue agent.

In FIG. 4, when the user 11 performs a start operation, at time t1, the server device 100 causes the dialogue agent to make an utterance 401, such as a greeting or ice-breaker, as a verbal response, and also to perform an ice-breaker 402, such as a bow or a smile, as a non-verbal response.

In response to this, at time t2, when the user speaks, the server device 100 acquires the linguistic information and non-verbal information of the user 11. At this time, the server device 100 may cause the dialogue agent to make a non-verbal response, such as a nod 403.

Here, the linguistic information of user 11 includes, for example, information indicating the content of user 11's utterance 411, which has been converted into text using voice recognition technology. Furthermore, the non-linguistic information of user 11 includes, for example, non-linguistic information such as the facial expression, gaze, posture, or emotions of user 11, obtained using image recognition technology or the like. Furthermore, the non-linguistic information of user 11 may include, for example, non-linguistic audio information (paralanguage) such as tone of voice, speaking speed, voice pitch, voice intensity, throat clearing, sighs, laughter, or silence, obtained from the audio contained in the video of user 11. In this way, non-linguistic information such as images and audio is utilized multimodally.

Linguistic information is information that conveys the content of an utterance through words. For example, it is information that conveys meaning based on clearly defined language rules and dictionaries, such as words, grammar, sentence structure, and context. For example, linguistic information includes information indicating the content of user 11's utterance 411 that has been converted into text using voice recognition technology.

Non-verbal information is information that is transmitted through means other than words. For example, it includes non-verbal information such as the facial expression, gaze, posture, or emotions of user 11 obtained by image recognition technology or the like. Non-verbal information of user 11 also includes non-verbal audio information (paralanguage) such as tone of voice, speaking speed, voice pitch, voice intensity, throat clearing, sighs, laughter, or silence obtained from the audio contained in the video of user 11. In this way, in this embodiment, non-verbal information such as images and audio is utilized multimodally.

In addition, the server device 100 interprets the intention of the user 11's speech based on the user's linguistic information and taking into account the user's non-linguistic information. This allows the server device 100 to improve the accuracy of intent interpretation compared to interpreting intent based on linguistic information alone.

Furthermore, the server device 100 generates a response content of the dialogue agent corresponding to the intention of the user 11's speech. This response content includes a verbal response that indicates the speech content uttered by the dialogue agent, and a non-verbal response that indicates, for example, the facial expression or gestures of the dialogue agent. Preferably, the server device 100 changes the non-verbal response of the dialogue agent according to the acquired non-verbal information of the user 11.

At time t3, the server device 100 controls the dialogue agent according to the generated response. For example, the server device 100 converts the generated verbal response into voice using a voice synthesis process, and causes the dialogue agent to speak 404. Preferably, the server device 100 moves the mouth of the dialogue agent (lip sync) in time with the dialogue agent's utterance 404. Furthermore, the server device 100 causes the dialogue agent to execute a non-verbal response, such as a facial expression or a gesture, according to the generated non-verbal response.

In this way, the dialogue system 1 according to this embodiment changes the response content (verbal response and non-verbal response) of the dialogue agent (virtual human 201, 301) according to non-verbal information of the user 11. Therefore, according to this embodiment, the dialogue system 1 that uses the dialogue agent to dialogue with the user can react more appropriately to the user 11.
<Hardware Configuration>
(Computer hardware configuration)
The server device 100 has, for example, a hardware configuration of a computer 500 as shown in Fig. 5. Alternatively, the server device 100 is composed of a plurality of computers 500. Furthermore, the terminal device 10 may have, for example, a hardware configuration of a computer 500 as shown in Fig. 5.

Figure 5 is a diagram showing an example of the hardware configuration of a computer according to an embodiment. For example, as shown in Figure 5, the computer 500 includes a CPU (Central Processing Unit) 501, a ROM (Read Only Memory) 502, a RAM (Random Access Memory) 503, a HD (Hard Disk) 504, a HDD (Hard Disk Drive) controller 505, a display 506, an external device connection I/F (Interface) 507, a network I/F 508, a keyboard 509, a pointing device 510, a DVD-RW (Digital Versatile Disk Rewritable) drive 512, a media I/F 514, and a bus line 515.

When the computer 500 is a terminal device 10, the computer 500 further includes a microphone 521, a speaker 522, an audio input/output I/F 523, a CMOS (Complementary Metal Oxide Semiconductor) sensor 524, and an image sensor I/F 525.

Of these, the CPU 501 controls the overall operation of the computer 500. The ROM 502 stores programs used to start up the computer 500, such as an IPL (Initial Program Loader). The RAM 503 is used, for example, as a work area for the CPU 501. The HD 504 stores programs such as an OS (Operating System), applications, and device drivers, as well as various data. The HDD controller 505 controls the reading and writing of various data from the HD 504, for example, under the control of the CPU 501. The HD 504 and the HDD controller 505 are examples of storage devices.

The display 506 displays various information such as a cursor, a menu, a window, characters, or an image. The display 506 may be provided outside the computer 500. The external device connection I/F 507 is an interface for connecting various external devices to the computer 500. The network I/F 508 is an interface for connecting the computer 500 to the communication network 2 and communicating with other devices.

The keyboard 509 is a type of input means having multiple keys for inputting characters, numbers, various instructions, etc. The pointing device 510 is a type of input means for selecting and executing various instructions, selecting a processing target, moving a cursor, etc. Note that the keyboard 509 and the pointing device 510 may be provided outside the computer 500.

The DVD-RW drive 512 controls the reading and writing of various data from the DVD-RW 511, which is an example of a removable recording medium. Note that the DVD-RW 511 is not limited to a DVD-RW, and may be another removable recording medium. The media I/F 514 controls the reading and writing (storing) of data from the media 513, such as a flash memory. The bus line 515 includes an address bus, a data bus, and various control signals for electrically connecting the above components.

The microphone 521 is a built-in circuit that converts sound into an electrical signal. The speaker 522 is a built-in circuit that converts the electrical signal into physical vibrations to produce sound such as music or voice. The sound input/output I/F 523 is a circuit that processes the input and output of sound signals between the microphone 521 and the speaker 522 under the control of the CPU 501.

The CMOS sensor 524 is a type of built-in imaging means that captures an image of a subject (e.g., a self-portrait) under the control of the CPU 501 to obtain image data. Note that the computer 500 may have an imaging means such as a CCD (Charge Coupled Device) sensor instead of the CMOS sensor 524. The imaging element I/F 525 is a circuit that controls the driving of the CMOS sensor 524.

(Example of Hardware Configuration of Terminal Device)
6 is a diagram illustrating an example of a hardware configuration of a terminal device according to an embodiment. Here, an example of the hardware configuration of the terminal device 10 will be described in the case where the terminal device 10 is an information terminal such as a smartphone or a tablet terminal.

In the example of FIG. 6, the terminal device 10 includes a CPU 601, a ROM 602, a RAM 603, a storage device 604, a CMOS sensor 605, an image sensor I/F 606, an acceleration/orientation sensor 607, a media I/F 609, and a GPS (Global Positioning System) receiver 610.

Of these, the CPU 601 controls the operation of the entire terminal device 10 by executing a specific program. The ROM 602 stores a program used to start up the CPU 601, such as an IPL. The RAM 603 is used as a work area for the CPU 601. The storage device 604 is a large-capacity storage device that stores the OS, programs such as apps, and various data, and is realized by, for example, an SSD (Solid State Drive) or a flash ROM.

The CMOS sensor 605 is a type of built-in imaging means that captures an image of a subject (mainly a self-portrait) under the control of the CPU 601 to obtain image data. The terminal device 10 may have an imaging means such as a CCD sensor instead of the CMOS sensor 605. The imaging element I/F 606 is a circuit that controls the driving of the CMOS sensor 605. The acceleration/direction sensor 607 is various sensors such as an electronic magnetic compass or gyrocompass that detects geomagnetism, and an acceleration sensor. The media I/F 609 controls the reading or writing (storage) of data to a media (storage media) 608 such as a flash memory. The GPS receiver 610 receives GPS signals (positioning signals) from GPS satellites.

The terminal device 10 also includes a long-distance communication circuit 611, an antenna 611a of the long-distance communication circuit 611, a CMOS sensor 612, an image sensor I/F 613, a microphone 614, a speaker 615, an audio input/output I/F 616, a display 617, an external device connection I/F 618, a short-distance communication circuit 619, an antenna 619a of the short-distance communication circuit 619, and a touch panel 620.

Of these, the long-distance communication circuit 611 is a circuit that communicates with other devices, for example, via the communication network 2. The CMOS sensor 612 is a type of built-in imaging means that captures an image of a subject and obtains image data under the control of the CPU 601. The image sensor I/F 613 is a circuit that controls the operation of the CMOS sensor 612. The microphone 614 is a built-in circuit that converts sound into an electrical signal. The speaker 615 is a built-in circuit that converts an electrical signal into physical vibrations to produce sounds such as music and voice. The sound input/output I/F 616 is a circuit that processes the input and output of sound wave signals between the microphone 614 and the speaker 615 under the control of the CPU 601.

The display 617 is a type of display means such as a liquid crystal or organic EL (Electro Luminescence) that displays an image of a subject, various icons, etc. The external device connection I/F 618 is an interface for connecting various external devices. The short-range communication circuit 619 includes a circuit for performing short-range wireless communication. The touch panel 620 is a type of input means that allows the user to operate the terminal device 10 by pressing the display 617.

The terminal device 10 also includes a bus line 621. The bus line 621 includes an address bus and a data bus for electrically connecting the components such as the CPU 601 shown in FIG. 6.

Note that the hardware configuration of the terminal device 10 shown in FIG. 6 is an example. The terminal device 10 may have other hardware configurations as long as it has a computer configuration, a communication circuit, a display, a microphone, a speaker, etc.

<Functional configuration>
FIG. 7 is a diagram illustrating an example of a functional configuration of a dialogue system according to an embodiment.

(Functional configuration of the server device)
The server device 100 realizes, for example, a functional configuration as shown in Fig. 7 by the computer 500 included in the server device 100 executing a predetermined program stored in a storage medium. In the example of Fig. 7, the server device 100 has a communication unit 701, a first acquisition unit 702, a second acquisition unit 703, a generation unit 704, a voice synthesis unit 711, a drawing unit 712, and an output unit 713. At least a part of the above functional configurations may be realized by hardware.

The server device 100 also implements a memory unit 710 using storage devices such as the HD 504 and the HDD controller 505. The memory unit 710 may also be implemented using a storage server provided outside the server device 100, a cloud service, or the like.

The communication unit 701, for example, uses the network I/F 508 to connect the server device 100 to the communication network N and executes communication processing to communicate with other devices such as the terminal device 10.

The first acquisition unit 702 executes a first acquisition process to acquire language information of the user 11 from a dialogue with the user 11 who uses the terminal device 10. For example, the first acquisition unit 702 detects speech segments from the video (moving image and audio) of the user 11 received from the terminal device 10 by the communication unit 701 using a technique such as VAD (Voice Activity Detection) to acquire the speech of the user 11. The first acquisition unit 702 also executes a speech recognition process on the acquired speech of the user 11 to convert the speech of the user 11 into text. Furthermore, the first acquisition unit 702 acquires the converted speech text of the user 11 as the language information of the user 11.

The second acquisition unit 703 executes a second acquisition process to acquire non-verbal information of the user 11 from a dialogue with the user 11 using the terminal device 10. For example, the second acquisition unit 703 acquires non-verbal information of the user 11, such as facial expressions, gaze, or emotions, by image processing from the video (moving images and audio) of the user 11 received from the terminal device 10 by the communication unit 701. The second acquisition unit 703 also acquires non-verbal information of the user 11, such as the volume of the voice, intonation, or tone of the voice, from the video (moving images and audio) of the user 11 received from the terminal device 10 by the communication unit 701.

The generation unit 704 executes a generation process to generate response contents including a verbal response (dialogue contents) of the dialogue agent and a non-verbal response (action of the dialogue agent, or paralanguage, etc.) based on the linguistic information of the user 11 acquired by the first acquisition unit 702 and the non-verbal information of the user 11 acquired by the second acquisition unit 703. For example, the generation unit 704 includes a dialogue control unit 705, an intention interpretation unit 706, and a response generation unit 707. Although not shown, a large amount of actual dialogue information (sound, image, etc.) is accumulated in the backend of the response generation unit 707, and the dialogue information is used to construct the response generation unit 707. When the response generation unit 707 is a machine learning model described later, the dialogue information is used as learning data to contribute to improving the accuracy of dialogue generation.

The dialogue control unit 705 executes dialogue control processing, including processing for inputting linguistic and non-verbal information of the user 11 and processing for outputting linguistic and non-verbal responses of the dialogue agent.

The intention interpretation unit 706 executes an intention interpretation process that interprets the intention of the user 11's utterance based on the linguistic information of the user 11 and taking into account the non-linguistic information of the user 11. For example, if the user 11 utters "That's good," it may be difficult to determine from the user 11's linguistic information (speech text) alone whether the user 11 intended it to be "good" or "unnecessary." Therefore, the intention interpretation unit 706 according to this embodiment uses not only the user 11's linguistic information (speech text) but also the user 11's non-linguistic information to interpret the intention of the user 11's utterance. This allows the intention interpretation unit 706 to improve the accuracy of the intention interpretation process.

For example, the intention interpretation unit 706 may input the linguistic information and non-linguistic information of user 11 into a machine learning model that has been trained in advance to interpret the user's intention using the linguistic information and non-linguistic information of multiple users as input data, and interpret the intention of the user's speech.

Here, machine learning refers to a technology that allows a computer to acquire human-like learning capabilities, in which the computer autonomously generates algorithms necessary for judgments such as data identification from training data that is previously loaded, and applies these to new data to make predictions. The learning method for machine learning may be any of supervised learning, unsupervised learning, semi-supervised learning, reinforcement learning, and deep learning, or may be a combination of these learning methods; any learning method for machine learning is acceptable.

The response generation unit 707 executes a response generation process to generate a response content of the dialogue agent corresponding to the speech intention of the user 11. This response content includes a verbal response that represents the speech content uttered by the dialogue agent, and a non-verbal response that represents, for example, the facial expression or gestures of the dialogue agent. Preferably, the server device 100 changes the verbal response and non-verbal response of the dialogue agent according to the acquired non-verbal information of the user 11.

For example, the response generation unit 707 changes the content of the action of the dialogue agent in response to the non-verbal information of the user 11. The response generation unit 707 also changes the timing of the action of the dialogue agent in response to the non-verbal information of the user 11.

For example, natural language processing based on rules or large-scale language models can be used to generate the response content. As an example of a large-scale language model, a sentence generation language model called GPT-3 (Generative Pre-trained Transformer 3) can be applied. In rule-based natural language processing, the response content of the dialogue agent is generated based on rules that predescribe the response content in response to the user's speech intention.

In addition, there are two types of response content progression: scenario type and slot-filling type (reference: https://goqsmile.com/function/about/).

The speech synthesis unit 711 executes speech synthesis processing to convert the linguistic response generated by the generation unit 704 into speech using speech synthesis technology.

The drawing unit 712 executes a drawing process to draw a dialogue screen depicting a dialogue agent according to the non-verbal response generated by the generation unit 704. For example, the drawing unit 712 reflects facial expressions, gaze, posture, emotions, etc., in a virtual human (dialogue agent) 201 as shown in FIG. 2 according to the non-verbal response.

Preferably, the drawing unit 712 also draws lip sync to move the dialogue agent's mouth in sync with the dialogue agent's speech.

The output unit 713 executes an output process to output a video including the voice of the dialogue agent vocalized by the voice synthesis unit 711 and the dialogue screen drawn by the drawing unit 712. For example, the output unit 713 transmits a video including the voice of the dialogue agent vocalized by the voice synthesis unit 711 and the dialogue screen drawn by the drawing unit 712 to the terminal device 10 via the communication unit 701.

The voice synthesis unit 711, the drawing unit 712, and the output unit 713 are an example of a control unit 714 that controls the dialogue agent based on the response content generated by the generation unit 704.

The storage unit 710 stores various information, data, programs, etc., such as machine learning models, rules, configuration information, and dialogue logs used by the server device 100.

(Functional configuration of terminal device)
The terminal device 10 may have any functional configuration as long as it can access the server device 100 using, for example, a web browser provided in the terminal device 10, display an interactive screen 200 such as that shown in FIG. 2, and transmit video of the user 11.

The system configuration of the dialogue system 1 shown in FIG. 7 is an example. For example, the dialogue system 1 may be configured by a single information processing device having the functional configuration of the server device 100 shown in FIG. 7. At least some of the functional configurations of the server device 100 may be included in the terminal device 10. For example, the terminal device 10 may have a first acquisition unit 702, a second acquisition unit 703, a voice synthesis unit 711, a drawing unit 712, and an output unit 713. In this case, the terminal device 10 may transmit linguistic information and non-linguistic information to the server device 100 and display a dialogue screen based on the linguistic response and non-linguistic response received from the server device 100.

<Processing flow>
Fig. 8 is a flowchart showing an outline of a dialogue process executed by a dialogue system according to an embodiment. This process shows an example of a process repeatedly executed by a dialogue system 1 having a functional configuration as shown in Fig. 7. It is assumed that at the start of the process in Fig. 8, a dialogue is already taking place between a user 11 using a terminal device 10 and a dialogue agent provided by a server device 100.

In step S801, the first acquisition unit 702 acquires user language information from the dialogue between the user 11 and the dialogue agent. For example, the first acquisition unit 702 acquires the speech of the user 11 from the video of the user 11 received from the terminal device 10 by the communication unit 701. The first acquisition unit 702 also performs a speech recognition process on the acquired speech of the user 11, and acquires the speech text (language information) of the user 11, which is a text version of the speech of the user 11.

In step S802, the second acquisition unit 703 acquires non-verbal information of the user 11 from the dialogue between the user 11 and the dialogue agent in parallel with the processing of step S801. For example, the second acquisition unit 703 acquires non-verbal information such as facial expression, gaze, or emotions of the user 11 through image processing from the video of the user 11 received by the communication unit 701 from the terminal device 10. In addition, the second acquisition unit 703 acquires non-verbal information such as the volume, intonation, or tone of the voice through audio processing from the video of the user 11 received by the communication unit 701 from the terminal device 10.

In step S803, the generation unit 704 interprets the intention of the user 11's speech based on the language information of the user 11 acquired by the first acquisition unit 702 and the non-verbal information of the user 11 acquired by the second acquisition unit 703.

In step S804, the generation unit 704 generates a verbal response and a non-verbal response that correspond to the speech intent of the user 11.

In step S805, the speech synthesis unit 711 synthesizes the speech of the dialogue agent based on the linguistic response generated by the generation unit 704.

In step S806, the drawing unit 712 draws a dialogue agent based on the non-verbal response generated by the generation unit 704 in parallel with the processing of step S8095.

In step S807, the output unit 713 outputs a dialogue screen including the spoken voice of the dialogue agent synthesized by the voice synthesis unit 711 and the dialogue agent drawn by the drawing unit 712. For example, the output unit 713 transmits the dialogue screen to the terminal device 10 using the communication unit 701.

By repeatedly executing the process of FIG. 8, the dialogue system 1 can change not only the dialogue agent's speech but also the dialogue agent's non-verbal responses based on the non-verbal information of the user 11. As a result, according to this embodiment, the dialogue system 1 that uses the dialogue agent to dialogue with the user 11 can react more appropriately to the user 11.

[First embodiment]
The dialogue system 1 according to the present embodiment can be adapted to various applications by changing the dialogue scenario. In the first embodiment, an example of dialogue processing adapted to a business negotiation application will be described.

<Functional configuration>
The dialogue system 1 according to the first embodiment has, for example, a functional configuration as shown in Fig. 7. Moreover, the generation unit 704 according to the first embodiment has, for example, a functional configuration as shown in Fig. 9.

Figure 9 is a diagram showing an example of the functional configuration of the generation unit according to the first embodiment. As shown in Figure 9, the dialogue control unit 705 of the generation unit 704 includes, for example, an input filter unit 901, a dialogue state management unit 902, and an output filter unit 903.

The input filter unit 901 has, for example, an input I/F function that accepts input of linguistic information and non-linguistic information from the user 11, a function to respond to misrecognition, and a function to detect inappropriate input. Note that the function to respond to misrecognition and the function to detect inappropriate input are optional and not essential.

The dialogue state management unit 902 has, for example, a function for recording input information, a function for storing the current negotiation stage, a function for controlling the negotiation stage, and a function for recording output information. The negotiation stage is an example of a negotiation progress level defined numerically.

The output filter unit 903 has, for example, an output I/F function that outputs the language and non-language responses of the dialogue agent, and a function for detecting inappropriate output. Note that the function for detecting inappropriate output is optional and not essential.

The intention interpretation unit 706 executes an intention interpretation process to interpret the intention of the user 11's speech based on the linguistic information and non-linguistic information of the user 11 received by the dialogue control unit 705. The intention interpretation unit 706 can estimate the intention of the user 11, for example, from the linguistic information and context of the user 11, but is more likely to be able to interpret the intention of the user 11 more accurately by taking into account the non-linguistic information of the user 11.

For example, the utterance "You're kidding, right?" by user 11 is often used as a negative response, but it can also be used when user 11 is happy and says "You're kidding, right?" when expectations are exceeded in a good way. In such a case, it is desirable for the intention interpretation unit 706 to use the non-verbal information of user 11 as a clue to more accurately interpret the intention of user 11.

For example, if the non-verbal information of the user 11 indicates that the user 11 has a high-pitched voice tone and a bright facial expression, the intention interpretation unit 706 may determine that the utterance of the user 11, "You're kidding, right?", is "positive (happily)." In this case, the generation unit 704 may, for example, set the facial expression of the dialogue agent to a smile and maintain the current dialogue scenario.

On the other hand, if the tone of the user 11's voice is low and the facial expression of the image of the user 11 is gloomy, the intention interpretation unit 706 may determine that the utterance of the user 11, "You're kidding, right?", is "negative." In this case, the generation unit 704 may, for example, reduce the gestures and hand movements of the dialogue agent and transition to a dialogue (product) scenario that includes more detailed examples (or transition to a scenario for another product).

In the business world of a business negotiation, it is difficult to reveal the emotions of the other party, so non-verbal information that is judged to be negative is important information that affects not only the progress of the negotiation but also the formation of a long-term impression that may affect the next negotiation, so it requires careful handling. For example, it is desirable to take into account the lowness of the tone and even the gloominess of the facial expression when deciding whether or not to transition to a different scenario.

The response generation unit 707 includes a plurality of dialogue scenarios 911-917 corresponding to a plurality of negotiation stages 1-7, a product recommendation unit 918, and a judgment unit 919. The product recommendation unit 918 and the judgment unit 919 may be provided outside the response generation unit 707.

The dialogue scenario 911 corresponding to the first stage is a dialogue scenario used when starting a business negotiation, and includes, for example, a greeting to start a business negotiation or a search for customer data. The dialogue scenario 912 corresponding to the second stage includes, for example, a dialogue such as exchanging business cards or small talk. The dialogue scenario 913 corresponding to the third stage includes, for example, a dialogue such as inquiring about the business content or the equipment used. The dialogue scenario 914 corresponding to the fourth stage includes, for example, a dialogue such as confirming actual needs or uncovering latent needs.

The dialogue scenario 911 corresponding to the fifth stage performs dialogues such as presenting recommended products, presenting catchy slogans that increase purchasing desire, deciding whether to postpone or end a business negotiation, etc. The dialogue scenario 912 corresponding to the sixth stage performs dialogues such as confirming delivery dates or inducing electronic contracts. The dialogue scenario 917 corresponding to the seventh stage performs interactions such as creating daily reports or generating and sending questionnaires, etc.

The dialogue state management unit 902 of the dialogue control unit 705 selects a dialogue scenario to use from a number of dialogue scenarios 911 to 917 depending on the current state of the negotiation. For example, the dialogue control unit 705 starts the negotiation from dialogue scenario 911 corresponding to the first stage, and increases the negotiation stage as the negotiation progresses. In addition, the dialogue control unit 705 decreases the negotiation stage if the user 11 is negative about the negotiation.

This allows the generation unit 704 to change the response content of the dialogue agent according to multiple pre-set negotiation stages. Note that the negotiation stage is an example of multiple pre-set dialogue stages.

The product recommendation unit 918 executes a product recommendation process for selecting a product to recommend to the user 11, for example, in the fifth stage, based on the dialogue content in the first to fourth stages. The judgment unit 919 executes a judgment process for determining, for example, in the fifth stage, based on the dialogue content in the first to fifth stages, whether to postpone the business negotiation, or whether to end the business negotiation.

Note that the number of multiple negotiation stages 1 to 7 shown in FIG. 9 is just an example, and may be any other number greater than or equal to two. Also, the dialogue content of the multiple dialogue scenarios 911 to 917 shown in FIG. 9 is just an example, and may be other content.

<Processing flow>
Fig. 10A is a flowchart showing an example of dialogue processing according to the first embodiment. This processing shows an example of dialogue processing executed by a dialogue system 1 having, for example, the functional configuration of a server device 100 as shown in Fig. 7 and the functional configuration of a generation unit 704 as shown in Fig. 9.

In step S1001, the dialogue system 1 starts a dialogue with the dialogue scenario 911 corresponding to the first stage, and determines whether or not there is customer data related to the user 11. If there is customer data, the dialogue system 1 transitions the process to step S1002. On the other hand, if there is no customer data, the dialogue system 1 transitions the process to step S1008.

Here, the processing of steps S1002 to S1005 and the processing of steps S1008 to S1011 are at similar stages of negotiation, but the dialogue scenarios used are different. For example, in the processing of steps S1002 to S1005, since the dialogue system 1 has customer data, it is desirable to use a dialogue scenario that advances negotiations based on past negotiation data. On the other hand, in the processing of steps S1008 to S1011, since the dialogue system 1 does not have customer data, it is desirable to use a dialogue scenario that carefully listens to the customer, including information necessary to create customer data. This makes it possible to prevent the dialogue agent from asking the user 11 the same questions every time.

When the process proceeds to step S1002, the dialogue system 1 performs a dialogue in the dialogue scenario 912 corresponding to the second stage, and determines whether or not business card exchange or small talk has occurred. If business card exchange or small talk has occurred, the dialogue system 1 transitions the process to step S1004. On the other hand, if business card exchange or small talk has not occurred, the dialogue system 1 ends the process (business negotiation) of FIG. 10A, for example. Preferably, the dialogue system 1 ends the business negotiation if business card exchange or small talk has not occurred even after a predetermined time has elapsed since the start of the dialogue in the dialogue scenario 912 corresponding to the second stage.

When the process proceeds to step S1003, the dialogue system 1 conducts a dialogue using the dialogue scenario 913 corresponding to the third stage, and determines whether or not, for example, the business content or the status of the equipment being used, etc., has been obtained. If the status has been obtained, the dialogue system 1 proceeds to step S1004. On the other hand, if the status has not been obtained, the dialogue system 1 ends the process (business negotiation) of FIG. 10A, for example. Preferably, the dialogue system 1 ends the business negotiation if the status has not been obtained even after a predetermined time has elapsed since the dialogue was started using the dialogue scenario 913 corresponding to the third stage.

When the process proceeds to step S1004, the dialogue system 1 conducts a dialogue using the dialogue scenario 914 corresponding to the fourth stage, and determines whether or not it has been able to hear needs, such as potential needs or predicted needs. If it has been able to hear needs, the dialogue system 1 transitions the process to step S1005. On the other hand, if it has not been able to hear needs, the dialogue system 1 returns the process to step S1003.

When the process proceeds to step S1005, the dialogue system 1 conducts a dialogue using the dialogue scenario 915 corresponding to the fifth stage, and determines whether or not the product has been proposed. If the product has been proposed, the dialogue system 1 proceeds to step S1006. On the other hand, if the product has not been proposed, the dialogue system 1 returns the process to step S1004 or step S1005.

For example, the dialogue system 1 uses the product recommendation unit 918 to select a product to propose to the user 11 based on the information acquired in steps S1003 and S1004. However, if the acquired information is insufficient and the product recommendation unit 918 cannot select a product to propose to the user 11, the dialogue system 1 returns the process to step S1004 or step S1005.

When the process proceeds to step S1006, the dialogue system 1 conducts a dialogue using the dialogue scenario 916 corresponding to the sixth stage, and determines whether or not a contract has been concluded. If a contract has been concluded, the dialogue system 1 proceeds to step S1007. On the other hand, if a contract has not been concluded, the dialogue system 1 returns the process to step S1005, for example.

When the process proceeds to step S1007, the dialogue system 1 conducts a dialogue using the dialogue scenario 917 corresponding to the seventh stage, and determines whether the business negotiations have been organized. If the business negotiations have been organized, the dialogue system 1 ends the process of FIG. 10A.

On the other hand, when the process moves from step S1001 to step S1008, the dialogue system 1 performs a dialogue in dialogue scenario 912 (for a new customer) corresponding to the second stage, and determines whether or not business card exchange or small talk has occurred. If business card exchange or small talk has occurred, the dialogue system 1 transitions the process to step S1009. On the other hand, if business card exchange or small talk has not occurred, the dialogue system 1 ends the process (business negotiation) of FIG. 10A. Preferably, the dialogue system 1 ends the business negotiation if business card exchange or small talk has not occurred even after a predetermined time has elapsed since starting the dialogue in dialogue scenario 912 (for a new customer) corresponding to the second stage.

When the process proceeds to step S1009, the dialogue system 1 conducts a dialogue using dialogue scenario 913 (for a new customer) corresponding to the third stage, and determines whether or not it has been able to hear about, for example, the business content or the status of the equipment being used. If it has been able to hear about the status, the dialogue system 1 transitions the process to step S1010. On the other hand, if it has not been able to hear about the status, the dialogue system 1 ends the process (business negotiation) in FIG. 10A. Preferably, if the dialogue system 1 has not been able to hear about the status even after a predetermined time has elapsed since starting the dialogue using dialogue scenario 913 (for a new customer) corresponding to the third stage, it ends the business negotiation.

When the process proceeds to step S1010, the dialogue system 1 conducts a dialogue using the dialogue scenario 914 (for a new customer) corresponding to the fourth stage, and determines whether or not it has been able to hear needs, such as potential needs or predicted needs. If it has been able to hear needs, the dialogue system 1 transitions the process to step S1011. On the other hand, if it has not been able to hear needs, the dialogue system 1 returns the process to step S1009.

When the process proceeds to step S1011, the dialogue system 1 conducts a dialogue using the dialogue scenario 915 (for a new customer) corresponding to the fifth stage, and determines whether or not the product has been proposed. If the product has been proposed, the dialogue system 1 transitions the process to step S1006. On the other hand, if the product has not been proposed, the dialogue system 1 returns the process to step S1010.

By the process of FIG. 10A, the dialogue system 1 can change the response content of the dialogue agent according to multiple pre-set dialogue stages.

The process in FIG. 10A is an example. For example, if a contract has not been concluded in step S1006, the dialogue system 1 may execute the processes in steps S1021 and S1022 in FIG. 10B.

FIG. 10B is a flowchart (2) showing an example of dialogue processing according to the first embodiment. If a contract has not been concluded in step S1006, the dialogue system 1 transitions the processing to step S1021.

When the process proceeds to step S1021, the dialogue system 1 determines whether the emotion analysis of the user 11 is positive. If the emotion analysis is positive, the dialogue system 1 returns the process to step S1005. On the other hand, if the emotion analysis is not positive (if it is negative), the dialogue system 1 proceeds to step S1022.

When the process proceeds to step S1022, the dialogue system 1, for example, issues a closing (or postponement) greeting and ends the process of FIG. 10B. For example, the dialogue system 1 may cause the dialogue agent to issue a closing greeting and bow.

Figure 11 is a diagram for explaining an example of the use of non-verbal information according to the first embodiment. For example, the dialogue system 1 acquires a direction vector 1101 indicating the direction in which the face of the user 11 is facing from an image 1100 of the user 11, and acquires gaze information indicating the gaze of the user 11 based on the acquired direction vector 1101 and the position 112 of the user's 11 pupils.

For example, if user 11 is interested in the product presented by the dialogue agent, user 11 tends to stare at the product displayed on the dialogue screen, and so the gaze does not fluctuate much (small variance), for example, as in gazes 1103a and 1103b. On the other hand, if user 11 is not interested in the product presented by the dialogue agent, their attention decreases, and so the gaze fluctuates (large variance), for example, as in gaze 1103c.

Therefore, for example, the dialogue system 1 may obtain gaze information representing the gaze of the user 11 after presenting a product to the user 11, and if the gaze variance is small, determine that the emotion analysis of the user 11 is positive (continue the business negotiation). Also, for example, the dialogue system 1 may obtain gaze information representing the gaze of the user 11 after presenting a product to the user 11, and if the gaze variance is large, determine that the emotion analysis of the user 11 is negative (end or postpone the business negotiation).

Note that this method is not limited to determining whether to end (or postpone) a negotiation, but may also be used, for example, to determine whether to move to a higher negotiation stage or return to a lower negotiation stage.

Second Embodiment
In the first embodiment, an example of dialogue processing corresponding to a nursing care application will be described. In the nursing care application, a dialogue scenario corresponding to reminiscence therapy can be used. Reminiscence therapy is a psychotherapy that can stabilize the mind of elderly people and improve their cognitive functions by talking about their past.

It is said that talking about fond memories using reminiscence therapy is a process in which the left brain verbalizes the images that arise in the right brain. A conversation with an introduction, development, twist and conclusion is called the "5W (When, Where, Who, What, Why) method of speaking," while a conversation that focuses on the situation and what it was like is called the "1H (How) method of speaking." It is said that talking about the situation and the scene at the time is twice as enjoyable as a story with an introduction, development, twist and conclusion.

Therefore, in the second embodiment, the dialogue system 1 uses a dialogue scenario based on the reminiscence method to create a dialogue scenario that overlaps 1H discourse in order to specifically deepen the dialogue as the dialogue progresses, and generates the response content of the dialogue agent based on that dialogue scenario.

The functional configuration of the dialogue system 1 according to the second embodiment may be the same as the functional configuration of the dialogue system 1 described in FIG. 7.

<Processing flow>
12 and 13 are diagrams showing an example of a transition of a dialogue scenario according to the second embodiment. These diagrams show an example of a transition of a dialogue scenario for reminiscence therapy. Note that since the actual transition changes depending on the utterance of the user 11, these diagrams show an example of a transition when the user 11 speaks, as shown in FIGS. 12 and 13.

For example, in state 1201, the dialogue agent utters, "Did you play any sports when you were a student?", and in state 1202, the user 11 utters, as an example, "I played sport A."

In this case, in the first step, the dialogue system 1 has the dialogue agent make an utterance that reflects on general knowledge of sport A. For example, in state 1203, the dialogue agent utters, "What position were you in?". Also, in state 1204, the user 11, as an example, utters, "I was in position B."

In this case, in the second stage, the dialogue system 1 causes the dialogue agent to make an utterance that delves deeper into the topic of sport A. For example, the dialogue agent randomly selects one state from states 1205, 1209, 1213, and 1215, and transitions to the selected state.

As an example, when transitioning to state 1205, the dialogue agent utters, "Have you ever played in a match?". Also, in state 1206, the user 11 utters, as an example, "I've played many times."

Here, as a third stage, the dialogue system 1 causes the dialogue agent to make an utterance that delves deeper into the topic of state 1205. For example, in state 1207, the dialogue agent utters, "Did you win any awards?". Also, in state 1208, as an example, the user 11 utters, "I participated in the prefectural tournament." Here, as an example, the dialogue system 1 transitions the state to state 1217.

As another example, when transitioning from state 1204 to state 1209, the dialogue agent utters, "How often did you play sport A?". In addition, in state 1210, the user 11 utters, as an example, "I played it more than three times a week."

Here, as a third stage, the dialogue system 1 causes the dialogue agent to make an utterance that delves deeper into the topic of state 1209. For example, in state 1211, the dialogue agent utters, "What did you like about sport A?". Also, in state 1212, as an example, the user 11 utters, "The fact that you can play in a team." Here, as an example, the dialogue system 1 transitions the state to state 1217.

As another example, when transitioning from state 1204 to state 1213, the dialogue agent utters "Did you like sport A?". Also, in state 1214, as an example, the user 11 utters "Yes." Here, as an example, the dialogue system 1 transitions the state to state 1211.

As another example, when transitioning from state 1204 to state 1215, the dialogue agent utters, "Do you ever watch sport A?". Also, in state 1216, as an example, the user 11 utters, "Yes." Here, as an example, the dialogue system 1 transitions the state to state 1217. In this way, the dialogue system 1 may omit the third stage of digging deeper.

When the state transitions to 1217, the dialogue agent utters, "Thank you for letting me know. I hear you're enjoying sport A. That's great.", and in state 1218, the user 11 utters, for example, "Yes."

At this point, the dialogue system 1 may, for example, end the dialogue, or may further transition the state to state 1301 in FIG. 13.

When the state transitions to 1301, the dialogue agent utters, for example, "Did you have a favorite team?". Also, in state 1302, it is assumed that the user 11 utters, for example, "I liked team C."

In this case, in the fourth stage, the dialogue system 1 causes the dialogue agent to make an utterance that delves deeper into the favorite team (or selection) in sport A. For example, in state 1303, the dialogue agent utters, "What did you like about team C?". Also, in state 1304, it is assumed that the user 11 utters, as an example, "their strengths." In this case, the dialogue system 1 causes the dialogue agent to make a closing greeting. For example, in state 1305, the dialogue agent ends the dialogue by uttering, "I see. Thank you for letting me know. Thank you for your time. I will end the dialogue." etc.

By using the transitions in Figures 12 and 13, the dialogue system 1 can have the dialogue agent converse by layering 1H speech to specifically deepen the dialogue as the dialogue progresses, using a dialogue scenario based on the reminiscence method.

[Third embodiment]
For example, in an interactive screen 300 as shown in FIG. 3, by adding auxiliary visual information in addition to the voice dialogue and the behavior of the virtual human 301, it becomes easier to deepen the dialogue in business negotiations or in nursing care.

Figure 14 is a diagram showing an example of a dialogue screen according to the third embodiment. In the example of Figure 14, in addition to a virtual human (dialogue agent) 1401 and a dialogue 302 using character strings, an illustration 1403, which is an image generated based on the dialogue content, is displayed on the dialogue screen 1400. This illustration 1403 allows the user 11 to easily visualize the image of cross-country skiing, which is the dialogue content. Note that the illustration 1403 may include sound information such as sound effects or voices separate from the dialogue content.

<Functional configuration>
Fig. 15 is a diagram showing an example of the functional configuration of the dialogue system according to the third embodiment. As shown in Fig. 15, the server device 100 according to the third embodiment has an image generation unit 1501 in addition to the functional configuration of the server device 100 described in Fig. 7.

The image generating unit 1501 is, for example, included in the generating unit 704, and executes an image generation process to generate an illustration 1403, which is an image generated based on the content of the dialogue with the user 11. For example, the image generating unit 1501 can generate the illustration 1403 using a trained machine learning model (e.g., DALL.E, DALL.E2, or Stable Diffusion, etc.) that generates images from text information. The image generating unit 1501 may also generate the illustration 1403, which is an image related to the content of the dialogue, based on at least one of the linguistic information and non-linguistic information of the user 11.

For example, the image generating unit 1501 may generate the illustration 1403 when it determines that the emotion analysis of the user 11 is "positive" based on the linguistic information of "cross-country skiing" spoken by the user 11 and the non-linguistic information of the user 11's voice being "high-pitched." This allows the dialogue system 1 to more effectively induce recollections from the user 11 and engage in dialogue.

Note that the functional configurations other than the image generation unit 1501 may be similar to the functional configurations of the dialogue system 1 according to one embodiment described in FIG. 7.

<Processing flow>
16 is a flowchart showing an example of dialogue processing according to the third embodiment. This processing shows an example of dialogue processing executed by the dialogue system 1 having the functional configuration shown in FIG.

In step S1601, the first acquisition unit 702 acquires the speech of the user 11. In addition, in step S1602, the first acquisition unit 702 executes a speech recognition process on the acquired speech of the user 11. As a result, the first acquisition unit 702 outputs language information of the user 11, which is the speech of the user 11 converted into text. Note that the processes of steps S1601 and S1602 may utilize, for example, the process of step S801 in FIG. 8.

In step S1603, the image generating unit 1501 extracts a summary, keywords, etc. from the speech of the user 11. In addition, in step S1604, the image generating unit 1501 generates an image such as the illustration 1403 described in FIG. 14 based on the extracted summary or keywords, etc.

In step S1604, the generation unit 704 generates, for example, a voice to be made to speak by the dialogue agent. Note that this process may utilize, for example, the processes of steps S803 and S804 in FIG. 8. Furthermore, when the image generation unit 1501 generates an illustration 1403 of cross-country skiing as shown in FIG. 14, the generation unit 704 may generate a voice to make the dialogue agent speak about cross-country skiing.

In step S1606, the generation unit 704 outputs the image generated by the image generation unit 1501 and the sound generated by the generation unit 704 to the interactive screen 1400. At this time, the interactive system 1 may cause the virtual human 1401 to perform an action to assist the displayed illustration 1403 (e.g., pointing with a finger, etc.).

By the processing of FIG. 16, the dialogue system 1 can display, for example, an illustration 1403, which is an image related to the dialogue content, on the dialogue screen 1400, as shown in FIG. 4.

[Fourth embodiment]
Fig. 17 is a diagram showing an example of the functional configuration of the dialogue system according to the fourth embodiment. As shown in Fig. 17, the server device 100 according to the fourth embodiment has a summarizing unit 1701 in addition to the functional configuration of the server device 100 described in Fig. 7.

The summarization unit 1701 is, for example, included in the generation unit 704, and performs a summarization process to summarize the dialogue log stored in the memory unit 710 by the dialogue control unit 705, for example, to create a report, etc.

When a dialogue takes place between the user 11 and the dialogue agent, the dialogue control unit 705 of the dialogue system 1 creates, for example, a dialogue log 1800 as shown in FIG. 18 and stores it in the memory unit 710, etc.

In the example of FIG. 18, the dialogue log 1800 includes information such as "timestamp," "speaker," "utterance text," and "file name" as items. "Timestamp" is information indicating the date and time when an utterance was made by the user 11 or the dialogue agent. "Speaker" is information indicating whether the utterance of "utterance text" was made by a user or a dialogue agent. "Utterance text" is information that converts the utterance of the user 11 or the dialogue agent into text. "File name" is information indicating the file name of the utterance of the user 11.

As shown in FIG. 18, the dialogue log 1800 is a complete record of the dialogue between the user 11 and the dialogue agent, so it is desirable to summarize it, for example, when submitting it as a report.

The summarization unit 1701 may, for example, summarize the dialogue log 1800 by applying a large-scale language model, or may summarize the dialogue log 1800 by using a cloud service that is publicly available as a text summarization AI (Artificial Intelligence).

Information that is important when summarizing includes, for example, 5W1H information such as date, time, location, user information (attributes, and whether the customer is new or existing, etc.), the issues or needs the user has, information on the proposed products, and action items or next plans. The summarizing unit 1701 summarizes the dialogue log 1800 and creates a report including this information, or minutes of the dialogue, etc.

The summarizing unit 1701 may also determine that the user 11 is interested in the product presented by the speech agent based on linguistic information such as "yes" spoken by the user 11 and non-linguistic information such as the "high tone" of the user 11's voice and the user 11's "cheerful expression." In this case, when creating the summary, the summarizing unit 1701 desirably creates the sentence so as not to omit any description of the product.

[Fifth embodiment]
Fig. 19 is a diagram showing an example of the functional configuration of the dialogue system according to the fifth embodiment. As shown in Fig. 19, the server device 100 according to the fifth embodiment has a catchphrase generation unit 1901 in addition to the functional configuration of the server device 100 described in Fig. 7.

The catch phrase generation unit 1901 executes a catch phrase generation process for generating a catch phrase to be presented to the user 11 together with a product recommendation in, for example, a dialogue scenario 915 corresponding to the fifth stage shown in FIG. 9. A catch phrase is an advertising copy or promotional copy that attracts people's attention, and in this case, is a character string for appealing to the user 11 about the product proposed to the user 11.

(Example of catchphrase 1)
As an example, assume that the outline of a product proposed to the user by the dialogue agent is a needs analysis service with the following contents.

"We support the quality and time of responses at support and call centers in industries such as retail and wholesale, food and beverages, manufacturing, information and communications, services, pharmaceuticals and cosmetics, and tourism. We also provide contextual analysis of the vast number of inquiries received from customers, helping to plan sales promotion measures and provide hints for developing new products and services."
However, in this state, it is difficult to convey the characteristics of the product to the user 11. Therefore, the catch phrase generating unit 1901 may generate, for example, the following catch phrase.
1) Support from customer service to policy planning!
AI that thoroughly analyzes customers
Alternatively, the catch phrase generating unit 1901 may generate a catch phrase such as the following, for example.
2) AI learns and analyzes accumulated customer feedback!
Providing the best possible solution in a timely manner

(Example of catchphrase 2)
As another example, assume that the outline of a product proposed to the user by the dialogue agent is a sales support service with the following contents.

"The accumulation of customer interaction history and sales know-how is dependent on individuals and is not shared within the team. When taking over, searching for scattered customer data takes time, which is inefficient. By sharing information in the sales field, which tends to be dependent on individuals, time-consuming search work can be reduced. For example, if reference information such as proposal documents for similar cases created by veterans can be shared, issues such as inconsistent document creation due to differences in skill can be resolved, and this contributes to the development of documents that lead to successful sales negotiations."
However, in this state, it is difficult to convey the characteristics of the product to the user 11. Therefore, the catch phrase generating unit 1901 may generate, for example, the following catch phrase.
3) Instant installation of what interests your customers!
AI to support successful business negotiations
Alternatively, the catch phrase generating unit 1901 may generate a catch phrase such as the following, for example.
4) AI learns personal sales styles!
AI recommends proposals based on customer interests. Such catchy slogans can be generated efficiently by using, for example, a large-scale language model. The catchy slogan generating unit 1901 may generate a catchy slogan by using a catchy slogan generating service provided by an external cloud service or the like.

<Processing flow>
Fig. 20 is a flowchart showing an example of an information provision process according to the fifth embodiment. This process shows an example of a process for generating a catchy slogan corresponding to a product to be proposed to a user 11 in a dialogue scenario 915 corresponding to the fifth stage as shown in Fig. 9.

In step S2001, the product recommendation unit 918 in FIG. 9 determines the product to be proposed to the user 11, for example, based on the content of the dialogue carried out in steps S1003 to S1004 in FIG. 10A.

In step S2002, the catchphrase generation unit 1901 in FIG. 19 acquires product data related to the determined product from the memory unit 710, etc.

In step S2003, the catch phrase generation unit 1901 uses the acquired product data to generate a catch phrase for the product determined by the product recommendation unit 918. As one example, the catch phrase generation unit 1901 may generate a catch phrase using a catch phrase generation service provided by an external cloud service or the like. As another example, the catch phrase generation unit 1901 may generate a catch phrase using a large-scale language model.

In step S2004, the dialogue system 1 presents the product proposed to the user 11 and a catchphrase for the product to the user 11. For example, the dialogue system 1 displays information about the proposed product and a catchphrase for the product on the display 202 displayed on the dialogue screen 200 as shown in FIG. 2.

The process shown in FIG. 20 is an example. For example, the product proposed to the user 11 may be a package product that combines multiple products. In this case, the catchphrase generation unit 1901 acquires product data of the multiple products in step S2002, and generates a catchphrase using the product data of the multiple products in step S2003.

The fifth embodiment enables the dialogue system 1 to communicate the value of a product to the user 11 in an easy-to-understand and concise manner.

Sixth embodiment
Fig. 21 is a diagram showing an example of the functional configuration of a dialogue system according to the sixth embodiment. As shown in Fig. 21, the server device 100 according to the sixth embodiment has (stores) a history DB (Database) 2101, input/output information of non-verbal information (hereinafter simply referred to as input/output information) 2102, etc. in the storage unit 710, in addition to the functional configuration of the server device 100 described in Fig. 7.

The past history DB2101 is a database that stores information such as the user 11's past dialogue logs, non-verbal information, and physical condition.

The input/output information 2102 includes, for example, information for determining whether the non-verbal information acquired (input) from the image and voice of the user 11 is positive or negative, as shown in FIG. 22. The input/output information 2102 also includes, for example, information indicating whether the non-verbal information expressed by the image and voice of the dialogue agent is positive or negative, as shown in FIG. 22.

This allows the intention interpretation unit 706 to easily determine whether the non-verbal information contained in the image and voice of the user 11 is positive or negative, using the input/output information 2102. Furthermore, the response generation unit 707 can use the input/output information 2102 to obtain examples of positive non-verbal information or negative non-verbal information of the dialogue agent.

Furthermore, the second acquisition unit 703 according to the sixth embodiment acquires non-verbal information (emotion system) and non-verbal information (personality system) when acquiring non-verbal information of the user 11 from a dialogue with the user 11 who uses the terminal device 10. Here, the non-verbal information (emotion system) includes non-verbal information that changes depending on the time, such as the user 11's emotions, attitude, words (strength, speed, intonation, etc.), physiological characteristics, or physical actions (gaze, facial expression, etc.). For example, the intention interpretation unit 706 can determine whether the user 11 is positive or negative based on the non-verbal information (emotion system) acquired by the second acquisition unit 703.

On the other hand, the non-verbal information (personality system) includes non-verbal information (attribute information) that does not change or changes little from time to time, such as the gender, age, physical characteristics, or appearance of the user 11. For example, the response generation unit 707 can generate a verbal response or a non-verbal response according to the user's attributes (for example, gender, age, or physique) based on the non-verbal information (personality system) acquired by the second acquisition unit 703. Note that the non-verbal information (personality system) is an example of non-verbal information indicating the attributes of the user 11.

Note that other functional configurations of the dialogue system 1 according to the sixth embodiment may be similar to the functional configuration of the dialogue system 1 described in FIG. 7.

<Processing flow>
Fig. 23 is a flowchart showing an example of a dialogue process according to the sixth embodiment. This process shows an example of a process executed by the dialogue system 1 shown in Fig. 21 after starting a dialogue between the user 11 and the dialogue agent. Note that a detailed description of the process contents similar to the outline of the dialogue process according to the embodiment explained in Fig. 8 will be omitted here.

In step S2301, the first acquisition unit 702 acquires user language information from the dialogue between the user 11 and the dialogue agent.

In steps S2302 and S2303, the second acquisition unit 703 acquires non-verbal information (emotions) and non-verbal information (personality) of the user 11 from the dialogue between the user 11 and the dialogue agent in parallel with the processing of step S2301.

In step S2304, the generation unit 704 interprets the intention of the user 11's speech based on the linguistic information acquired by the first acquisition unit 702 and the non-linguistic information (emotion-related) acquired by the second acquisition unit 703.

In step S2305, the generation unit 704 generates a linguistic response (dialogue) corresponding to the intention of the user 11's speech by referring to the non-verbal information (personality-related) acquired by the second acquisition unit 703 or the past history DB 2101. For example, the generation unit 704 determines the gender, hobbies, body type, etc. of the user 11 from the past dialogue history with the user 11 in the past history DB 2101, and generates a different linguistic response (dialogue) according to the gender, hobbies, body type, etc. of the user 11.

If there is no past history of the user 11, the generation unit 704 may, for example, detect a face area from an image of the user 11 and estimate the gender, age, etc. of the user 11 using age/gender estimation AI (Artificial Intelligence) or the like. The generation unit 704 may also estimate the body type of the user 11 from an image of the user 11 using body type estimation AI or the like. Furthermore, the generation unit 704 may determine the hobbies, etc. of the user 11 from the language information of the user 11. The generation unit stores the estimated gender, age, body type, etc. of the user 11 in the past history DB 2101.

As a specific example, assume that during a business negotiation, the generation unit 704 determines from the linguistic and non-linguistic information of the user 11 that the user 11 is a woman in her 40s whose hobby is cosmetics (hereinafter referred to as cosmetics). In this case, the generation unit 704 may determine that it is worth introducing or proposing cosmetic products for people in their 40s, and may generate a linguistic response introducing specific products, for example.

As another example, the generation unit 704 may estimate the body type of the user 11 from an image of the user 11 during a business negotiation, and compare it with the history of the user 11's past body types to track the progress of the user 11's body type or compare it with past body types. As a result, the generation unit 704 may generate a verbal response for the user 11 who has recently gained weight, for example, to recommend a product such as low-carbohydrate food ingredients or a weight management application.

As another example, the generation unit 704 may estimate the fashion sense of the user 11's clothing from an image of the user 11 during a business negotiation, and compare this with the user 11's past fashion sense of clothing. As a result, the generation unit 704 may generate a verbal response introducing specific products to the user 11 who is determined to be worthy of preferentially introducing clothing-related products.

As another example, the generation unit 704 may estimate the body type of the user 11 from an image of the user 11 during a business negotiation, and determine whether it is necessary to check the physical condition of the user 11 based on medical history information from past history, etc. As a result, the generation unit 704 may generate a verbal response to check the current physical condition of the user 11 who is determined to need to have their physical condition checked.

In step S2306, the generation unit 704 determines the paralanguage of the dialogue agent (e.g., tone of voice, speaking rate, voice pitch, voice intensity, throat clearing, sigh, laughter, silence, etc.) based on the generated linguistic response and further on the non-linguistic information of the user 11. For example, when the generation unit 704 refers to the input/output information 2102 as shown in FIG. 22 and determines that the emotion analysis of the user 11 is positive, it may acquire the positive non-linguistic information (paralanguage) of the dialogue agent from the input/output information 2102. Similarly, when the generation unit 704 refers to the input/output information 2102 as shown in FIG. 22 and determines that the emotion analysis of the user 11 is negative, it may acquire the negative non-linguistic information (paralanguage) of the dialogue agent from the input/output information 2102.

Note that the input/output information 2102 shown in FIG. 22 is an example. In the input/output information 2102, various positive non-verbal information and negative non-verbal information of the user 11, and positive non-verbal information and negative non-verbal information of the dialogue agent are registered in advance.

In step S2307, the control unit 714 synthesizes a response voice of the dialogue agent based on the linguistic response generated by the generation unit 704 and the paralanguage determined by the generation unit 704.

In addition, the server device 100 executes the processes of steps S2308 and S2309 in parallel with the processes of steps S2306 and S2307.

In step S2308, the generation unit 704 determines the facial expression, gaze, or behavior of the dialogue agent based on the non-verbal information of the user 11. For example, when the generation unit 704 refers to the input/output information 2102 as shown in FIG. 22 and determines that the emotion analysis of the user 11 is positive, the generation unit 704 acquires the positive non-verbal information (facial expression, gaze, or behavior, etc.) of the dialogue agent from the input/output information 2102. Similarly, when the generation unit 704 refers to the input/output information 2102 as shown in FIG. 22 and determines that the emotion analysis of the user 11 is negative, the generation unit 704 acquires the negative non-verbal information (facial expression, gaze, or behavior, etc.) of the dialogue agent from the input/output information 2102.

In step S2309, the generation unit 704 determines the motion of the dialogue agent based on the facial expression, gaze, or gesture of the determined dialogue agent.

As a specific example, if the generation unit 704 determines that the emotion analysis of the user 11 is positive during a business negotiation, the generation unit 704 may, for example, make the dialogue agent smile and make larger hand gestures. Also, if the generation unit 704 determines that the emotion analysis of the user 11 is negative, the generation unit 704 may, for example, make the dialogue agent have a sad face and nod, bow, etc.

In addition to positive/negative judgments, the behavior (motion) of the dialogue agent may be based on non-verbal information (personality system). For example, in the case of a positive judgment, the dialogue agent may be made to perform a motion that matches (is similar to) the user's non-verbal information (personality system), such as hand gestures or crossed arms recorded in the past history DB, or the pace and rhythm of the conversation.

In step S2310, the control unit 714 draws a dialogue agent based on the operation of the dialogue agent determined by the generation unit 704, and outputs a dialogue screen including the drawn dialogue agent and the synthesized response voice. For example, the output unit 713 transmits the dialogue screen to the terminal device 10 using the communication unit 701.

For example, by repeatedly executing the process of FIG. 8, the dialogue system 1 can react more appropriately to the user 11 based on the non-verbal information (personality-related) of the user 11 or the past history DB 2101, etc.

<Examples of usage scenarios>
Next, an example of a usage scene of the dialogue system 1 according to this embodiment will be described.

(Usage scene A)
Fig. 24 is a diagram showing an example of a system configuration of a usage scene A according to an embodiment. Usage scene A shows an example in which the terminal device 10 in Fig. 1 is a digital signage signage terminal 2400. In the example of Fig. 24, the signage terminal 2400 includes an input device 2401 such as a camera and a microphone, and a hardware configuration of a computer.

Figure 25 is a flowchart showing an example of a dialogue start process for usage scene A according to one embodiment.

In step S2501, the dialogue system 1 detects the face of the user 11 from an image captured by the input device 2401 provided in the signage terminal 2400. As a specific example, the dialogue system 1 extracts a person's facial image from the image captured by the input device 2401, and performs facial recognition on the extracted facial image. Furthermore, if the extracted facial image passes facial recognition, the dialogue system 1 determines that it has detected the face of the user 11.

In step S2502, the dialogue system 1 determines whether face detection has continued for a predetermined time. For example, the dialogue system 1 determines whether the state in which the face of the user 11 has been detected has continued for a predetermined time (e.g., 5 seconds). If face detection has continued for the predetermined time, the dialogue system 1 transitions the process to step S2503. On the other hand, if face detection has not continued for the predetermined time, the dialogue system 1 returns the process to step S2501. Note that the processes of steps S2501 and S2502 may be performed by the signage terminal 2400 or by the server device 100.

When the process proceeds to step S2503, the server device 100 determines whether there is past history for the user 11. For example, the server device 100 refers to the past history DB 2101, etc., and if there is a past dialogue log for the user 11, it determines that there is past history for the user 11. If there is past history, the server device 100 proceeds to step S2504. On the other hand, if there is no past history, the server device 100 proceeds to step S2505.

When the process proceeds to step S2504, the server device 100 determines a scenario to be used in the dialogue process from the past history of the user 11 (past dialogue logs, etc.). This allows the dialogue system 1 to prevent the same question or utterance from being repeatedly made to the same user 11.

When the process proceeds to step S2505, the server device 100 selects a standard scenario (e.g., a scenario for a new customer) as the scenario to be used for the dialogue processing.

When the process proceeds to step S2506, the dialogue system 1 executes, for example, the dialogue processing described in Figs. 1 to 23 between the dialogue system 1 and the signage terminal 2500. Through the processing in Fig. 25, the dialogue system 1 can provide a dialogue service to the user 11 using the signage terminal 2500. Furthermore, the dialogue system 1 can change the dialogue content provided to the user 11 based on the user 11's past dialogue history, etc. Note that the processing in steps S2703 to S2705 is optional and not required. For example, the dialogue system 1 may determine a scenario to be used for the dialogue during the dialogue processing in step S2506.

(Usage scene B)
Fig. 26 is a diagram showing an example of a system configuration of a usage scene B according to an embodiment. Usage scene B shows an example in which the terminal device 10 in Fig. 1 is a display terminal 2600 for the metaverse. The display terminal 2600 includes, for example, a head-mounted display or a spatial reproduction display for the metaverse, and a computer configuration. In addition, the dialogue system 1 provides a dialogue service to the user 11 by using a dialogue agent in the virtual space.

Figure 27 is a flowchart showing an example of a dialogue start process for usage scene B according to one embodiment.

In step S2701, the dialogue system 1 detects the approach of the avatar of the user 11 in the virtual space. For example, the dialogue system 1 detects whether the avatar of the user 11 has approached within a predetermined distance (e.g., 1 m) from the login information of the user 11, the coordinates of the avatar of the user 11 in the virtual space, and the coordinates of the dialogue agent.

In step S2702, the dialogue system 1 determines whether the state in which the avatar of the user 11 approaches within a predetermined distance (e.g., 1 m) continues for a predetermined time (e.g., 5 seconds). If the approach of the avatar of the user 11 continues for the predetermined time, the dialogue system 1 transitions the process to step S2703. On the other hand, if the approach of the avatar of the user 11 does not continue for the predetermined time, the dialogue system 1 returns the process to step S2701.

When the process proceeds to step S2703, the server device 100 determines whether there is past history for the user 11. For example, the server device 100 refers to the past history DB 2101, etc., and if there is a past dialogue log for the user 11, it determines that there is past history for the user 11. If there is past history, the server device 100 proceeds to step S2704. On the other hand, if there is no past history, the server device 100 proceeds to step S2705.

When the process proceeds to step S2704, the server device 100 determines a scenario to be used in the dialogue process from the past history of the user 11 (past dialogue logs, etc.). On the other hand, when the process proceeds to step S2705, the server device 100 selects a standard scenario (e.g., a scenario for a new user) as the scenario to be used in the dialogue process.

When the process proceeds to step S2706, the dialogue system 1 executes, for example, the dialogue processing described in Figs. 1 to 23 in the virtual space. Through the processing in Fig. 27, the dialogue system 1 can provide a dialogue service to the user 11 in the virtual space using the metaverse display terminal 2600.

(Usage scene C)
28 is a diagram showing an example of a system configuration of a usage scene C according to an embodiment. Usage scene C shows an example in which a user 11 uses a terminal device 10 to hold a web conference with a dialogue agent provided by a server device 100. Note that the user 11 may participate in a web conference provided by a conference server 2810 or the like outside the system, or the server device 100 may provide the web conference.

Figure 29 is a flowchart showing an example of a dialogue start process for usage scene C according to one embodiment.

In step S2901, it is assumed that the user 11 uses the terminal device 10 to participate in the same web conference as the dialogue agent provided by the dialogue system 1. For example, the user 11 uses the terminal device 10 to access a link for participating in the web conference with the dialogue agent, thereby participating in the web conference.

In step S2902, the dialogue system 1 determines whether or not a dialogue start operation by the user 11 has been accepted during the web conference. If a dialogue start operation by the user 11 has been accepted, the dialogue system 1 transitions the process to step S2903. On the other hand, if a dialogue start operation by the user 11 has not been accepted, the dialogue system 1, for example, repeatedly executes the process of step S2902.

When the process proceeds to step S2903, the server device 100 determines whether there is past history for the user 11. For example, the server device 100 refers to the past history DB 2101, etc., and if there is a past dialogue log for the user 11, it determines that there is past history for the user 11. If there is past history, the server device 100 proceeds to step S2904. On the other hand, if there is no past history, the server device 100 proceeds to step S2905.

When the process proceeds to step S2904, the server device 100 determines a scenario to be used in the dialogue process from the past history of the user 11 (past dialogue logs, etc.). On the other hand, when the process proceeds to step S2905, the server device 100 selects a standard scenario (e.g., a scenario for a new user) as the scenario to be used in the dialogue process.

When the process proceeds to step S2906, the dialogue system 1 executes, for example, the dialogue processing described in FIGS. 1 to 23 on the web conference. Through the processing of FIG. 29, the dialogue system 1 can provide a dialogue service to the user 11 using the web conference.

As described above, according to each embodiment of the present invention, a dialogue system 1 that uses a dialogue agent to engage in dialogue with a user 11 can provide a more appropriate reaction to the user 11.

<Additional Information>
Each function of each embodiment described above can be realized by one or more processing circuits. Here, the term "processing circuit" in this specification includes a processor programmed to execute each function by software, such as a processor implemented by an electronic circuit, and devices such as an ASIC (Application Specific Integrated Circuit), a DSP (digital signal processor), an FPGA (field programmable gate array), and a conventional circuit module designed to execute each function described above.

Furthermore, the devices described in the examples are merely illustrative of one of multiple computing environments for implementing the embodiments disclosed herein. In one embodiment, the server apparatus 100 includes multiple computing devices, such as a server cluster. The multiple computing devices are configured to communicate with each other via any type of communication link, including a network, shared memory, etc., and perform the processes disclosed herein.

Furthermore, each functional configuration of the server device 100 may be integrated into one server device, or may be divided into multiple devices. Furthermore, at least some of the functional configurations of the server device 100 may be possessed by the terminal device 10.

<Additional Notes>
This specification discloses the following dialogue systems, dialogue control methods, and programs.
(Section 1)
A dialogue system that uses a dialogue agent to dialogue with a user,
a first acquisition unit that acquires language information of the user from the dialogue;
a second acquisition unit that acquires non-verbal information of the user from the dialogue;
a generation unit that generates a response content including a verbal response and a non-verbal response of the dialogue agent based on the verbal information and the non-verbal information of the user;
a control unit that controls the dialogue agent based on the response content generated by the generation unit;
A dialogue system comprising:
(Section 2)
the response content of the dialogue agent includes a non-verbal response of the dialogue agent;
the generation unit changes the non-verbal response of the dialogue agent in response to non-verbal information of the user.
2. A dialogue system as defined in claim 1.
(Section 3)
3. The dialogue system according to claim 2, wherein the generation unit changes the content of the action of the dialogue agent in response to non-verbal information of the user.
(Section 4)
4. The dialogue system according to claim 2, wherein the generation unit changes a timing of an action of the dialogue agent in response to non-verbal information of the user.
(Section 5)
5. The dialogue system according to any one of claims 1 to 4, wherein the non-verbal information of the user includes facial expression, gaze, posture, or emotional information obtained from an image of the user.
(Section 6)
6. The dialogue system according to any one of claims 1 to 5, wherein the non-verbal information of the user includes information on the volume, intonation, or timbre of the voice obtained from the user's voice.
(Section 7)
7. The dialogue system according to any one of claims 1 to 6, wherein the generation unit changes the response content of the dialogue agent in accordance with a scenario of the dialogue.
(Section 8)
8. The dialogue system according to any one of claims 1 to 7, wherein the generation unit changes the response content of the dialogue agent in accordance with a plurality of dialogue stages set in advance.
(Section 9)
The dialogue system according to claim 8, wherein the generation unit changes the dialogue stage based on gaze information of the user.
(Article 10)
an image generating unit that generates an image related to the dialogue content based on the language information of the user;
Engaging in a dialogue with the user using the dialogue agent and the image.
10. The dialogue system according to any one of claims 1 to 9.
(Article 11)
11. The dialogue system according to any one of claims 1 to 10, further comprising a summarizing unit that summarizes the dialogue based on a dialogue log of the dialogue.
(Article 12)
The interaction is a business negotiation with the user,
Proposing a product to be proposed to the user based on the content of the dialogue of the business negotiation;
12. The dialogue system according to any one of claims 1 to 11.
(Article 13)
A catchphrase for the merchandise is presented based on the contents of the dialogue during the business negotiation.
Dialogue system according to clause 12.
(Section 14)
A database storing a past history of the dialogue is provided,
14. The dialogue system according to any one of claims 1 to 13, wherein the generation unit changes a scenario of the dialogue based on a past history of the dialogue.
(Article 15)
A database storing a past history of the dialogue is provided,
the generation unit generates a linguistic response of the dialogue agent by referring to a past history of the dialogue;
15. The dialogue system according to any one of claims 1 to 14.
(Article 16)
The second acquisition unit acquires non-verbal information indicating an attribute of the user from the dialogue;
16. The dialogue system according to any one of claims 1 to 15, wherein the generation unit generates the linguistic response or the non-linguistic response according to an attribute of the user.
(Section 17)
In a dialogue system that uses a dialogue agent to have a dialogue with a user,
The computer
acquiring language information of the user from the dialogue;
A process of acquiring non-verbal information of the user from the dialogue;
a generation process for generating a response content including a verbal response and a non-verbal response of the dialogue agent based on the verbal information and the non-verbal information of the user;
A process of controlling the dialogue agent based on the response content generated in the generation process;
The method for controlling dialogue is as follows.
(Article 18)
In a dialogue system that uses a dialogue agent to have a dialogue with a user,
On the computer,
acquiring language information of the user from the dialogue;
A process of acquiring non-verbal information of the user from the dialogue;
a generation process for generating a response content including a verbal response and a non-verbal response of the dialogue agent based on the verbal information and the non-verbal information of the user;
A process of controlling the dialogue agent based on the response content generated in the generation process;
A program to execute.

Although the embodiment of the present invention has been described above, the present invention is not limited to such a specific embodiment, and various modifications and applications are possible within the scope of the gist of the present invention described in the claims.

Reference Signs List 1 Dialogue system 10 Terminal device 100 Server device 200, 300 Dialogue screen 201, 301, 1401 Virtual human (dialogue agent)
500 Computer 702 First acquisition unit 703 Second acquisition unit 704 Generation unit 714 Control unit 1501 Image generation unit 1701 Summary unit 1901 Catchphrase generation unit

JP 2022-093479 A

Claims (18)

A dialogue system that uses a dialogue agent to dialogue with a user,
a first acquisition unit that acquires language information of the user from the dialogue;
a second acquisition unit that acquires non-verbal information of the user from the dialogue;
a generation unit that generates a response content including a verbal response and a non-verbal response of the dialogue agent based on the verbal information and the non-verbal information of the user;
a control unit that controls the dialogue agent based on the response content generated by the generation unit;
A dialogue system comprising: the response content of the dialogue agent includes a non-verbal response of the dialogue agent;
the generation unit generates a non-verbal response of the dialogue agent in response to non-verbal information of the user.
The dialogue system according to claim 1 . The dialogue system according to claim 2, wherein the generation unit changes the content of the action of the dialogue agent according to non-verbal information of the user. The dialogue system according to claim 2, wherein the generation unit changes the timing of the action of the dialogue agent according to non-verbal information of the user. The dialogue system according to any one of claims 1 to 4, wherein the non-verbal information of the user includes facial expression, gaze, posture, or emotional information obtained from an image of the user. The dialogue system according to claim 5, wherein the non-verbal information of the user includes information on the volume, intonation, or timbre of the voice obtained from the user's voice. The dialogue system according to claim 1, wherein the generation unit changes the response content of the dialogue agent according to the dialogue scenario. The dialogue system according to claim 1, wherein the generation unit changes the response content of the dialogue agent according to a plurality of preset dialogue stages. The dialogue system according to claim 8, wherein the generation unit changes the dialogue stage based on the user's gaze information. an image generating unit that generates an image related to the content of the dialogue based on at least one of the linguistic information and the non-linguistic information of the user;
interacting with the user using the dialogue agent and the image;
The dialogue system according to claim 1 . The dialogue system according to claim 1, further comprising a summarizing unit that summarizes the dialogue based on a dialogue log of the dialogue. The interaction is a business negotiation with the user,
Proposing a product to be proposed to the user based on the content of the dialogue of the business negotiation;
The dialogue system according to claim 1 . A catchphrase for the merchandise is presented based on the contents of the dialogue during the business negotiation.
Dialogue system according to claim 12. A database storing a past history of the dialogue is provided,
The dialogue system according to claim 7 , wherein the generation unit changes a scenario of the dialogue based on a past history of the dialogue. A database storing a past history of the dialogue is provided,
the generation unit generates a linguistic response of the dialogue agent by referring to a past history of the dialogue;
The dialogue system according to claim 1 . The second acquisition unit acquires non-verbal information indicating an attribute of the user from the dialogue;
The dialogue system according to claim 1 , wherein the generation unit generates the linguistic response or the non-linguistic response according to an attribute of the user. In a dialogue system that uses a dialogue agent to have a dialogue with a user,
The computer
acquiring language information of the user from the dialogue;
A process of acquiring non-verbal information of the user from the dialogue;
a generation process for generating a response content including a verbal response and a non-verbal response of the dialogue agent based on the verbal information and the non-verbal information of the user;
A process of controlling the dialogue agent based on the response content generated in the generation process;
The method for controlling dialogue is as follows. In a dialogue system that uses a dialogue agent to have a dialogue with a user,
On the computer,
acquiring language information of the user from the dialogue;
A process of acquiring non-verbal information of the user from the dialogue;
a generation process for generating a response content including a verbal response and a non-verbal response of the dialogue agent based on the verbal information and the non-verbal information of the user;
A process of controlling the dialogue agent based on the response content generated in the generation process;
A program to execute.

Images