KR102681147B1 - Method and apparatus for generating appropriate responses based on the user intent in an ai chatbot through retrieval-augmented generation - Google Patents

Abstract

This specification provides a method for a server to generate an answer to a chatbot user's inquiry, comprising: creating a flow for generating the answer; As, the flow includes representative example sentences related to the query; Entering a prompt for generating an example sentence based on the representative example sentence into a Large Language Model (LLM); Obtaining the example sentence from the large language model; and adding the example sentence to the flow.

Description

Method and device for implementing answer generation appropriate for artificial intelligence chatbot user intent through search augmented generation { METHOD AND APPARATUS FOR GENERATING APPROPRIATE RESPONSES BASED ON THE USER INTENT IN AN AI CHATBOT THROUGH RETRIEVAL-AUGMENTED GENERATION }

This specification relates to a method and device for extracting answers suitable for chatbot queries from knowledge documents uploaded by an administrator and generating them according to the context.

Currently, the generative AI chat GPT developed by OpenAI is becoming a hot topic around the world. However, one of the main problems with these models is the phenomenon of “hallucinations,” which generate information that is irrelevant or factual, out of context of the question. Because of this, models such as ChatGPT may have limitations as simple conversation bots.

Therefore, chatbots for task-oriented dialogue to create value are required in all industries. Purpose-driven conversations are about industry and business-related questions, such as making restaurant reservations and placing orders, rather than simple questions like “How’s the weather today?” Because these conversations vary depending on the unique data, work manuals, and work knowledge that each industry and company has, it may be difficult for a generative chatbot to respond directly. Therefore, the ability to create chatbots tailored to each industry and company, understand user questions, and automate the relevant tasks is essential.

To meet these needs, the chatbot builder currently available is a tool that automates tasks according to flows defined in advance by chatbot managers of individual companies. It can understand natural language and automatically determine which flow should be used. However, machine learning techniques such as supervised learning can be used to automatically classify flows, and for this, training example data is required. For example, you need to train a machine learning model using sample question data such as “I want to make a reservation at a restaurant” or “Can I make a reservation at a restaurant?”

These machine learning-based chatbot services have several problems. First, it requires cost and time investment to build query example data. For example, chatbot managers must build sample query data themselves, which requires considerable time and effort. Generating large-scale query example candidate data requires a lot of cost and time.

Second, there are difficulties in establishing pre-flow and conversation intent. The conversation flow must be defined in advance along with a sample query, but it may be difficult to predict whether the defined flow matches the actual user query. As a result, chatbot managers may have to create unnecessary flows in advance or perform duplicate tasks.

Additionally, operation and monitoring costs are incurred. Separate monitoring personnel are required to maintain the quality of chatbot services. This results in additional operating costs, and monitoring personnel must check new content of user queries and create new flows and query examples/answers. This is a factor that increases the workload of chatbot managers and increases costs.

As such, efficiently managing and operating chatbot services is a complex task, and automated methods are required to overcome these problems.

The purpose of this specification is to provide a method and device for automatically extracting answers suitable for chatbot queries from knowledge documents uploaded by an administrator and generating them according to the context.

The technical problems to be achieved by this specification are not limited to the technical problems mentioned above, and other technical problems not mentioned will be clear to those skilled in the art from the detailed description of the specification below. It will be understandable.

One aspect of the present specification is a method for a server to implement the generation of an answer to a chatbot user's inquiry, comprising: generating a flow for generating the answer; As, the flow includes representative example sentences related to the query; Entering a prompt for generating an example sentence based on the representative example sentence into a Large Language Model (LLM); Obtaining the example sentence from the large language model; and adding the example sentence to the flow.

Additionally, adding the example sentence to the flow includes transferring the example sentence to a machine learning model and vectorizing it; Returning a primary key related to the example sentence; and adding the example sentence to the flow using the primary key. may include.

Additionally, if there is no flow matching the query, uploading a knowledge file for extracting the answer; extracting text from the knowledge file; and vectorizing the text; It may further include.

Additionally, dividing the extracted text into chunks; As, the chunk is a set of texts with similar meaning; Further comprising: vectorizing the text; vectorizing the chunk; and storing vectors of the chunks; It may further include.

Additionally, loading the knowledge file based on the vector of the stored chunk; It may further include.

Additionally, receiving the query from a user terminal; Vectorizing the input query; determining the degree of similarity between the query and the representative example sentence or the example sentence; and returning the flow based on the similarity. It may further include.

Another aspect of the present specification is a server for implementing response generation to a chatbot user's inquiry, comprising: a memory containing a machine learning model; A communication unit for communicating with external electronic devices; and a processor for functionally controlling the memory and the communication unit. It includes, the processor generates a flow for generating the answer, the flow includes a representative example sentence related to the query, and based on the representative example sentence in a Large Language Model (LLM), an example sentence You can enter a prompt for creation, obtain the example sentence from the large language model, and add the example sentence to the flow.

Another aspect of the present specification is a method for an administrator terminal to generate an answer to a chatbot user's inquiry, comprising: receiving, from an administrator, a flow creation button for generating the answer; In response to an input of the create button, transmitting a command to create the flow to a server; As, the flow creation command includes the name of the flow and a representative example sentence related to the query; In response to the flow creation command, receiving a message indicating that the flow has been created; transmitting, to the server, a command for generating example sentences related to the representative example sentences; Receiving candidates of the example sentences obtained from a Large Language Model (LLM); displaying candidates for the example sentences to the manager; selecting a final example sentence from the manager based on the candidate examples; delivering the final example sentence to the server; may include.

According to an embodiment of the present specification, an answer suitable for a chatbot query can be automatically extracted from a knowledge document uploaded by an administrator and generated according to the context.

The effects that can be obtained in this specification are not limited to the effects mentioned above, and other effects not mentioned can be clearly understood by those skilled in the art from the description below. .

1 is a block diagram for explaining an electronic device related to this specification.
Figure 2 is a block diagram of an AI device according to an embodiment of the present specification.
Figure 3 is an example of a pipeline for flow creation applicable to this specification.
4 to 6 are examples of manager terminals for flow creation that can be applied to this specification.
Figure 7 illustrates representative example sentence data to which this specification can be applied.
Figure 8 is an example of a pipeline for generating flow examples that can be applied to this specification.
Figure 9 is an example of the activation procedure of a knowledge file to which this specification can be applied.
Figure 10 illustrates a knowledge screen to which this specification can be applied.
11 illustrates an activation indication of a knowledge file to which this specification can be applied.
Figure 12 illustrates knowledge data to which this specification can be applied.
Figure 13 illustrates a procedure for classifying a chatbot user's inquiry into a flow to which this specification can be applied.
Figure 14 illustrates providing an answer based on a knowledge file to which this specification can be applied.
15 illustrates a prompt for generating an answer to which this disclosure may be applied.
Figure 16 is an embodiment of a server to which this specification can be applied.
17 and 18 are examples of a manager terminal to which this specification can be applied.
The accompanying drawings, which are included as part of the detailed description to aid understanding of the present specification, provide embodiments of the present specification and explain technical features of the present specification together with the detailed description.

Hereinafter, embodiments disclosed in the present specification will be described in detail with reference to the attached drawings. However, identical or similar components will be assigned the same reference numerals regardless of reference numerals, and duplicate descriptions thereof will be omitted. The suffixes “module” and “part” for components used in the following description are given or used interchangeably only for the ease of preparing the specification, and do not have distinct meanings or roles in themselves. Additionally, in describing the embodiments disclosed in this specification, if it is determined that detailed descriptions of related known technologies may obscure the gist of the embodiments disclosed in this specification, the detailed descriptions will be omitted. In addition, the attached drawings are only for easy understanding of the embodiments disclosed in this specification, and the technical idea disclosed in this specification is not limited by the attached drawings, and all changes included in the spirit and technical scope of this specification are not limited. , should be understood to include equivalents or substitutes.

Terms containing ordinal numbers, such as first, second, etc., may be used to describe various components, but the components are not limited by the terms. The above terms are used only for the purpose of distinguishing one component from another.

When a component is said to be “connected” or “connected” to another component, it is understood that it may be directly connected or connected to the other component, but that other components may exist in between. It should be. On the other hand, when it is mentioned that a component is “directly connected” or “directly connected” to another component, it should be understood that there are no other components in between.

Singular expressions include plural expressions unless the context clearly dictates otherwise.

In this application, terms such as “comprise” or “have” are intended to designate the presence of features, numbers, steps, operations, components, parts, or combinations thereof described in the specification, but are not intended to indicate the presence of one or more other features. It should be understood that this does not exclude in advance the possibility of the existence or addition of elements, numbers, steps, operations, components, parts, or combinations thereof.

1 is a block diagram for explaining an electronic device related to this specification.

The electronic device 100 includes a wireless communication unit 110, an input unit 120, a sensing unit 140, an output unit 150, an interface unit 160, a memory 170, a control unit 180, and a power supply unit 190. ), etc. may be included. The components shown in FIG. 1 are not essential for implementing an electronic device, so the electronic device described in this specification may have more or fewer components than the components listed above.

More specifically, among the above components, the wireless communication unit 110 is used between the electronic device 100 and the wireless communication system, between the electronic device 100 and another electronic device 100, or between the electronic device 100 and an external server. It may include one or more modules that enable wireless communication between the devices. Additionally, the wireless communication unit 110 may include one or more modules that connect the electronic device 100 to one or more networks.

This wireless communication unit 110 may include at least one of a broadcast reception module 111, a mobile communication module 112, a wireless Internet module 113, a short-range communication module 114, and a location information module 115. .

The input unit 120 includes a camera 121 or an image input unit for inputting an image signal, a microphone 122 or an audio input unit for inputting an audio signal, and a user input unit 123 for receiving information from a user, for example. , touch keys, push keys (mechanical keys, etc.). Voice data or image data collected by the input unit 120 may be analyzed and processed as a user's control command.

The sensing unit 140 may include one or more sensors for sensing at least one of information within the electronic device, information on the surrounding environment surrounding the electronic device, and user information. For example, the sensing unit 140 includes a proximity sensor (141), an illumination sensor (142), a touch sensor, an acceleration sensor, a magnetic sensor, and a gravity sensor. Sensor (G-sensor), gyroscope sensor, motion sensor, RGB sensor, IR sensor (infrared sensor), fingerprint scan sensor, ultrasonic sensor , optical sensors (e.g., cameras (see 121)), microphones (see 122), battery gauges, environmental sensors (e.g., barometers, hygrometers, thermometers, radiation detection sensors, It may include at least one of a heat detection sensor, a gas detection sensor, etc.) and a chemical sensor (e.g., an electronic nose, a healthcare sensor, a biometric sensor, etc.). Meanwhile, the electronic device disclosed in this specification can utilize information sensed by at least two of these sensors by combining them.

The output unit 150 is for generating output related to vision, hearing, or tactile sense, and includes at least one of a display unit 151, an audio output unit 152, a haptip module 153, and an optical output unit 154. can do. The display unit 151 can implement a touch screen by forming a layered structure or being integrated with the touch sensor. This touch screen functions as a user input unit 123 that provides an input interface between the electronic device 100 and the user, and can simultaneously provide an output interface between the electronic device 100 and the user.

The interface unit 160 serves as a passageway for various types of external devices connected to the electronic device 100. This interface unit 160 connects devices equipped with a wired/wireless headset port, an external charger port, a wired/wireless data port, a memory card port, and an identification module. It may include at least one of a port, an audio input/output (I/O) port, a video input/output (I/O) port, and an earphone port. In response to the external device being connected to the interface unit 160, the electronic device 100 may perform appropriate control related to the connected external device.

Additionally, the memory 170 stores data supporting various functions of the electronic device 100. The memory 170 may store a plurality of application programs (application programs or applications) running on the electronic device 100, data for operating the electronic device 100, and commands. At least some of these applications may be downloaded from an external server via wireless communication. Additionally, at least some of these applications may be present on the electronic device 100 from the time of shipment for basic functions of the electronic device 100 (e.g., incoming and outgoing calls, receiving and sending functions). Meanwhile, the application program may be stored in the memory 170, installed on the electronic device 100, and driven by the control unit 180 to perform an operation (or function) of the electronic device.

In addition to operations related to the application program, the control unit 180 typically controls the overall operation of the electronic device 100. The control unit 180 can provide or process appropriate information or functions to the user by processing signals, data, information, etc. input or output through the components discussed above, or by running an application program stored in the memory 170.

Additionally, the control unit 180 may control at least some of the components examined with FIG. 1 in order to run an application program stored in the memory 170. Furthermore, the control unit 180 may operate at least two of the components included in the electronic device 100 in combination with each other in order to run the application program.

The power supply unit 190 receives external power and internal power under the control of the control unit 180 and supplies power to each component included in the electronic device 100. This power supply unit 190 includes a battery, and the battery may be a built-in battery or a replaceable battery.

At least some of the components may cooperate with each other to implement operation, control, or a control method of an electronic device according to various embodiments described below. Additionally, the operation, control, or control method of the electronic device may be implemented on the electronic device by running at least one application program stored in the memory 170.

In this specification, servers and terminals may include electronic devices 100.

Figure 2 is a block diagram of an AI device according to an embodiment of the present specification.

The AI device 20 may include an electronic device including an AI module capable of performing AI processing or a server including the AI module. Additionally, the AI device 20 may be included in at least a portion of the electronic device 100 shown in FIG. 1 and may be equipped to perform at least a portion of AI processing.

The AI device 20 may include an AI processor 21, memory 25, and/or a communication unit 27.

The AI device 20 is a computing device capable of learning a neural network, and may be implemented in various electronic devices such as servers, desktop PCs, laptop PCs, tablet PCs, etc.

The AI processor 21 can learn a neural network using a program stored in the memory 25. In particular, the AI processor 21 can generate an artificial intelligence model (and/or Large Language Model (LLM)) for natural language processing on the memory 25, knowledge documents uploaded by the administrator, and/or the terminal. These artificial intelligence models can be trained using data that can be collected through .

The AI processor 21 may be a general-purpose processor (e.g., CPU), or may be an AI-specific processor (e.g., GPU, graphics processing unit) for artificial intelligence learning.

The memory 25 can store various programs and data necessary for the operation of the AI device 20. The memory 25 can be implemented as non-volatile memory, volatile memory, flash-memory, hard disk drive (HDD), or solid state drive (SDD). The memory 25 is accessed by the AI processor 21, and reading/writing/modifying/deleting/updating data by the AI processor 21 can be performed. Additionally, the memory 25 may store a neural network model (eg, deep learning model) generated through a learning algorithm for data classification/recognition according to an embodiment of the present specification.

Meanwhile, the AI processor 21 may include a data learning unit that learns a neural network for data classification/recognition. For example, the data learning unit may learn a deep learning model by acquiring training data to be used for learning and applying the acquired training data to the deep learning model.

The communication unit 27 can transmit the results of AI processing by the AI processor 21 to an external electronic device.

Here, external electronic devices may include other terminals and servers.

Meanwhile, the AI device 20 shown in FIG. 2 has been described as functionally divided into an AI processor 21, a memory 25, a communication unit 27, etc., but the above-described components are integrated into one module to form an AI module. Alternatively, it may be referred to as an artificial intelligence (AI) model.

Figure 3 is an example of a pipeline for flow creation applicable to this specification.

Referring to Figure 3, a pipeline for automatically creating the flow of a chatbot builder for a Task Oriented Dialog is illustrated. A flow may represent a series of steps and processes through which the server 310 interacts with a user to provide answers to the user's queries.

For example, when the chatbot manager uploads a knowledge document, the server 310 can find an answer that matches the chatbot user's inquiry based on this and create it according to the context. Additionally, when the server 310 receives a new, undefined query from a chatbot user, it automatically creates a new flow and can be used to classify the intent of similar queries in the future.

More specifically, the server 310 may include a flow creation module for creating a flow from a user query. The administrator can manage the flow creation module through the administrator terminal 320.

1. The manager terminal 320 transmits a flow creation command to the server 310. For example, the manager terminal 320 pairs a flow name and a representative example sentence and transmits them to the server 310. Information about the flow name and representative example sentences may be input from the administrator or may be preset data.

4 to 6 are examples of manager terminals for flow creation that can be applied to this specification.

Referring to FIG. 4, the manager can create a flow by inputting the flow creation button 420 through the manager terminal 320. The flow name 400 may be an ID indicating a flow into which a customer's inquiry can be classified, and the representative example sentence 410 may be an example of a query representing the flow. For example, if the flow name (400) is “Customer Center Operating Hours,” the representative example sentence (410) is “What time is the customer center currently open?” It can be set to . The flow created in this way is expressed as a flow chart in which nodes are connected, and the manager can check it (430) through the manager terminal (320).

Referring to Figure 5, a flow may include a start node and a message node. For example, the start node represents a representative example sentence (query), and when example sentence candidates described later are added, the example sentence candidates may be managed in dependence on the start node. The administrator can click the start node through the administrator terminal 320 and directly input example sentence candidates (510) through the settings screen 500. A message node is an answer corresponding to a user query, and the administrator can set an answer corresponding to the query in the flow.

Referring to FIG. 6, when the manager terminal 320 receives the flow creation button 420, it can display a panel for creating a new flow to the manager. Through this, the administrator can create a new flow by entering the flow name and representative example sentences.

2. Referring again to FIG. 3, the server 310 may transmit a representative example sentence to the machine learning model. The machine learning model can convert representative example sentences into multidimensional vectors and transmit them to the server 310. For example, a machine learning model can convert a representative example sentence into a 768-dimensional vector. The dimension of the output may be different depending on which Tokenizer or Vectorizer the machine learning model uses.

3. The server 310 creates metadata, such as the flow creation date and the person who created it, and adds it to the database. Through this, administrators can update and manage flows.

4-1. For example, the server 310 may store metadata about flows in a relational database (RDB).

4-2. For example, the server 310 may store the converted vector and related metadata in a vector database through a machine learning model.

Figure 7 illustrates representative example sentence data to which this specification can be applied.

Referring to FIG. 7, when the representative example sentence is “I want to order,” the metadata, ID, and converted vector corresponding to the representative example sentence are illustrated. Such data can be generated and managed for each representative example sentence, and the manager can check it through the manager terminal 320.

5. Referring again to FIG. 3, the server 310 may return the success/failure of the above-described procedures to the manager terminal 320. The administrator can check whether the flow was successfully created through the administrator terminal 320.

Figure 8 is an example of a pipeline for generating flow examples that can be applied to this specification.

Referring to FIG. 8, the server 310 can automatically generate flow example sentences through LLM. For example, a Large Language Model is a type of artificial intelligence (AI) system that has the ability to understand and generate human-like language by learning large amounts of text data. These models can perform text generation, text completion, language translation, text summarization, and a variety of other natural language processing tasks. A well-known example of a large-scale language model is GPT-3 developed by OpenAI. If necessary, the server 310 can be connected to an LLM provided by an external server.

However, such use of LLM may incur excessive costs. For example, the cost of hardware and cloud services required to train a model, the cost of using the learned model in an operational environment, and the cost of preprocessing and management of data used for model training and inference may be excessive. there is.

Accordingly, in this specification, the server 310 exemplifies a method of efficiently generating example sentence candidates for a flow by minimally using LLM.

Flow example sentences are sentences for finding a user's inquiry. The more example sentences are set in the server 310, the better the chatbot provided by the server 310 can perform.

For example, if the flow name is "Customer Center Operating Hours" and the representative example sentence is "What hours is the Customer Center currently open?", one or more example sentences related to queries similar to the representative example sentence will be "Customer Center Operating Hours". Can be set in flow.

1. The administrator terminal 320 transmits a command to automatically generate a flow example sentence to the server 310. For example, the administrator may input a button for automatically generating flow examples through the administrator terminal 320 and transmit a command for automatically generating flow examples to the server 310.

2. The server 310 queries representative examples of the flow through the Vector Database. For example, if the administrator requests automatic generation of an example sentence for the “Customer Center Operating Hours” flow, the server 310 can obtain the representative example sentence by querying “What time is the customer center currently open?”

3. The server 310 is the LLM and inputs a prompt for generating an example sentence that is contextually similar to the representative example sentence. The LLM can generate candidate example sentences as a response to the prompt and transmit them to the server 310. For example, LLM can generate five example sentence candidates in Python list format as responses to a prompt.

4. The server 310 delivers the generated example sentence candidates to the manager terminal 320. The manager terminal 320 may display example sentence candidates in a table form to the manager. The administrator can check example sentences similar to the generated representative example sentences through the example sentence candidates.

5. The manager terminal 320 delivers the final example sentence to the server 310 based on the generated example sentence candidates. For example, the administrator can multiple select the final example sentence through the administrator terminal 320. When the administrator terminal 320 receives a button “Apply as an example sentence” from the administrator, the selected example sentences can be transmitted to the server 310.

6. The server 310 can transmit the final example sentences to the machine learning model and vectorize them through a natural language processing vectorizer.

7. The server 310 may generate metadata of the final example sentences (e.g., which flow the example sentence belongs to, when it was created, etc.), sort the data according to the schema of the database, and add it to the database. .

7-1. The server 310 stores the converted vector, original text, and metadata of the final example sentences in Vector DB and returns an automatically generated primary key in response to the added example sentences. For example, a primary key can be created as a unique identifier. This identifier can be used as a unique key to represent an example sentence.

7-2. The server 310 may add the returned primary key to the corresponding flow in the RDB. The primary key can serve to establish a connection between different databases (Vector DB and RDB). To establish this relationship, the server 310 can use the returned primary key within the corresponding flow of the RDB to connect the example sentence with the flow. This serves as a kind of reference and can be used to identify which flow the example belongs to. This can play a similar role to RDB's Foreign Key.

8. The server 310 may return the success/failure of the above-described procedures to the manager terminal 320. The administrator can check whether the example sentences of the flow have been successfully created through the administrator terminal 320.

Through the above-described procedures of FIGS. 3 to 8, the server 310 can build a flow to receive inquiries from users.

Through this, in this specification, the server 310 automatically generates example sentences that are contextually similar to the query through a large language model (LLM), so it saves time and input manpower to build example sentence data for chatbot users' queries. Costs can be reduced.

Figure 9 is an example of the activation procedure of a knowledge file to which this specification can be applied.

In this specification, knowledge refers to manufacturing, distribution, finance, communication, and medicine. education. It can refer to a document file containing business regulations, guidelines, or educational content for all industries, including legal services and public institutions.

The server 310 can generate and provide answers to user inquiries using knowledge files.

1. The server 310 acquires a knowledge file. For example, the administrator can upload a file containing knowledge for use in the chatbot service to the server through the administrator terminal 320. The administrator can directly select a knowledge file from the 'knowledge' screen displayed through the administrator terminal 320 or upload it to the server 310 by dragging and dropping it.

Figure 10 illustrates a knowledge screen to which this specification can be applied.

Referring to FIG. 10, the administrator can upload knowledge files to the knowledge file upload field 1010 of the knowledge screen by dragging and dropping. The administrator can activate (1000) the necessary knowledge files. For example, knowledge files uploaded by an administrator can be stored and managed through a cloud server.

2. Referring again to FIG. 9, the server 310 may confirm the administrator's authentication and store the uploaded file in the storage of the group to which the administrator belongs.

3. The server 310 is commanded to activate knowledge. For example, to load a knowledge file, the administrator may input the knowledge activation button 1000 corresponding to the knowledge file to be loaded on the knowledge screen through the administrator terminal 320.

11 illustrates an activation indication of a knowledge file to which this specification can be applied.

Referring to FIG. 11, when the administrator inputs the activation button 1000 and the conversion/activation process of the corresponding knowledge file is completed, the administrator terminal 320 may display the activation status to the administrator.

4. Again, referring to FIG. 9, the server 310 may load the knowledge file from storage. The server 310 may switch the knowledge activation column of the knowledge table corresponding to the loaded knowledge file to “activated.”

5. The server 310 extracts text from the knowledge file and parses it.

6. The server 310 may perform preprocessing on the extracted text by file format (eg, Word, HWP, PDF, etc.). This preprocessing process can be performed to split meaningful chunks. For example, the server 310 may perform preprocessing on a text file in PDF format. More specifically, the server 310 can use regular expressions to replace three or more line breaks with two line breaks and two or more spaces with one space.

7. The server 310 may split the text into chunks. In this specification, a chunk may be a meaningful set of texts with similar context. For example, the server 310 may define one chunk as one paragraph. The standard for such a paragraph may be one or more line breaks, and even if it is the same paragraph, when moving to the next page, the paragraph can be divided into two based on the page. However, sentences cut from the next page may be incorporated into the previous paragraph.

8. The server 310 can vectorize chunks through a machine learning model.

9. The server 310 may store the converted vector, text before conversion, and/or metadata (e.g., chunk extracted from which file page, file creation date, etc.) in Vector DB.

Figure 12 illustrates knowledge data to which this specification can be applied.

Referring to FIG. 12, when the knowledge is "Places that have introduced it also appear to be speeding up service advancement to enhance quality," the metadata, ID, and converted vector corresponding to the knowledge are exemplified. Such data can be created and managed for each knowledge, and the manager can check it through the manager terminal 320.

10. Referring again to FIG. 9, the server 310 may return the success/failure of the above-described procedures to the manager terminal 320. The administrator can check whether knowledge activation has been successfully performed through the administrator terminal 320.

Figure 13 illustrates a procedure for classifying a chatbot user's inquiry into a flow to which this specification can be applied.

Referring to FIG. 13, the server may set a flow and include knowledge data through the above-described procedure. For example, the server may generate a flow by performing the operation of FIG. 3 described above, and may generate one or more example sentences in the flow according to the operation of FIG. 8 described above. Additionally, according to the operation of FIG. 9 described above, the knowledge file may be in an activated state.

The server receives a query from the user terminal (S1310). For example, a user can use a chatbot service provided by the server through the user terminal. For example, these chatbot services may be provided in web/app format, and users can enter queries in text form through a chat room provided by the chatbot service.

The server vectorizes the input query (S1320). The server can vectorize queries through a natural language processing vectorizer. For example, the server can convert the query text into a 768-dimensional vector and use a vectorizer such as the vectorizer described above.

The server determines the similarity between the query and the example sentences (S1330). For example, the server can compare the vectors of example sentences included in the Vector DB with the vector of the query to determine the degree of similarity between the vectors. In more detail, the server can compare the similarity between vectors of representative example sentences and/or example sentences added by performing the operation of FIG. 7 and vectors that vectorize the query.

The server returns a flow based on similarity (S1340).

For example, the server can obtain the three vectors with the highest similarity and then vote on the flow. For example, voting may refer to a form of ensemble learning. Ensemble learning is used to combine several different machine learning models to provide stronger predictive power than a single model. Voting is a method of ensemble and can refer to a method of performing the final prediction by combining predictions from multiple models. For example, if the flows of three vectors were voted as [flow A, flow B, flow A], the query can be classified as flow A.

The server delivers a response to the user terminal based on the returned flow (S1350). For example, the server can deliver the queried answer to the user terminal according to the returned flow, and the user terminal can display the answer to the user as the answer provided by the chatbot.

Figure 14 illustrates providing an answer based on a knowledge file to which this specification can be applied.

Referring to FIG. 14, if the server fails to return an appropriate flow in S1340, it may provide an answer to the user based on the knowledge file. For example, the server compares the vectors of example sentences and the vector of the query to determine the similarity between the vectors, but if the similarity does not reach the threshold or the voting result is a tie, the server determines the similarity between the vectors and the vector of the query. Based on the knowledge file, an answer can be provided to the user.

The server searches for knowledge based on the query (S1410). For example, the server can search for a knowledge vector similar to the query vector from Vector DB, where knowledge files are vectorized and stored. Knowledge files are vectorized in chunks, so knowledge can correspond to chunks.

The server determines the similarity between the query and knowledge (S1420). For example, the server may determine knowledge with the highest similarity, or knowledge with a similarity exceeding a threshold.

The server returns knowledge based on similarity (S1430). For example, the server can extract the chunks with the highest similarity (which can be multiple depending on administrator settings) and create a prompt that is input into the LLM.

The server delivers an answer to the user terminal based on the returned knowledge (S1440). The server can generate a prompt to generate an answer based on the returned chunk. The server can input the generated prompt into the LLM, generate an answer, and deliver it to the administrator terminal.

15 illustrates a prompt for generating an answer to which this disclosure may be applied.

Referring to FIG. 15, the server may provide two chunks (paragraphs) returned to the LLM. For example, the server can register a query in LLM as a question (user_input). If LLM can find a query from two chunks, it can find the answer, create it according to the context, and return it in Python dictionary format {'answer': ____ }. If LLM cannot find an answer, it may return {'answer': 'No answer'}.

Figure 16 is an embodiment of a server to which this specification can be applied.

Referring to FIG. 16, the server may perform the above-described operations to implement answer generation to the chatbot user's inquiry. The server can use LLM to add example sentences to the flow to improve flow classification prediction accuracy.

The server creates a flow to generate an answer (S1610). For example, the flow may include representative example sentences related to the user's query, and/or answers to the query.

The server inputs a prompt for generating an example sentence into the LLM (S1620). For example, the server can generate a prompt for generating example sentences similar to representative example sentences and input them into the LLM.

The server obtains an example sentence from the LLM (S1630). For example, the server can obtain example sentences generated by LLM according to the prompt.

The server adds an example sentence to the flow (S1640). For example, the server can vectorize and store example sentences through a machine learning model. Afterwards, the server returns the primary key related to the example sentence, and using the primary key, the example sentence can be added to the flow.

The server can receive a query from a user terminal, vectorize it, and determine the degree of similarity to a representative example sentence or example sentences in the flow. The server returns a flow according to the determined similarity and can provide an answer to the user according to the flow.

If there is no flow that can match the query, the server can upload a knowledge file to extract the answer. The server can extract text from knowledge files, vectorize it, and store it. More specifically, the server can split the extracted text into chunks, vectorize them, and store them.

The server can load knowledge files through vectors of stored chunks. Vectors of these loaded chunks can be used to generate answers by comparing them to vectorized queries.

17 and 18 are examples of a manager terminal to which this specification can be applied.

Referring to FIG. 17, the administrator can obtain an example sentence from LLM through the administrator terminal and add the example sentence to the flow. For example, the administrator can check the flow for which an example sentence is to be added through the administrator terminal. When the administrator clicks the start node, the administrator terminal displays the settings screen 500, and when the example sentence generation button 1700 is input from the administrator, a command for automatically generating a flow example sentence can be transmitted to the server. The server may obtain example sentences through the procedures of S1620 and S1630 and transmit the obtained example sentences to the administrator terminal. The administrator terminal can display a list of automatically generated example sentences to the administrator. Based on the list of example sentences, the administrator may select (1720) one or more example sentences to be applied.

Referring to FIG. 18, the manager terminal can add the example sentence selected by the manager to the example sentence list 1810 of the flow. For example, when the administrator selects an example sentence by inputting a checkbox (1720) and inputs the “Apply as example sentence” (1730) button, the administrator terminal adds the selected example sentence to the example sentence list (1810), can be displayed. The administrator can check the example sentence list (1810) and enter the “Apply” (1800) button, the administrator terminal transmits the final example sentence to the server, and the server can add the final example sentence to the flow through operation S1640. there is.

The above-mentioned specification can be implemented as computer-readable code on a program-recorded medium. Computer-readable media includes all types of recording devices that store data that can be read by a computer system. Examples of computer-readable media include HDD (Hard Disk Drive), SSD (Solid State Disk), SDD (Silicon Disk Drive), ROM, RAM, CD-ROM, magnetic tape, floppy disk, optical data storage device, etc. This also includes those implemented in the form of carrier waves (e.g., transmission via the Internet). Accordingly, the above detailed description should not be construed as restrictive in all respects and should be considered illustrative. The scope of this specification should be determined by reasonable interpretation of the appended claims, and all changes within the equivalent scope of this specification are included in the scope of this specification.

In addition, although the above description focuses on the services and embodiments, this is only an example and does not limit the present specification, and those of ordinary skill in the field to which this specification pertains will understand that it does not deviate from the essential characteristics of the services and embodiments. It can be seen that various modifications and applications not exemplified above are possible. For example, each component specifically shown in the embodiments can be modified and implemented. And the differences related to these modifications and applications should be construed as being included in the scope of the present specification as defined in the attached claims.

Claims (8)

In a method for a server to generate an answer to a chatbot user's inquiry,
creating a flow for generating the answer; As, the flow includes representative example sentences related to the query;
Entering a prompt for generating an example sentence similar to the representative example sentence in a Large Language Model (LLM) based on the representative example sentence;
Obtaining the example sentence from the large language model; and
adding the example sentence to the flow;
Includes,
The example sentence is a sentence that is dependent on the flow and can match the query,
How to implement it.
According to paragraph 1,
The step of adding the example sentence to the flow is
Passing the example sentence to a machine learning model and vectorizing it;
Returning a primary key related to the example sentence; and
Adding the example sentence to the flow using the primary key;
Implementation method including.
According to paragraph 1,
If there is no flow matching the query, uploading a knowledge file for extracting the answer;
extracting text from the knowledge file; and
vectorizing the text;
Implementation method further comprising:
According to clause 3,
dividing the extracted text into chunks; as,
The chunk is a set of texts with similar meaning;
It further includes,
The step of vectorizing the text is
vectorizing the chunk; and
storing vectors of the chunks;
Implementation method further comprising:
According to clause 4,
loading the knowledge file based on the vector of the stored chunk;
Implementation method further comprising:
According to paragraph 1,
Receiving the query from a user terminal;
Vectorizing the input query;
determining the degree of similarity between the query and the representative example sentence or the example sentence; and
returning the flow based on the similarity;
Implementation method further comprising:
In the server for implementing answers to chatbot users' inquiries,
Memory with machine learning models;
A communication unit for communicating with external electronic devices; and
a processor for functionally controlling the memory and the communication unit;
Including,
The processor is
Create a flow for generating the answer, and the flow includes representative example sentences related to the query,
Enter a prompt to generate an example sentence similar to the representative example sentence based on the representative example sentence in a Large Language Model (LLM),
Obtaining the example sentences from the large language model,
Add the example sentence to the flow,
The example sentence is a sentence that is dependent on the flow and can match the query.
In a method for an administrator terminal to generate an answer to a chatbot user's inquiry,
Receiving, from an administrator, a flow creation button for generating the answer;
In response to an input of the create button, transmitting a command to create the flow to a server; As, the flow creation command includes the name of the flow and a representative example sentence related to the query;
In response to the flow creation command, receiving a message indicating that the flow has been created;
transmitting, to the server, a command to generate an example sentence similar to the representative example sentence;
Receiving candidates of the example sentences obtained from a Large Language Model (LLM);
displaying candidates for the example sentences to the manager;
selecting a final example sentence from the manager based on the candidate examples;
delivering the final example sentence to the server;
Includes,
The final example above is
An implementation method, which is a sentence that is dependent on the flow and can match the query.