TWI740103B - Customer service assiting method based on artifical intelligence - Google Patents

Abstract

A customer service assisting method based on the artificial intelligence, and the method comprises: receiving a first facial image including a plurality of first feature points and a first voice message including a first emotion information by a processor, and generating a second emotional state based on the first emotional information by the processor. When the first emotional state is the same as the second emotional state, the processor uses the first emotional state as a current emotional state. On the contrary, the processor performs a sentiment analysis program. When the first voice message includes a transaction information and an auxiliary information, the processor generates a transaction reply according to a financial transaction saved in a service database. On the contrary, the processor performs a problem analysis program. The processor sends the response example to the display interface based on at least of the transaction reply and the current emotional state.

Description

Customer service auxiliary device and method based on artificial intelligence

The present invention relates to a customer service assistant device and method, in particular to a customer service assistant device and method based on artificial intelligence.

As the financial services provided by banks continue to increase, more and more people rely on online customer service to complete pending financial services. Therefore, the quality of online customer service is even more important.

In order to improve the quality of online customer service, facial recognition can now be used to assist customer service in judging the customer’s emotions, so that the customer service can determine whether to provide additional financial product sales. Generally, when customers cannot clearly express their problems, the customer service staff can only deal with them based on their own experience. However, due to the high turnover rate of customer service personnel in this position, it is often difficult for customer service personnel to pass on and accumulate their own experience. Therefore, it is not enough in practice to assist customer service staff to respond by distinguishing emotions.

Therefore, there is still a need for a customer service assistance device and method based on artificial intelligence to improve the above-mentioned problems.

The present invention is to provide a customer service assistant device and method based on artificial intelligence, which can judge the emotional state of customers based on facial expressions and voices, and can analyze the questions asked by customers at the same time, and integrate the results of problem analysis and emotional analysis to provide The customer service staff serves as a reference for the response.

The present invention provides a customer service assistance method based on artificial intelligence, including: receiving a first facial image and a first voice message with a processor, wherein the first facial image includes a plurality of first feature points, and the first The voice message includes a first emotional information; the processor generates a first emotional state according to the first feature points, and a second emotional state is generated according to the first emotional information; the processor determines the first emotion Whether the state is the same as the second emotional state; when the first emotional state is the same as the second emotional state, use the first emotional state as an instant emotional state; when the first emotional state is different from the second emotional state , Using the processor to execute an emotion analysis program to generate the real-time emotional state; using the processor to determine whether the first voice message includes a transaction information and an auxiliary information; when the processor determines that the first voice message includes the transaction Information and the auxiliary information, the processor generates a transaction reply based on a financial transaction in a business database corresponding to the transaction information and the auxiliary information; when the processor determines that the first voice message only contains the transaction information Or the auxiliary information, the processor executes a problem analysis program to generate the transaction response; and the processor generates a response example based on at least the transaction response and the real-time emotional state, and sends it to a display interface to present the response example .

The present invention provides a customer service assistant device based on artificial intelligence, including: a business database storing multiple financial transactions; a processor connected to the business database for receiving a first facial image and a first A voice message, wherein the first face image includes a plurality of first feature points, the first voice message includes a first emotion information, the processor generates a first emotional state according to the feature points, and according to the first Emotional information generates a second emotional state, and determines whether the first emotional state is the same as the second emotional state, and determines whether the first voice message includes a transaction information and an auxiliary information; when the first emotional state and the second emotional state are the same If the second emotional state is the same, the processor uses the first emotional state of the face as an instant emotional state; when the processor determines that the first emotional state is different from the second emotional state, the processor performs an emotional analysis Program; when the processor determines that the first voice message contains the transaction information and the auxiliary information, the processor generates a transaction response based on one of the financial transactions in the business database; the processor determines the first If a voice message contains only the transaction information or the auxiliary information, the processor executes a problem analysis process; wherein the processor generates a response example based on at least the transaction response and the real-time emotional state; and a display interface for the processing The device is electrically connected to receive and present the response example.

The present invention is to provide a customer service assistance device and method based on artificial intelligence. In addition to judging the emotional state of customers based on facial expressions and voices, it can also analyze the questions asked by customers at the same time, and integrate the results of problem analysis and emotional analysis to provide Provide customer service staff as a reference for response and effectively improve the quality of customer service.

The above description of the disclosure and the following description of the embodiments are used to demonstrate and explain the spirit and principle of the present invention, and to provide a further explanation of the scope of the patent application of the present invention.

The detailed features and advantages of the present invention will be described in detail in the following embodiments. The content is sufficient to enable anyone familiar with the relevant art to understand the technical content of the present invention and implement it accordingly, and according to the content disclosed in this specification, the scope of patent application and the drawings. Anyone who is familiar with relevant skills can easily understand the purpose and advantages of the present invention. The following examples further illustrate the viewpoints of the present invention in detail, but do not limit the scope of the present invention by any viewpoint.

Please refer to Figure 1A, Figure 1B to Figure 2. Figures 1A to 1B are schematic diagrams of the use of an artificial intelligence-based customer service assistant device 1 according to an embodiment of the present invention, and Figure 2 is an artificial intelligence-based customer service assistant device according to an embodiment of the present invention 1 structure diagram. The customer service assistant device 1 includes a memory 11, a processor 13 and a display interface 15.

The memory 11 can be a cloud hard disk, a physical hard disk, or implemented by any component with a storage function. The memory 11 includes a business database 111 and a response database 113. The business database 111 stores multiple financial transactions and threshold values, and the response database 113 stores multiple preset sentence patterns and multiple preset responses. Specifically, financial affairs can provide information required by banks to handle financial services, such as personal information that needs to be prepared for opening an account, or the process of reporting the loss of a credit card. The default sentence pattern is based on questions such as "what" and "how" to answer questions with unclear information. For example, when a customer says "I have something missing", the corresponding default sentence pattern can be "What is missing?"; or when the customer says "I want to manage money", the corresponding default sentence pattern can be " How do you manage your finances?".

The processor 13 may be implemented by a central processing unit (CPU) or other components with sufficient computing functions. When the memory 11 is implemented as a physical hard disk, the processor 13 is electrically connected to the memory 11; on the other hand, when the memory 11 is implemented as a cloud hard disk, the processor 13 communicates with the memory 11 through a network connect. The processor 13 can search for relevant data from the business database 111 and the response database 113 of the memory 11 according to the facial images and voices of the customer to generate the real-time emotional state ES and response sample A, and will respond to sample A and marketing suggestions S is sent to the display interface 15 for reference by the customer service staff for answering. On the other hand, when the instant emotional state ES exceeds the threshold value, the processor 13 may also generate a marketing suggestion S. The detailed operation of the processor 13 will be described in subsequent paragraphs.

The display interface 15 can be implemented by any screen, and is electrically connected to the processor 13. The display interface 15 can present the face image of the customer, and can play back the customer's voice message through the microphone. While presenting the facial image and the voice message, the display interface 15 can present the response example A at the same time, and when the real-time emotional state ES exceeds the threshold value, it receives and presents the marketing suggestion S. Therefore, through the display interface 15, the user can observe the customer's facial image and voice message, and refer to the response example A and the marketing suggestion S at the same time, to determine the content of the subsequent response. In addition, through the user's self-setting, the display interface 15 can also display the above-mentioned real-time emotional state ES to provide a more complete analysis result.

To describe in detail the process of the processor 13 generating the instant emotional state ES, please refer to FIG. 3, and continue to refer to FIGS. 1A and 1B. FIG. 3 is a flowchart of an artificial intelligence-based customer service assistance method according to an embodiment of the present invention. When a customer’s facial image and voice message are sent to the customer service assistance device 1, please refer to step S101: the processor 13 receives the first facial image I1 and the first voice message V1, where the first facial image I1 includes A plurality of first feature points F1, and the first voice message V1 includes the first emotion information E1. The aforementioned multiple first feature points F1 may be facial features used to determine emotions, such as the corners of the mouth, the corners of the eyes, eyebrows, or facial muscle changes, and the first emotion information E1 may be the intonation change of the first voice message V1 Or emotional words. Please refer to step S103: the processor 13 generates a first emotional state ES1 based on the first feature points F1, and generates a second emotional state ES2 based on the first emotional information E1. Simply put, the first emotional state ES1 is generated based on the first facial image I1, and the second emotional state ES2 is generated based on the first voice message V1. After the processor 13 generates the first emotional state ES1 and the second emotional state ES2, please refer to the determination formula D1: the processor 13 determines whether the first emotional state ES1 and the second emotional state ES2 are the same. When the first emotional state ES1 and the second emotional state ES2 are the same, refer to step S105: use the first emotional state ES1 as the immediate emotional state ES. This real-time emotional state ES is the current emotional state of the customer, and the user can choose whether to present it on the display interface 15 or not. Conversely, when the first emotional state ES1 and the second emotional state ES2 are different, please refer to step S107: the processor 13 executes the emotional analysis program to generate the instant emotional state ES. The above sentiment analysis procedure will be explained in subsequent paragraphs.

Please continue to refer to Figure 3, Figure 1A and Figure 1B. As shown in FIG. 1B, when the customer's immediate emotional state ES has improved (for example, from angry to stable or happy), refer to the judgment formula D2: the processor 13 determines whether the immediate emotional state ES exceeds the threshold. When the instant emotional state ES has exceeded the threshold, please refer to step S109 in FIG. 3: the processor 13 generates a marketing recommendation S according to the financial transaction corresponding to the transaction response. The transaction reply is related to the financial affairs inquired by the customer, such as information required for opening an account or applying for a credit card, etc., and is generated after the processor 13 analyzes the voice message. The detailed process of the processor 13 generating a transaction reply will be described in subsequent paragraphs. Conversely, when the instant emotional state ES does not exceed the threshold value, it means that the current mood of the customer has not yet calmed down. Therefore, please proceed to step S111: the processor 13 does not generate a marketing recommendation S. Simply put, through the threshold, the processor 13 can further determine whether the customer's real-time emotional state ES is suitable for promoting financial products, so that the customer service staff can more accurately grasp the timing of the promotion. Step S1 is the final step of all sentiment analysis and problem analysis, so it will not be described first in this paragraph.

To describe in detail the process of the processor 13 generating a transaction reply, please refer to FIG. 4, and continue to refer to FIGS. 1A and 1B. FIG. 4 is a flowchart of an artificial intelligence-based customer service assistance method according to an embodiment of the present invention. After the processor 13 receives the first facial image I1 including a plurality of first feature points F1 and the first voice message V1 including the first emotion information E1 (that is, step S101 in FIG. 3), please refer to the determination formula D3: The processor 13 determines whether the first voice message V1 includes transaction information and auxiliary information; the transaction information is related to the financial item inquired by the customer, and the auxiliary information is the to-do list of the aforementioned financial item. For example, when the first voice message V1 is "My credit card is missing", "Credit card" is transaction information, and "Lost" is auxiliary information. When the first voice message V1 includes transaction information and auxiliary information, please refer to step S121: the processor 13 generates a transaction response based on the financial transaction in the business database 111; the above-mentioned financial transaction corresponds to the transaction information and auxiliary information. Continuing the previous example, the transaction reply can be the procedures and required information for reporting the loss of the credit card. On the other hand, when the first voice message V1 only contains transaction information or auxiliary information, please refer to step S123: the processor 13 executes the problem analysis program QA to generate a transaction reply. The above-mentioned problem analysis program QA will be described in detail in FIG. 6. In addition, when the first voice message V1 does not contain business information or auxiliary information at the same time, it is generally a daily greeting, so there is no need to deal with this situation.

To illustrate the sentiment analysis procedure, please refer to Figure 5, and continue to refer to Figures 1A and 1B. FIG. 5 is a detailed flowchart of an artificial intelligence-based customer service assistance method according to an embodiment of the present invention. According to the judgment formula D1 of FIG. 3, the emotion analysis program includes a sampling subroutine EA1 and a judgment subroutine EA2. The processor 13 executes the sampling subroutine EA1 first, and then executes the judgment subroutine EA2. In detail, the sampling subroutine EA1 further includes steps S1071 and S1072, and the judging subroutine EA2 includes step S1073. Please refer to step S1071: the processor 13 receives a second facial image I2 including a plurality of second feature points F2, and receives a second voice message V2 including the second emotion information E2. To put it simply, the second facial image I2 continues to appear after the first facial image I1, and the second voice message V2 continues to appear after the first voice message V1. Please refer to step S1072: the processor 13 generates a third emotional state ES3 according to the second feature points F2, and generates a fourth emotional state ES4 according to the second emotional information E2. Specifically, the third emotional state ES3 is connected to the first emotional state ES1 and is determined by facial features, and the fourth emotional state ES4 is connected to the second emotional state ES2 and is determined by voice messages. After the processor 13 generates the third emotional state ES3 and the fourth emotional state ES4, it will further determine whether the third emotional state ES3 and the fourth emotional state ES4 are the same (that is, the judgment formula D4). When the third emotional state ES3 is the same as the fourth emotional state ES4, please refer to step S1073: use the third emotional state ES3 as the immediate emotional state ES. Conversely, when the third emotional state ES3 and the fourth emotional state ES4 are different, the processor 13 executes the sampling subroutine EA1 and the subsequent determination subroutine EA2 again, and repeatedly compares the facial features and voice messages obtained each time. Until the customer’s immediate emotional state ES can be obtained.

To illustrate the problem analysis program QA in detail, please refer to Figure 6, and continue to refer to Figures 1A and 1B. FIG. 6 is a detailed flowchart of an artificial intelligence-based customer service assistance method according to an embodiment of the present invention. The judgment formula D3 has been described in FIG. 4, so the description will not be repeated here. When the customer's question is not complete enough that the first voice message V1 does not contain auxiliary information or business information at the same time, please refer to the judgment formula D5: the processor 13 determines that the first voice message V1 contains auxiliary information or business information. After the processor 13 determines that the first voice message V1 contains auxiliary information or business information, please refer to step S1231: when the first voice message V1 only contains auxiliary information, the processor 13 generates it according to the preset sentence pattern in the response database 113 Business reply, in which the above-mentioned default sentence pattern corresponds to auxiliary information. As in the example in the preceding paragraph, when a customer says "I have something missing", "Something is missing" is auxiliary information. In this case, the corresponding default sentence pattern is "what", and the processor 13 can reply "what is missing?" accordingly to assist the customer service staff in asking for further information. On the other hand, please refer to step S1232: when the first voice message V1 only contains transaction information, the processor 13 generates a transaction response based on the default response in the response database 113; wherein the above-mentioned default response corresponds to the transaction information. For example, when a customer says "I want to ask a question about a credit card", the "credit card" is the transaction information, and the processor 13 can find the corresponding default response from the response database 113 based on the "credit card" (for example: Do you want to apply for a credit card?) in order to help customer service staff to respond quickly to customers and further assist customers to provide more information.

For the presentation result of the customer service assistance method, please refer to Figure 7, and continue to refer to Figures 1A and 1B. FIG. 7 is a partial flowchart of an artificial intelligence-based customer service assistance method according to an embodiment of the present invention. After the processor 13 has generated the real-time emotional state ES, the marketing suggestion S (depending on the real-time emotional state ES is selectively generated), and the transaction response, please refer to step S1: use the processor 13 to generate at least the transaction response and the real-time emotional state ES Response sample A is sent to the display interface 15 to present response sample A. To put it simply, the processor 13 integrates the customer's real-time emotional state ES and the transaction reply into the response example A, and presents it on the display interface 15 to assist the customer service staff in responding in the most appropriate way. When the customer’s immediate emotional state ES is angry, response example A can express the transaction response in a concise and clear way. Conversely, when the customer’s immediate emotional state ES is happy, the response example A may further include a marketing suggestion S to assist the customer service staff in promoting financial products.

In summary, the present invention is to provide a customer service assistance device and method based on artificial intelligence. In addition to judging the emotional state of customers based on facial expressions and voices, it can also analyze the questions asked by customers at the same time, and analyze the problems and emotions. The results are integrated and provided to the customer service staff as a reference for response, effectively improving the quality of customer service.

Although the present invention is disclosed in the foregoing embodiments, it is not intended to limit the present invention. All changes and modifications made without departing from the spirit and scope of the present invention fall within the scope of the patent protection of the present invention. For the scope of protection defined by the present invention, please refer to the attached scope of patent application.

1: Customer service assistance device based on artificial intelligence 11: Memory 111: Business Database 113: Response Database 13: processor 15: display interface F1: the first feature point F2: second feature point I1: The first facial image I2: Second facial image E1: First Emotional Information E2: Second Emotional Information V1: The first voice message V2: Second voice message ES1: the first emotional state ES2: second emotional state ES3: The third emotional state ES4: Fourth emotional state ES: Instant emotional state EA1: sampling subroutine EA2: Judgment subroutine A: Response example S: Marketing advice

1A to 1B are schematic diagrams of using an artificial intelligence-based customer service assistant device according to an embodiment of the present invention. FIG. 2 is a structural diagram of an artificial intelligence-based customer service assistance device according to an embodiment of the present invention. FIG. 3 is a flowchart of an artificial intelligence-based customer service assistance method according to an embodiment of the present invention. FIG. 4 is a flowchart of an artificial intelligence-based customer service assistance method according to an embodiment of the present invention. FIG. 5 is a detailed flowchart of an artificial intelligence-based customer service assistance method according to an embodiment of the present invention. FIG. 6 is a detailed flowchart of an artificial intelligence-based customer service assistance method according to an embodiment of the present invention. FIG. 7 is a partial flowchart of an artificial intelligence-based customer service assistance method according to an embodiment of the present invention.

1: Customer service assistance device based on artificial intelligence

11: Memory

111: Business Database

113: Response Database

13: processor

15: display interface

F2: second feature point

I2: Second facial image

V2: Second voice message

ES3: The third emotional state

ES4: Fourth emotional state

ES: Instant emotional state

A: Response example

S: Marketing advice

Claims (10)

A customer service assistance method based on artificial intelligence includes: receiving a first facial image and a first voice message with a processor, wherein the first facial image includes a plurality of first feature points, and the first voice message includes A first emotional information; use the processor to generate a first emotional state according to the first feature points, and generate a second emotional state based on the first emotional information; use the processor to determine the first emotional state and the Whether the second emotional state is the same; when the first emotional state is the same as the second emotional state, use the first emotional state as an instant emotional state; when the first emotional state is different from the second emotional state, use the The processor executes an emotion analysis program to generate the real-time emotional state; uses the processor to determine whether the first voice message includes a transaction information and an auxiliary information; when the processor determines that the first voice message includes the transaction information and the Auxiliary information, the processor generates a transaction reply based on a financial transaction in a business database corresponding to the transaction information and the auxiliary information; when the processor determines that the first voice message only contains the transaction information or the auxiliary information Information, the processor executes a problem analysis process to generate the transaction response; and the processor generates a response example based on at least the transaction response and the real-time emotional state, and sends it to a display interface to present the response example, wherein the The emotion analysis program includes a judging subroutine, and the judging subroutine includes: Use the processor to determine whether a third emotional state is the same as a fourth emotional state; and when the processor determines that the third emotional state is the same as the fourth emotional state, use the third emotional state as the instant emotional state , Wherein the third emotional state follows the first emotional state and is determined by facial features, and the fourth emotional state follows the second emotional state and is determined by voice messages, wherein the question The analysis procedure includes: judging by the processor that the first voice message only contains the auxiliary information or the business information; A predetermined sentence pattern in a response database of the information generates the transaction reply, and the predetermined sentence pattern corresponds to the auxiliary information. The customer service assistance method according to claim 1, wherein the emotion analysis program includes: executing a sampling subroutine with the processor; and after the processor executes the sampling subroutine, executing the judging subroutine with the processor. The customer service assistance method according to claim 2, wherein the sampling subroutine includes: receiving, by the processor, a second facial image including a plurality of second feature points and a second voice message including a second emotion information And using the processor to generate the third emotional state according to the second feature points, and generate the fourth emotional state according to the second emotional information. The customer service assistance method according to claim 3, wherein the judging subroutine further includes: when the processor judges that the third emotional state is different from the fourth emotional state, the processor executes the sampling subroutine again. The customer service assistance method according to claim 1, wherein the problem analysis program further includes: when the processor determines that the first voice message only contains the transaction information, the processor uses the response database corresponding to the transaction information One of the default answers in the replies to the transaction. In the customer service assistance method described in claim 1, an example of using the processor to generate the response based on at least the transaction response and the real-time emotional state includes: determining whether the real-time emotional state exceeds a threshold by the processor; when the real-time emotional state When the state exceeds the threshold, the processor generates a marketing recommendation based on the financial transaction corresponding to the transaction response; and the processor generates the response example based on the marketing recommendation. A customer service assistance device based on artificial intelligence includes: a business database storing multiple financial transactions; a processor connected to the business database for receiving a first facial image and a first voice message, The first facial image includes a plurality of first feature points, the first voice message includes a first emotion information, the processor generates a first emotional state according to the first feature points, and according to the first emotion The information generates a second emotional state, and it is determined whether the first emotional state is the same as the second emotional state, and whether the first voice message includes a transaction information and Auxiliary information; when the first emotional state is the same as the second emotional state, the processor uses the first emotional state as an instant emotional state; when the processor determines the first emotional state and the second emotional state Differently, the processor executes a sentiment analysis program; when the processor determines that the first voice message contains the transaction information and the auxiliary information, the processor generates it according to one of the financial transactions in the business database A transaction reply; the processor determines that the first voice message only contains the transaction information or the auxiliary information, then the processor executes a problem analysis procedure, wherein the processor generates a response at least based on the transaction reply and the real-time emotional state Example; a display interface electrically connected to the processor to receive and present the response example, wherein when the processor executes the emotion analysis program, the processor is further used to determine a third emotional state and a fourth emotional state If the emotional state is the same, the third emotional state is taken as the instant emotional state, where the third emotional state is continued from the first emotional state and is judged by facial features, and the fourth emotional state is continued from The second emotional state is determined by the voice message; and a response database is electrically connected to the processor and stores a plurality of preset sentence patterns, wherein when the processor determines that the first voice message only contains For the auxiliary information, the processor generates the business response according to the predetermined sentence pattern corresponding to the auxiliary information, and the predetermined sentence pattern corresponds to the auxiliary information. The customer service assistance device according to claim 7, wherein the response database further includes a plurality of preset responses, wherein when the processor determines that the first voice message only includes the transaction information, the processor corresponds to the transaction The default reply to the information generates a reply to the transaction. The customer service assistance device according to claim 7, wherein the business database further includes a threshold value, and when the processor determines that the real-time emotional state exceeds the threshold value, the processor generates a response based on the financial transaction corresponding to the transaction response One-line marketing suggestions and sent to the display interface. The customer service assistance device according to claim 7, wherein when the processor executes the emotion analysis program, the processor is further configured to execute a sampling subroutine to receive a second facial image including a plurality of second feature points And a second voice message containing a second emotional information, and generate the third emotional state based on the second feature points, and generate the fourth emotional state based on the second emotional information, when the processor determines the first emotional state If the third emotional state is different from the fourth emotional state, the processor executes the sampling subroutine again.

Images