JP7324354B2 - Systems and methods for intent discovery from multimedia conversations - Google Patents

Description

The present disclosure relates to machine learning procedures for multimedia conversations and natural language processing, and in particular to discovery intent from multimedia conversations such as audio, web chat, and video.

The intent discovery system may utilize supervised machine learning to perform natural language processing. Supervised machine learning training procedures rely on subject matter experts (SMEs) manually identifying individual audio data and labeling them with intentions, which are then used by machine learning models used to adjust the weights, biases, and other metrics of Training often requires considerable effort by subject matter experts to achieve acceptable model performance. Fatigue, user error, and other human limitations can constrain model performance, prolong training, or make training infeasible.

The present disclosure relates to systems and methods for discovering intent from multimedia conversations.

In one embodiment, a method of intent discovery is disclosed. The method may include receiving a multimedia conversation. The method may include obtaining optimal parameter settings based on the topic relevance quantifier. The topic relevance quantifier includes a machine learning model trained to predict optimal settings for ensuring relevance of extracted topics. The method includes the steps of: converting multimedia speech into text information based on optimal parameter settings; normalizing the text information into a document corpus based on the optimal parameter settings by a natural language processing mechanism; and extracting topics from the document corpus to process the multimedia conversation based on the optimal parameter settings. The extracted topics relate to groups of topic words. The method may include determining separate contextual metrics for each of the topics. The method may comprise selecting relevant topics for intent discovery from the extracted topics based on individual contextual metrics. The method may include obtaining documents related to the relevant topic from the document corpus. The method may include displaying information of documents related to relevant topics via a user interface. The method may include receiving feedback data regarding documents related to the relevant topic. The method may include determining intent information related to the relevant topic based on the feedback data.

In another embodiment, a system for intent discovery is disclosed. The system may receive multimedia conversations. The system may obtain optimal parameter settings based on the topic relevance quantifier. A topic relevance quantifier may include a machine learning model trained to predict topic relevance. The system may convert multimedia conversations into textual information based on optimal parameter settings. The system may normalize the textual information into the document corpus based on optimal parameter settings. The system may extract topics from the document corpus based on optimal parameter settings. A topic may relate to a group of topic words. The system may determine separate contextual metrics for each of the topics. The system may select relevant topics for intent discovery from the extracted topics based on individual contextual metrics. The system may obtain documents related to related topics from the document corpus. The system may display information of documents related to relevant topics via a user interface. The system may receive feedback data regarding documents related to relevant topics. The system may determine intent information related to related topics based on the feedback data.

In another embodiment, a non-transitory computer-readable storage medium contains computer-executable instructions. The computer-executable instructions, when implemented by a processor, cause the processor to perform the methods described above.

The above embodiments and other aspects and options of their implementation are explained in more detail in the drawings, the description and the appended claims.

Embodiments may be better understood with reference to the following drawings and description. Components in the drawings are not necessarily to scale. Further, in the drawings, the same reference numerals indicate corresponding parts throughout the different drawings.

A first example of a system of intent discovery is shown. 4 shows an example of audio call processing. Examples of topic evaluation and selection are provided. Examples of rule formulation and intent refinement are given. An example of optimal parameter setting search is shown. Examples of training a topic relevance quantifier model and a parameterized relevance quantifier model are shown. Figure 2 shows a flow diagram of the system; A second example of a system of intent discovery is shown.

In various examples, systems and methods may discover intent from a collection of multimedia conversations. This is done in various stages. In the first stage, topics are extracted from the text or transcription of the multimedia conversation. Related topics are then selected from the extracted topics and used for discovery intent. Optionally, the selection of related topics is used to create rules used to discover refined intents. The optimal parameters of the system for topic extraction are found during the training phase. Typical implementations of the various stages of the system are described in FIGS. 2-6 and related descriptions.

The intent discovery systems and methods described herein may provide various technical advances. As an example, the intent discovery system described herein may provide a framework for discovering intent from audio calls to support analysis of customer needs. An intent discovery system can discover intent from unlabeled audio or text content based on unsupervised detection of high-proximity and well-balanced topics, supplemented by supervised rule-based intent elaboration.

Another example of the technological advances achieved by the systems and methods described herein is that intent is discovered by promoting expert oversight in a more useful manner. The most informative topics are selected based on metrics detailed below. The SME or business analyst then uses the information provided by the selected topic to discover intent or uses rules to discover more granular or refined intent. can search for documents.

The systems and methods described herein provide improvements over existing market solutions. Further advantages, efficiencies, and improvements over existing market solutions are demonstrated in the systems and methods described below.

FIG. 1 shows a first example of system 100 . System 100 may discover intent from multimedia conversations. Multimedia conversations may include audio, voice chat, web chat, web transcription, video, and any other form of suitable multimedia information. A collection of audio calls 102 is used herein as a typical example of a multimedia conversation to describe various embodiments. Audio call 102 may include content that represents speaker intent, such as a recording of a customer phone service conversation between a customer and a business representative. Audio call 102 can be stored electronically in a variety of ways, such as audio files, bulk data in databases, and the like. System 100 may include conversation processor 110 , topic evaluator 130 , topic selector 140 , intent definer 145 , rule formulator 150 , intent refiner 160 , intent statistics component 170 , and parameter retriever 180 .

Conversation processor 110 of system 100 may process audio call 102 to extract topics from audio call 102 based on the parameter settings file. The functionality of conversation processor 110 is described in detail with reference to FIG.

As shown in FIG. 2 , conversation processor 110 may include speech-to-text engine 212 , natural language processing (NLP) normalization component 214 , and topic extractor 216 . Parameter settings 204 may include all processing parameters used by conversation processor 110 and values for the parameters. Parameter settings 204 may be configured using a structured or semi-structured parameter settings file containing default settings. Additionally or alternatively, parameter settings 204 may be defined by a data scientist. Additionally or alternatively, parameter settings 204 may be provided by parameter retriever 180, which is described in detail below.

Speech-to-text engine 212 may perform speech recognition on audio call 102 to convert speech information to text information based on the parameters of speech-to-text engine 212 set in parameter settings 204 . Parameters of the speech-to-text engine 212 may include, by way of example, sample rate and audio filters based on the length of the audio call, and the like. Textual information may be stored in corpus database 202, as an example.

NLP normalization component 214 normalizes NLP textual information based on the NLP normalization component 214 parameters set in parameter settings 204 to generate a corpus of normalized speech documents corresponding to audio call 102 . procedures may apply. Normalization NLP procedures may include removing stop words and noise words, adding part-of-speech (PoS) tags, recognizing entities, and the like.

In some implementations, the NLP normalization component 214 may also extract multidimensional features, also called word embeddings, for each word in the spoken document. Word embeddings can express the meaning of words in a distributive semantic space. Word embeddings and other annotations may be stored in corpus 202 or other database in dictionary format. Parameters for the normalized NLP procedure and word embedding extraction are, by way of example, selection of filters based on regular expressions and PoS tags, minimum count value to consider words, window size for embeddings, type of embeddings and size, as well as the like.

Topic extractor 216 may process the normalized dialogue documents to extract topics 206 based on the parameters of topic extractor 216 set in parameter settings 204 . Parameters may include, as examples, the number of topics to be extracted, the number of iterations for topic extraction, the number of passes for topic extraction, and the like. Topics 206 may include groups of words in the normalized dialogue document. Topics 206 may be stored in a topic store, which may contain information about the topic associated with each topic word and each normalized conversational document.

Topic evaluator 130 and topic selector 140 work together to select relevant topics for intent discovery from the topics extracted by topic extractor 216 . Specifically, topic evaluator 130 may measure contextual metrics for each extracted topic 206 . Contextual metrics may include proximity scores and balance scores. A proximity score may indicate how spatially close the topic words of a topic in the normalized dialogue document are. That is, how relevant are the groups of words within a given topic in a conversational document. A balanced score may indicate how well the topic words of a topic express enough information to understand the specificity of the intent within the context of the normalized dialogue document. Topic selector 140 may select relevant topics from the topics extracted by topic extractor 216 based on the proximity scores and balance scores of the extracted topics. Exemplary functional details of topic evaluator 130 and topic selector 140 are described below with reference to FIG.

As shown in FIG. 3, topic evaluator 130 may include topic proximity evaluator 320 and topic balance evaluator 330 . Extracted topics 206 may be stored in topic store 301 . Topic proximity evaluator 320 reads topics 206 from topic store 301 and, based on the word embeddings extracted by NLP normalization component 214 and the word uniqueness scores in unique word dictionary 302, determines the proximity for each of the topics. Sex score may be measured. Topic proximity evaluator 320 may store topic proximity scores in topic store 301 .

An example proximity score between two words w _i and w _j may be calculated as follows.

During the ceremony,
Coh(w _i , w _j ) is the cosine similarity between the embeddings of w _i and w _j ,
RIDF(w) is the Residual Inverse Document Frequency (RIDF) of word w;
C ₂ ^N is the binomial coefficient.
The proximity score of a topic T is a combination of the above scores calculated among the Top-K words associated with T, eg using the average function as a combination.

Unique word dictionary 302 may be provided by word uniqueness evaluator 360 . In one implementation, word uniqueness evaluator 360 reads normalized speech documents from corpus database 202 and computes a uniqueness score for each word in the normalized conversational documents for its domain uniqueness. obtain. Uniqueness scores can be used to understand what words within a topic are relevant to the business domain. Finally, word uniqueness evaluator 360 stores the scores in unique word dictionary 302 . An example of a word uniqueness score for a topic T is computed based on the uniqueness of the Top-K words belonging to T, an example uniqueness measure is RIDF used in natural language processing.

Topic balance evaluator 330 may retrieve topics 206 from topic store 301 and calculate a balance score for each of the topics based on unique word dictionary 302 and frequent related word dictionary 304 . A balance score measures how the topic words within a given topic are balanced with respect to domain-specific words and frequently related words. Topic balance evaluator 330 may store topic balance scores in topic store 301 .

An example of a balanced score for topic T is described as follows.
Balance(T) = S_TF_RIDF + boost_IDF + boost_Impact
During the ceremony,
S_TF_RIDF is the sum of the TF-RIDF scores of the Top-K words, TF-RIDF is the multiplication between the Term Frequency (TF) and the RIDF score;
boost_IDF is the ratio of the Inverse Document Frequency (IDF) averaged among the Top-K words;
boost_Impact is the percentage of the impact score, such as the Prevalent Association Lift (PAL) score, for example detailed below, which indicates how strong the impact of the Top K words of topic T is in the conversation data. Take into consideration.

The frequent related word dictionary 304 may be provided by the joint context extractor 370 and word impact evaluator 380 . In some implementations, the context extractor 370 may extract a sliding window of words for each sentence in the conversation. The extracted window is used to compute the context of each word, the context being created by the set of words close to the word selected given the extracted window. Context from the normalized conversational documents is used with predetermined parameter settings to calculate an impact score.

Word influence evaluator 380 may utilize the context of each word extracted by context extractor 370 to measure the influence the word has in the normalized dialogue document. A high impact score means that the word directs the flow of the conversation. Word influence evaluator 380 may calculate influence scores by measuring the association between the two closest words in the extracted context. Words with higher influence are frequent words that have stronger associations with other words in the conversation. Finally, word impact evaluator 380 may store word impact scores in frequently related word dictionary 304 .

An example impact score is based on the PAL below.

During the ceremony,
S(w) is the support of word w, equal to count(w)/N, where N is the number of associations extracted from the context;
S(w _l →w _r ) is the support of the association (w _l →w _r ), which is equal to count(w _l →w _r )/N, where N is the number of associations extracted from the context. Yes, and count(w _l →w _r ) means the number of times w _l follows w _r .
As an example, the impact score could be calculated by using the PAL score and selecting the left word with the highest PAL value in it.

Continuing to refer to FIG. Topic selector 140 may include topic visualization component 340 . The topic visualization component 340 may implement interactive search for users such as data scientists, illustratively via a graphical user interface. As an example, a user may first enter a text query. A text query can be automatically calculated by user selection of words in the topic, using word part-of-speech tags, uniqueness scores, and influence scores. Alternatively, the text query may be defined directly by the user. In response to receiving the text query, topic visualization component 340 may utilize the text query to search the normalized dialogue document for text segments that contain the words in the text query. Topic visualization component 340 may then display the text segments associated with the topic via a graphical user interface. Displaying text segments related to a topic may help a user make a determination as to whether the topic is a related topic. Topic visualization component 340 may receive user decisions via a graphical user interface.

Here, the topic visualization component 340 provides an interactive search for data scientists to retrieve segments of text using words within the topic. Reading text segments, along with topical contextual metrics, helps data scientists make better selections of relevant topics.

Alternatively or additionally, topic selector 140 may include best topic selector 350 . As an example, the best topic selector 350 reads extracted topics from the topic store 301 and displays information for all extracted topics, including topic proximity scores and balance scores, via a graphical user interface. may Both the proximity score and the balance score facilitate the user's understanding of the topic. In this way, users may rank topics based on proximity scores and balance scores. The best topic selector 350 may select topics with rankings above a predetermined threshold as relevant topics. Alternatively, the user may directly select a portion of the topic as a related topic based on displayed information about the topic. Best topic selector 350 may receive such selections via a user graphical interface.

As described above with reference to FIG. 3, another example of the technical advances achieved by the systems and methods described herein is to provide intent discovery information based on topic proximity scores and balance scores. A selection of related topics to offer.

Return to FIG. The intent definer 145 may determine the association of intent information with related topics. As an example, intent definer 145 displays relevant topics to users, such as SMEs, via a graphical user interface. A user may enter an intent name for each of the related topics. Accordingly, intent definer 145 may label related topics with corresponding intent names.

Optionally, a related topic with an intent name such as "baggage inquiry" may be related to a large amount of normalized dialogue documents, such as 10% of all normalized dialogue documents. For example, intent definer 145 may display related topics and conversational documents related to related topics to the user via a graphical user interface. By reviewing related topics and related conversational documents, users may determine that intent refinement is necessary to discover sub-intents, also called refined intents, of related topics. As a result, intent definer 145 may trigger the intent refinement process performed by intent formulator 150 and intent refiner 160 .

The rule formulator 150 and the intent refiner 160 work together to refine the intent of the relevant topic. Specifically, rule formulator 150 may obtain a set of query rules corresponding to related topics. A query rule may be defined as a set of topic words within related topics combined with logical operators such as and, or, not. The intent refiner 160 utilizes each query rule as the query text to retrieve the individual conversational documents corresponding to the query rule from a corpus of normalized conversational documents or original conversational transcripts, and the business analyst Or, with the help of a user, such as an SME in a business, it may identify the refined intent contained in the retrieved individual conversational documents. In addition, intent statistics component 170 may perform various statistical operations on intents identified based on query rules. Exemplary functional details of rule formulator 150, intent refiner 160, and intent statistics component 170 are described below with reference to FIG.

In embodiments, rule formulator 150 may be implemented as rule discovery component 410 . Specifically, rule discovery component 410 may find a set of query rules that describe each of the topics based on their PoS tags and scores such as influence and proximity. Query rules can be augmented with semantically related words such as synonyms extracted from the ontology store 408, which may contain domain information. An example of an extracted rule would be a logical combination of related words extracted from the ontology store 408 in addition to the selected word.

In another embodiment, rule formulator 150 may be implemented as rule definition component 420 . Specifically, the rule definition component 420 may interact with users such as business analysts or SMEs, as examples. A user may manually define query rules for each of the related topics based on the information contained in the related topics. The rule definition component 420 may then receive the query rules defined by the user, illustratively via a user graphical interface.

Intent refiner 160 may include rule enforcement component 440 and intent labeler 450, as shown in FIG. The rule execution component 440 may utilize each query rule obtained from the rule formulator 150 to retrieve the conversational document 402 corresponding to the query rule from the indexed corpus of conversational documents 406 . As an example, rule execution component 440 may retrieve conversational documents 402 from indexed corpus 406 utilizing topic words in query rules as keywords or phrases. In some implementations, the rule execution component 440 further determines the retrieved conversational document 402 based on how relevant the query words in the query rule are to the retrieved conversational document 403. You can rank.

Indexed corpus 406 may be provided by indexer 430 . Specifically, indexer 430 indexes each of the normalized dialogue documents and original dialogue transcripts in corpus 202 by the words in the documents to produce indexed corpus 406 of dialogue documents. may be generated.

Intent labeler 450 may enlist the help of a user, such as an SME or business analyst, to label query rules with the name of the corresponding refined intent. As an example, the intent labeler 450 displays to the user, via the user graphical interface, three query rules describing related topics labeled with the intent name "baggage inquiry." The user specifies different refined intents for the three query rules, eg baggage restrictions, lost baggage, and damaged baggage. As a result, intent labeler 450 labels the three query rules with "baggage restrictions," "missing baggage," and "damaged baggage," respectively.

Through intent refinement, vague intents may be broken down into finer-grained intents. As an example, the "baggage inquiry" intent is split into three refined intents: "baggage restricted", "baggage lost", and "baggage damaged". In addition, query rule execution helps to understand how conversational documents pertaining to the vague "baggage inquiry" intent map to the refined intent. This creates an intent hierarchy.

Since a conversational document can represent more than one intent, it is possible that any individual conversational document 402 can be retrieved by executing more than one rule query. Accordingly, the intent document assigner 460 may assign the conversational document 402 to multiple intents and store the conversational document 402 in the intent store 404 along with the intent assignment information for the multiple intents. In some implementations, where rule execution component 440 provides a ranking score for each query rule execution, the ranking score of conversational document 402 for each intent is also stored in intent store 404 . This makes it easier to compute useful statistics, such as how many conversations in an audio collection are related to each intent, and which intent should be considered the matching primary intent of the conversation. will be.

Intentional utterance classifier 470 is an interactive utterance classifier for users, such as SMEs, to inspect read-out conversational document 402 for intent and to highlight sets of text segments in conversational document 402 that clearly express such intent. means may be implemented. Thus, the intentional utterance classifier 470 uses the retrieved speech document 402 and the emphasized speech document 402 such that the text classifier can classify all text segments in the retrieved speech document 402 that contain intent-related utterances. A text classifier may be trained using the text features of the extracted text segments.

As described above with reference to FIG. 4, another example of the technological advances achieved by the systems and methods described herein is intent gathering with an informed teacher.

Return to FIG. The parameter retriever 180 may provide optimal parameter settings to the conversation processor 110 so that the conversation processor 110 can use the optimal parameter settings to extract more useful topics in intent discovery. As an example, a topic containing the words "account, balance" has stronger relevance for intent discovery as compared to a topic containing the words "him, her, that". From an intent discovery perspective, topics containing the words “he, she, that” are considered noise and are not expected to be extracted by the conversation processor 110 . The search for optimal parameter settings by parameter searcher 180 during the training phase is described in detail with reference to FIG.

As shown in FIG. 5, parameter retriever 180 includes optimal parameter selector 510, conversation processor 520, topic relevance quantification component 530, relevance quantification result tracker 540, configuration feature extractor 550, and A topic feature extractor 560 may be included.

Configuration feature extractor 550 may extract a set of features representing values of parameters defined in parameter space 506 .

Optimal parameter selector 510 may use the parameter features extracted by setting feature extractor 550 and setting relevance quantifier 502 to search parameter space 506 to extract a set of candidate parameter settings 503. . Setting relevance quantifier 502 includes a machine learning model trained to predict the effect of candidate parameter settings on the relevance of topics extracted using such parameter settings. The setting relevance quantifier 502 is trained based on previously explored parameter settings. The mechanism for training the preference relevance quantifier 502 is detailed later.

By using the setting relevance quantifier 502, instead of searching through all possible parameter settings or simply making a random selection of possible parameter settings, the optimal parameter selector 510 finds the most likely relevant It would be nice to be able to predict which parameter settings could be used to extract topics.

Conversation processor 520 functions in a manner similar to conversation processor 110 of FIG. Specifically, the conversation processor 520 converts the audio collection 102 into text information based on the individual candidate parameter settings 503, normalizes the text information into the document corpus based on the individual candidate parameter settings 503, and converts the text information into the document corpus based on the individual candidate parameter settings 503. and extract a set of topics from the document corpus based on individual parameter setting candidates 503 .

A topic feature extractor 560 may extract multi-dimensional features for each topic extracted by each conversation processor 520 . In particular, topic feature extractor 560 extracts such multi-dimensional features of a topic, illustratively by aggregating word embeddings associated with each word belonging to the topic.

As shown in FIG. 5, using as input topic/setting feature objects 505 containing topics extracted by each conversation processor 520, multidimensional features of the extracted topics, and candidate parameter settings 503, topic The relevance quantifier component 530 may utilize the topic relevance quantifier 504 to measure the relevance of the extracted topics, where the relevance is expressed as an integer interval between 0 and 5 as an example. It may be expressed as a relevance score, where 0 means not relevant and 5 means very relevant. Topic relevance quantifier 504 includes a machine learning model trained to predict topic relevance for intent discovery based on parameter settings. Topic relevance quantifier 504 is trained based on previously extracted topics from audio collection 102 . The mechanism for training topic relevance quantifier 504 is described in detail later.

For each of the candidate parameter settings 503, the relevance quantification result tracker 540 compares the relevance scores of the extracted topics based on the candidate parameter settings and the predetermined relevance scores to determine whether the relevance criteria are met. may be compared to sex criteria. As an example, if 90% of the relevance scores reach the target relevance threshold, the relevance quantification result tracker 540 may determine that the candidate parameter settings met the relevance criteria. As a result, among the parameters of the parameter settings that meet the relevance criteria, the parameter setting candidate with the highest relevance score can be selected as the optimal parameter setting 508 for intent discovery in the system 100 .

On the other hand, if relevance quantification result tracker 540 determines that none of the candidate parameter settings 503 satisfy the relevance criteria, it directs optimal parameter selector 510 to select another set of candidate parameter settings 503. trigger. This forms a closed loop to automatically search the parameter space 506 for candidate parameter settings until a relevance criterion is met.

Additionally, relevance quantification result tracker 540 may check whether topic relevance quantifier 504 can still classify topics with good accuracy. If the relevance quantification results of the extracted topics start to deviate from the reference accuracy, i.e. if the topic relevance quantifier 504 cannot generalize well enough to new topics, relevance quantification result tracking Device 540 stops the closed loop. As a result, topic relevance quantifier 504 and setting relevance quantifier 502 may require further training to improve accuracy.

As described above with reference to FIG. 5, another example of the technical advances achieved by the systems and methods described herein is to determine optimal parameter settings for topic extraction and their impact on topic-relevant Finding by predicting using a relevance quantifier and a parameterized relevance quantifier.

A mechanism for training topic relevance quantifier 504 and setting relevance quantifier 502 is described below with reference to FIG.

As shown in FIG. 6, annotation executor 610 is a metacomponent that includes a plurality of speech processing units, speech processing units 1 through N, and user annotator 615 . Multiple speech processing units can run in parallel to process the audio collection 102 with separate parameterization files, namely parameterization files 1 through N. These parameter settings files may be selected by a user, such as a data scientist, at the start of an annotation run. Alternatively or additionally, these parameter setting files may be pre-selected according to predetermined selection rules.

Speech processing units 1-N function in a manner analogous to speech processing unit 110 of FIG. Taking the dialogue processing unit 1 as an example, the dialogue processing unit 1 converts the audio collection 102 into training text information based on the parameter setting file 1, and transforms the training text information into a training document corpus based on the parameter setting file 1. It may be normalized to generate a training set of word embeddings for the words in the training document corpus, and then extract a training set of topics from the training document corpus based on the parameter settings file1.

All topics extracted by dialogue processing units 1 through N may be input to user annotator 615 . User annotator 615 may, by way of example, display these topics to users, such as SMEs, via a graphical user interface. In doing so, the user may annotate the relevance of each topic to intent discovery. By way of example, a user may assign each topic a numerical score that expresses the topic's relevance to intent discovery.

Note that the annotation executor 610 may continue to operate using a different parameter setting file, if desired. In doing so, annotation executor 610 may output more topics annotated by the user for subsequent training of parameterization quantifier 502 and topic relevance quantifier 504 .

Functioning similarly to topic feature extractor 560 of FIG. Multidimensional features may be extracted. Additionally, functioning in a manner similar to configuration feature extractor 550 of FIG. 5, configuration feature extractor 630 may extract a set of features representing the values of the parameters defined in parameter configuration files 1 through N. .

Using the topics annotated with scores by the user annotator 615, the topic features extracted by the topic feature extractor 560, and the parameter setting file features extracted by the setting feature extractor 550, parameter setting and topic association The relevance quantifier trainer 640 trains the topic relevance quantifier 504 and the setting relevance quantifier 502 of the two machine learning models by learning the annotated topic relevance and corresponding parameter setting files. You can train. Various applicable machine learning models including, by way of example, artificial neural networks, decision trees, support vector machines, and the like, to train topic relevance quantifier 504 and setting relevance quantifier 502. can be adopted for

Specifically, using annotated topic scores and topic features as input, topic relevance quantifier 504 may be trained to learn whether a set of topics is relevant. . Using the scores of the annotated topics, the topic's contextual metrics stored in the topic store 301, and the parameter features of the corresponding parameter settings file as inputs, the setting relevance quantifier 502 extracts relevant topics from the audio collection 102. It may be trained to learn if a parameter settings file is available for extraction.

Parameter setting and topic relevance quantifier trainer 640 may include a loop in which model topic relevance quantifier 504 and model setting relevance quantifier 502 determine that their classification accuracy reaches a predetermined threshold. May be released when reached. Otherwise, the annotation executor 610 extracts more topics for annotation by the user, and then extracts various parameter configuration file to instantiate more dialogue processing units.

Here, rather than requiring users to label individual topics with intent as is done in traditional intent discovery techniques, the present disclosure simply determines whether topics are relevant for intent discovery. The user is asked to determine whether, such as by annotating each topic with a relevance score, as described above by way of example. An example relevance score may be an integer between 0 and 5, where 0 means not relevant and 5 means very relevant. As discussed with reference to FIG. 5, utilizing model topic relevance quantifier 504 and setting relevance quantifier 502 in searching for optimal parameter settings 508 incorporates human insight into the optimal parameter selection process. can be

Various components of the system 100 shown in FIGS. 1-6 have been described above. Note that system 100 may be implemented with additional, different, or fewer components than shown. Each component may include additional, different, or fewer components.

FIG. 7 shows a flow diagram of system 100 . Each step may include additional, different, or fewer operations than shown in FIG. Each step may be performed in a different order than shown in FIG.

In the embodiment shown in FIG. 7, system 100 may receive (710) a multimedia conversation. A multimedia conversation may be, by way of example, an audio call containing content that represents the speaker's intent, such as a recording of a customer telephone service conversation between a customer and a business representative. Multimedia conversations may be communicated to system 100 via application programming interfaces, message brokers, Representational State Transfer (REST), and/or any other suitable communication technology.

System 100 may obtain optimal parameter settings based on the topic relevance quantifier (720). Optimal parameter settings may include various parameters that can be used to extract relevant topics from multimedia conversations. A topic relevance quantifier includes a machine learning model trained to predict topic relevance. As an example, system 100 performs the optimal parameter setting search process described above with reference to FIG. During this process, the topic relevance quantifier measures the relevance of the extracted topics through the candidate parameter settings. If the topic's relevance meets the relevance criteria, the system 100 sets the candidate parameter setting as the optimal parameter setting. Otherwise, system 100 searches for the next candidate parameter setting until it meets the relevance criteria.

System 100 may process (730) a multimedia conversation, such as an audio call, based on the optimal parameter settings by performing steps 732-736. As an example, system 100 utilizes a speech-to-text engine to convert 732 the audio call to text based on optimal parameter settings. System 100 then normalizes 734 the text into the document corpus based on the optimal parameter settings using the natural language processing engine. The system 100 renders text by applying standard normalization NLP procedures, including, by way of example, removing stopwords and noise words, assigning part-of-speech tags, recognizing entities, and the like. may be processed. System 100 then extracts topics from the document corpus based on the optimal parameter settings (736). Topics may be represented as a topic data structure containing information about the topic related to each topic word and each document.

System 100 may then determine individual contextual metrics for each of the topics extracted at step 736 . By way of example, contextual metrics include topic proximity scores and balance scores. A proximity score indicates how spatially close the topic words of a topic in the document corpus are. A balanced score indicates how well the topic words of a topic express enough information to understand the specificity of the intent within the context of the document corpus.

System 100 may select 750 relevant topics for intent discovery from the extracted topics based on the contextual metrics. As an example, system 100 may select topics with the highest proximity and balance scores as relevant topics. As another example, the system 100 may display topics with their proximity and balance scores to a user, such as a data scientist, and with the user's help determine which topics are relevant topics. .

System 100 may obtain 760 documents related to relevant topics from the document corpus normalized in step 734 . As an example, system 100 may search a document corpus for documents containing topic words related to related topics.

System 100 may display information about documents related to the relevant topic via a user graphical user interface (770). As an example, system 100 displays the entire text of each document. As another example, system 100 displays multiple text segments of each document. A plurality of text segments includes topic words of related topics.

System 100 may receive 780 feedback data regarding conversational documents related to relevant topics. By way of example, inspection of the displayed text or text segments allows the user to determine whether the document clearly expresses its intent to correspond to the relevant topic. If such an intent appears, the user enters the name of the intent as feedback data. Optionally, if the related topic involves a large amount of conversational documents, for example 10% of all conversational documents, the intent corresponding to the related topic is vague and the intent refinement to discover sub-intents of the related topic. Indicates that the conversion is required. As described above with reference to FIG. 4, by reviewing related topics and displayed conversation documents related to related topics, the user may define a set of query rules as subtopics of related topics as an example. may specify different sub-intents for the query rule. Thus, feedback data may include the names of sub-intents.

System 100 may determine intent information related to related topics based on the feedback data (790). As an example, system 100 labels related topics with intent names. Alternatively, or additionally, when system 100 performs intent refinement on related topics, system 100 labels each of the query rules corresponding to the related topic with a distinct subordinate intent name. Additionally, the system 100 may find relevant utterances that represent intent or sub-intent in the document. As a further example, system 100 can calculate various useful statistics such as how many audio calls are targeted for this intent.

FIG. 8 shows another example of system 100 . System 100 may include communication interface 812 , input interface 828 , and/or system circuitry 814 . System circuitry 814 may include a processor 816 or multiple processors. Alternatively or additionally, system circuitry 814 may include memory 820 .

Processor 816 may be in communication with memory 820 . In some examples, processor 816 may also communicate with additional components such as communication interface 812, input interface 828, and/or user interface 818. Examples of processor 816 include general purpose processors, central processing units, logical CPUs/arrays, microcontrollers, servers, application specific integrated circuits (ASICs), digital signal processors. , a field programmable gate array (FPGA), and/or digital circuitry, analog circuitry, or some combination thereof.

Processor 816 may be one or more devices operable to execute logic. The logic is stored in memory 820 or other memory and, when executed by processor 816, provides processor 816 with conversation processor 110, topic evaluator 130, topic selector 140, rule formulator 150, intent refiner 160, It may include computer-executable instructions or computer code that cause operations of intent statistics component 170, parameter search 180, and/or system 100 to be performed. Additionally, the processor 816 includes the operations of the velocity textization engine 212, the NLP normalization component 214, and/or the topic extractor 216 for audio call processing, the topic proximity evaluator 320 for topic selection, the topic balance evaluation Rule discovery component for operation, rule formulation and intent refinement of generator 330, topic visualization component 340, best topic selector 350, word uniqueness evaluator 360, context extractor 370, and/or word impact evaluator 380 410, operation of rule definition component 420, rule execution component 440, indexer 430, intent labeler 450, intent document assigner 460, and/or intent utterance classifier 470, configuration feature extractor 550 for optimal parameter configuration search, optimal Operation of Parameter Selector 510, Conversation Processor 520, Topic Relevance Quantifier Component 530, Relevance Quantification Results Tracker 540, and/or Topic Feature Extractor 560, Parameterized Relevance Quantifier and Topic Relevance Quantifier may cause the operations of annotation executor 610, user annotator 615, topic feature extractor 620, configuration feature extractor 630, and/or parameter configuration and topic relevance quantifier trainer 640 to train an organ. . The computer code may include instructions executable using processor 816 .

Memory 820 may be any device for storing and retrieving data, or any combination thereof. Memory 820 may be random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), or flash memory. Non-volatile and/or volatile memory may be included. Alternatively, or in addition, memory 820 may include optical, magnetic (hard drives), solid state drives, or any other form of data storage device. Memory 820 stores at least one of conversation processor 110, topic evaluator 130, topic selector 140, rule formulator 150, intent refiner 160, intent statistics component 170, parameter search 180, and/or system 100. may include Alternatively or additionally, memory may include any other components or subcomponents of system 100 described herein.

User interface 818 may include any interface that displays graphical information. System circuitry 814 and/or communication interface(s) 812 may communicate signals or commands to user interface 818 that cause the user interface to display graphical information. Alternatively or additionally, user interface 818 may be remote to system 100, with system circuitry 814 and/or communication interface(s) communicating instructions, such as HTML, to the user interface to The interface may display, compile, and/or render information content. In some examples, the content displayed by user interface 818 may be interactive or responsive to user input. By way of example, user interface 818 may communicate signals, messages, and/or information back to communication interface 812 or system circuitry 814 .

System 100 can be implemented in many different ways. In some examples, system 100 may be implemented using one or more logical components. By way of example, the logical components of system 100 may be hardware or a combination of hardware and software. The logic component may be conversation processor 110, topic evaluator 130, topic selector 140, intent definer 145, rule formulator 150, intent refiner 160, intent statistics component 170, parameter search 180, or any It may contain components or subcomponents. In some examples, each logic component is an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), a digital logic circuit, an analog circuit, separate circuits or gates, or any other type of combination of hardware. , or combinations thereof. Alternatively or additionally, each component includes memory hardware, such as a portion of memory 820 that includes instructions executable by processor 816 or other processors to implement one or more of the features of the logic component. It's okay. If any one of the logical components includes a portion of memory containing instructions executable using processor 816, the component may or may not include processor 816. FIG. In some examples, each logical component is simply a portion of memory 820 or other physical memory that contains instructions executable using processor 816 or other processor(s), and other hardware. The features of the corresponding component may be implemented without the component containing any of the Each component may be interchangeably referred to as a hardware component because each component includes at least some hardware, even if the included hardware includes software.

Some features are shown stored on a computer-readable storage medium (eg, as logic implemented as computer-executable instructions or as data structures in memory). The system and its logic and data structures, in whole or in part, may be stored on, distributed over, and read from one or more types of computer-readable storage media. Examples of computer-readable storage media include hard disks, floppy disks, CD-ROMs, flash drives, cache, volatile memory, non-volatile memory, RAM, flash memory, or any other type of one or more computer-readable storage. A medium can be included. The computer-readable storage medium may include any type of non-transitory computer-readable medium such as CD-ROM, volatile memory, non-volatile memory, ROM, RAM, or any other suitable storage device.

The processing power of the system may be distributed among multiple entities, such as among multiple processors and memories, optionally including multiple distributed processing systems. Parameters, databases, and other data structures may be stored and managed separately, may be incorporated into a single memory or database, and may be logically and physically organized in many different ways. , linked lists, hash tables, or implicit storage mechanisms. Logic, such as programs or circuitry, may be combined or split between multiple programs, distributed across several memories and processors, and shared libraries (e.g., dynamic link libraries (DLLs) link library).

Regardless of the particular implementation described, all descriptions are exemplary in nature rather than limiting. By way of example, selected aspects, features, or components of each implementation are presented as being stored in memory(s), although all or part of one or more systems may be stored on a hard disk by way of example. , flash memory drives, floppy disks, and secondary storage devices such as CD-ROMs. Moreover, the various logic units, circuitry, and screen display functionality are only examples of such functionality, and any other configuration containing similar functionality is possible.

Discrete logic, software, or instructions for implementing the processes, methods, and/or techniques described above may be provided on a computer-readable storage medium. The functions, acts, or tasks illustrated in the drawings or described herein may be performed in response to one or more sets of logic or instructions stored in or on computer readable media. good. Any function, operation, or task may be independent of any particular type of instruction set, storage medium, processor, or processing strategy and may be implemented in software, hardware, integrated circuits, firmware, microcode, and software operating singly or in combination. can be performed by the like. Likewise, processing strategies may include multiprocessing, multitasking, parallel processing, and the like. In one example, the instructions are stored on a removable media device for reading by a local or remote system. In other examples, the logic or instructions are stored at a remote location for transfer over a computer network or over a telephone line. In yet another example, the logic or instructions are stored within a given computer and/or central processing unit (“CPU”).

Additionally, although specific components are described above, the methods, systems, and articles of manufacture described herein may include additional, fewer, or different components. By way of example, a processor may be implemented as a microprocessor, microcontroller, application specific integrated circuit (ASIC), discrete logic, or a combination of other types of circuits or logic. Likewise, the memory may be DRAM, SRAM, Flash, or any other type of memory. Flags, data, databases, tables, entities, and other data structures may be stored and managed separately, incorporated into a single memory or database, distributed, or distributed in many different forms. may be organized logically and physically in Components may operate independently or may be part of the same device executing the same or different programs. The components may reside on separate hardware such as separate removable circuit boards or may share common hardware such as the same memory and processor to implement instructions from memory. The programs may be part of a single program, separate programs, or distributed across several memories and processors.

A second action is sometimes said to be “in response to” a first action, regardless of whether the second action results directly or indirectly from the first action. The second action may occur at a significantly later time than the first action and still be in response to the first action. Similarly, the second action causes the second action to be performed directly by one or more of the intervening actions, even if an intervening action occurs between the first action and the second action. is sometimes said to be in response to the first action. As an example, if a first action sets a flag, and a third action initiates a second action after each time the flag is set, then the second action is triggered in response to the first action. can be assumed.

To clarify the use and thereby publicly inform, the phrase "at least one of <A>, <B>, ... and <N>" or "<A>, <B>, ... <N>, or at least one of the combinations thereof", or "<A>, <B>, . One or more members selected from the group comprising A, B, . means In other words, the phrase means any combination of one or more of the components A, B, . and other components may further include additional components not listed in combination.

While various embodiments have been described, it will be appreciated by those skilled in the art that many more embodiments and implementations are possible. Accordingly, the embodiments described herein are examples and not the only possible embodiments and implementations.

Claims (7)

A method of intent discovery, comprising:
receiving a multimedia conversation by a processor in communication with a memory ;
selecting , by the processor, a first set of candidate parameter settings from a parameter repository;
determining , by the processor, whether a relevance criterion is met for each candidate parameter setting in the first set of candidate parameter settings, comprising:
converting , by the processor, the multimedia conversation into text information based on the candidate parameter settings;
normalizing , by the processor, the textual information into a document corpus based on the candidate parameter settings by means of a natural language processing mechanism;
generating , by the processor, a set of word embeddings for words in the document corpus;
extracting , by the processor, a set of topics from the document corpus based on the candidate parameter settings;
obtaining , by the processor, individual topic features for each topic in the set of topics based on the set of word embeddings;
by the processor a separate relevance metric for each topic in the set of topics based on the topic features and a topic relevance quantifier comprising a machine learning model trained to predict topic relevance; and
determining , by the processor, whether the candidate parameter setting satisfies a relevance criterion by comparing the relevance metric to the relevance criterion; determining whether the relevance criteria are met for a candidate configuration;
In response to the relevance criterion being satisfied by at least one candidate parameter setting in the first set of candidate parameter settings, the processor causes the at least one candidate parameter setting in the first set of candidate parameter settings to selecting, from the candidate parameter settings, the candidate parameter setting that best satisfies the relevance criteria as the optimal parameter setting;
determining , by the processor, relevant topics from a set of topics extracted from the document corpus based on the optimal parameter settings;
determining , by the processor, intent information related to the determined relevant topic;
method including. The step of selecting the first set of candidate parameter settings from the parameter repository comprises:
selecting , by the processor, the first set of candidate parameter settings from a parameter repository based on a setting relevance quantifier, wherein the setting relevance quantifier determines the relevance of a topic to the intent discovery by: 2. The method of claim 1, comprising a machine learning model trained to predict the impact of candidate settings. The step of obtaining the optimal parameter settings comprises:
In response to the relevance criteria not being met by any candidate parameter settings in the first set of candidate parameter settings,
selecting , by the processor, a second set of candidate parameter settings from the parameter repository based on the setting relevance quantifier;
3. The method of claim 2, further comprising: determining , by the processor , for each candidate parameter setting in the second set of candidate parameter settings whether the relevance criterion is met. obtaining , by the processor, a plurality of training parameter settings;
Steps below:
converting , by the processor, the multimedia dialogue into training textual information based on the training parameter settings;
normalizing , by the processor, by the natural language processing mechanism, the training text information into a training document corpus based on the training parameter settings;
generating , by the processor, a training set of word embeddings for words in the training document corpus; and
extracting , by the processor, a training set of topics from the training document corpus based on the training parameter settings; and processing , by the processor, the multimedia conversation based on each of the training parameter settings individually. a step;
obtaining , by the processor, individual training topic features for each topic in the training set of topics based on the training set of word embeddings;
displaying , by the processor, a separate training set of topics via a user interface;
receiving , by the processor, an individual score input for each of the topics, the score input indicating the relevance of the topic to the intent discovery;
3. The method of claim 2, further comprising training, by the processor, a machine learning model based on the score input and the training topic features to obtain the topic relevance quantifier. extracting by the processor a separate training setting feature from each of the training parameter settings, the training setting feature indicating a value of a parameter in the training parameter setting; said extracting training configuration features;
5. The method of claim 4, further comprising training, by the processor, a machine learning model based on the score input and the training preference features to obtain the preference relevance quantifier. A system for intent discovery comprising a processor, the processor comprising:
receiving a multimedia conversation;
selecting a first set of candidate parameter settings from a parameter repository;
For each candidate parameter setting in the first set of candidate parameter settings,
converting the multimedia conversation into text information based on the candidate parameter settings;
normalizing the text information into a document corpus based on the parameter setting candidates;
generating a set of word embeddings for words in the document corpus;
extracting a set of topics from the document corpus based on the candidate parameter settings;
obtaining individual topic features for each topic in the set of topics based on the set of word embeddings;
Determining a separate relevance metric for each topic in the set of topics based on the topic features and a topic relevance quantifier comprising a machine learning model trained to predict topic relevance. and determining whether the candidate parameter setting satisfies a relevance criterion by comparing the relevance metric to the relevance criterion;
from the at least one candidate parameter setting in the first set of candidate parameter settings in response to the relevance criterion being satisfied by at least one candidate parameter setting in the first set of candidate parameter settings. , selecting the candidate parameter setting that best satisfies the relevance criterion as the optimal parameter setting;
determining relevant topics from a set of topics extracted from the document corpus based on the optimal parameter settings;
determining intent information related to the determined relevant topic;
A system configured to A non-transitory computer-readable storage medium containing computer-executable instructions, the instructions comprising:
receiving a multimedia conversation;
selecting a first set of candidate parameter settings from a parameter repository;
For each candidate parameter setting in the first set of candidate parameter settings,
converting the multimedia conversation into text information based on the candidate parameter settings;
normalizing the text information into a document corpus based on the parameter setting candidates;
generating a set of word embeddings for words in the document corpus;
extracting a set of topics from the document corpus based on the candidate parameter settings;
obtaining individual topic features for each topic in the set of topics based on the set of word embeddings;
Determining a separate relevance metric for each topic in the set of topics based on the topic features and a topic relevance quantifier comprising a machine learning model trained to predict topic relevance. and determining whether the candidate parameter setting satisfies a relevance criterion by comparing the relevance metric to the relevance criterion;
from the at least one candidate parameter setting in the first set of candidate parameter settings in response to the relevance criterion being satisfied by at least one candidate parameter setting in the first set of candidate parameter settings. , selecting the candidate parameter setting that best satisfies the relevance criterion as the optimal parameter setting;
determining relevant topics from a set of topics extracted from the document corpus based on the optimal parameter settings;
determining intent information related to the determined relevant topic;
A non-transitory computer-readable storage medium executable by a processor for

Images