CN113902952B - Feature evaluation method, device, electronic equipment and storage medium - Google Patents
Feature evaluation method, device, electronic equipment and storage medium Download PDFInfo
- Publication number
- CN113902952B CN113902952B CN202010642000.7A CN202010642000A CN113902952B CN 113902952 B CN113902952 B CN 113902952B CN 202010642000 A CN202010642000 A CN 202010642000A CN 113902952 B CN113902952 B CN 113902952B
- Authority
- CN
- China
- Prior art keywords
- feature
- sample data
- model
- evaluated
- features
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000011156 evaluation Methods 0.000 title claims abstract description 74
- 238000013210 evaluation model Methods 0.000 claims abstract description 87
- 238000012549 training Methods 0.000 claims abstract description 71
- 238000000034 method Methods 0.000 claims abstract description 65
- 230000008569 process Effects 0.000 claims abstract description 46
- 238000012360 testing method Methods 0.000 claims description 75
- 238000012216 screening Methods 0.000 claims description 28
- 238000004364 calculation method Methods 0.000 claims description 9
- 239000013598 vector Substances 0.000 description 19
- 238000010586 diagram Methods 0.000 description 8
- 230000008901 benefit Effects 0.000 description 6
- 230000007246 mechanism Effects 0.000 description 3
- 238000012795 verification Methods 0.000 description 3
- 230000006978 adaptation Effects 0.000 description 2
- 238000004590 computer program Methods 0.000 description 2
- 238000007477 logistic regression Methods 0.000 description 2
- 238000013528 artificial neural network Methods 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000008859 change Effects 0.000 description 1
- 238000013500 data storage Methods 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 239000000284 extract Substances 0.000 description 1
- 238000001914 filtration Methods 0.000 description 1
- 230000006870 function Effects 0.000 description 1
- 238000010801 machine learning Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/21—Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation
- G06F18/214—Generating training patterns; Bootstrap methods, e.g. bagging or boosting
Landscapes
- Engineering & Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Theoretical Computer Science (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Bioinformatics & Cheminformatics (AREA)
- Bioinformatics & Computational Biology (AREA)
- Artificial Intelligence (AREA)
- Evolutionary Biology (AREA)
- Evolutionary Computation (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Life Sciences & Earth Sciences (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The embodiment of the disclosure relates to a feature evaluation method, a feature evaluation device, electronic equipment and a storage medium. The method comprises the following steps: acquiring target sample data; wherein the target sample data comprises first sample data comprising basic features and second sample data comprising features to be evaluated; the basic features are features which are currently applied to the recommendation system, and the features to be evaluated are features which are not currently applied to the recommendation system; training a feature evaluation model based on target sample data, wherein in the model training process, intermediate data generated based on the first sample data is used as intermediate input quantity in the second sample data training process; and generating evaluation information of the feature to be evaluated based on the output result of the feature evaluation model. The technical scheme provided by the embodiment of the disclosure realizes simple and rapid evaluation of the effectiveness of the features to be evaluated by taking the basic features as reference basis, thereby further realizing low-cost, high-efficiency and automatic feature selection of the recommendation system.
Description
Technical Field
The embodiment of the disclosure relates to the technical field of computers, in particular to a feature evaluation method, a device, electronic equipment and a storage medium.
Background
In the field of service recommendation, service recommendation data is typically trained based on machine learning to generate a service recommendation model, and then related services recommended to a user are determined based on the service recommendation model. However, in order to obtain a better recommendation result during training of the service recommendation model, the technician needs to select a proper recommendation model and perform super-parameter tuning, and also needs to extract features from service recommendation sample data. The selection of the service recommendation features directly determines the accuracy of the prediction result of the service recommendation model. Therefore, the evaluation of the validity of the service recommendation feature becomes critical.
In the related art, in order to verify the validity of one service recommendation feature, multiple tests are generally required, a large amount of computing resources are input, and it takes a long time to verify the validity of the service recommendation feature.
Disclosure of Invention
The embodiment of the disclosure provides a feature evaluation method, a device, electronic equipment and a storage medium, which can simply and rapidly accurately evaluate the effectiveness of a feature to be evaluated. The technical scheme of the present disclosure is as follows:
according to a first aspect of the embodiments of the present disclosure, there is provided a feature evaluation method, including:
Acquiring target sample data; wherein the target sample data comprises first sample data comprising basic features and second sample data comprising features to be evaluated; the basic features are features which are currently applied to a recommendation system, and the features to be evaluated are features which are not currently applied to the recommendation system;
Training a feature evaluation model based on the target sample data, wherein in a model training process, intermediate data generated based on the first sample data is used as intermediate input quantity in a second sample data training process;
and generating evaluation information of the feature to be evaluated based on the output result of the feature evaluation model.
Optionally, the feature evaluation model includes a base sub-model and a test sub-model;
Training a feature evaluation model based on the target sample data, wherein in a model training process, intermediate data generated based on the first sample data is used as an intermediate input quantity in a second sample data training process, and the method comprises the following steps:
inputting first sample data into a basic sub-model in a feature evaluation model, and simultaneously inputting second sample data into a test sub-model in the feature evaluation model so as to train the feature evaluation model;
in the model training process, data output by any network layer of the basic sub-model is used as intermediate data generated based on the first sample data, and the intermediate data is input to any network layer of the test sub-model, so that the intermediate data is used as intermediate input quantity in the second sample data training process.
Optionally, the step of generating the evaluation information of the feature to be evaluated based on the output result of the feature evaluation model includes:
respectively obtaining the area under a first curve output by the basic submodel and the area under a second curve output by the test submodel;
calculating the area gain of the area under the second curve relative to the area under the first curve;
and generating evaluation information of the feature to be evaluated according to the area gain.
Optionally, the step of generating the evaluation information of the feature to be evaluated according to the area gain includes:
When the area gain is in a preset interval range, determining the feature to be evaluated as an effective feature;
and when the area gain is out of a preset interval range, determining the feature to be evaluated as an invalid feature.
Optionally, before training the feature evaluation model based on the target sample data, the method further includes:
acquiring feature screening sample data;
Determining feature statistical information of the feature to be evaluated based on the feature screening sample data;
and screening the characteristics to be evaluated according to the characteristic statistical information.
Optionally, the feature statistics include feature total amount, average feature amount and feature coverage.
Optionally, the feature to be evaluated comprises a single feature to be evaluated and/or a combination of features to be evaluated.
According to a second aspect of the embodiments of the present disclosure, there is provided a feature evaluation apparatus, including:
an acquisition unit configured to perform acquisition of target sample data; wherein the target sample data comprises first sample data comprising basic features and second sample data comprising features to be evaluated; the basic features are features which are currently applied to a recommendation system, and the features to be evaluated are features which are not currently applied to the recommendation system;
A training unit configured to perform training of a feature evaluation model based on the target sample data, in a model training process, taking intermediate data generated based on the first sample data as an intermediate input amount in the second sample data training process;
And a generating unit configured to perform generation of evaluation information of the feature to be evaluated based on an output result of the feature evaluation model.
Optionally, the feature evaluation model includes a base sub-model and a test sub-model;
the training unit is configured to perform:
inputting first sample data into a basic sub-model in a feature evaluation model, and simultaneously inputting second sample data into a test sub-model in the feature evaluation model so as to train the feature evaluation model;
in the model training process, data output by any network layer of the basic sub-model is used as intermediate data generated based on the first sample data, and the intermediate data is input to any network layer of the test sub-model, so that the intermediate data is used as intermediate input quantity in the second sample data training process.
Optionally, the generating unit includes:
An area acquisition subunit configured to perform acquisition of a first area under a curve output by the base submodel and a second area under a curve output by the test submodel, respectively;
A gain calculation subunit configured to perform calculation of an area gain of the area under the second curve relative to the area under the first curve;
and an information generation subunit configured to perform generation of evaluation information of the feature to be evaluated according to the area gain.
Optionally, the information generating subunit is configured to perform:
When the area gain is in a preset interval range, determining the feature to be evaluated as an effective feature;
and when the area gain is out of a preset interval range, determining the feature to be evaluated as an invalid feature.
Optionally, the feature evaluation device further includes:
a sample data acquisition unit configured to perform acquisition of feature screening sample data before training a feature evaluation model based on the target sample data;
a statistical information determination unit configured to perform determination of feature statistical information of the feature to be evaluated based on the feature screening sample data;
And the screening unit is configured to perform screening on the characteristics to be evaluated according to the characteristic statistical information.
Optionally, the feature statistics include feature total amount, average feature amount and feature coverage.
Optionally, the feature to be evaluated comprises a single feature to be evaluated and/or a combination of features to be evaluated.
According to a third aspect of embodiments of the present disclosure, there is provided an electronic device, comprising:
A processor;
A memory for storing executable instructions of the processor;
wherein the processor is configured to execute the instructions to implement the feature evaluation method described in the first aspect.
According to a fourth aspect of embodiments of the present disclosure, there is provided a storage medium, which when executed by a processor of an electronic device, causes the electronic device to perform the feature evaluation method of any embodiment of the present disclosure.
According to a fifth aspect of embodiments of the present disclosure, there is provided a computer program product, which when executed by a processor of an electronic device, implements the feature evaluation method according to any of the embodiments of the present disclosure.
The technical scheme provided by the embodiment of the disclosure at least brings the following beneficial effects: the feature evaluation model is trained based on the first sample data containing the basic features which are currently applied to the recommendation system and the second sample data containing the features to be evaluated which are not currently applied to the recommendation system, in the model training process, intermediate data generated based on the first sample data are used as intermediate input quantity in the second sample data training process, evaluation information of the features to be evaluated is generated according to the output result of the feature evaluation model, the technical problems that in the prior art, the calculation resource investment is large and the time is long in verification of the feature effectiveness are solved, simple and quick evaluation is carried out on the effectiveness of one or more features to be evaluated by taking the basic features as reference basis, and therefore feature selection of the recommendation system is further achieved with low cost, high efficiency and automation.
It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the disclosure.
Drawings
The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the disclosure and together with the description, serve to explain the principles of the disclosure and do not constitute an undue limitation on the disclosure.
FIG. 1 is a flow chart illustrating a feature evaluation method according to an exemplary embodiment.
FIG. 2 is a flow chart illustrating another feature evaluation method according to an example embodiment.
Fig. 3 is a schematic diagram illustrating a structure of a service recommendation characteristic evaluation model according to an exemplary embodiment.
Fig. 4 is a schematic diagram showing the structure of a feature evaluation system according to an exemplary embodiment.
Fig. 5 is a block diagram of a feature evaluation device, according to an example embodiment.
Fig. 6 is a block diagram illustrating a configuration of an electronic device according to an exemplary embodiment.
Detailed Description
In order to enable those skilled in the art to better understand the technical solutions of the present disclosure, the technical solutions of the embodiments of the present disclosure will be clearly and completely described below with reference to the accompanying drawings.
It should be noted that the terms "first," "second," and the like in the description and claims of the present disclosure and in the foregoing figures are used for distinguishing between similar objects and not necessarily for describing a particular sequential or chronological order. It is to be understood that the data so used may be interchanged where appropriate such that the embodiments of the disclosure described herein may be capable of operation in sequences other than those illustrated or described herein. The implementations described in the following exemplary examples are not representative of all implementations consistent with the present disclosure. Rather, they are merely examples of apparatus and methods consistent with some aspects of the present disclosure as detailed in the accompanying claims.
The difficulty of automatic feature selection is that the number of single and combined service recommendation features selectable in the recommended service is too large, and if each service recommendation feature is tried once, the cost is not acceptable, and how to quickly and massively evaluate the effectiveness of each service recommendation feature becomes important. In the related art, in a scheme for evaluating the effect of candidate features in batches based on a file-wise LR (Logistic Regression, LR, logistic regression model), an existing feature is first used to train a basic LR model, then a sub-LR model is trained for each candidate feature, the sub-LR model is added with the basic LR model as output, the sub-LR model is trained, the weights of the existing feature are fixed, and only the weights of the candidate features are updated.
However, the above technical solution has the following disadvantages: 1. only an LR model is supported, and most network models adopted during service recommendation are DNN (Deep Neural Network ) models, excellent characteristics are represented in the LR model, advantages are not necessarily maintained in the DNN model, poor characteristics are represented in the LR, and high effectiveness is possible in the DNN model. 2. The feature combination method only supports the Cross combination method, but cannot support the combination method of embedding vector multiplication embedding multiply and the combination method based on the attention mechanism attention which are common in DNN. 3. The basic model needs to be trained first, then the sub-model needs to be trained, the sample data needs to be trained for multiple times, the process is complex, and the efficiency is low.
Fig. 1 is a flowchart illustrating a feature evaluation method according to an exemplary embodiment, which is used in an electronic device as shown in fig. 1, and includes the following steps.
In step S11, target sample data is acquired; wherein the target sample data comprises first sample data comprising basic features and second sample data comprising features to be evaluated; the basic features are features that have been currently applied to a recommendation system, and the feature to be evaluated is a feature that has not been currently applied to the recommendation system.
The recommendation system can be understood as a system for service recommendation. The basic features may be features that have been applied to the recommendation system at present, that is, the basic features may be one or more based on the effective features that have been used when the recommendation system makes service recommendations. In addition, the base features may comprise single features and/or combined features. Specifically, the basic features may be preset by a technician according to experience, or may be effective features determined according to the feature evaluation method provided by the embodiment of the present invention. The feature to be evaluated is a feature that is not currently applied to the recommendation system, i.e. a feature for evaluating the validity, and may also be understood as a feature for evaluating whether or not the service recommendation based on the recommendation system is valid. The feature to be evaluated may include at least one group of features to be evaluated, and each group of features to be evaluated may be one or more. Optionally, the feature to be evaluated comprises a single feature to be evaluated and/or a combination of features to be evaluated. The advantage of this is that when the feature to be evaluated is a single feature to be evaluated, the validity of the single feature to be evaluated can be evaluated; when the feature to be evaluated is a combination feature to be evaluated, the validity of the combination feature to be evaluated can be evaluated; when the feature to be evaluated is a single feature to be evaluated and a combined feature to be evaluated, the validity of the single feature to be evaluated and the combined feature to be evaluated can be rapidly evaluated at the same time. The feature to be evaluated can be a combination of two single features or a combination of a plurality of single features, and the combined feature obtained by combining N (N is an integer greater than or equal to 2) single features can be called an N-order combined feature.
In one implementation of the disclosed embodiments, the step of obtaining the target sample data includes: determining basic characteristics and characteristics to be evaluated; extracting feature values corresponding to the basic features from a large amount of service recommendation data based on the basic features, and extracting feature values corresponding to the features to be evaluated from a large amount of service recommendation data based on the features to be evaluated; taking the basic features and the feature values corresponding to the basic features as first sample data, and taking the features to be evaluated and the feature values corresponding to the features to be evaluated as second sample data; the first sample data and the second sample data constitute target sample data. Illustratively, the basic feature and the feature to be evaluated input by the user are determined, for example, a feature evaluation instruction input by the user is received, and the feature evaluation instruction includes the basic feature and the feature to be evaluated. And then, acquiring service recommendation data, wherein the electronic equipment can acquire the service recommendation data from a local recommendation system and can also acquire the service recommendation data in a certain time period from a server. And respectively extracting characteristic values corresponding to the basic characteristics and the characteristics to be evaluated from the service recommendation data, taking the characteristic values corresponding to the basic characteristics and the basic characteristics as first sample data, and taking the characteristic values corresponding to the characteristics to be evaluated and the characteristics to be evaluated as second sample data.
For example, in the short video recommended service, when the electronic device evaluates whether a feature is effective in the short video recommendation, click request data of a short video stored locally may be used as service recommendation data, or click request data of the short video received by a server providing a short video service may be obtained from the server within a preset period of time, and the received click request data of the short video may be used as service recommendation data. And respectively extracting data (namely, feature values) corresponding to the basic features and the features to be evaluated from click request data (namely, service recommendation data) of the short video. For example, click request data for a short video may include: the user clicks and plays short videos by using the mobile terminal at 8 th night after 30 years old, the playing time of the short videos is 5 minutes, and the types of the short videos are entertainment videos. Also exemplary, the underlying features may include: the feature values corresponding to the basic features extracted from the service recommendation data are respectively as follows: the female, entertainment class, i.e. the first sample data, can be expressed as: { age of user, 30; sex of the user, female; short video type, entertainment class }; the feature values corresponding to the recommended features to be evaluated, which are extracted from the service recommendation sample data, are respectively: 5 minutes, 30 women; the second sample data may be expressed as: { short video playback duration, 5 minutes; age + gender, 30 girl }.
In step S12, the feature evaluation model is trained based on the target sample data, and in the model training process, intermediate data generated based on the first sample data is used as an intermediate input amount in the second sample data training process.
Illustratively, the target sample data is input into a feature evaluation model, which is trained based on the target sample data. The feature evaluation model comprises a plurality of network layers, and in the process of training the feature evaluation model, each network layer can process the first sample data and the second sample data respectively to generate intermediate data, and the intermediate data generated based on the first sample data is used as intermediate input quantity in the process of training the second sample data. For example, the data output by the network layer corresponding to the first sample data is used as intermediate data generated based on the first sample data, and the intermediate data generated based on the first sample data is input to a certain network layer corresponding to the second sample data, so that the intermediate data generated based on the first sample data is used as an intermediate input amount in the training process of the second sample data.
Optionally, the feature evaluation model includes a base sub-model and a test sub-model; training a feature evaluation model based on the target sample data, wherein in a model training process, intermediate data generated based on the first sample data is used as an intermediate input quantity in a second sample data training process, and the method comprises the following steps: inputting first sample data into a basic sub-model in a feature evaluation model, and simultaneously inputting second sample data into a test sub-model in the feature evaluation model so as to train the feature evaluation model; in the model training process, data output by any network layer of the basic sub-model is used as intermediate data generated based on the first sample data, and the intermediate data is input to any network layer of the test sub-model, so that the intermediate data is used as intermediate input quantity in the second sample data training process. The method has the advantages that the intermediate data generated based on the first sample data is used as the intermediate input quantity in the training process of the second sample data, so that the effectiveness of the characteristics to be evaluated is rapidly evaluated based on the basic characteristics.
By way of example, a feature evaluation model may be understood as a learning model that quickly determines the validity of a feature to be evaluated after inputting target sample data. Alternatively, the feature evaluation model includes a base sub-model and a test sub-model, i.e., the feature evaluation model is a hybrid network model constructed based on the base sub-model and the test sub-model. In one implementation of the disclosed embodiments, first sample data is input into a base sub-model in a feature evaluation model, while second sample data is input into a test sub-model in the feature evaluation model to train the feature evaluation model based on the first sample data and the second sample data. In the process of training the feature evaluation model, data output by any network layer of the basic sub-model is used as intermediate data generated based on the first sample data, and the intermediate data generated based on the first sample data is input to any network layer of the test sub-model, so that the intermediate data generated based on the first sample data is used as intermediate input quantity in the process of training the second sample data. It will be appreciated that any network layer of the base sub-model is connected to any network layer of the test sub-model to input data output by any network layer of the base sub-model to any network layer of the test sub-model.
Wherein the number of test sub-models is the same as the number of sets of features to be evaluated, e.g. the features to be evaluated comprise at least one set of features to be evaluated, the feature evaluation model comprises a base sub-model and at least one test sub-model. Any network layer of the base sub-model is forward connected to any of the same network layers of the respective test sub-models. The advantage of this arrangement is that the basic sub-model is independently trained and will not be affected by each test sub-model, since the output of a certain network layer of the basic sub-model is forward connected to a certain network layer of each test sub-model, the output result of the network layer of the basic sub-model will be referred to during the training of each test sub-model, and each test sub-model will not deliver gradients to the basic sub-model during back propagation. The basic sub-model and each test sub-model may be the same or different. Alternatively, the base sub-model and each test sub-model may both be DNN network models.
Optionally, the connection mode between the basic sub-model and the test sub-model includes a splicing connection mode and a summation connection mode. For example, if the second network layer of the basic sub-model is connected to the third network layer of the test sub-model in the forward direction, the splicing connection manner may be understood as that the vector output by the second network layer of the basic sub-model is spliced with the vector to be input to the third network layer of the test sub-model, and the spliced vector is input to the third network layer of the test sub-model. For example, when the vector output by the second network layer of the basic sub-model and the vector to be input to the third network layer of the test sub-model are both 5-dimensional vectors, and the basic sub-model and the test sub-model are connected in a splicing manner, the vector input to the third network layer of the test sub-model is a10 (5+5) -dimensional vector. The summing connection may be understood as a summation operation of a vector output from the second network layer of the basic sub-model and a vector to be input to the third network layer of the test sub-model, and inputting the summed vector to the third network layer of the test sub-model. For example, the vector output by the second network layer of the basic sub-model and the vector to be input to the third network layer of the test sub-model are both 10-dimensional vectors, and when the basic sub-model and the test sub-model are connected in a summation manner, the vector input to the third network layer of the test sub-model is a 10-dimensional vector.
Alternatively, the target sample data may be preprocessed before being input into the feature evaluation model, and the preprocessed target sample data may be input into the feature evaluation model. For example, if the basic feature or the feature to be evaluated is a continuous feature, discrete feature change is performed on the target sample data, so that the feature value corresponding to the basic feature or the feature to be evaluated in the target sample data becomes a discrete value.
In step S13, evaluation information of the feature to be evaluated is generated based on the output result of the feature evaluation model.
In one implementation of the disclosed embodiments, after the target sample data is input to the feature evaluation model, the feature evaluation model is trained based on the target sample data, and an output result of the feature evaluation model is obtained. The output result of the feature evaluation model may include AUC (Area Under Curve) and a loss function value, among others. The AUC may be determined based on ROC (Receiver Operating Characteristic, subject work feature) curves or PR (Precision Recall) curves, among others. And then, generating evaluation information of the feature to be evaluated according to the output result of the feature evaluation model. The evaluation information of the feature to be evaluated can comprise effective and ineffective, and when the feature to be evaluated is determined to be the effective feature, the addition of the feature to be evaluated can effectively improve the accuracy of service recommendation based on the recommendation system; when the feature to be evaluated is determined to be an invalid feature, the addition of the feature to be evaluated has little influence on improving the accuracy of the service recommendation based on the recommendation system, and even reduces the accuracy of the service recommendation based on the recommendation system.
According to the feature evaluation method provided by the embodiment of the disclosure, the feature evaluation model is trained based on the first sample data containing the basic features which are currently applied to the recommendation system and the second sample data containing the features to be evaluated which are not currently applied to the recommendation system, in the model training process, the intermediate data generated based on the first sample data are used as the intermediate input quantity in the second sample data training process, and the evaluation information of the features to be evaluated is generated according to the output result of the feature evaluation model, so that the technical problems of large calculation resource investment and long time consumption in feature validity verification in the prior art are solved, and the validity of one or more features to be evaluated is simply and quickly evaluated by taking the basic features as reference basis, so that feature selection of the recommendation system is further realized with low cost, high efficiency and automation.
It should be noted that, the feature evaluation scheme provided by the embodiment of the present disclosure may also be suitable for application scenarios in which validity of features is evaluated in a non-service recommendation field, and at this time, features and data in a recommendation system may be directly replaced with related features and related data of other application scenarios.
In an optional embodiment, the step of generating the evaluation information of the feature to be evaluated based on the output result of the feature evaluation model includes: respectively obtaining the area under a first curve output by the basic submodel and the area under a second curve output by the test submodel; calculating the area gain of the area under the second curve relative to the area under the first curve; and generating evaluation information of the feature to be evaluated according to the area gain. For example, since the feature evaluation model includes a base sub-model and a test sub-model, the output result of the feature evaluation model may include the output result of the base sub-model and the output result of the test sub-model. When the output result of the feature evaluation model is AUC, the output result of the feature evaluation model specifically includes a first AUC output by the basic submodel and a second AUC output by the test submodel. The AUC may be the area under the ROC curve, that is, the area enclosed by the ROC curve and the coordinate axis. Then, area gains of the second AUC relative to the first AUC are calculated respectively, and evaluation information of the feature to be evaluated is generated according to the area gains. The area gain of the second AUC relative to the first AUC is understood as the difference between the second AUC and the first AUC.
Optionally, the step of generating the evaluation information of the feature to be evaluated according to the area gain includes: when the area gain is in a preset interval range, determining the feature to be evaluated as an effective feature; and when the area gain is out of a preset interval range, determining the feature to be evaluated as an invalid feature. The method includes determining whether the area gain is within a preset interval range, determining the feature to be evaluated as an effective feature when the area gain is within the preset interval range, and determining the feature to be evaluated as an ineffective feature when the area gain is not within the preset interval range, that is, the area gain is outside the preset interval range. When the feature to be evaluated is at least one group of feature to be evaluated, the corresponding feature evaluation model comprises a basic sub-model and at least one test sub-model, and the number of the test sub-models is the same as the number of the groups of the feature to be evaluated. Thus, the area gain of the second AUC output by each test sub-model relative to the first AUC output by the base sub-model can be calculated, and then the evaluation information of each set of features to be evaluated can be generated according to each area gain. Illustratively, the features to be evaluated include five sets of features to be evaluated, and the output results of the test sub-models determined by the above scheme are shown in the following table:
Where auc_diff represents area gain, AUC in the above table represents AUC of the second ROC curve outputted from each test sub-model, and feature_num represents data amount of the second sample data (i.e., data to be evaluated). If the preset interval range is set to [0,0.02], the 1 st and 3 rd sets of features to be evaluated may be determined as valid features, and the 2 nd, 4 th and 5 th sets of features to be evaluated may be determined as invalid features.
In an alternative embodiment, before training the feature evaluation model based on the target sample data, the method further includes: acquiring feature screening sample data; determining feature statistical information of the feature to be evaluated based on the feature screening sample data; and screening the characteristics to be evaluated according to the characteristic statistical information. The method has the advantages that invalid characteristics to be evaluated can be filtered, the data size of target sample data is reduced, and the evaluation efficiency of the effectiveness of the characteristics to be evaluated based on the characteristic evaluation model is improved.
Illustratively, feature screening sample data is obtained, where feature screening sample data may be understood as small batch data. And determining feature statistical information of the feature to be evaluated according to the feature screening sample data, wherein the feature statistical information can comprise feature total amount, average feature amount and feature coverage. The method has the advantages that the feature to be evaluated is screened through the feature statistical information such as the feature total amount, the average feature amount and the feature coverage, the accuracy of screening the feature to be evaluated can be effectively improved, invalid features to be evaluated can be ensured to be screened out as much as possible, the data size of target sample data is reduced, and the speed of evaluating the feature to be evaluated is improved. The feature total amount may be understood as the total sum of feature values corresponding to a certain feature to be evaluated extracted from the feature screening sample data, and the average feature amount may be understood as an average value of feature total amounts of individual single features of the combined feature when the certain feature to be evaluated is the combined feature. Feature coverage may be understood as the size of the amount of sample data in the feature screening sample data that contains a feature to be evaluated. And screening the characteristics to be evaluated according to the characteristic statistical information, and filtering invalid characteristics to be evaluated. For example, the features to be evaluated with the feature coverage smaller than a certain threshold may be filtered out, and features to be evaluated with the feature total amount or the average feature amount larger than a preset threshold may be filtered out. Alternatively, the feature whose feature amount has not been changed after combination, such as the combination of the user ID and the sex, may be filtered out in advance, since the sex of the same user is fixed, the feature after combination is actually equivalent to the single feature of the user ID.
FIG. 2 is a flowchart illustrating another feature evaluation method, as shown in FIG. 2, for an electronic device, according to an exemplary embodiment, comprising the steps of:
In step S21, target sample data is acquired; wherein the target sample data comprises first sample data comprising basic features and second sample data comprising features to be evaluated; the basic features are features that have been currently applied to the recommendation system, and the features to be evaluated are features that have not been currently applied to the recommendation system.
It should be noted that, when the basic feature or the feature to be evaluated includes the combined feature, the combined feature data may be generated based on the target sample data, where the combined feature data includes the combined feature and a combined feature value corresponding to the combined feature.
In step S22, the first sample data is input into the basic sub-model in the feature evaluation model, while the second sample data is input into the test sub-model in the feature evaluation model, to train the feature evaluation model.
In step S23, in the model training process, data output from any network layer of the basic sub-model is used as intermediate data generated based on the first sample data, and the intermediate data is input to any network layer of the test sub-model to use the intermediate data as an intermediate input amount in the training process for the second sample data.
FIG. 3 is a schematic diagram of a feature evaluation model, according to an example embodiment. As shown in fig. 3, the feature evaluation model includes one basic sub-model 31 and N test sub-models 32, wherein the number of test sub-models is equal to the number of sets of features to be evaluated. As shown in fig. 3, any network layer of the basic sub-model 31 is forward connected to any same network layer of each test sub-model 32, and the basic sub-model 31 and each test sub-model 32 share one label, where label may be understood as a learning target for indicating a model to be evaluated. For example, in the short video recommendation service, the validity of the short video recommendation is generally determined by whether the user clicks on the recommended short video, and thus, when evaluating the validity of the short video recommendation feature, the clicked and un-clicked can be used as tag information of the base data and the data to be evaluated.
In one implementation of the disclosed embodiment, the Base data (Base Slot) is input into the Base sub-model 31, and at the same time, each set of data to be evaluated (Test Slot 1-Test Slot N) is respectively input into the corresponding Test sub-model 32, where the Test sub-models are in one-to-one correspondence with the data to be evaluated. Alternatively, the base sub-model 31 and each test sub-model 32 may each be a DNN network model. If the feature to be evaluated contained in the data to be evaluated involves combining the feature to be evaluated, the single feature may be combined at the feature level, such as a cross-cross combination, or may be combined within the network layer of the test sub-model 32, such as an embedded vector multiplication embedding multiply combination or a combination based on the attention mechanism attention. When the feature to be evaluated involves combining the feature to be evaluated, and combining the feature to be evaluated in the network layer of the test sub-model (such as embedding vector multiplication combination or combination based on attention mechanism), if the basic feature and the feature to be evaluated both involve the same single feature, the basic data input to the basic sub-model and the data to be evaluated input to the test sub-model may be the same, i.e. share the same data.
In step S24, the area under the first curve output by the basic submodel and the area under the second curve output by the test submodel are obtained, respectively.
In step S25, the area gain of the area under the second curve relative to the area under the first curve is calculated.
In step S26, when the area gain is within the preset interval range, the feature to be evaluated is determined as a valid feature.
In step S27, when the area gain is outside the preset interval range, the feature to be evaluated is determined as an invalid feature.
According to the feature evaluation method provided by the embodiment of the disclosure, the feature evaluation model is trained based on the first sample data containing the basic features which are currently applied to the recommendation system and the second sample data containing the features to be evaluated which are not currently applied to the recommendation system, in the model training process, the intermediate data generated based on the first sample data are used as the intermediate input quantity in the second sample data training process, and the evaluation information of the features to be evaluated is generated according to the output result of the feature evaluation model, so that the technical problems of large calculation resource investment and long time consumption in feature validity verification in the prior art are solved, and the validity of one or more features to be evaluated is simply and quickly evaluated by taking the basic features as reference basis, so that feature selection of the recommendation system is further realized with low cost, high efficiency and automation.
Fig. 4 is a schematic diagram showing the structure of a feature evaluation system according to an exemplary embodiment. As shown in fig. 4, a feature pool FeaturePool in the feature evaluation system stores preconfigured service recommendation features (including basic service recommendation features and service recommendation features to be evaluated) and a network model corresponding to the service recommendation features, a data reading module DATAREADER reads sample data from service recommendation sample data TRAINING DATA, a sample generator SampleGenerator extracts feature values corresponding to the service recommendation features according to the service recommendation features in FeaturePool to generate service recommendation data (including basic service recommendation data and service recommendation data to be evaluated), a trainer Trainer trains the service recommendation data based on the network model corresponding to the preconfigured service recommendation features in FeaturePool, during the training, stores the generated model parameters in a parameter server PARAMETER SERVER, and invokes a DNN engine DNN ENGINE to calculate DNN, and a feature collecting module FeatureMetric is used for collecting output results of the service recommendation feature evaluation model as an index of feature evaluation, such as AUC or Loss.
Fig. 5 is a block diagram of a feature evaluation device, according to an example embodiment. Referring to fig. 5, the apparatus may be applied to an electronic device including an acquisition unit 51, a training unit 52, and a generation unit 53.
An acquisition unit 51 configured to perform acquisition of target sample data; wherein the target sample data comprises first sample data comprising basic features and second sample data comprising features to be evaluated; the basic features are features which are currently applied to a recommendation system, and the features to be evaluated are features which are not currently applied to the recommendation system;
a training unit 52 configured to perform training of a feature evaluation model based on the target sample data, in which intermediate data generated based on the first sample data is used as an intermediate input amount in the training of the second sample data;
A generating unit 53 configured to generate evaluation information of the feature to be evaluated based on an output result of the feature evaluation model.
Optionally, the feature evaluation model includes a base sub-model and a test sub-model;
the training unit is configured to perform:
inputting first sample data into a basic sub-model in a feature evaluation model, and simultaneously inputting second sample data into a test sub-model in the feature evaluation model so as to train the feature evaluation model;
in the model training process, data output by any network layer of the basic sub-model is used as intermediate data generated based on the first sample data, and the intermediate data is input to any network layer of the test sub-model, so that the intermediate data is used as intermediate input quantity in the second sample data training process.
Optionally, the generating unit includes:
An area acquisition subunit configured to perform acquisition of a first area under a curve output by the base submodel and a second area under a curve output by the test submodel, respectively;
A gain calculation subunit configured to perform calculation of an area gain of the area under the second curve relative to the area under the first curve;
and an information generation subunit configured to perform generation of evaluation information of the feature to be evaluated according to the area gain.
Optionally, the information generating subunit is configured to perform:
When the area gain is in a preset interval range, determining the feature to be evaluated as an effective feature;
and when the area gain is out of a preset interval range, determining the feature to be evaluated as an invalid feature.
Optionally, the feature evaluation device further includes:
a sample data acquisition unit configured to perform acquisition of feature screening sample data before training a feature evaluation model based on the target sample data;
a statistical information determination unit configured to perform determination of feature statistical information of the feature to be evaluated based on the feature screening sample data;
And the screening unit is configured to perform screening on the characteristics to be evaluated according to the characteristic statistical information.
Optionally, the feature statistics include feature total amount, average feature amount and feature coverage.
Optionally, the feature to be evaluated comprises a single feature to be evaluated and/or a combination of features to be evaluated.
The specific manner in which the various modules perform the operations in the apparatus of the above embodiments have been described in detail in connection with the embodiments of the method, and will not be described in detail herein.
Fig. 6 is a block diagram illustrating a configuration of an electronic device according to an exemplary embodiment. As shown in fig. 6, the electronic device 60 includes a processor 61; a memory 62 for storing executable instructions of the processor 61, the memory 62 may include a RAM and a ROM; wherein the processor 61 is configured to execute the instructions to implement the above-described method.
In an exemplary embodiment, a storage medium is also provided, such as a memory (62) storing executable instructions, that are executable by a processor (61) of the electronic device to perform the above method. Alternatively, the storage medium may be a non-transitory computer readable storage medium, which may be, for example, ROM, random Access Memory (RAM), CD-ROM, magnetic tape, floppy disk, optical data storage device, and the like.
In an exemplary embodiment, a computer program product is also provided, which when executed by a processor of an electronic device, implements the above-described method.
Other embodiments of the disclosure will be apparent to those skilled in the art from consideration of the specification and practice of the disclosure disclosed herein. This application is intended to cover any adaptations, uses, or adaptations of the disclosure following, in general, the principles of the disclosure and including such departures from the present disclosure as come within known or customary practice within the art to which the disclosure pertains. It is intended that the specification and examples be considered as exemplary only, with a true scope and spirit of the disclosure being indicated by the following claims.
It is to be understood that the present disclosure is not limited to the precise arrangements and instrumentalities shown in the drawings, and that various modifications and changes may be effected without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.
Claims (14)
1. A feature evaluation method, comprising:
Acquiring target sample data; wherein the target sample data comprises first sample data comprising basic features and second sample data comprising features to be evaluated; the basic features are features which are currently applied to a recommendation system, and the features to be evaluated are features which are not currently applied to the recommendation system; the target sample data comprise user age, user gender, short video type and short video playing duration;
Training a feature evaluation model based on the target sample data, wherein in a model training process, intermediate data generated based on the first sample data is used as intermediate input quantity in a second sample data training process;
generating evaluation information of the feature to be evaluated based on an output result of the feature evaluation model;
the feature evaluation model comprises a basic sub-model and a test sub-model;
Training a feature evaluation model based on the target sample data, wherein in a model training process, intermediate data generated based on the first sample data is used as an intermediate input quantity in a second sample data training process, and the method comprises the following steps:
inputting first sample data into a basic sub-model in a feature evaluation model, and simultaneously inputting second sample data into a test sub-model in the feature evaluation model so as to train the feature evaluation model;
in the model training process, data output by any network layer of the basic sub-model is used as intermediate data generated based on the first sample data, and the intermediate data is input to any network layer of the test sub-model, so that the intermediate data is used as intermediate input quantity in the second sample data training process.
2. The feature evaluation method according to claim 1, wherein the step of generating evaluation information of the feature to be evaluated based on the output result of the feature evaluation model includes:
respectively obtaining the area under a first curve output by the basic submodel and the area under a second curve output by the test submodel;
calculating the area gain of the area under the second curve relative to the area under the first curve;
and generating evaluation information of the feature to be evaluated according to the area gain.
3. The feature evaluation method according to claim 2, wherein the step of generating evaluation information of the feature to be evaluated from the area gain includes:
When the area gain is in a preset interval range, determining the feature to be evaluated as an effective feature;
and when the area gain is out of a preset interval range, determining the feature to be evaluated as an invalid feature.
4. The feature evaluation method according to claim 1, further comprising, before training a feature evaluation model based on the target sample data:
acquiring feature screening sample data;
Determining feature statistical information of the feature to be evaluated based on the feature screening sample data;
and screening the characteristics to be evaluated according to the characteristic statistical information.
5. The feature evaluation method according to claim 4, wherein the feature statistical information includes a feature total amount, an average feature amount, and a feature coverage.
6. The feature evaluation method according to any one of claims 1 to 5, characterized in that the feature to be evaluated comprises a single feature to be evaluated and/or a combination of features to be evaluated.
7. A feature evaluation apparatus, characterized by comprising:
an acquisition unit configured to perform acquisition of target sample data; wherein the target sample data comprises first sample data comprising basic features and second sample data comprising features to be evaluated; the basic features are features which are currently applied to a recommendation system, and the features to be evaluated are features which are not currently applied to the recommendation system; the target sample data comprise user age, user gender, short video type and short video playing duration;
A training unit configured to perform training of a feature evaluation model based on the target sample data, in a model training process, taking intermediate data generated based on the first sample data as an intermediate input amount in the second sample data training process;
a generation unit configured to perform generation of evaluation information of the feature to be evaluated based on an output result of the feature evaluation model;
the feature evaluation model comprises a basic sub-model and a test sub-model;
the training unit is configured to perform:
inputting first sample data into a basic sub-model in a feature evaluation model, and simultaneously inputting second sample data into a test sub-model in the feature evaluation model so as to train the feature evaluation model;
in the model training process, data output by any network layer of the basic sub-model is used as intermediate data generated based on the first sample data, and the intermediate data is input to any network layer of the test sub-model, so that the intermediate data is used as intermediate input quantity in the second sample data training process.
8. The feature evaluation device according to claim 7, wherein the generation unit includes:
An area acquisition subunit configured to perform acquisition of a first area under a curve output by the base submodel and a second area under a curve output by the test submodel, respectively;
A gain calculation subunit configured to perform calculation of an area gain of the area under the second curve relative to the area under the first curve;
and an information generation subunit configured to perform generation of evaluation information of the feature to be evaluated according to the area gain.
9. The feature evaluation device of claim 8, wherein the information generation subunit is configured to perform:
When the area gain is in a preset interval range, determining the feature to be evaluated as an effective feature;
and when the area gain is out of a preset interval range, determining the feature to be evaluated as an invalid feature.
10. The feature evaluation device according to claim 7, characterized in that the feature evaluation device further comprises:
a sample data acquisition unit configured to perform acquisition of feature screening sample data before training a feature evaluation model based on the target sample data;
a statistical information determination unit configured to perform determination of feature statistical information of the feature to be evaluated based on the feature screening sample data;
And the screening unit is configured to perform screening on the characteristics to be evaluated according to the characteristic statistical information.
11. The feature evaluation device according to claim 10, wherein the feature statistical information includes a feature total amount, an average feature amount, and a feature coverage.
12. The feature evaluation device of any one of claims 7-11, wherein the feature to be evaluated comprises a single feature to be evaluated and/or a combination of features to be evaluated.
13. An electronic device, comprising:
A processor;
a memory for storing executable instructions of the processor;
Wherein the processor is configured to execute the instructions to implement the feature evaluation method of any one of claims 1 to 6.
14. A storage medium, which when executed by a processor of a server, enables the server to perform the feature evaluation method of any one of claims 1 to 6.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010642000.7A CN113902952B (en) | 2020-07-06 | 2020-07-06 | Feature evaluation method, device, electronic equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010642000.7A CN113902952B (en) | 2020-07-06 | 2020-07-06 | Feature evaluation method, device, electronic equipment and storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN113902952A CN113902952A (en) | 2022-01-07 |
| CN113902952B true CN113902952B (en) | 2024-09-20 |
Family
ID=79186674
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010642000.7A Active CN113902952B (en) | 2020-07-06 | 2020-07-06 | Feature evaluation method, device, electronic equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113902952B (en) |
Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108805332A (en) * | 2018-05-07 | 2018-11-13 | 北京奇艺世纪科技有限公司 | A kind of feature evaluation method and apparatus |
| CN111291266A (en) * | 2020-02-13 | 2020-06-16 | 腾讯科技（北京）有限公司 | Artificial intelligence based recommendation method and device, electronic equipment and storage medium |
-
2020
- 2020-07-06 CN CN202010642000.7A patent/CN113902952B/en active Active
Patent Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108805332A (en) * | 2018-05-07 | 2018-11-13 | 北京奇艺世纪科技有限公司 | A kind of feature evaluation method and apparatus |
| CN111291266A (en) * | 2020-02-13 | 2020-06-16 | 腾讯科技（北京）有限公司 | Artificial intelligence based recommendation method and device, electronic equipment and storage medium |
Also Published As
| Publication number | Publication date |
|---|---|
| CN113902952A (en) | 2022-01-07 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN109583594B (en) | Deep learning training method, device, equipment and readable storage medium | |
| CN108337656A (en) | A kind of mobile intelligent perception motivational techniques | |
| CN111563077B (en) | Power grid voltage data missing filling method, system, terminal and storage medium | |
| CN113918826B (en) | Processing method of release information, and training method and device of resource prediction model | |
| CN113256335A (en) | Data screening method, multimedia data delivery effect prediction method and device | |
| CN113902952B (en) | Feature evaluation method, device, electronic equipment and storage medium | |
| CN111984519A (en) | Test method and device for service system | |
| CN113450158A (en) | Bank activity information pushing method and device | |
| CN114519520B (en) | Model evaluation method, device and storage medium | |
| CN114564523B (en) | Big data vulnerability analysis method and cloud AI system for intelligent virtual scene | |
| CN116755890A (en) | Multi-scene business data collaborative handling method and system based on big data platform | |
| CN117056478A (en) | Control method and device for electric power customer service system | |
| CN113094589B (en) | Intelligent service recommendation method and device | |
| CN112036607B (en) | Wind power output fluctuation prediction method and device based on output level and storage medium | |
| CN113128794A (en) | Quantization evaluation method and device | |
| CN113159926A (en) | Loan transaction repayment date determination method and device | |
| CN111882347A (en) | Model performance detection method, device, computer equipment and storage medium | |
| CN116610308B (en) | Code management method and device, electronic equipment and storage medium | |
| CN117815674B (en) | Game information recommendation method and device, computer readable medium and electronic equipment | |
| CN118796632A (en) | Test method, device and equipment for training performance of artificial intelligent accelerator card | |
| CN109614328B (en) | Method and apparatus for processing test data | |
| CN116822578A (en) | Method, device, equipment and storage medium for distributing crowd sensing tasks based on CCV-MAB | |
| CN113781231A (en) | Automatic transaction data generation method and device | |
| CN115115410A (en) | Advertisement recall method, apparatus, electronic device, program product, and medium | |
| CN118113421A (en) | Cloud resource pool capacity planning method and device for cloud workstation and computer equipment |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |