CN113689040B

CN113689040B - Control method and control system for measurement model and computer readable medium

Info

Publication number: CN113689040B
Application number: CN202110981298.9A
Authority: CN
Inventors: 法提·奥尔梅兹
Original assignee: Yangtze Memory Technologies Co Ltd
Current assignee: Yangtze Memory Technologies Co Ltd
Priority date: 2021-08-25
Filing date: 2021-08-25
Publication date: 2024-03-15
Anticipated expiration: 2041-08-25
Also published as: CN113689040A

Abstract

The present disclosure relates to a control method and a control system for a measurement model, wherein the measurement model includes a training model and a prediction model, wherein the trained training model is used for updating the prediction model, and the prediction model is used for predicting one or more products manufactured by a machine, the control method comprising: detecting application prediction error degree of a prediction model for one or more products manufactured by the machine and running anomaly degree of the prediction model; and issuing an alarm and/or at least partially changing the operational state of the predictive model in response to at least one of the following conditions being met: the application prediction error degree is greater than the reference application prediction error degree, and the running anomaly degree is greater than the reference running anomaly degree.

Description

Control method and control system for measurement model and computer readable medium

Technical Field

The present application relates to virtual measurement, and more particularly, to a control method and control system for a virtual measurement model.

Background

In semiconductor manufacturing, inspection of products at various stages of a production line is an important component of product yield management. In the conventional inspection method, a plurality of measuring stations are required to be introduced into a production line to actually measure the product. Such actual measurements may affect the production of the production line and may not be available at some stage. Virtual measurement is a measurement means for predicting or estimating the quality of a product based on the related data collected from, for example, a production machine by using a virtual measurement model, with which the influence of measurement on the actual production of a production line can be reduced.

In the semiconductor manufacturing process, when the virtual measurement technique is used to predict the quality of the product on-line or off-line or control the batch in real time, small changes in the manufacturing process, the recipe of the raw materials, the running condition of the machine, the connection condition between the prediction model and the database, etc. may have a large influence on the actual application of the prediction model. In addition, in practical applications, conditions such as periodic maintenance of the production machine, sudden failure of the production machine, and abnormal raw material state may occur, and these conditions may also affect the prediction accuracy of the virtual model. Thus, the predictive model of the virtual measurement needs to be monitored by a computer or directly by an engineer to discover and eliminate or reduce such adverse effects in time.

On the other hand, problems such as the lower prediction capacity, the old training set, communication faults with the database and the like can also occur in the training process of the prediction model. These problems can lead to poor training of the predictive model, which in turn can affect the final prediction accuracy of the predictive model. Thus, the training process of the predictive model also needs to be monitored, and the above problems discovered and eliminated in time.

Accordingly, there is a need for a method and system that can monitor the training and prediction of a virtual measurement model and take corresponding action when an anomaly occurs in the virtual measurement model.

It should be appreciated that this background section is intended to provide a useful background for understanding the technology in part and is not meant to imply that such content was previously known to those skilled in the art prior to this application.

Disclosure of Invention

The present disclosure provides a control method of a measurement model, wherein the measurement model includes a training model and a prediction model, wherein the trained training model is used to update the prediction model, and the prediction model is used to predict one or more machine-manufactured products, the method comprising: detecting application prediction error degree of a prediction model for one or more products manufactured by the machine and running anomaly degree of the prediction model; and issuing an alarm and/or at least partially changing the operational state of the predictive model in response to at least one of the following conditions being met: the application prediction error degree is greater than the reference application prediction error degree, and the running anomaly degree is greater than the reference running anomaly degree.

The present disclosure also provides a control method of a measurement model, the measurement model including a training model and a prediction model, wherein the trained training model is used to update the prediction model, and the prediction model is used to predict one or more machine-manufactured products, the method including: detecting training prediction accuracy of a training model for a training set and associated anomaly of a sample in the training set and a machine for manufacturing the sample, wherein the training set comprises products manufactured by one or more machines; and issuing an alarm and/or at least partially changing the operational state of the predictive model in response to at least one of the following conditions being met: the training prediction accuracy is less than the reference training prediction accuracy, and the associated anomaly is greater than the reference associated anomaly.

The present disclosure also provides a control system of a measurement model, the control system comprising: a processor; and a memory having one or more programs stored thereon, which when executed by the processor, cause the processor to perform the method of: detecting application prediction error degree of a prediction model for one or more products manufactured by the machine and running anomaly degree of the prediction model; and issuing an alarm and/or at least partially changing the operational state of the predictive model in response to at least one of the following conditions being met: the application prediction error degree is greater than the reference application prediction error degree, and the running anomaly degree is greater than the reference running anomaly degree.

The present disclosure also provides a control system of a measurement model, the control system comprising: a processor; and a memory having one or more programs stored thereon, which when executed by the processor, cause the processor to perform the method of: detecting training prediction accuracy of a training model for a training set and associated anomaly of a sample in the training set and a machine for manufacturing the sample, wherein the training set comprises products manufactured by one or more machines; and issuing an alarm and/or at least partially changing the operational state of the predictive model in response to at least one of the following conditions being met: the training prediction accuracy is less than the reference training prediction accuracy, and the associated anomaly is greater than the reference associated anomaly.

The present disclosure also provides a non-transitory computer-readable storage medium storing computer instructions for causing a computer to perform the method of: detecting application prediction error degree of a prediction model for one or more products manufactured by the machine and running anomaly degree of the prediction model; and issuing an alarm and/or at least partially changing the operational state of the predictive model in response to at least one of the following conditions being met: the application prediction error degree is greater than the reference application prediction error degree, and the running anomaly degree is greater than the reference running anomaly degree.

The present disclosure also provides a non-transitory computer-readable storage medium storing computer instructions for causing a computer to perform the method of: detecting training prediction accuracy of a training model for a training set and associated anomaly of a sample in the training set and a machine for manufacturing the sample, wherein the training set comprises products manufactured by one or more machines; and issuing an alarm and/or at least partially changing the operational state of the predictive model in response to at least one of the following conditions being met: the training prediction accuracy is less than the reference training prediction accuracy, and the associated anomaly is greater than the reference associated anomaly.

The present disclosure also provides a computer program product comprising a computer program which, when executed by a processor, implements the method of: detecting application prediction error degree of a prediction model for one or more products manufactured by the machine and running anomaly degree of the prediction model; and issuing an alarm and/or at least partially changing the operational state of the predictive model in response to at least one of the following conditions being met: the application prediction error degree is greater than the reference application prediction error degree, and the running anomaly degree is greater than the reference running anomaly degree.

The present disclosure also provides a computer program product comprising a computer program which, when executed by a processor, implements the method of: detecting training prediction accuracy of a training model for a training set and associated anomaly of a sample in the training set and a machine for manufacturing the sample, wherein the training set comprises products manufactured by one or more machines; and issuing an alarm and/or at least partially changing the operational state of the predictive model in response to at least one of the following conditions being met: the training prediction accuracy is less than the reference training prediction accuracy, and the associated anomaly is greater than the reference associated anomaly.

Drawings

The above and other advantages and features of the present disclosure will become more apparent by describing in detail exemplary embodiments thereof with reference to the attached drawings.

Fig. 1 shows a schematic diagram of a control method and an application environment of a control system of a measurement model according to an embodiment of the present application.

Fig. 2 shows a control method of a measurement model according to an embodiment of the present application.

Fig. 3 shows a control method of a measurement model according to another embodiment of the present application.

FIG. 4 shows a diagram for illustrating real-time prediction R according to an embodiment of the present application ² Schematic diagram of the calculation method of the score.

Detailed Description

Exemplary embodiments of the present invention will now be described more fully hereinafter with reference to the accompanying drawings, in which preferred embodiments of the invention are shown. This invention may, however, be embodied in different forms and should not be construed as limited to the exemplary embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the invention to those skilled in the art.

It will also be understood that when an element or layer is referred to as being "on," "connected to," or "coupled to" another element or layer, it can be directly on or connected to the other element or layer or intervening elements or layers may be present. In contrast, when an element or layer is referred to as being "directly on," "directly connected to," or "directly coupled to" another element or layer, there are no intervening elements or layers present. To this extent, the term "connected" can refer to a physical, electrical, and/or fluid connection with or without intervening elements.

Like reference numerals refer to like elements throughout the specification. In the drawings, the thickness of layers and regions are exaggerated for clarity.

Although the terms "first," "second," etc. may be used herein to describe various elements, these elements should not be limited by these terms. These terms may be used to distinguish one element from another element. Thus, a first element discussed below could be termed a second element without departing from the teachings of one or more embodiments. Describing an element as a "first" element may not require or imply the presence of a second element or other element. The terms "first," "second," and the like may also be used herein to distinguish between different classes or groups of elements. For simplicity, the terms "first," "second," and the like may refer to "a first class (or group)", "a second class (or group)", and the like, respectively.

The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting. As used herein, the term "and/or" includes any and all combinations of one or more of the associated listed items. It will be further understood that the terms "comprises" and/or "comprising," when used in this specification, specify the presence of stated features, regions, steps, operations, elements, and/or components, but do not preclude the presence or addition of one or more other features, regions, steps, operations, elements, components, and/or groups thereof.

Furthermore, relative terms, such as "lower" or "bottom" and "upper" or "top," may be used herein to describe one element's relationship to another element as illustrated in the figures. It will be understood that relative terms are intended to encompass different orientations of the device in addition to the orientation depicted in the figures. In an exemplary embodiment, when the device in one of the figures is turned over, elements described as being on the "lower" side of other elements would then be oriented on the "upper" side of the other elements. Thus, the exemplary term "lower" may encompass both an orientation of "lower" and "upper", depending on the particular orientation of the figure. Similarly, when the device in one of the figures is turned over, elements described as "below" or "beneath" other elements would then be oriented "above" the other elements. Thus, the exemplary term "below" or "beneath" can encompass both an orientation of above and below.

As used herein, "about" or "approximately" includes the values as well as averages within acceptable deviation limits of the particular values as determined by one of ordinary skill in the art in view of the measurement in question and the error associated with the particular amount of measurement (i.e., limitations of the measurement system). For example, "about" may mean within one or more standard deviations.

Unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs. It will be further understood that terms, such as those defined in commonly used dictionaries, should be interpreted as having a meaning that is consistent with their meaning in the context of the relevant art and the present invention and will not be interpreted in an idealized or overly formal sense unless expressly so defined herein.

As is conventional in the art, some example implementations are described and illustrated in the figures for functional blocks, units, and/or modules. Those skilled in the art will appreciate that the blocks, units, and/or modules are physically implemented by electrical (or optical) circuits such as logic circuits, discrete components, microprocessors, hardwired circuits, memory elements, wired connectors, or the like, which may be formed using semiconductor-based manufacturing techniques or other manufacturing techniques. Where the blocks, units, and/or modules are implemented by a microprocessor or other similar hardware, they may be programmed and controlled by software (e.g., microcode) to perform the various functions recited herein, and optionally driven by firmware and/or software. It is also contemplated that each block, unit, and/or module may be implemented with dedicated hardware, or may be implemented as a combination of dedicated hardware for performing some functions and a processor (e.g., one or more programmed microprocessors and associated circuitry) for performing other functions. Additionally, each block, unit, and/or module in some example embodiments may be physically separated into two or more interactive and discrete blocks, units, and/or modules without departing from the scope of the inventive concept. Furthermore, blocks, units, and/or modules in some example embodiments may be physically combined into more complex blocks, units, and/or modules without departing from the scope of the inventive concepts.

As shown in fig. 1, an application environment of a control method and a control system of a measurement model (i.e., a virtual measurement model) according to an embodiment of the present application may include a machine, a controller 120, a measurement model 130, a measurement model control unit 133, and a database 140, wherein the machine may include one or more machines, for example, a first machine 111 to an nth machine 112 (N is a natural number), and the measurement model 130 may include a training model 131 and a prediction model 132. The measurement model 130, the measurement model control unit 133 and the database 140 may be implemented by instructions stored in a computer readable medium, for example, the functions of the measurement model 130, the measurement model control unit 133 and the database 140 may be implemented when the above instructions are executed by a computer processor.

The first through nth stages 111 through 112 may be used to generate products, such as semiconductor devices, wafers, and the like. The first through nth stations 111-112 may communicate with the database 140 to send data related to the stations (e.g., temperature, production speed, etc.) and data related to the products (e.g., such as product photos, temperature, etc.) to the database 140.

The controller 120 may control the operations of the first to nth stages 111 to 112, for example, may stop the operations of one or more of the first to nth stages 111 to 112. The controller 120 may communicate with the predictive model 132 to request the predictive model 132 to predict a product, such as that manufactured by the first machine 111, or to stop or resume operation, such as the first machine 111, based on the prediction by the predictive model 132. The controller 120 may include a plurality of controllers, where each controller may correspond to a respective machine. Although the controller is shown as being separate from the machine in fig. 1, the controller of the present application is not limited thereto, and it may be integrated in the machine.

The training model 131 is a predictive model for training. The training model 131 may be used to update the predictive model 132 after training, e.g., the predictive model 132 may be replaced with the trained training model 131. The training method of the training model 131 is not limited, for example, when the training model 131 and the prediction model 132 are constructed based on a convolutional neural network, a back propagation method may be used to compare the prediction result of the training model 131 with the reference label of the sample in the training set to obtain the difference between the two, adjust the parameters of the convolutional neural network according to the difference, and then perform the next prediction and comparison until the difference between the prediction result and the reference label is no longer reduced. During training, training model 131 may communicate with database 140 to obtain training set data.

The prediction model 132 is used to predict products produced by the corresponding machine in response to a prediction request from the controller 120 to determine, for example, whether the product quality is acceptable, whether a defect exists, and the like. The predictive model 132 may communicate with the database 140 to obtain relevant data for the corresponding machine and/or product to predict. For example, the predictive model 132 may obtain various parameters of the machine (e.g., temperature, production speed, etc.) from the database 140, and may also obtain data related to the product (e.g., such as product photographs, temperature, etc.).

The measurement model control unit 133 is configured to perform a measurement model control method and may communicate with the training model 131 and the prediction model 132 to control the training model 131 and the prediction model 132 according to a training prediction accuracy of the training set by the training model, an anomaly degree of a training process, an anomaly degree of association of a sample in the training set with a machine for manufacturing the same, an application prediction error degree of the prediction model to a product manufactured by the machine, an operation anomaly degree of the prediction model, and the like, wherein the training prediction accuracy reflects a prediction accuracy of the training model for the sample in the training set, and an actual prediction accuracy of the prediction model is directly affected due to an excessively low training prediction accuracy due to the training model being used for updating the prediction model; the anomaly degree of the training process reflects the severity of anomaly occurring in the training process of the training model, such as the time for completing one training, the time interval between two successful training, etc., and the overlarge indexes can reflect that the connection of the training model or the measurement model control unit and the database in the training is possibly problematic, or the data itself is problematic; the degree of correlation anomaly between the sample in the training set and the machine for manufacturing the sample reflects the degree that the sample can represent the product manufactured by the machine, if the degree of correlation anomaly is too low, the sample used cannot well represent the whole condition of the product manufactured by the machine, and further the model trained by using the sample cannot accurately predict the product manufactured by the machine; the application of the prediction error degree reflects the prediction accuracy of the prediction model for the actual product, and the application of the excessive prediction error degree reflects the fact that the prediction model cannot accurately predict the product; the degree of operation abnormality reflects the severity of abnormality occurring in the prediction process, and the excessive degree of operation abnormality reflects the possibility of excessively high load of the server, problems occurring in the data used for prediction, insufficient coverage rate of prediction for each machine station, and the like. The measurement model control unit 133 may also determine whether to issue an alarm, stop training, stop the machine operation, or the like based on the above information. The measurement model control unit 133 may communicate with the database 140, for example, to read history data about the training model 131 and the prediction model 132, or to store control information or the like to the database 140.

The database 140 is used to store data, for example, data about the machine and the product from the machine, a training set of predictive models, control information of a measurement model control unit, logs, and the like.

The control method of the measurement model will be described in detail with reference to fig. 2.

As shown in fig. 2, a control method of a measurement model according to an embodiment of the present application may include: detecting training prediction accuracy of a training model for a training set (S211) and associated anomaly degree of a sample in the training set and a machine for manufacturing the sample (S221); and in response to at least one of the following conditions being met, issuing an alarm and/or at least partially changing the operational state of the predictive model (S250): the training prediction accuracy is lower than the reference training prediction accuracy (S212) and the associated anomaly is greater than the reference associated anomaly (S222).

At step S211, the control method of the measurement model according to the embodiment of the present application may detect the training modelAccuracy of training predictions for a training set. Training the prediction accuracy may use R ² Score is represented, wherein R ² The score may be obtained using equation 1 below:

Wherein, for the predicted value of the training model on the ith sample in the training set, y _i E for the reference value of the ith sample _i Representing the difference between the predicted value of the i-th sample and the reference value. R is R ² The score reflects the fitting capacity of the training model to the sample data, and the larger the value is, the stronger the fitting capacity is, and the higher the corresponding prediction accuracy is. By aiming at R ² The score sets a reference value (e.g. reference R ² Score) as a reference training prediction accuracy, a corresponding operation may be performed when the training prediction accuracy of the training model is lower than the reference training prediction accuracy. For example, when training the R of the model ² Score less than reference R ² The score, which indicates that the fit ability of the training model is low, may affect the prediction accuracy, and may immediately inform the user of the anomaly and/or halt the prediction for all machines. For example, by the display device displaying alarm information to the user and/or by the measurement model control unit 133 shown in fig. 1, an instruction is issued to the prediction model 132 to suspend prediction for all the stations.

In the present embodiment, R of the model is trained ² The score may include a simple R ² Score (also may be referred to as "first R ² Score "), median R ² Score (may also be referred to as "second R ² Score ") and real-time prediction R ² Score (may also be referred to as "third R ² Score ").

Simple R ² The score may be R calculated for all samples in the training set ² Score.

Median value R ² The score may be obtained by the following method: randomly dividing the training set into a first training set and a second training set; training a training model using a first training set; predicting the second training set using the trained training model; obtaining R for the second training set for the trained training model ² A score; dividing the training set into a new first training set and a new second training set again at random; repeating the above steps a predetermined number of times and obtaining R ² Median of the scores as the second R ² Score.

For example, 80% of the samples in the training set may be randomly divided into training sample sets and the remaining 20% of the samples are divided into test sample sets, the training model is trained using the training sample sets, the samples in the test sample sets are predicted using the trained training model, and the R of the training model for the training is obtained ² Score. Then, 80% of the samples in the training set are divided into new training sample sets again, the remaining 20% of the samples are divided into new test sample sets, and the original training sample sets and the test sample sets are replaced by the new training sample sets and the new test sample sets respectively. The above procedure can be repeated, for example, 1000 times, and 1000R's are obtained ² Score, finally, the 1000R ² Median score of scores as median R of training model ² Score of

Predicting R in real time ² The score may be obtained by the following method: sequencing the products in the training set in order of production completion time from first to last; selecting a predetermined number as a selection window; selecting a third training set using a selection window starting from the first sample in the training set; training a training model by using a third training set; predicting a sample positioned at the rear position of the selection window in the training set by using the trained training model; moving the selection window one sample step back in the training set; updating the third training set by using the samples in the moved selection window; repeating the above steps until the last sample in the training setThe method comprises the steps of predicting the cost; calculating R for all predictors ² Score as real-time prediction R ² Score.

FIG. 3 shows a diagram for illustrating real-time prediction R according to an embodiment of the present application ² Schematic diagram of the calculation method of the score.

In this embodiment, the samples in the training set may be the products that are detected and labeled with the detection results and are produced by the first machine 111 to the nth machine 112 shown in fig. 1. As shown in fig. 3, the samples in the training set may be arranged in a sequence of from first to last generation completion times, e.g., the generation completion time of sample 301 is earlier than the generation completion time of sample 305, and the generation completion time of sample 305 is earlier than the generation completion time of sample 306. For example, four samples may be selected as the selection window and the third training set 310 may be selected using the selection window starting with the first sample 301 in the training set. Samples 305 in the training set that are located one after the selection window are then predicted using the trained training model. After the prediction is completed, the selection window may be moved back one sample step to select four samples after sample 301 and the third training set 310 is updated with the four samples to third training set 320, then the training model is trained with third training set 320 and the samples 306 are predicted using the trained training model. And so on, when the last sample is predicted, the training process ends. Finally, R is calculated for all predicted results in the above process ² Score as real-time prediction R ² Score.

In the present embodiment, R is simply ² The score may be greater than the median R ² Score, and median R ² The score may be greater than the real-time predicted R ² Score.

In the present embodiment, simple R can be respectively ² Score, median R ² Fractional and real-time prediction R ² Score is used as training prediction accuracy of training model and is respectively matched with the first R ² Score, second R ² Score and third R ² R corresponding to fraction ² The fractional reference value is used as a reference to train the prediction accuracy. The measurement model control method may include: when (when)Simple R ² Score, median R ² Fractional and real-time prediction R ² When at least one of the scores is below the corresponding reference training prediction accuracy (i.e., S212, the training prediction accuracy is less than the reference training prediction accuracy), an alert may be raised and/or training of the training model for all of the machines may be suspended (S250). The reference training prediction accuracy may be determined experimentally or based on historical data.

In some embodiments, the control method of the measurement model may further include: and detecting the abnormal degree of the training process of the training model aiming at the training set. The training process anomaly reflects the severity of anomalies that occur during the training of the training model.

In the training process, abnormal data flow between the training model and the database or problems of the training data can cause the increase of the interval time between two times of successful training of the training model, and the monitoring of the interval time between two times of successful training can discover the problems in time and take corresponding measures. Thus, in some embodiments, a method of detecting training process anomalies of a training model for a training set may include: detecting an interval between successive successful training of the training model; and using the interval time to represent the abnormal degree of the training process, and using the corresponding interval time reference value as the reference training process abnormal degree. The measurement model control method may include: and sending out an alarm and/or suspending training of the training models for all the machines in the condition that the interval time is larger than the interval time reference value and exceeds the preset time.

For example, the timing may be started when the training model successfully completes one training, and the timing may be stopped until the next time the training is successfully completed, where the time counted is the interval time, when the interval time is greater than the interval time reference value and continues to exceed the predetermined time, it may be determined that the training of the training model is abnormal, and at this time, an alarm may be issued and/or the training of the training model for all the machine stations may be suspended. The interval reference value may be determined experimentally or based on historical data.

Problems with the data connection between the training model and the database may cause excessive training time for the training model, for example, excessive connection latency between the training model and the database may result in excessive training time, and monitoring of the training time may enable timely discovery of such problems and take corresponding action. Thus, in some embodiments, a method of detecting training process anomalies for a training model may include: detecting the training time for completing the training of the training model each time; and using the training time to represent the training process anomaly degree, and using the corresponding training time reference value as the reference training process anomaly degree. The measurement model control method may include: and sending out an alarm and/or suspending training of the training models for all the machines in the condition that the training time is greater than the training time reference value and exceeds the preset time.

For example, the timing may be started each time training of the training model is started, and stopped when training is completed, and the counted time is the training time. When the training time exceeding the training time reference value exists in the obtained training time and lasts for more than a preset time, the training of the training model can be determined to be abnormal, and at the moment, an alarm can be sent out and/or the training of the training model for all the machine stations can be stopped. The training time reference value may be determined experimentally or based on historical data

Referring again to fig. 2, in step S221, the associated anomalies of the samples in the training set and the machine from which the samples were made may be detected. The associated anomaly may represent an anomaly level of correlation between the samples in the training set and the machine.

As described above, the samples in the training set may be the detected products actually produced by each machine, so that in order to ensure that the trained model can be universally applied to each machine, it is generally expected that the number of samples from each machine in the sample set is substantially the same, and the difference between the numbers of samples from each machine can reflect the associated anomaly between the samples in the training set and the machine. Thus, in some embodiments, detecting the associated anomalies of a sample in a training set with a machine that manufactured the sample may include: detecting the number of samples from each of a plurality of machines in a training set; calculating a difference in the number of samples from each pair of the plurality of stations; the maximum value of the differences in the number of samples is used as the associated anomaly, and the difference reference value of the corresponding number of samples is used as the reference associated anomaly. The measurement model control method may include: and when the maximum value in the difference of the sample numbers of each pair of machines is larger than the difference reference value of the sample numbers, giving out an alarm and/or suspending training of the training model for the larger of the two machines corresponding to the maximum value.

For example, samples from each station in the training set may be counted, and the number of samples of each station may be compared with the number of samples of other stations, respectively, to obtain a difference in the number of samples between each pair of stations, and then a maximum value thereof may be taken as the associated anomaly, and a difference reference value of the corresponding number of samples may be used as the reference associated anomaly. When the maximum value in the difference of the sample numbers exceeds the difference reference value of the sample numbers, the correlation degree of the training set to a certain machine is too large, which leads to too large dependence of the trained model to a certain machine, and further influences the prediction of other machines. Therefore, an alarm may be issued at this time, or training of a training model for the machine concerned may be suspended. For example, training of the training model for the first machine may be stopped when the number of samples from the first machine is greater than the number of samples from the second machine, and the differences between them constitute the maximum of the differences in the number of samples between all machines. The difference reference value for the number of samples may be determined experimentally or based on historical data.

After a product is manufactured, the degree of association with the manufacturing tool decreases over time due to changes in the condition of the manufacturing tool or the product recipe, particularly after periodic maintenance of the tool or adjustment of the product recipe. Thus, if a sample in the training set has been manufactured for a longer period of time, its effect on the training model may be weak. Thus, in some embodiments, detecting the associated anomalies of the samples in the training set with the machine that manufactured the samples may include: detecting the manufacturing completion time of samples in the training set; the manufacturing completion time is utilized as the associated anomaly, and the corresponding reference manufacturing completion time is utilized as the reference associated anomaly. The measurement model control method may include: when the manufacturing completion time is earlier than the reference manufacturing completion time, an alarm is sent out and/or training of a training model for a corresponding machine is suspended, wherein the corresponding machine is a machine corresponding to a sample with the manufacturing completion time earlier than the reference manufacturing completion time.

For example, the manufacturing completion time for each sample in the training set may be obtained from the database and compared to the reference manufacturing completion time, and when the manufacturing completion time for a sample is greater than the reference manufacturing completion time, it may be indicated that the sample may have a weaker association with the corresponding station, and may indicate that the sample has a higher anomaly associated with the reference of the station, at which time an alarm may be raised and/or training of the training model for the station that manufactured the sample may be suspended. The reference manufacturing completion time may be determined experimentally or based on historical data.

In the training process, if a training set is used for training over a larger time span, the association degree between the sample set and the machine is also reduced. Thus, in some embodiments, detecting the associated anomalies of the samples in the training set with the machine that manufactured the samples may include: detecting a training time span of a training set; the training time span is utilized as the associated anomaly and the corresponding reference training time span is utilized as the reference associated anomaly. The measurement model control method may include: and when the training time span is larger than the reference training time span, an alarm is sent out and/or training of the training models for all the machines is suspended.

For example, the time counted from when training using a set of samples is started may be counted as the training time span. When the training time span is greater than the reference training time span, the association degree of the instruction book sample set and the machine is reduced, and correspondingly, the association degree of abnormality is increased and exceeds the reference association degree of abnormality. At this point, an alarm may be raised and/or training of the training model for all of the machines may be suspended.

Referring to fig. 4, a control method of a measurement model according to another embodiment of the present application may include: detecting an application prediction error degree of the prediction model for one or more machine-manufactured products (S231), and an operation anomaly degree of the prediction model (S241); and in response to at least one of the following conditions being met, issuing an alarm and/or at least partially changing the operational state of the predictive model (S250): the application prediction error degree is lower than the reference application prediction error degree (S232), and the running anomaly degree is higher than the reference running anomaly degree (S242).

It should be noted that, the control method of the measurement model described with reference to fig. 4 and the control method of the measurement model described with reference to fig. 3 may be used alone or in combination. For example, the measurement model may be monitored using only the method shown in fig. 3 to take corresponding measures when an abnormality occurs, or the measurement model may be monitored using only the method shown in fig. 4 to take corresponding measures when an abnormality occurs, and furthermore, the method shown in fig. 3 and the method shown in fig. 4 may be used simultaneously to monitor both the training model and the measurement model.

In step S231, a prediction error degree of the prediction model applied to the product manufactured by the machine may be detected. The prediction accuracy may reflect the accuracy of the predicted result of the prediction model.

In a production environment, the machine often needs to be maintained at an irregular period, and a product produced by the machine before maintenance and a product produced by the machine after maintenance may have a large difference, which may cause that the prediction model is not suitable for the machine after maintenance. Errors that may manifest themselves in the detected data as predicted results of the predictive model are outliers, e.g., significantly excessive errors. If the proportion of the outlier is large, the prediction model may not be suitable for most machines, and the prediction accuracy is reduced. Thus, in some embodiments, detecting the applied prediction error degree of the prediction model for the product manufactured by the machine may include: detecting a prediction error outlier ratio of predictions performed by the prediction model, wherein the prediction error constant ratio is represented by a ratio at which the prediction error constant occurs in a predetermined number of predictions made recently by the prediction model; the prediction error degree is applied using the prediction error constant ratio as the application prediction error degree, and the corresponding reference prediction error constant ratio is used as the reference. The measurement model control method may include: when the prediction error outlier ratio is greater than the reference prediction error outlier constant ratio and remains for more than a predetermined time, an alarm is raised and/or prediction of the prediction model for all of the machines is stopped.

In the generation process, the actual detection can be performed on part of the products, and the prediction error of the prediction model can be obtained by comparing the actual detection result of the products with the prediction result of the prediction model. In this embodiment, the rate at which the prediction error difference constant occurs in the last 10 predictions of the model can be detected. The constant prediction error value may be a prediction error exceeding a predetermined threshold, for example, 10% of the predetermined threshold, and when the prediction error of a prediction is 15% of the prediction error of a certain time, the prediction error may be regarded as the constant prediction error value. When the ratio of prediction error constant values occurring in the last 10 predictions of the model is greater than the corresponding reference prediction error outlier ratio (e.g., 20%), it can be considered that the prediction accuracy of the prediction model will be affected. The measurement model control unit may alert and/or stop the prediction of the prediction model for all the stations if the prediction error constant value ratio exceeds the reference prediction error abnormal value ratio for a predetermined time. The reference prediction error constant ratio may be determined experimentally or based on historical data.

The median prediction error value can reflect an average of prediction errors of the prediction model, and thus, in some embodiments, detecting an applied prediction error degree of the prediction model for a product manufactured by the machine may include: detecting a predicted median of prediction errors of predictions performed by a prediction model; the prediction error degree is applied using the prediction error median as the application prediction error degree and the corresponding reference prediction error median as the reference. The measurement model control method may include: and when the prediction error median value is larger than the reference prediction error median value and is maintained for more than a preset time, an alarm is sent out and/or prediction of the prediction model for all machines is stopped.

For example, prediction errors of, for example, the last 15 predictions performed by the prediction model may be ranked, and a prediction error (for example, 20%) located at a middle position (8 th bit) may be used as a median of the prediction errors of the prediction model. When the median prediction error value is greater than the corresponding median reference prediction error value (e.g., 15%), it can be considered that the prediction accuracy of the prediction model will be affected. The measurement model control unit may alert and/or stop the prediction of the prediction model for all the machines if the median prediction error exceeds the median reference prediction error for a predetermined time. The median reference prediction error value may be determined experimentally or based on historical data.

The change rate of the prediction error can reflect the change trend of the prediction error of the prediction model, and is beneficial to timely finding out the state abnormality of the prediction model. Thus, in some embodiments, detecting the applied prediction error degree of the prediction model for the product manufactured by the machine may include: detecting a prediction error change rate of prediction performed by the prediction model; the prediction error rate is utilized as an applied prediction error degree, and the corresponding reference prediction error rate is utilized as a reference applied prediction error degree. The measurement model control method may include: when the prediction error rate of change is greater than the reference prediction error rate of change and remains for more than a predetermined time, an alarm is raised and/or prediction of the prediction model for all of the tools is stopped.

For example, the prediction error of the last 10 predictions made by the prediction model may be compared with the prediction error of its respective previous predictions to obtain 10 prediction error change rates, and the average of the 10 prediction error change rates may be taken as the prediction error change rate of the prediction model. In another embodiment, the prediction error of the last prediction performed by the prediction model may be compared with the prediction error of the previous prediction to obtain a prediction error change rate, and the prediction error change rate is used as the prediction error change rate of the prediction model. When the prediction error change rate of the prediction model exceeds the reference prediction error change rate and exceeds the predetermined time, indicating that the prediction error of the prediction model is continuously increasing, it can be considered that the prediction accuracy of the prediction model will be affected, at which time the measurement model control unit can alert and/or stop the prediction of the prediction model for all the machines. The reference prediction error rate of change may be determined experimentally or based on historical data.

Referring again to fig. 4, in step S241, an operational anomaly of the prediction model may be detected, which may represent a degree to which the prediction model is abnormal in the prediction process.

When the prediction model runs, the prediction model may suffer from problems of database connection or excessive load of a prediction server, and the problems may cause the prediction model to complete one-time prediction to be excessively long, so that the running abnormality of the prediction model is increased. Thus, detecting operational anomalies of the predictive model may include: detecting a predicted time for the prediction model to complete prediction in response to a predicted request by a controller of one or more machines; the predicted time is utilized as the running anomaly, and the corresponding reference predicted time is utilized as the reference running anomaly. The measurement model control method may include: an alarm is issued when the predicted time is greater than the reference predicted time.

For example, the timer may be started when the controller of a certain machine sends a prediction request to the prediction model, and the timer may be ended when the prediction model obtains a prediction result, and the counted time is the prediction time. Comparing the predicted time with the corresponding reference predicted time, and when the predicted time is greater than the reference predicted time, indicating that the predicted model operates abnormally, namely, the operation abnormality of the predicted model is greater than the reference operation abnormality, and sending out an alarm. The reference prediction time may be determined experimentally or based on historical data.

When the prediction model runs, the prediction model may encounter data integrity problems in a database, and loss of partial data may cause the prediction model to fail, thereby reducing the prediction success rate of the prediction model. Thus, in some embodiments, detecting the operational anomaly of the predictive model may include: detecting a prediction success rate of the prediction model to successfully predict in response to a prediction request of a controller of one or more machines; the predicted success rate is utilized as the running anomaly, and the corresponding reference predicted success rate is utilized as the reference running anomaly. The measurement model control method may include: and when the predicted success rate is smaller than the reference predicted success rate, an alarm is sent out.

For example, in the case where the reference prediction success rate is 70%, when the prediction model performs the nth prediction, the success rate of the N-10 th to nth predictions by the prediction model may be calculated, and if the success rate is greater than 70%, the current prediction model may be considered to be normal in operation, and the prediction success rate after the detection may be continued. When the prediction model makes the next prediction (i.e., when the n+1th prediction is made), the prediction success rate from the N-9 th prediction to the n+1th prediction can be detected, and if the prediction success rate at this time is less than 70%, the prediction model can be considered to be abnormal in operation, and an alarm can be issued. In other words, in the present embodiment, the predicted success rate of the last 10 predictions of the prediction model may be always detected, and an alarm may be issued when the predicted success rate is smaller than the reference predicted success rate. Although the reference prediction success rate is described as 70% in the present embodiment and the prediction success rate of detecting the last 10 predictions is described, the present application is not limited thereto, and the number of times of detection and the reference prediction success rate may be set to other values, wherein the reference value prediction success rate may be determined by experiments or based on historical data.

In order to make full use of the predictive model and apply virtual testing as widely as possible to each machine, it is beneficial to monitor the number of predictive requests from each machine. Thus, in some embodiments, detecting the operational anomaly of the predictive model may include: detecting a predicted number of requests from each of the one or more machines; calculating a difference in predicted request times from each pair of one or more machines; the maximum value of the differences between the predicted request times is used as the running anomaly, and the difference reference value of the corresponding predicted request times is used as the reference running anomaly. The measurement model control method may include: and when the maximum value in the difference between the predicted request times is larger than the reference value of the difference between the predicted request times, an alarm is sent out.

For example, it is possible to count the predicted requests from each station and compare the number of predicted requests of each station with the number of predicted requests of other stations, respectively, to obtain the difference in the number of predicted requests between each pair of stations, and then take the maximum value thereof as the running anomaly degree and use the difference reference value of the corresponding number of predicted requests as the reference running anomaly degree. When the maximum value of the difference between the predicted request times exceeds the difference reference value of the predicted request times, the fact that a certain machine station occupies excessive prediction model resources is indicated, and therefore a part of the machine stations cannot be detected fully. Therefore, an alarm can be issued at this time. The difference reference value of the predicted number of requests may be determined experimentally or based on historical data.

In some embodiments, when at least one of the training prediction accuracy is smaller than the reference training prediction accuracy, the associated anomaly degree is greater than the reference associated anomaly degree, the application prediction error degree is greater than the reference application prediction error degree, and the operation anomaly degree is greater than the reference operation anomaly degree, it is indicated that the state of the measurement model is abnormal, the prediction accuracy may be affected, and further the product quality detection of the production line is affected, and at this time, one or more machine work measures may be stopped. For example, the operation of the machine that may be affected by the measurement model anomaly may be stopped. The machine which stops working can resume working after the abnormal measurement model is discharged.

The various embodiments described above may be implemented as software comprising instructions that may be stored in a machine-readable storage medium, which may be read by a machine (e.g., a computer or control system). A machine (or control system) is a device that invokes instructions stored in a storage medium, which includes a processor and a memory storing one or more programs that, when executed by the processor, may perform the method described above with reference to fig. 3 or the method described with reference to fig. 4, or perform the method described with reference to fig. 3 and the method described with reference to fig. 4.

If the instructions are executed by a processor, the processor may perform the functions corresponding to the instructions by itself or by using other elements under the control of the processor. The instructions may include code that is generated or executed by a compiler or an interpreter. For example, the control method of the measurement model described above may be performed when instructions stored in the memory are executed by the processor.

The machine-readable storage medium may be provided in the form of a non-transitory storage medium.

According to an embodiment, a method according to the various embodiments described above may be provided as comprised in a computer program product. The computer program product may be transacted as a product between a seller and a consumer. The computer program product may be distributed in the form of a machine readable storage medium, e.g. a compact disc ROM (CD-ROM), or online through an application store. In the case of online distribution, at least a portion of the computer program product may be at least temporarily stored or temporarily generated in a storage medium of a memory of the manufacturer's server, the application store's server, or a relay server, for example.

While the present disclosure has been shown and described with reference to various embodiments thereof, it will be understood by those skilled in the art that various changes in form and details may be made therein without departing from the spirit and scope of the invention as defined by the appended claims and their equivalents.

Claims

1. A method of controlling a measurement model, wherein the measurement model comprises a training model and a prediction model, wherein the trained training model is used to update the prediction model, the prediction model is used to predict one or more machine-manufactured products, the method comprising:

detecting an application prediction error degree of the prediction model for products manufactured by the one or more machines and an operation anomaly degree of the prediction model;

detecting training prediction accuracy of the training model for a training set and associated anomaly of a sample in the training set and a machine for manufacturing the sample, wherein the training set is a product manufactured by one or more machines; and

issuing an alarm and/or at least partially changing the operating state of the predictive model in response to at least one of the following conditions being met:

the applied prediction error degree is greater than the reference applied prediction error degree,

the operational anomaly is greater than the reference operational anomaly,

the training prediction accuracy is less than the reference training prediction accuracy

The associated anomaly is greater than the reference associated anomaly.

2. The control method of claim 1, wherein detecting operational anomalies of the predictive model comprises:

Detecting a predicted time for the prediction model to complete prediction in response to a prediction request of the one or more machines;

the predicted time is utilized as the running anomaly and a corresponding reference predicted time is utilized as the reference running anomaly.

3. The control method according to claim 2, characterized in that the control method further comprises:

and when the predicted time is greater than the reference predicted time, an alarm is sent out.

4. The control method of claim 1, wherein detecting operational anomalies of the predictive model comprises:

detecting a predicted success rate of the prediction model for successfully predicting in response to the prediction request of the one or more machines, wherein the predicted success rate is represented by a recent, predetermined number of predicted success rates;

the running anomalies are calculated based on the predicted success rates, and the reference running anomalies are calculated based on corresponding reference predicted success rates.

5. The control method according to claim 4, characterized in that the control method further comprises:

and when the predicted success rate is smaller than the reference predicted success rate, determining that the running anomaly is greater than the reference running anomaly, and sending out an alarm.

6. The control method of claim 1, wherein detecting operational anomalies of the predictive model comprises:

detecting a predicted number of requests from each of the one or more machines;

calculating a difference in predicted request times from each pair of the one or more machines;

the maximum value of the difference between the predicted request times is used as the running anomaly, and the difference reference value of the corresponding predicted request times is used as the reference running anomaly.

7. The control method according to claim 6, characterized in that the control method further comprises:

and sending out an alarm when the maximum value in the difference between the predicted request times is larger than the difference reference value of the predicted request times.

8. The control method of claim 1, wherein detecting an applied prediction error degree of the predictive model for the one or more machine-manufactured products comprises:

detecting a prediction error outlier ratio of predictions performed by the prediction model, wherein the prediction error outlier ratio is represented by a ratio of occurrence of a prediction error constant in a predetermined number of predictions made recently by the prediction model, wherein the prediction error constant is a prediction error exceeding a predetermined threshold;

The prediction error outlier ratio is utilized as the applied prediction error degree, and a corresponding reference prediction error constant ratio is utilized as the reference applied prediction error degree.

9. The control method according to claim 8, characterized in that the control method further comprises:

when the prediction error outlier ratio is greater than the reference prediction error outlier ratio and remains for more than a predetermined time, an alert is issued and/or prediction of the prediction model for all machines is stopped.

10. The control method of claim 1, wherein detecting an applied prediction error degree of the predictive model for the one or more machine-manufactured products comprises:

detecting a predicted median of prediction errors of predictions performed by the prediction model, wherein the median of prediction errors is represented by a median of prediction errors of a predetermined number of predictions made recently by the prediction model;

the prediction error median is utilized as the applied prediction error degree, and the corresponding reference prediction error median is utilized as the reference applied prediction error degree.

11. The control method according to claim 10, characterized in that the control method further comprises:

And when the prediction error median value is larger than the reference prediction error median value and is maintained for more than a preset time, an alarm is sent out and/or the prediction of the prediction model for all machines is stopped.

12. The control method of claim 1, wherein detecting an applied prediction error degree of the predictive model for the one or more machine-manufactured products comprises:

detecting a predicted prediction error rate of change of the prediction performed by the prediction model, wherein the predicted error rate of change is represented by a degree of change in a predicted error of a predetermined number of predictions made recently by the prediction model;

the prediction error rate is utilized as the applied prediction error degree, and a corresponding reference prediction error rate is utilized as the reference applied prediction error degree.

13. The control method according to claim 12, characterized in that the control method further comprises:

and when the prediction error change rate is larger than the reference prediction error change rate and is maintained for more than a preset time, an alarm is sent out and/or the prediction of the prediction model for all machines is stopped.

14. The control method of claim 1, wherein detecting training prediction accuracy of the training model for the training set comprises:

By R ² The score represents the training prediction accuracy, wherein the R ² The score includes a first R ² Score and second R ² Score of the first R ² The score is R calculated for all samples in the training set ² Score of the second R ² The score is R calculated for samples in multiple subsets of the training set ² Median score of the scores.

15. The control method of claim 14, wherein the second R ² The score is obtained by:

randomly dividing the training set into a first training set and a second training set;

training the training model using the first training set;

predicting the second training set using the trained training model;

obtaining R for the second training set for the trained training model ² A score;

dividing the training set into a new first training set and a new second training set again at random; and

repeating the above steps for a predetermined number of times to obtain R ² Median of the scores as the second R ² Score.

16. The control method of claim 14, wherein R ² The score also includes a third R ² Score of the third R ² The score comprises: training model based on sample training of temporally preceding production, R for sample of temporally succeeding production ² Score.

17. The control method of claim 16, wherein the third R ² The score is obtained by:

sequencing samples in the training set in order of production completion time from first to last;

selecting a predetermined number as a selection window for determining a consecutive, predetermined number of data when selecting data from the training set;

determining the third R according to the training set and the selection window ² Score.

18. The control method of claim 17, wherein the third R is determined based on the training set and the selection window ² A score, comprising:

selecting a third training set using the selection window starting from the first sample in the training set;

training the training model using the third training set;

predicting a sample positioned at the rear position of the selection window in the training set by using a training model trained by the third training set;

moving the selection window one sample step back in the training set;

updating the third training set by using samples in the moved selection window;

Repeating the above steps until the last sample in the training set is predicted; and

calculating R for all prediction results ² Score as the third R ² Score.

19. The control method of claim 16, wherein the first R ² A score greater than the second R ² Score, and the second R ² A score greater than the third R ² Score.

20. The control method of claim 16, wherein detecting training prediction accuracy of the training model for the training set further comprises:

respectively utilizeThe first R ² Score, the second R ² Score and the third R ² The score represents the training prediction accuracy and is utilized separately from the first R ² Score, the second R ² Score and the third R ² R corresponding to fraction ² The fractional reference value is used as the reference to train the prediction accuracy.

21. The control method according to claim 20, characterized in that the control method further comprises:

when said first R ² Score, the second R ² Score and the third R ² At least one of the scores is lower than the corresponding R ² And when the score reference value is obtained, determining that the training prediction accuracy is lower than the reference training prediction accuracy, and sending out an alarm and/or suspending training of the training model for all the machines.

22. The control method according to claim 1, characterized in that the method further comprises:

detecting the abnormal degree of the training process of the training model; and

the operating state of the predictive model is at least partially changed in response to the training process anomaly being greater than a reference training process anomaly.

23. The control method of claim 22, wherein detecting the training process anomalies of the training model comprises:

detecting an interval between successive successful training of the training model;

and using the interval time to represent the abnormal degree of the training process, and using the corresponding interval time reference value as the abnormal degree of the reference training process.

24. The control method according to claim 23, characterized in that the control method further comprises:

and sending out an alarm and/or suspending training of the training models for all the machines under the condition that the interval time is larger than the interval time reference value and exceeds a preset time.

25. The control method of claim 22, wherein detecting the training process anomalies of the training model comprises:

detecting training time used for training the training model each time;

And using the training time to represent the anomaly degree of the training process, and using the corresponding training time reference value as the reference training process anomaly degree.

26. The control method according to claim 25, characterized in that the control method further comprises:

and sending out an alarm and/or suspending training of the training models for all the machines under the condition that the training time is greater than the training time reference value and exceeds a preset time.

27. The control method of claim 1, wherein detecting an association anomaly of a sample in the training set with a station from which the sample was made when the sample in the training set is from a plurality of stations comprises:

detecting the number of samples from each of the plurality of machines in the training set;

calculating a difference in the number of samples from each pair of the plurality of stations;

the maximum value of the differences in the sample numbers is used as the associated anomaly degree, and the difference reference value of the corresponding sample numbers is used as the reference associated anomaly degree.

28. The control method according to claim 27, characterized in that the control method further comprises:

when the maximum value of the differences between the sample numbers of each pair of machines is larger than the difference reference value of the sample numbers, an alarm is sent out and/or training of a training model for the larger sample number of the two machines corresponding to the maximum value is suspended.

29. The control method of claim 1, wherein detecting an association anomaly of a sample in the training set with a station from which the sample was made when the sample in the training set is from a plurality of stations comprises:

detecting the manufacturing completion time of the samples in the training set;

the manufacturing completion time is utilized as the associated anomaly, and a corresponding reference manufacturing completion time is utilized as the reference associated anomaly.

30. The control method according to claim 29, characterized in that the control method further comprises:

when the manufacturing completion time is earlier than the reference manufacturing completion time, an alarm is issued and/or training of a training model for a corresponding machine is suspended, wherein the corresponding machine is a machine that manufactures a sample having the manufacturing completion time earlier than the reference manufacturing completion time.

31. The control method of claim 1, wherein detecting an association anomaly of a sample in the training set with a station from which the sample was made when the sample in the training set is from a plurality of stations comprises:

detecting a training time span of the training set, the training time span representing a time counted from when the training set is used to train the training model;

The training time span is utilized as the associated anomaly and a corresponding reference training time span is utilized as the reference associated anomaly.

32. The control method according to claim 31, characterized in that the control method further comprises:

when the training time span is greater than the reference training time span, an alarm is issued and/or training of the training model for all the machines is suspended.

33. The control method according to any one of claims 1 to 13, wherein issuing an alarm includes:

the application predicts information related to the degree of error and the degree of operational anomaly.

34. The control method of any one of claims 1-32, wherein issuing an alarm comprises:

information related to the training prediction accuracy and the associated anomaly is displayed by a display device.

35. The control method according to any one of claims 1 to 13, characterized in that the control method further comprises:

and stopping at least one of the one or more machines in response to at least one of the following conditions being met:

the application prediction error degree is larger than the reference application prediction error degree, and

The operational anomaly is greater than the reference operational anomaly.

36. The control method according to any one of claims 1 to 32, characterized in that the control method further comprises:

The associated anomaly is greater than the reference associated anomaly.

37. A method of controlling a measurement model, the measurement model comprising a training model and a prediction model, wherein the trained training model is used to update the prediction model, and the prediction model is used to predict one or more machine-manufactured products, the method comprising:

detecting training prediction accuracy of the training model for a training set and associated anomaly of a sample in the training set and a machine for manufacturing the sample, wherein the training set comprises products manufactured by one or more machines; and

The training prediction accuracy is less than the reference training prediction accuracy, and the associated anomaly is greater than the reference associated anomaly.

38. A control system for a measurement model, comprising:

a processor; and

a memory having one or more programs stored thereon, which when executed by the processor, cause the processor to implement the method of any of claims 1-36.

39. A control system for a measurement model, comprising:

a processor; and

a memory having one or more programs stored thereon, which when executed by the processor, cause the processor to implement the method of claim 37.

40. A non-transitory computer-readable storage medium storing computer instructions for causing the computer to execute the control method according to any one of claims 1-36.

41. A non-transitory computer-readable storage medium storing computer instructions for causing the computer to execute the control method according to claim 37.

42. A computer program product comprising a computer program which, when executed by a processor, implements the control method according to any one of claims 1-36.

43. A computer program product comprising a computer program which, when executed by a processor, implements the control method according to claim 37.