CN111337452A - Method for verifying feasibility of spectral data model transfer algorithm - Google Patents
Method for verifying feasibility of spectral data model transfer algorithm Download PDFInfo
- Publication number
- CN111337452A CN111337452A CN202010282589.4A CN202010282589A CN111337452A CN 111337452 A CN111337452 A CN 111337452A CN 202010282589 A CN202010282589 A CN 202010282589A CN 111337452 A CN111337452 A CN 111337452A
- Authority
- CN
- China
- Prior art keywords
- predicted
- model
- slave
- model transfer
- transfer algorithm
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000012546 transfer Methods 0.000 title claims abstract description 41
- 230000003595 spectral effect Effects 0.000 title claims abstract description 27
- 238000000034 method Methods 0.000 title claims abstract description 17
- 238000013499 data model Methods 0.000 title claims abstract description 9
- 238000001228 spectrum Methods 0.000 claims abstract description 33
- 238000012937 correction Methods 0.000 claims abstract description 18
- 238000012545 processing Methods 0.000 claims abstract description 4
- 241000208125 Nicotiana Species 0.000 description 16
- 235000002637 Nicotiana tabacum Nutrition 0.000 description 16
- 239000000843 powder Substances 0.000 description 14
- 230000000694 effects Effects 0.000 description 7
- 229960002715 nicotine Drugs 0.000 description 7
- SNICXCGAKADSCV-JTQLQIEISA-N (-)-Nicotine Chemical compound CN1CCC[C@H]1C1=CC=CN=C1 SNICXCGAKADSCV-JTQLQIEISA-N 0.000 description 6
- SNICXCGAKADSCV-UHFFFAOYSA-N nicotine Natural products CN1CCCC1C1=CC=CN=C1 SNICXCGAKADSCV-UHFFFAOYSA-N 0.000 description 6
- 239000011159 matrix material Substances 0.000 description 4
- 238000005259 measurement Methods 0.000 description 3
- 239000000126 substance Substances 0.000 description 3
- 238000012360 testing method Methods 0.000 description 3
- 239000000463 material Substances 0.000 description 2
- 239000008186 active pharmaceutical agent Substances 0.000 description 1
- 230000006978 adaptation Effects 0.000 description 1
- 238000004458 analytical method Methods 0.000 description 1
- 238000013459 approach Methods 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000005056 compaction Methods 0.000 description 1
- 235000013305 food Nutrition 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000010238 partial least squares regression Methods 0.000 description 1
- 238000003672 processing method Methods 0.000 description 1
- 238000010183 spectrum analysis Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G01—MEASURING; TESTING
- G01N—INVESTIGATING OR ANALYSING MATERIALS BY DETERMINING THEIR CHEMICAL OR PHYSICAL PROPERTIES
- G01N21/00—Investigating or analysing materials by the use of optical means, i.e. using sub-millimetre waves, infrared, visible or ultraviolet light
- G01N21/17—Systems in which incident light is modified in accordance with the properties of the material investigated
- G01N21/25—Colour; Spectral properties, i.e. comparison of effect of material on the light at two or more different wavelengths or wavelength bands
- G01N21/31—Investigating relative effect of material at wavelengths characteristic of specific elements or molecules, e.g. atomic absorption spectrometry
- G01N21/35—Investigating relative effect of material at wavelengths characteristic of specific elements or molecules, e.g. atomic absorption spectrometry using infrared light
- G01N21/359—Investigating relative effect of material at wavelengths characteristic of specific elements or molecules, e.g. atomic absorption spectrometry using infrared light using near infrared light
-
- G—PHYSICS
- G01—MEASURING; TESTING
- G01N—INVESTIGATING OR ANALYSING MATERIALS BY DETERMINING THEIR CHEMICAL OR PHYSICAL PROPERTIES
- G01N21/00—Investigating or analysing materials by the use of optical means, i.e. using sub-millimetre waves, infrared, visible or ultraviolet light
- G01N21/17—Systems in which incident light is modified in accordance with the properties of the material investigated
- G01N21/25—Colour; Spectral properties, i.e. comparison of effect of material on the light at two or more different wavelengths or wavelength bands
- G01N21/31—Investigating relative effect of material at wavelengths characteristic of specific elements or molecules, e.g. atomic absorption spectrometry
- G01N21/35—Investigating relative effect of material at wavelengths characteristic of specific elements or molecules, e.g. atomic absorption spectrometry using infrared light
- G01N21/3563—Investigating relative effect of material at wavelengths characteristic of specific elements or molecules, e.g. atomic absorption spectrometry using infrared light for analysing solids; Preparation of samples therefor
-
- G—PHYSICS
- G01—MEASURING; TESTING
- G01N—INVESTIGATING OR ANALYSING MATERIALS BY DETERMINING THEIR CHEMICAL OR PHYSICAL PROPERTIES
- G01N2201/00—Features of devices classified in G01N21/00
- G01N2201/12—Circuits of general importance; Signal processing
- G01N2201/129—Using chemometrical methods
Landscapes
- Physics & Mathematics (AREA)
- Spectroscopy & Molecular Physics (AREA)
- Health & Medical Sciences (AREA)
- Life Sciences & Earth Sciences (AREA)
- Chemical & Material Sciences (AREA)
- Analytical Chemistry (AREA)
- Biochemistry (AREA)
- General Health & Medical Sciences (AREA)
- General Physics & Mathematics (AREA)
- Immunology (AREA)
- Pathology (AREA)
- Investigating Or Analysing Materials By Optical Means (AREA)
Abstract
The invention discloses a method for verifying feasibility of a spectral data model transfer algorithm, which comprises the following steps: dividing a sample to be detected into a correction set and a prediction set; the host machine detects to obtain host machine correction set spectrum data and host machine predicted spectrum data, and the slave machine detects to obtain slave machine measured spectrum data; performing PLS algorithm modeling by using the spectral data of the host correction set, and predicting the spectral data of the host prediction set by using the model to obtain a content predicted value M; processing the predicted spectrum data of the slave by adopting a model transfer algorithm to obtain brand new predicted spectrum data of the slave, and predicting by using a model to obtain a content predicted value N; and calculating the difference value between the predicted value M and the predicted value N, wherein if the absolute value of the difference value is less than a threshold value, the model transfer algorithm is feasible. The method can quickly verify the feasibility of the model transfer algorithm, and selects the optimal model transfer algorithm for different samples and different near-infrared spectrometers.
Description
Technical Field
The invention relates to the technical field of spectrum data model transfer, in particular to a method for verifying feasibility of a spectrum data model transfer algorithm.
Background
The spectrum contains abundant material information, and the spectral analysis technology has the advantages of no damage, high speed and the like, and has wide application in the fields of agriculture, food, industry and the like. The spectral data and the related chemical values are adopted for modeling, the prediction of the chemical values of the unknown samples can be quickly realized, but the spectral model has certain limitation on the prediction of the unknown samples and can only predict the unknown samples in a certain range. Different temperatures, different instruments, different measurement conditions, and different sample spectra from different regions all result in inaccurate prediction results. To solve this problem, one approach is to collect the sample spectra and chemical values to re-model, which is time and labor consuming; the other method carries out model transfer on the original model to solve the problems of model non-adaptation and the like, and simply and quickly improves the prediction result of the unknown sample. Direct correction (DS), direct piecewise correction (PDS), orthogonal signal method (OSC), Wavelet Transform (WT) and other algorithms are mainly used for model transfer under different temperatures, different instruments and different measurement conditions, and although the above algorithms can solve the problem that the model is affected by instrument performance changes, analysis time, measurement conditions and other aspects in some cases. However, the prediction effect of the revised spectral data obtained by adopting different model transfer algorithms is different for different samples in different regions and different near infrared devices. At present, in order to improve the prediction capability after model transfer in the market, most processing methods are optimized from the model transfer algorithm, however, the problem that the prediction effect is not good after model transfer cannot be solved essentially because the model transfer algorithm may not adapt to the current model is not considered, and a method for rapidly verifying the feasibility of the model transfer algorithm does not appear in the market at present.
Disclosure of Invention
The invention aims to provide a method for verifying feasibility of a spectral data model transfer algorithm, which is used for solving the problem of poor prediction effect caused by the fact that an optimized model transfer algorithm is adopted to improve the prediction capability after model transfer but the matching problem of the model transfer algorithm and the current model is not considered in the prior art.
The invention solves the problems through the following technical scheme:
a method of verifying feasibility of a spectral data model transfer algorithm, comprising:
step S100: dividing samples to be detected into two groups, namely a correction set and a prediction set; one near-infrared spectrometer is used as a host to detect a sample to be detected, so that spectrum data of a host correction set and spectrum data predicted by the host are respectively obtained, and the other near-infrared spectrometer is used as a slave to detect a prediction set, so that spectrum data measured by the slave are obtained;
step S200: performing PLS algorithm modeling by using the spectral data of the host correction set, and predicting the spectral data of the host prediction set by using the model to obtain a predicted value M of the component content of the sample to be tested;
the partial least squares regression (PLS) method can not only process the spectrum matrix, but also process the sample content matrix in the same way, so that the noise information in the spectrum matrix and the sample content matrix can be eliminated, and a good prediction effect can be obtained.
Step S300: processing the predicted spectrum data of the slave by adopting a model transfer algorithm to obtain brand new predicted spectrum data of the slave; the actual role of the model transfer algorithm is: and converting the spectral data collected by the slave equipment into spectral data consistent with the master equipment so that the model built by the master can be directly used for slave prediction.
Step S400: predicting the brand-new predicted spectral data of the slave by using the model to obtain a predicted value N of the component content of the sample to be tested;
step S500: and calculating the difference value between the predicted value M and the predicted value N, wherein if the absolute value of the difference value is always smaller than a preset threshold value, the model transfer algorithm is feasible.
And (5) replacing different model transfer algorithms, repeatedly executing the step S300, verifying the feasibility of the different model transfer algorithms, and finding out the best and most matched model transfer algorithm.
Compared with the prior art, the invention has the following advantages and beneficial effects:
the method can quickly verify the feasibility of the model transfer algorithm, selects the optimal model transfer algorithm for different samples and different near-infrared spectrometers, and is suitable for large-scale popularization and application.
Drawings
FIG. 1 is a flow chart of the present invention.
Detailed Description
The present invention will be described in further detail with reference to examples, but the embodiments of the present invention are not limited thereto.
Example (b):
referring to fig. 1, a method for verifying feasibility of a spectral data model transfer algorithm includes:
step S100: dividing samples to be detected into two groups, namely a correction set and a prediction set; one near-infrared spectrometer is used as a host to detect a sample to be detected, so that spectrum data of a host correction set and spectrum data predicted by the host are respectively obtained, and the other near-infrared spectrometer is used as a slave to detect a prediction set, so that spectrum data measured by the slave are obtained;
for example: the tobacco powder sample is selected as the sample to be tested, the tobacco content required to be predicted is the nicotine content, the tobacco powder sample is simple to prepare and convenient to test, and the sample error is extremely small, so that the tobacco powder sample is an excellent test material. The method comprises the steps of firstly selecting 500 parts of tobacco powder samples as samples to be tested, numbering the samples according to the number of 1-500, wherein the tobacco powder samples 1-400 are used as a correction set, the tobacco powder samples 401-500 are used as a prediction set, the samples to be tested are respectively placed into a special tool set for tobacco testing, the tobacco powder samples are enabled to be flat in surface and have certain thickness by downward compaction, and light rays emitted by a near-infrared spectrometer can form a good diffuse reflection effect through the tobacco powder samples, so that sample errors can be effectively reduced, and data accuracy is improved.
Step S200: performing PLS algorithm modeling by using the spectral data of the host correction set, and predicting the spectral data of the host prediction set by using the model to obtain a predicted value M of the component content of the sample to be tested;
partial Least Squares (PLS) to No. 1 mainframe spectrometerModeling collected tobacco powder spectral data of No. 1-400 and known tobacco powder nicotine content of No. 1-400, establishing a spectral model of spectrum-nicotine content, then predicting nicotine content of tobacco powder of No. 401-500 collected by a host spectrometer by using the spectral model, and respectively obtaining predicted values M of nicotine content of tobacco powder of No. 401-500401,M402,M403……M500。
Step S300: processing the predicted spectrum data of the slave by adopting a model transfer algorithm to obtain brand new predicted spectrum data of the slave;
step S400: predicting the brand-new predicted spectral data of the slave by using the model to obtain a predicted value N of the component content of the sample to be tested; the model established by the host machine carries out nicotine content prediction on the spectral data of No. 401-500 tobacco powder processed by the slave machine model transfer algorithm, and predicted values N of the nicotine content of No. 401-500 tobacco powder are obtained respectively401,N402,N403……N500。
Step S500: and calculating the difference value between the predicted value M and the predicted value N, wherein if the absolute value of the difference value is always smaller than a preset threshold value, the model transfer algorithm is feasible.
In the present embodiment, (N) is calculated separately401-M401|,N402-M402|,……|N500-M500If the value of |) is always smaller than a certain threshold, it indicates that the prediction effect of the spectrum model established by the master correction set on the master predicted collection spectrum data is consistent with the prediction effect of the spectrum model established by the master correction set on the prediction set spectrum data processed by the slave model transfer algorithm after the slave spectrum data is processed by the model transfer algorithm, and further indicates that the spectrum model established by the master correction set can be used for the slave prediction after the model transfer, thereby proving that the model transfer algorithm is feasible.
Although the present invention has been described herein with reference to the illustrated embodiments thereof, which are intended to be preferred embodiments of the present invention, it is to be understood that the invention is not limited thereto, and that numerous other modifications and embodiments can be devised by those skilled in the art that will fall within the spirit and scope of the principles of this disclosure.
Claims (1)
1. A method for verifying feasibility of a spectral data model transfer algorithm, comprising:
step S100: dividing samples to be detected into two groups, namely a correction set and a prediction set; one near-infrared spectrometer is used as a host to detect a sample to be detected, so that spectrum data of a host correction set and spectrum data predicted by the host are respectively obtained, and the other near-infrared spectrometer is used as a slave to detect a prediction set, so that spectrum data measured by the slave are obtained;
step S200: performing PLS algorithm modeling by using the spectral data of the host correction set, and predicting the spectral data of the host prediction set by using the model to obtain a predicted value M of the component content of the sample to be tested;
step S300: processing the predicted spectrum data of the slave by adopting a model transfer algorithm to obtain brand new predicted spectrum data of the slave;
step S400: predicting the brand-new predicted spectral data of the slave by using the model to obtain a predicted value N of the component content of the sample to be tested;
step S500: and calculating the difference value between the predicted value M and the predicted value N, wherein if the absolute value of the difference value is always smaller than a preset threshold value, the model transfer algorithm is feasible.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010282589.4A CN111337452A (en) | 2020-04-08 | 2020-04-08 | Method for verifying feasibility of spectral data model transfer algorithm |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010282589.4A CN111337452A (en) | 2020-04-08 | 2020-04-08 | Method for verifying feasibility of spectral data model transfer algorithm |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN111337452A true CN111337452A (en) | 2020-06-26 |
Family
ID=71180911
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010282589.4A Pending CN111337452A (en) | 2020-04-08 | 2020-04-08 | Method for verifying feasibility of spectral data model transfer algorithm |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111337452A (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112461770A (en) * | 2020-11-17 | 2021-03-09 | 山东省科学院海洋仪器仪表研究所 | Method for acquiring performance of spectrometer |
| CN113624717A (en) * | 2021-09-14 | 2021-11-09 | 四川启睿克科技有限公司 | Model establishment and use method for predicting sample components based on near infrared spectrum data |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2006014123A1 (en) * | 2004-07-27 | 2006-02-09 | Vladimir Aleksandrovich Zubkov | Method for producing independent multidimensional calibrating patterns |
| CN105738317A (en) * | 2016-02-16 | 2016-07-06 | 广州纤维产品检测研究院 | Textile near-infrared model transfer method |
| CN106932557A (en) * | 2017-04-12 | 2017-07-07 | 山东省科学院海洋仪器仪表研究所 | A kind of soil nutrient Model transfer method between different regions recommended based on many algorithms |
| CN106951720A (en) * | 2017-04-12 | 2017-07-14 | 山东省科学院海洋仪器仪表研究所 | Soil nutrient model transfer method based on canonical correlation analysis and linear interpolation |
| CN109444066A (en) * | 2018-10-29 | 2019-03-08 | 山东大学 | Model transfer method based on spectroscopic data |
-
2020
- 2020-04-08 CN CN202010282589.4A patent/CN111337452A/en active Pending
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2006014123A1 (en) * | 2004-07-27 | 2006-02-09 | Vladimir Aleksandrovich Zubkov | Method for producing independent multidimensional calibrating patterns |
| CN105738317A (en) * | 2016-02-16 | 2016-07-06 | 广州纤维产品检测研究院 | Textile near-infrared model transfer method |
| CN106932557A (en) * | 2017-04-12 | 2017-07-07 | 山东省科学院海洋仪器仪表研究所 | A kind of soil nutrient Model transfer method between different regions recommended based on many algorithms |
| CN106951720A (en) * | 2017-04-12 | 2017-07-14 | 山东省科学院海洋仪器仪表研究所 | Soil nutrient model transfer method based on canonical correlation analysis and linear interpolation |
| CN109444066A (en) * | 2018-10-29 | 2019-03-08 | 山东大学 | Model transfer method based on spectroscopic data |
Non-Patent Citations (2)
| Title |
|---|
| 刘翠玲等: "直接标准化算法在食用油酸值和过氧化值上的近红外光谱模型转移的研究", 《光谱学与光谱分析》 * |
| 林振兴等: "PDS用于不同温度下的近红外光谱模型传递研究", 《分析测试学报》 * |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112461770A (en) * | 2020-11-17 | 2021-03-09 | 山东省科学院海洋仪器仪表研究所 | Method for acquiring performance of spectrometer |
| CN112461770B (en) * | 2020-11-17 | 2022-11-29 | 山东省科学院海洋仪器仪表研究所 | Method for acquiring performance of spectrometer |
| CN113624717A (en) * | 2021-09-14 | 2021-11-09 | 四川启睿克科技有限公司 | Model establishment and use method for predicting sample components based on near infrared spectrum data |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN104089911B (en) | Spectral model transmission method based on one-variable linear regression | |
| CN103854305B (en) | A kind of Model Transfer method based on multi-scale Modeling | |
| CN105300923A (en) | Correction method of measuring point free temperature compensation model during process of online application of near infrared spectrum analyzer | |
| CN117312968A (en) | Method for predicting organic matter content of saline-alkali farmland soil | |
| CN111337452A (en) | Method for verifying feasibility of spectral data model transfer algorithm | |
| CN104596975A (en) | Method for measuring lignin of reconstituted tobacco by paper-making process by virtue of near infrared reflectance spectroscopy technique | |
| CN104596979A (en) | Method for measuring cellulose of reconstituted tobacco by virtue of near infrared reflectance spectroscopy technique | |
| CN102128805A (en) | Method and device for near infrared spectrum wavelength selection and quick quantitative analysis of fruit | |
| CN105223140A (en) | The method for quickly identifying of homology material | |
| CN110231328B (en) | Raman spectrum quantitative analysis method based on half-peak height distance method | |
| TWI428581B (en) | Method for identifying spectrum | |
| CN111141809B (en) | Soil nutrient ion content detection method based on non-contact type conductivity signal | |
| CN111896497B (en) | Spectral data correction method based on predicted value | |
| CN113095192B (en) | Dynamic load spectrum compiling method based on time domain extrapolation technology | |
| CN109709060B (en) | Method for measuring asphalt softening point, penetration degree and mass loss | |
| CN112362609A (en) | Method for identifying oil stain smoke pollution source based on infrared spectrum technology | |
| CN114611582A (en) | Method and system for analyzing substance concentration based on near infrared spectrum technology | |
| CN107169161B (en) | Soil nutrient model transfer method for segmented direct correction and slope and intercept correction | |
| CN109521002B (en) | Fuel characteristic measuring method for solid fuel particle flow | |
| CN109668862B (en) | Aluminum electrolyte molecular ratio detection method based on laser-induced breakdown spectroscopy | |
| CN112683816B (en) | Spectrum identification method for spectrum model transmission | |
| CN111044504B (en) | Coal quality analysis method considering uncertainty of laser-induced breakdown spectroscopy | |
| CN106872397A (en) | A kind of method based on existing calibration model quick detection agricultural product chemical constituent | |
| CN112861412A (en) | Biomass volatile component content measurement and modeling method based on near infrared spectrum principal component and neural network | |
| CN109145887B (en) | Threshold analysis method based on spectral latent variable confusion discrimination |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20200626 |
|
| RJ01 | Rejection of invention patent application after publication |