CN116071237B - Video hyperspectral imaging method, system and medium based on filter sampling fusion - Google Patents
Video hyperspectral imaging method, system and medium based on filter sampling fusion Download PDFInfo
- Publication number
- CN116071237B CN116071237B CN202310179342.3A CN202310179342A CN116071237B CN 116071237 B CN116071237 B CN 116071237B CN 202310179342 A CN202310179342 A CN 202310179342A CN 116071237 B CN116071237 B CN 116071237B
- Authority
- CN
- China
- Prior art keywords
- hyperspectral
- video
- filter
- image frame
- video stream
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000000034 method Methods 0.000 title claims abstract description 58
- 238000000701 chemical imaging Methods 0.000 title claims abstract description 57
- 230000004927 fusion Effects 0.000 title claims abstract description 42
- 238000005070 sampling Methods 0.000 title claims abstract description 33
- 230000003595 spectral effect Effects 0.000 claims abstract description 57
- 238000001228 spectrum Methods 0.000 claims abstract description 33
- 238000010586 diagram Methods 0.000 claims abstract description 25
- 238000007670 refining Methods 0.000 claims abstract description 11
- 238000001914 filtration Methods 0.000 claims description 25
- 230000006870 function Effects 0.000 claims description 20
- 238000005457 optimization Methods 0.000 claims description 20
- 238000012549 training Methods 0.000 claims description 15
- 238000012545 processing Methods 0.000 claims description 14
- 238000004590 computer program Methods 0.000 claims description 12
- 230000003287 optical effect Effects 0.000 claims description 10
- 238000003860 storage Methods 0.000 claims description 6
- 230000004913 activation Effects 0.000 claims description 4
- 238000003384 imaging method Methods 0.000 abstract description 18
- 238000013135 deep learning Methods 0.000 abstract description 4
- 230000008569 process Effects 0.000 abstract description 4
- 230000004044 response Effects 0.000 description 9
- 238000012795 verification Methods 0.000 description 4
- 238000005516 engineering process Methods 0.000 description 3
- 238000011156 evaluation Methods 0.000 description 3
- 238000002474 experimental method Methods 0.000 description 3
- 238000013507 mapping Methods 0.000 description 3
- 230000007246 mechanism Effects 0.000 description 3
- 238000005316 response function Methods 0.000 description 3
- 230000000694 effects Effects 0.000 description 2
- 238000009432 framing Methods 0.000 description 2
- 238000004519 manufacturing process Methods 0.000 description 2
- 239000011159 matrix material Substances 0.000 description 2
- 230000005855 radiation Effects 0.000 description 2
- 230000006978 adaptation Effects 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 230000018109 developmental process Effects 0.000 description 1
- 238000000605 extraction Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 230000001537 neural effect Effects 0.000 description 1
- 230000009467 reduction Effects 0.000 description 1
- 238000004088 simulation Methods 0.000 description 1
- 238000012360 testing method Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T3/00—Geometric image transformations in the plane of the image
- G06T3/40—Scaling of whole images or parts thereof, e.g. expanding or contracting
- G06T3/4053—Scaling of whole images or parts thereof, e.g. expanding or contracting based on super-resolution, i.e. the output image resolution being higher than the sensor resolution
-
- G—PHYSICS
- G01—MEASURING; TESTING
- G01J—MEASUREMENT OF INTENSITY, VELOCITY, SPECTRAL CONTENT, POLARISATION, PHASE OR PULSE CHARACTERISTICS OF INFRARED, VISIBLE OR ULTRAVIOLET LIGHT; COLORIMETRY; RADIATION PYROMETRY
- G01J3/00—Spectrometry; Spectrophotometry; Monochromators; Measuring colours
- G01J3/28—Investigating the spectrum
- G01J3/2823—Imaging spectrometer
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T3/00—Geometric image transformations in the plane of the image
- G06T3/40—Scaling of whole images or parts thereof, e.g. expanding or contracting
- G06T3/4046—Scaling of whole images or parts thereof, e.g. expanding or contracting using neural networks
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N5/00—Details of television systems
- H04N5/222—Studio circuitry; Studio devices; Studio equipment
- H04N5/262—Studio circuits, e.g. for mixing, switching-over, change of character of image, other special effects ; Cameras specially adapted for the electronic generation of special effects
- H04N5/265—Mixing
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N7/00—Television systems
- H04N7/01—Conversion of standards, e.g. involving analogue television standards or digital television standards processed at pixel level
- H04N7/0135—Conversion of standards, e.g. involving analogue television standards or digital television standards processed at pixel level involving interpolation processes
-
- G—PHYSICS
- G01—MEASURING; TESTING
- G01J—MEASUREMENT OF INTENSITY, VELOCITY, SPECTRAL CONTENT, POLARISATION, PHASE OR PULSE CHARACTERISTICS OF INFRARED, VISIBLE OR ULTRAVIOLET LIGHT; COLORIMETRY; RADIATION PYROMETRY
- G01J3/00—Spectrometry; Spectrophotometry; Monochromators; Measuring colours
- G01J3/28—Investigating the spectrum
- G01J3/2823—Imaging spectrometer
- G01J2003/2826—Multispectral imaging, e.g. filter imaging
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02A—TECHNOLOGIES FOR ADAPTATION TO CLIMATE CHANGE
- Y02A40/00—Adaptation technologies in agriculture, forestry, livestock or agroalimentary production
- Y02A40/10—Adaptation technologies in agriculture, forestry, livestock or agroalimentary production in agriculture
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- Artificial Intelligence (AREA)
- Evolutionary Computation (AREA)
- Signal Processing (AREA)
- Multimedia (AREA)
- Spectroscopy & Molecular Physics (AREA)
- Computational Linguistics (AREA)
- Mathematical Physics (AREA)
- Data Mining & Analysis (AREA)
- General Health & Medical Sciences (AREA)
- Molecular Biology (AREA)
- Computing Systems (AREA)
- General Engineering & Computer Science (AREA)
- Biophysics (AREA)
- Software Systems (AREA)
- Biomedical Technology (AREA)
- Life Sciences & Earth Sciences (AREA)
- Health & Medical Sciences (AREA)
- Image Processing (AREA)
- Image Analysis (AREA)
Abstract
The invention discloses a video hyperspectral imaging method, a system and a medium based on filter sampling fusion, which comprise the steps of obtaining RGB video stream and filter video stream obtained by a filter, extracting RGB image frame I and filter image frame F; upsampling RGB image frame I into shallow spectral feature map X 0 Reconstructing the spectrum into a grain refining characteristic diagram V; optimizing the filter curve of the filter image frame F to obtain a hyperspectral image X 1 The method comprises the steps of carrying out a first treatment on the surface of the The grain refining characteristic diagram V and the hyperspectral image X 1 Fused into hyperspectral image frames X that make up a hyperspectral video stream. The invention uses RGB video stream and filter video stream obtained by filter to process video hyperspectral fusion imaging, combines advanced deep learning technique to fully mine the characteristics among RGB image frame, filter image frame and hyperspectral image frame, can realize rapid imaging of hyperspectral video stream, and can be used for realizing small-volume high-spatial resolution video hyperspectral imaging equipment.
Description
Technical Field
The invention relates to the technical field of video image processing, in particular to a video hyperspectral imaging method, a system and a medium based on optical filter sampling fusion.
Background
Compared with the traditional RGB image, the hyperspectral image has higher spectral resolution and rich spectral information, and has irreplaceable effects in the fields of aerospace, medical treatment, industry and the like. However, due to limitations of imaging principles and imaging sensors, the time-space-spectral resolution of the existing hyperspectral imaging methods are mutually restricted, and thus it is difficult to acquire high-resolution hyperspectral videos. On the other hand, spectral imaging equipment is expensive to manufacture, and the application of hyperspectral is limited to a large extent. Currently, there are great advances and developments in the technology of obtaining multispectral images by means of a specific-response multi-filter combination. These filters are often narrow-band pass high-transmission filters, and the correlation between the filters is weak, so that it is difficult to recover the spectral information of the multispectral image. Meanwhile, the existing imaging apparatus can rapidly obtain an RGB image or an RGB video of high spatial resolution. Unfortunately, RGB images may lack much spectral information than hyperspectral images. The realization of video hyperspectral imaging by recovering spectra from RGB video has become a key technical problem to be solved urgently.
Disclosure of Invention
The invention aims to solve the technical problems: aiming at the problems in the prior art, the invention provides a video hyperspectral imaging method, a system and a medium based on filter sampling fusion, which are used for carrying out video hyperspectral fusion imaging by using RGB video stream and filter video stream obtained by a filter, fully excavating the characteristics among RGB image frames, filter image frames and hyperspectral image frames by combining with advanced deep learning technology, can realize rapid imaging of hyperspectral video stream, and can be used for realizing small-volume high-spatial-resolution video hyperspectral imaging equipment.
In order to solve the technical problems, the invention adopts the following technical scheme:
a video hyperspectral imaging method based on filter sampling fusion comprises the following steps:
s101, acquiring an RGB video stream and a filter video stream obtained through a filter, extracting an RGB image frame I from the RGB video stream in frames, and extracting a filter image frame F from the filter video stream in frames;
s102, up-sampling the RGB image frame I to obtain a shallow spectral feature map X 0 Performing spectrum reconstruction to obtain a fine grain characteristic diagram V; the filtering curve optimization is carried out on the filtering image frame F to obtain a hyperspectral image X 1 ；
S103, the grain refining characteristic diagram V and the hyperspectral image X 1 Fused into hyperspectral image frames X that make up a hyperspectral video stream.
Optionally, in step S102, the RGB image frame I is up-sampled to obtain a shallow spectral feature map X 0 Comprising the following steps:
s201, subtracting a G wave band from an R wave band of an RGB image frame I to obtain RG wave band characteristics, subtracting a B wave band from a G wave band of the RGB image frame I to obtain GB wave band characteristics, and respectively taking the R wave band, the G wave band, the B wave band, the RG wave band characteristics and the GB wave band characteristics of the RGB image frame I as a group of characteristics in a combined characteristic diagram H to obtain a combined characteristic diagram H;
s202, performing spectral dimension up-sampling on the combined feature map H by adopting a trained spectral self-similarity guiding network layer to obtain a shallow spectral feature map X 0 。
Optionally, the functional expression of the trained spectral self-similarity guided network layer in step S202 is:
M i =Conv 3×3 (Conv 3×3 (Conv 3×3 (H i )))，
X 0 =Conv 3×3 ((ReLu(CAT[M 1 ,M 2 ,M 3 ,M 4 ,M 5 ))，
in the above, M i Is the firstiGroup characteristic H i Conv 3×3 For convolution with a convolution kernel size of 3×3, reLu represents the ReLu activation function, CAT represents stacking in the channel dimension, M 1 ～M 5 Respectively 1 st to 5 th group of characteristics H i And shallow spectral feature map X 0 The same number of spectral bands as the hyperspectral image frame X.
Optionally, performing spectral reconstruction in step S102 to obtain a fine-grained feature map refers to subjecting the shallow spectral feature map X to 0 And inputting a trained spectrum reconstruction layer to perform spectrum reconstruction to obtain a grain refining characteristic diagram V, wherein the function expression of the spectrum reconstruction layer is as follows:
V=Res ×N (MAB(Res ×M (X 0 )))，
in the above, res ×M M residual network blocks representing a spectral reconstruction layer, MAB representing a hybrid attention module, res, of the spectral reconstruction layer ×N Representing N residual network blocks of the spectral reconstruction layer.
Optionally, in step S102, the filtered image frame F is subjected to filter curve optimization to obtain a hyperspectral imageX 1 The method is that the filtered image frame F is input into a trained filtering curve optimization layer to perform filtering curve optimization to obtain a hyperspectral image X 1 The function expression of the filter curve optimization layer is as follows:
X 1 = Conv 3×3 (Res ×K (Conv 3×3 (F)))，
in the above, conv 3×3 For increasing convolution with a convolution kernel size of 3×3 for maintaining consistency of spectral channel numbers, res ×K For K residual network blocks to increase nonlinearity, F is the filtered image frame.
Optionally, the grain refinement feature map V and the hyperspectral image X are refined in step S103 1 The hyperspectral image frame X fused into the hyperspectral video stream is the grain-refining characteristic image V and hyperspectral image X 1 Inputting a trained fusion layer to obtain a hyperspectral image frame X forming a hyperspectral video stream, wherein the fusion layer has a function expression as follows:
X= Res ×K (V+X 1 )
in the above, res ×K For K residual network blocks.
Optionally, step S101 further includes: and acquiring RGB video stream samples and filtering video stream samples obtained through a filter, performing end-to-end supervised training on a video hyperspectral imaging network formed by a spectrum self-similarity guiding network layer, a spectrum reconstruction layer and a filtering curve optimization layer by utilizing training samples formed by the RGB video stream samples and the filtering video stream samples, wherein a loss function adopted during training is an average absolute error, and the average absolute error is the sum of absolute differences between a label value of the supervised training and an output value of the video hyperspectral imaging network.
In addition, the invention also provides a video hyperspectral imaging system based on the optical filter sampling fusion, which comprises a first video acquisition device, a second video acquisition device and a video hyperspectral imaging processing device, wherein the first video acquisition device is used for acquiring RGB video streams, the second video acquisition device is used for acquiring filtered video streams after passing through the optical filter, the output ends of the first video acquisition device and the second video acquisition device are respectively connected with the video hyperspectral imaging processing device, the video hyperspectral imaging processing device comprises a microprocessor and a memory which are connected with each other, and the microprocessor is programmed or configured to execute the video hyperspectral imaging method based on the optical filter sampling fusion.
In addition, the invention also provides a video hyperspectral imaging system based on the filter sampling fusion, which comprises a microprocessor and a memory which are connected with each other, wherein the microprocessor is programmed or configured to execute the video hyperspectral imaging method based on the filter sampling fusion.
Furthermore, the present invention provides a computer readable storage medium having stored therein a computer program for programming or configuring by a microprocessor to perform a filter sample fusion based video hyperspectral imaging method.
Compared with the prior art, the invention has the following advantages:
1. the invention uses RGB video stream and filter video stream obtained by filter to process video hyperspectral fusion imaging, based on the mapping relation between RGB video stream and filter video stream obtained by filter and hyperspectral image, and can effectively simulate the response parameter of filter set, and fully uses the internal correlation characteristic of the two, with interpretability (see the imaging principle part description below, in particular), the invention combines advanced deep learning technology to fully mine the characteristics of RGB image frame, filter image frame and hyperspectral image frame, and can realize rapid imaging of hyperspectral video stream, and can be used for realizing small-volume high-space resolution video hyperspectral imaging equipment.
2. The invention has better universality and robustness without changing the structure and parameters of the network when the invention is used for different types of filter slice groups and RGB images.
Drawings
FIG. 1 is a schematic diagram of a basic flow of a method according to an embodiment of the present invention.
Fig. 2 is a schematic diagram of a network structure of a video hyperspectral imaging network according to an embodiment of the present invention.
Fig. 3 is a schematic diagram of a network structure of a spectral self-similarity guiding network layer according to an embodiment of the present invention.
FIG. 4 is a comparison of the results of four imaging methods on a CAVE dataset hyperspectral image in an embodiment of the present invention.
Detailed Description
As shown in fig. 1, the video hyperspectral imaging method based on filter sampling fusion in this embodiment includes:
s101, acquiring an RGB video stream and a filter video stream obtained through a filter, extracting an RGB image frame I from the RGB video stream in frames, and extracting a filter image frame F from the filter video stream in frames;
s102, up-sampling the RGB image frame I to obtain a shallow spectral feature map X 0 Performing spectrum reconstruction to obtain a fine grain characteristic diagram V; the filtering curve optimization is carried out on the filtering image frame F to obtain a hyperspectral image X 1 ；
S103, the grain refining characteristic diagram V and the hyperspectral image X 1 Fused into hyperspectral image frames X that make up a hyperspectral video stream.
The imaging principle of the video hyperspectral imaging method based on the filter sampling fusion in the embodiment is as follows:
first, for the relationship between the hyperspectral image frame X and the RGB image frame I, the following mapping relationship can be established:
I c (u,v)=∫ Λ P c (λ)X(u,v,λ)dλ，（1）
in the above-mentioned method, the step of,I c (u,v) The channel c position for RGB image frame I is%u,v) Wherein lambda is the wavelength interval,P c (λ) As a function of the spectral response of channel c at wavelength lambda,X(u,v,λ) The position is [ ] under the wavelength lambdau,v) Spectral radiation at that time; channel c satisfies c epsilon { R, G, B }, wherein R, G, B are three channels (wave bands) of RGB image respectively, (-), the following are allu,v) Representing a certain position in space of the RGB image/hyperspectral image. In practical applications the wavelength interval is sampled discretely, so (1) can be rewritten as:
I c (u,v)=∑ n P c (λ)X(u,v,λ n )，（2）
in the above-mentioned method, the step of,nrepresenting the number of spectral bands,X(u,v,λ n ) Is of discrete wavelengthλ n The lower position is%u,v) Spectral radiation at that time.
Spectral response functionP c (λ) Is determined by the camera sensor and is related to the physical parameters. That is, in the case where the light source is necessarily noiseless, the hyperspectral image frame X and the RGB image frame I are in a linear relationship. In practice, however, when an RGB image is photographed, noise increases with an increase in exposure time and temperature, so that the hyperspectral image frame X and the RGB image frame I are not in a linear relationship in a real case.
Secondly, for the relation between the hyperspectral image frame X and the filtered image frame F, the following mapping relation can be established:
F=PBX
in the above formula, F is a filtered image, the size of the filtered image is w×h, and the number of wave bands is s; x is a hyperspectral image, the size of the hyperspectral image is w multiplied by h, and the wave band is S; b is a filter response matrix, the specification size of which is S×S, which can be essentially a band-pass filter, P is a spectral response matrix, the dimensions of which are s×S, w, h are the width and height of the hyperspectral image or the filtered image, respectively.
And thirdly, based on the relation between the hyperspectral image frame X and the RGB image frame I and the relation between the hyperspectral image frame X and the filter image frame F, the RGB image frame I can be extracted from the RGB video stream in a framing way, and the filter image frame F can be extracted from the filter video stream in a framing way, so that the hyperspectral image frame X of the corresponding imaging hyperspectral video stream can be realized.
In this embodiment, in step S102, the RGB image frame I is up-sampled to obtain a shallow spectral feature map X 0 Comprising the following steps:
s201, subtracting a G band from an R band of an RGB image frame I to obtain RG band characteristics, subtracting a B band from a G band of the RGB image frame I to obtain GB band characteristics, and respectively using the R band, the G band, the B band, the RG band characteristics and the GB band characteristics of the RGB image frame I as a group of characteristics in a combined characteristic diagram H to obtain a combined characteristic diagram H:
H =[R, RG, G,GB, B]；
s202, performing spectral dimension up-sampling on the combined feature map H by adopting a trained spectral self-similarity guiding network layer to obtain a shallow spectral feature map X 0 。
In this embodiment, the functional expression of the trained spectral self-similarity guiding network layer in step S202 is:
M i =Conv 3×3 (Conv 3×3 (Conv 3×3 (H i )))，
X 0 =Conv 3×3 ((ReLu(CAT[M 1 ,M 2 ,M 3 ,M 4 ,M 5 ))，
in the above, M i Is the firstiGroup characteristic H i Conv 3×3 For convolution with a convolution kernel size of 3×3, reLu represents the ReLu activation function, CAT represents stacking in the channel dimension, M 1 ～M 5 Respectively 1 st to 5 th group of characteristics H i And shallow spectral feature map X 0 The same number of spectral bands as the hyperspectral image frame X. As shown in fig. 3, the spectral self-similarity guiding network layer includes five convolution channels, each of which includes three concatenated convolution Conv with a convolution kernel size of 3×3 3×3 For extracting the firstiGroup characteristic H i Intermediate feature M of (2) i The method comprises the steps of carrying out a first treatment on the surface of the Then sequentially passing through a channel dimension stacking module to obtain results M of five convolution channels 1 ～M 5 Stacked in the channel dimension, then activated by a ReLu activation layer, and finally by a convolution kernel of size 3×Convolution Conv of 3 3×3 Outputting a shallow spectral feature map X 0 。
In this embodiment, the spectrum reconstruction in step S102 to obtain the fine grained feature map means that the shallow spectrum feature map X 0 And inputting a trained spectrum reconstruction layer to perform spectrum reconstruction to obtain a fine-grained feature map V, wherein the function expression of the spectrum reconstruction layer is as follows:
V= Conv 3×3 (Res ×N (MAB(Res ×M (Conv 3×3 (X 0 )))))，
in the above, conv 3×3 For convolutions with a convolution kernel size of 3 x 3 for dimension reduction or increase, res ×M M residual network blocks representing a spectral reconstruction layer, MAB representing a hybrid attention module, res, of the spectral reconstruction layer ×N Representing N residual network blocks of the spectral reconstruction layer. Referring to fig. 2, the spectrum reconstruction layer includes M residual network blocks (pre-layer), a mixed attention module MAB and N residual network blocks (post-layer) which are sequentially cascaded, so that hyperspectral features can be better learned. Where M and N may be selected as desired, for example m=4 and n=3 in the present embodiment. The spectrum reconstruction layer is combined with the residual error network block, so that the network has good convergence, a mixed attention mechanism is provided, the spatial spectrum characteristics of the hyperspectral image can be better learned, the parameter quantity is less, the video hyperspectral imaging speed is high, the precision is high, and better mobility is achieved.
In this embodiment, in step S102, the filtered image frame F is subjected to filter curve optimization to obtain the hyperspectral image X 1 The method is that the filtered image frame F is input into a trained filtering curve optimization layer to perform filtering curve optimization to obtain a hyperspectral image X 1 The function expression of the filter curve optimization layer is as follows:
X 1 = Conv 3×3 (Res ×K (Conv 3×3 (F)))，
in the above, conv 3×3 For increasing convolution with a convolution kernel size of 3×3 for maintaining consistency of spectral channel numbers, res ×K In order to increase the nonlinear K residual network blocks, F is a filtered image frame, and the filtered image frame can be better simulated through multi-layer nonlinear convolution OR operationAnd (5) filtering the curve. Where K may be selected as desired, for example, in this embodiment, K has a value of k=3.
In the present embodiment, the grain refining feature map V and the hyperspectral image X are performed in step S103 1 The hyperspectral image frame X fused into the hyperspectral video stream is the grain-refining characteristic image V and hyperspectral image X 1 Inputting a trained fusion layer to obtain a hyperspectral image frame X forming a hyperspectral video stream, wherein the fusion layer has a function expression as follows:
X= Res ×T (V+X 1 )
in the above, res ×T For K residual network blocks. Wherein T may be selected as required, for example, in this embodiment, T takes a value of t=3.
In this embodiment, the step S101 further includes: and acquiring RGB video stream samples and filtering video stream samples obtained through a filter, performing end-to-end supervised training on a video hyperspectral imaging network formed by a spectrum self-similarity guiding network layer, a spectrum reconstruction layer and a filtering curve optimization layer by utilizing training samples formed by the RGB video stream samples and the filtering video stream samples, wherein a loss function adopted during training is an average absolute error, and the average absolute error is the sum of absolute differences between a label value of the supervised training and an output value of the video hyperspectral imaging network.
Because hyperspectral video can be divided into a plurality of frames of hyperspectral images, 32 disclosed by CAVE is utilized to carry out verification experiments on data sets in the embodiment, and in order to carry out validity verification on the video hyperspectral imaging method based on filter sampling fusion in the embodiment, a single frame of hyperspectral image is selected to carry out verification. We selected a hyperspectral public dataset CAVE dataset with a hyperspectral image band number of 31 and a spatial dimension of 512 x 512. In experiments, the hyperspectral image in the dataset was treated as a high resolution hyperspectral image, and a set of RGB images was downsampled as input images using a known spectral response function, while the filter response function was simulated according to the characteristics of the filtered images, and a set of filtered images was downsampled as input images. In the actual process, 20 pairs of data in the CAVE data set are used as training sets, 2 pairs of data are used as verification sets, 10 pairs of data are used as test sets, and 4 typical single RGB hyperspectral imaging methods are compared. The evaluation indexes of the fusion image are 4, namely a Spectrum Angle (SAM), a Root Mean Square Error (RMSE), a Unified Image Quality Index (UIQI) and a Structural Similarity (SSIM). Wherein a larger value of UIQI and SSIM indicates a better quality of the high resolution image and a larger value of SAM and RMSE indicates a worse quality of the high resolution image. Table 1 shows objective evaluation indices of imaging experiments on CAVE data sets for 4 typical imaging methods (Arad, HSCNN-R, AWAN+, HSRnet) and the method proposed in this example (TRFS), with the best numerical results being blackened.
Table 1: the method of this embodiment and four typical hyperspectral imaging methods provide objective performance metrics on the CAVE dataset.
FIG. 4 shows the results of three exemplary imaging methods HSCNN-R, AWAN+, HSRnet and the method proposed in this example (FRFN) hyperspectral image imaging in a CAVE dataset, wherein the first row is the case hyperspectral image artwork in the CAVE dataset, wherein (a 1) is the case hyperspectral image artwork in the CAVE dataset, and (a 2) is the hyperspectral image error result plot of the HSCNN-R method in the CAVE dataset; wherein (b 1) is the original hyperspectral image of the case in the CAVE data set, and (b 2) is the error result graph of the hyperspectral image of the AWAN+ method in the CAVE data set; wherein (c 1) is the original hyperspectral image of the case in the CAVE data set, and (c 2) is the error result graph of the hyperspectral image of the HSRnet method in the CAVE data set; wherein (d 1) is the case hyperspectral image artwork in the CAVE dataset, and (d 2) is the hyperspectral image error result graph result of the method (FRFN) proposed by the embodiment in the CAVE dataset; wherein (e 1) is the original image of the hyperspectral image of the case in the CAVE data set, and (e 2) is the ideal error result image. As can be seen from table 1 and fig. 4, all objective evaluation indexes of the method (FRFN) proposed in this embodiment are superior to those of other methods, because the FRFN introduces the filtered image obtained by sampling the optical filter to form a fused image with the RGB image, so that the advantages of the two images can be combined, and a hyperspectral image with better effect can be reconstructed. More importantly, the adopted mixed attention mechanism can better learn the spatial spectrum characteristics of the hyperspectral image and save the spatial and spectral details of the image.
In summary, the video hyperspectral imaging method based on the filter sampling fusion in the method of the embodiment includes inputting an RGB video stream sampled by the filter and an RGB video stream photographed by a sensor; and then the hyperspectral video X after fusion is obtained through a filter response curve optimization layer, a spectral feature extraction layer, a reconstruction network layer, a fusion layer and the like frame by frame, so that hyperspectral video X with high spatial resolution can be effectively obtained based on filter sampling fusion, filter response parameter simulation and hyperspectral video reconstruction are realized, and the hyperspectral video X has strong universality and fewer network parameters. The video hyperspectral imaging method based on the filter sampling fusion utilizes the strong learning capability of the deep neural convolution network, and takes the filtered video obtained through the filter sampling as an auxiliary condition to realize hyperspectral video imaging. Firstly, carrying out spectrum up-sampling on a single frame image in an RGB video stream to obtain a shallow spectrum characteristic image, then sending the shallow spectrum characteristic image into a trained hyperspectral reconstruction network layer to further excavate potential association of RGB and hyperspectrum, and simultaneously designing a mixed attention mechanism to better extract the empty spectrum characteristic of the hyperspectral image. And simultaneously, carrying out spectral response curve optimization on a single frame image in the filtered video stream to obtain a hyperspectral image mapped with the filtered image, then sending the hyperspectral image and a shallow spectral feature image obtained through RGB image frames into a fusion layer, finally obtaining a high-resolution hyperspectral image frame, and then obtaining a multi-frame hyperspectral image and combining to obtain a hyperspectral video. The method has the advantages that extra hyperspectral data are not needed for training, no priori knowledge is needed, training is only needed on an RGB video and a filtered video data set which are easier to obtain, a filter response curve is simulated through a deep learning module, the method can be suitable for hyperspectral data of different types, the noise interference resistance is high, compared with other high-performance single RGB video hyperspectral imaging methods, hyperspectral videos obtained by the video hyperspectral imaging method based on the filter sampling fusion have better quality, and compared with the other high-performance single RGB video hyperspectral imaging method, hyperspectral videos obtained by multi-frame combination have better quality, have stronger noise interference resistance, and have strong universality and robustness for filtered videos and RGB videos which pass through different types of filter sets of different types, and the network structure and parameters do not need to be changed too much.
In addition, the embodiment also provides a video hyperspectral imaging system based on optical filter sampling fusion, which comprises a first video acquisition device, a second video acquisition device and a video hyperspectral imaging processing device, wherein the first video acquisition device is used for acquiring RGB video streams, the second video acquisition device is used for acquiring filtered video streams after passing through optical filters, the output ends of the first video acquisition device and the second video acquisition device are respectively connected with the video hyperspectral imaging processing device, the video hyperspectral imaging processing device comprises a microprocessor and a memory which are connected with each other, and the microprocessor is programmed or configured to execute the video hyperspectral imaging method based on the optical filter sampling fusion.
In addition, the embodiment also provides a video hyperspectral imaging system based on the filter sampling fusion, which comprises a microprocessor and a memory which are connected with each other, wherein the microprocessor is programmed or configured to execute the video hyperspectral imaging method based on the filter sampling fusion. Furthermore, the present embodiment also provides a computer readable storage medium having a computer program stored therein, the computer program being configured or programmed by a microprocessor to perform the aforementioned video hyperspectral imaging method based on filter sample fusion.
It will be appreciated by those skilled in the art that embodiments of the present application may be provided as a method, system, or computer program product. Accordingly, the present application may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present application may take the form of a computer program product embodied on one or more computer-readable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and the like) having computer-usable program code embodied therein. The present application is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the application. It will be understood that each flow and/or block of the flowchart illustrations and/or block diagrams, and combinations of flows and/or blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks. These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks. These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
The above description is only a preferred embodiment of the present invention, and the protection scope of the present invention is not limited to the above examples, and all technical solutions belonging to the concept of the present invention belong to the protection scope of the present invention. It should be noted that modifications and adaptations to the present invention may occur to one skilled in the art without departing from the principles of the present invention and are intended to be within the scope of the present invention.
Claims (6)
1. The video hyperspectral imaging method based on the optical filter sampling fusion is characterized by comprising the following steps of:
s101, acquiring an RGB video stream and a filter video stream obtained through a filter, extracting an RGB image frame I from the RGB video stream in frames, and extracting a filter image frame F from the filter video stream in frames;
s102, up-sampling the RGB image frame I to obtain a shallow spectral feature map X 0 Performing spectrum reconstruction to obtain a fine grain characteristic diagram V; the filtering curve optimization is carried out on the filtering image frame F to obtain a hyperspectral image X 1 The method comprises the steps of carrying out a first treatment on the surface of the The spectrum reconstruction to obtain a fine grain characteristic image refers to the shallow spectrum characteristic image X 0 And inputting a trained spectrum reconstruction layer to perform spectrum reconstruction to obtain a grain refining characteristic diagram V, wherein the function expression of the spectrum reconstruction layer is as follows:
V=Res ×N (MAB(Res ×M (X 0 )))，
in the above, res ×M M residual network blocks representing a spectral reconstruction layer, MAB representing a hybrid attention module, res, of the spectral reconstruction layer ×N N residual network blocks representing a spectral reconstruction layer; the filtering curve optimization is carried out on the filtering image frame F to obtain a hyperspectral image X 1 The method is that the filtered image frame F is input into a trained filtering curve optimization layer to perform filtering curve optimization to obtain a hyperspectral image X 1 The function expression of the filter curve optimization layer is as follows:
X 1 = Conv 3×3 (Res ×K (Conv 3×3 (F)))，
in the above, conv 3×3 For increasing convolution with a convolution kernel size of 3×3 for maintaining consistency of spectral channel numbers, res ×K For K residual network blocks to increase nonlinearity, F is the filtered image frame;
s103, the grain refining characteristic diagram V and the hyperspectral image X 1 Fusing into hyperspectral image frames X forming a hyperspectral video stream;
in step S102, the RGB image frame I is up-sampled to obtain a shallow spectral feature map X 0 Comprising the following steps:
s201, subtracting a G wave band from an R wave band of an RGB image frame I to obtain RG wave band characteristics, subtracting a B wave band from a G wave band of the RGB image frame I to obtain GB wave band characteristics, and respectively taking the R wave band, the G wave band, the B wave band, the RG wave band characteristics and the GB wave band characteristics of the RGB image frame I as a group of characteristics in a combined characteristic diagram H to obtain a combined characteristic diagram H;
s202, performing spectral dimension up-sampling on the combined feature map H by adopting a trained spectral self-similarity guiding network layer to obtain a shallow spectral feature map X 0 The method comprises the steps of carrying out a first treatment on the surface of the Wherein the functional expression of the spectral self-similarity guiding network layer is as follows:
M i = Conv 3×3 (Conv 3×3 (Conv 3×3 (H i )))，
X 0 = Conv 3×3 ((ReLu(CAT[M 1 ,M 2 ,M 3 ,M 4 ,M 5 ))，
in the above, M i Is the firstiGroup characteristic H i Conv 3×3 For convolution with a convolution kernel size of 3×3, reLu represents the ReLu activation function, CAT represents stacking in the channel dimension, M 1 ～M 5 Respectively 1 st to 5 th group of characteristics H i And shallow spectral feature map X 0 The same number of spectral bands as the hyperspectral image frame X.
2. The method for hyperspectral imaging of video based on filter sample fusion as claimed in claim 1, wherein the fine-grained feature map V, hyperspectral image X are processed in step S103 1 The hyperspectral image frame X fused into the hyperspectral video stream is the grain-refining characteristic image V and hyperspectral image X 1 Inputting a trained fusion layer to obtain a hyperspectral image frame X forming a hyperspectral video stream, wherein the fusion layer has a function expression as follows:
X= Res ×K (V+X 1 )
in the above, res ×K For K residual network blocks.
3. The method of video hyperspectral imaging based on filter sample fusion as claimed in claim 1, further comprising, prior to step S101: and acquiring RGB video stream samples and filtering video stream samples obtained through a filter, performing end-to-end supervised training on a video hyperspectral imaging network formed by a spectrum self-similarity guiding network layer, a spectrum reconstruction layer and a filtering curve optimization layer by utilizing training samples formed by the RGB video stream samples and the filtering video stream samples, wherein a loss function adopted during training is an average absolute error, and the average absolute error is the sum of absolute differences between a label value of the supervised training and an output value of the video hyperspectral imaging network.
4. The utility model provides a video hyperspectral imaging system based on light filter sampling fuses, its characterized in that includes first video acquisition device, second video acquisition device and video hyperspectral imaging processing equipment, first video acquisition device is used for gathering RGB video stream, second video acquisition device is used for gathering the filtering video stream after the light filter, the output of first video acquisition device, second video acquisition device links to each other with video hyperspectral imaging processing equipment respectively, video hyperspectral imaging processing equipment includes interconnect's microprocessor and memory, microprocessor is programmed or is configured in order to carry out the video hyperspectral imaging method based on light filter sampling fuses of any one of claims 1 ~ 3.
5. A filter sample fusion based video hyperspectral imaging system comprising a microprocessor and a memory connected to each other, wherein the microprocessor is programmed or configured to perform the filter sample fusion based video hyperspectral imaging method of any one of claims 1 to 3.
6. A computer readable storage medium having a computer program stored therein, wherein the computer program is for programming or configuring by a microprocessor to perform the filter sample fusion based video hyperspectral imaging method of any one of claims 1 to 3.
Priority Applications (2)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310179342.3A CN116071237B (en) | 2023-03-01 | 2023-03-01 | Video hyperspectral imaging method, system and medium based on filter sampling fusion |
| PCT/CN2023/102544 WO2024178879A1 (en) | 2023-03-01 | 2023-06-27 | Optical filter sampling and fusion-based video hyperspectral imaging method and system, and medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310179342.3A CN116071237B (en) | 2023-03-01 | 2023-03-01 | Video hyperspectral imaging method, system and medium based on filter sampling fusion |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN116071237A CN116071237A (en) | 2023-05-05 |
| CN116071237B true CN116071237B (en) | 2023-06-20 |
Family
ID=86182012
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202310179342.3A Active CN116071237B (en) | 2023-03-01 | 2023-03-01 | Video hyperspectral imaging method, system and medium based on filter sampling fusion |
Country Status (2)
| Country | Link |
|---|---|
| CN (1) | CN116071237B (en) |
| WO (1) | WO2024178879A1 (en) |
Families Citing this family (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116071237B (en) * | 2023-03-01 | 2023-06-20 | 湖南大学 | Video hyperspectral imaging method, system and medium based on filter sampling fusion |
Family Cites Families (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2018165605A1 (en) * | 2017-03-09 | 2018-09-13 | Northwestern University | Hyperspectral imaging sensor |
| WO2020199205A1 (en) * | 2019-04-04 | 2020-10-08 | 合刃科技（深圳）有限公司 | Hybrid hyperspectral image reconstruction method and system |
| US11860034B2 (en) * | 2020-05-08 | 2024-01-02 | Raytheon BBN Technologies, Corp. | Systems, devices, and methods for hyperspectral imaging |
| CN113222823B (en) * | 2021-06-02 | 2022-04-15 | 国网湖南省电力有限公司 | Hyperspectral image super-resolution method based on mixed attention network fusion |
| CN113743001B (en) * | 2021-08-13 | 2023-12-12 | 湖南大学 | Spectral super-resolution reconstruction-oriented optical filter design method, optical filter and system |
| CN114757831B (en) * | 2022-06-13 | 2022-09-06 | 湖南大学 | High-resolution video hyperspectral imaging method, device and medium based on intelligent space-spectrum fusion |
| CN114998109B (en) * | 2022-08-03 | 2022-10-25 | 湖南大学 | Hyperspectral imaging method, system and medium based on dual RGB image fusion |
| CN115345909B (en) * | 2022-10-18 | 2023-01-24 | 无锡学院 | Hyperspectral target tracking method based on depth space spectrum convolution fusion characteristics |
| CN115700727A (en) * | 2023-01-03 | 2023-02-07 | 湖南大学 | Spectral super-resolution reconstruction method and system based on self-attention mechanism |
| CN116071237B (en) * | 2023-03-01 | 2023-06-20 | 湖南大学 | Video hyperspectral imaging method, system and medium based on filter sampling fusion |
-
2023
- 2023-03-01 CN CN202310179342.3A patent/CN116071237B/en active Active
- 2023-06-27 WO PCT/CN2023/102544 patent/WO2024178879A1/en unknown
Also Published As
| Publication number | Publication date |
|---|---|
| WO2024178879A1 (en) | 2024-09-06 |
| CN116071237A (en) | 2023-05-05 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| Anwar et al. | Densely residual laplacian super-resolution | |
| CN109064396B (en) | Single image super-resolution reconstruction method based on deep component learning network | |
| CN110009590B (en) | High-quality color image demosaicing method based on convolutional neural network | |
| CN113139898B (en) | Light field image super-resolution reconstruction method based on frequency domain analysis and deep learning | |
| Luo et al. | Lattice network for lightweight image restoration | |
| WO2024027095A1 (en) | Hyperspectral imaging method and system based on double rgb image fusion, and medium | |
| CN112801881B (en) | High-resolution hyperspectral calculation imaging method, system and medium | |
| CN112561799A (en) | Infrared image super-resolution reconstruction method | |
| CN116152120A (en) | Low-light image enhancement method and device integrating high-low frequency characteristic information | |
| CN111951164A (en) | Image super-resolution reconstruction network structure and image reconstruction effect analysis method | |
| CN116071237B (en) | Video hyperspectral imaging method, system and medium based on filter sampling fusion | |
| CN110717947B (en) | High-quality spectrum reconstruction method based on external and internal training | |
| CN112184553A (en) | Hyperspectral image super-resolution method based on depth prior information | |
| CN114841856A (en) | Image super-pixel reconstruction method of dense connection network based on depth residual channel space attention | |
| CN114266957A (en) | Hyperspectral image super-resolution restoration method based on multi-degradation mode data augmentation | |
| CN114926337A (en) | Single image super-resolution reconstruction method and system based on CNN and Transformer hybrid network | |
| Yang et al. | RSAMSR: A deep neural network based on residual self-encoding and attention mechanism for image super-resolution | |
| CN107203968A (en) | Single image super resolution ratio reconstruction method based on improved subspace tracing algorithm | |
| CN113743001B (en) | Spectral super-resolution reconstruction-oriented optical filter design method, optical filter and system | |
| CN110895790A (en) | Scene image super-resolution method based on posterior degradation information estimation | |
| CN116433548A (en) | Hyperspectral and panchromatic image fusion method based on multistage information extraction | |
| CN116823662A (en) | Image denoising and deblurring method fused with original features | |
| CN116205809A (en) | Image restoration model and method for progressive guided decoding network | |
| CN116612009A (en) | Multi-scale connection generation countermeasure network medical image super-resolution reconstruction method | |
| CN113793262A (en) | Image demosaicing method based on residual error feature aggregation attention block |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |