JP2013108788A - Multispectral image information acquisition device and multispectral image information acquisition method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To decrease the number of imaging sensors and enable one-shot photographing by acquiring an image with a small number of bands and an image of a multispectral filter array so that the images make a set and restoring a spectral image by signal processing.SOLUTION: In relation to a subject image, first image sensors 13R, 13G, 13B acquire image information of a first image which has a first color number which is three or more and a first pixel number, corresponding to the spectral sensitivity characteristic in the broadband. In relation to the same subject image, a second image sensor 14 acquires image information of a second image which has a second pixel number, a second color number corresponding to different spectral sensitivity characteristics according to the pixels, a spectral sensitivity characteristic which, in the narrowband, overlaps the spectral sensitivity characteristic in the broadband, and a third color number which is larger than the first color number when all the pixels are put together. On the basis of the image information of the first image and the image information of the second image which are acquired, image information of a third image which has the third color number and the first pixel number is generated as multispectral image information in relation to the subject image.

Description

  The present invention relates to a multispectral image information acquisition apparatus and a multispectral image information acquisition method for realizing highly accurate color reproduction based on subject spectral information.

  Traditionally, multispectral images have been useful for identifying what an object to be imaged belongs to, and for examining what state the object is in, such as remote sensing, biotechnology, medicine, and machine vision. It is used in the field.

  However, for example, as shown in FIG. 22, in order to collect a spectral image composed of a total of three dimensions of two dimensions in the spatial direction and one dimension in the wavelength direction, a special large-sized device or a spatial direction or wavelength direction may be used. Scanning is required.

  As a conventional spectral image acquisition device, a spectral image pickup device (for example, refer to Patent Document 1) including a tunable filter and a monochrome image sensor that can electrically change the center wavelength of transmitted light, a slit, There is a spectral image acquisition device (for example, refer to Patent Document 2) that acquires a spectral image for one line observed through a scene while scanning the scene.

  In addition, there is an attempt to restore a spectrum image by signal processing from a small amount of sensing data in order to realize a reduction in size of the apparatus. For example, a method has been proposed in which a spectral image with a high wavelength resolution is acquired from data with a low resolution in the wavelength direction by signal processing (see, for example, Patent Document 3).

  Further, in the field of remote sensing, a method of acquiring spectral image data having two different characteristics and outputting a spectral image having high spatial resolution and wavelength resolution by video signal processing is being studied. Specifically, there is an example in which data is acquired by a high spatial resolution monochrome sensor and a low spatial resolution spectral image sensor, and a high spatial resolution spectral image is output by signal processing (see, for example, Non-Patent Document 1). In addition, for the purpose of color reproduction based on spectrum information, there is an example of acquiring a three-channel image and multi-point measurement spectrum information (low-resolution spectrum image information) and outputting a high-resolution spectral image by signal processing (for example, (See Patent Documents 4 and 5 and Non-Patent Documents 2 and 3).

  On the other hand, in color image collection composed of three-channel images of R (red), green (G), and blue (B), instead of the three-plate type using three imaging sensors for R, G, and B In addition, for example, as shown in FIG. 23A, data is obtained by superimposing R, G, B, and 3 types of color filters in an array on an imaging sensor, and a color image is generated by interpolation processing. The plate method is widely used. When this is applied to N-band spectral image collection, as shown in FIG. 23B, a multispectral filter array in which N types of color filters are spatially arranged is used. Studies on multispectral filter array design methods and demosaicing methods have also been conducted (see, for example, Non-Patent Documents 4 and 5).

JP 2001-228024 A JP 2001-296180 A JP 2006-255324 A JP 2009-237817 A JP 2009-260411 A

M. T. Eismann and R. C. Hardie, “Hyperspectral resolution enhancement using high-resolution multispectral imagery with arbitrary response functions,” IEEE Trans. Image Processing, 43, 455-465 (2005). Y. Murakami, K. Ietomi, M. Yamaguchi and N. Ohyama, “MAP estimation of spectral reflectance from color image and multipoint spectral measurements,” Applied Optics, 46, 7068-7082 (2007). Y. Murakami, M. Yamaguchi, and N. Ohyama, “Piecewise Wiener estimation for reconstruction of spectral reflectance image by multipoint spectral measurements,” Applied Optics, 48 (11), 2188-2202 (2009). Raju Shrestha, Jon Yngve Hardeberg, and Rahat Khan, “Spatial arrangement of color filter array for multispectral image acquisition,” Proc. SPIE, 7875, pp. 787503-787503-9 (2011). Lian Miao, Hairong Qi, Rajeev Ramanath, and Wesley E. Snder, “Binary tree-based generic demosaicking algorithm for multispectral filter arrays,” IEEE Trans. Image Processing,! 5 (11), 3550-3558 (2006).

  However, since the apparatus described in Patent Document 1 and Patent Document 2 is a method for directly acquiring a spectral image without using signal processing, there is a trade-off relationship between spatial resolution, wavelength resolution, and time resolution. is there. Basically, still images are targeted. Furthermore, since data that can be acquired by one imaging sensor is two-dimensional, this is temporally scanned to acquire third-dimensional data. Therefore, it is difficult to shoot with moving subjects and videos. Further, in the case of a device that requires mechanical driving, an increase in the size of the device is inevitable.

  Therefore, there is a demand for a spectral image collecting apparatus that can be photographed in a small size and with one shot.

  Further, as in Patent Document 3, a method for acquiring a spectral image with a high wavelength resolution by signal processing from data with a low resolution in the wavelength direction is limited when the target is limited like an endoscope. By optimizing the signal processing method, relatively high accuracy can be expected. However, when targeting various subjects, sufficient accuracy in the wavelength direction cannot be obtained.

  Further, as in Non-Patent Documents 2 to 3, there are many problems in the method of acquiring multipoint measurement spectrum information.

  Further, as described in Non-Patent Documents 4 and 5, in the method using a multispectral filter array in which N types of color filters are spatially arranged, the spatial resolution and the wavelength resolution are in a trade-off relationship. In practice, it is difficult to apply when N is large (for example, N> 10).

  Therefore, in view of the conventional problems as described above, an object of the present invention is to collect multispectral image information in a small size and one shot by using a multispectral filter array to perform efficient spectral image sensing. An object of the present invention is to provide a multispectral image information acquisition apparatus and a multispectral image information acquisition method.

  Other objects of the present invention and specific advantages obtained by the present invention will become more apparent from the description of embodiments described below.

  When the spectral image is limited to a small area in the spatial direction, the image of each wavelength of the spectral image can often be expressed with high accuracy by a linear sum of images with a small number of channels (RGB image, 4-channel image, etc.). However, the coefficient of the linear sum differs for each region and for each wavelength. Furthermore, the coefficient of the linear sum for each wavelength can be calculated based on the sensing data of the decimal point in the spatial direction in the corresponding wavelength image. The latter sensing can be realized with a multispectral filter array.

  Therefore, in the present invention, instead of performing high-density sensing in both the spatial direction and the wavelength direction, sensing with high density in the spatial direction and sparse in the wavelength direction, and sensing with high density in the wavelength direction and sparse in the spatial direction From the data obtained by the above, a high-density spectral image is restored in both directions. In this case, the latter sensing uses a multispectral filter array (MSFA) that spatially arranges multiple color filters with different transmission wavelength bands to collect multispectral image information in a small and one-shot manner. To do.

  That is, the present invention is a multispectral image information acquisition apparatus, which has a first color number of three or more colors corresponding to a broadband spectral sensitivity characteristic and a first pixel for a subject image formed by a photographing lens. Image information of a first image having a number and a subject image having a second pixel number and a second color number corresponding to a different spectral sensitivity characteristic for each pixel, the spectral sensitivity The characteristic of the second image is narrow and overlaps with the spectral sensitivity characteristic of the wide band, and the number of colors when all the pixels are collected is the third number of colors larger than the first number of colors. Based on image information for outputting information, image information of the first image obtained by the image pickup means, and image information of the second image, the third color number and the first pixel number are included. Multispectral image calculation means for generating image information of the third image The provided, and generates the image information of the multi-spectral as image information the third image with respect to the subject image.

  In the multispectral image information acquisition apparatus according to the present invention, the imaging unit converts the subject image formed by the photographing lens into an image having the same number as the total number of colors of the first color number and the second color number. A branching optical system to be duplicated, and the same number of image sensors as the first number of colors arranged on the number of image planes equal to the number of the first colors duplicated by the branching optical system, and Each image sensor has a first number of pixels, and the spectral sensitivity characteristic of the image sensor is arranged on the first image sensor having a broadband spectral sensitivity characteristic, and on the same number of image planes as the second color number. The number of image sensors is the same as the second number of colors, and each image sensor has a second number of pixels, and each pixel of the image sensor has a different spectral sensitivity characteristic. Spectral sensitivity The number of colors is narrow and overlaps with the spectral sensitivity characteristics of the wide band, and the number of colors for collecting all the pixels included in the same number of image sensors as the second number of colors is larger than the first number of colors. A second image sensor having 3 colors may be provided.

  In the multispectral image information acquisition apparatus according to the present invention, the first image sensor is three image sensors for capturing a red image, a green image, and a blue image, and the second image sensor is a pixel. A color filter may be provided for each, and the image sensor may be a single image sensor having the second number of pixels color-coded by the third color filter.

  Also, in the multispectral image information acquisition apparatus according to the present invention, the imaging means includes a branching optical system that replicates the subject image formed by the photographing lens into two images, and one that is replicated by the branching optical system. One image sensor arranged on the image plane of the subject image, wherein a color filter is arranged for each pixel, and color coding is performed by the color filter of the first color number corresponding to the broadband spectral sensitivity characteristic. A first image sensor for acquiring image information of the first image having the first number of pixels, and one sheet arranged on the image plane of the other subject image copied by the branching optical system. An image sensor having a color filter for each pixel and color-coded by the color filter having the third color number corresponding to the narrow-band spectral sensitivity characteristic A first pixel corresponding to a broadband spectral sensitivity characteristic obtained by the first image sensor, wherein the second image sensor has a second image sensor for acquiring image information of the second image having two pixels. A demosaicing unit that generates image information of the image having the first number of colors and the fourth number of pixels from a number of pieces of image information by demosaicing processing; Image information of the second number of pixels corresponding to the narrow band spectral sensitivity characteristics obtained by the image sensor, and an image having the first number of colors and the fourth number of pixels demosaiced by the demosaicing unit. The image information of the image having the third color number and the fourth pixel number can be calculated based on the image information.

  In the multispectral image information acquisition apparatus according to the present invention, the imaging unit is a single image sensor arranged on an image plane on which a subject image is formed by the imaging lens, and a color filter for each pixel. The color coding is performed by the color filter having the first color number corresponding to the spectral sensitivity characteristic in the wide band and the color filter having the third color number corresponding to the spectral sensitivity characteristic in the narrow band. An image sensor having a number of pixels, and further, image information obtained by the imaging means is converted into pixel information corresponding to the broadband spectral sensitivity characteristic and pixel information corresponding to the narrow band spectral sensitivity characteristic. From the image information separation unit to be separated and the pixel information corresponding to the broadband spectral sensitivity characteristics separated by the image information separation unit, by demosaicing processing, A multi-spectral image calculation means comprising a demosaicing unit for generating image information of an image having the first number of colors and a fourth number of pixels, wherein the multispectral image calculation means is a narrow-band spectral sensitivity characteristic separated by the image information separation unit , The third color number, and the third color number based on the image information of the image having the first color number and the fourth pixel number demosaiced by the demosaicing unit. Image information of an image having the number of pixels of 4 can be calculated.

  Further, in the multispectral image information acquisition apparatus according to the present invention, the imaging means is a pixel in which the pixels on the image sensor are a collection of adjacent pixels of the number of first colors and the number of second colors. Each group has one color filter of the first color number corresponding to the broadband spectral sensitivity characteristic, and a first corresponding to the narrow band spectral sensitivity characteristic overlapping with the broadband spectral sensitivity characteristic. A color filter of the second color number among the color filters of the number of colors of 3, and a number corresponding to a number obtained by dividing the third color number by the second color number, The pixel groups may be arranged so that all of the narrowband color filters of the third color number are included in the group of adjacent pixel groups.

  Furthermore, in the multispectral image information acquisition apparatus according to the present invention, the multispectral image calculation means includes all the first small regions for each first small region image obtained by dividing the first image. A second small area image having the same number of pixels as the first small area image is obtained by multiplying each color image of the first small area image by a weighting coefficient and adding them to the image. A process of calculating, wherein a value of a third small area image cut out from the second image as an area corresponding to the first small area image, and the value of the third small area image included in the second small area image The weighting factor is determined so as to minimize the difference between the small region image 3 and the corresponding value, and the same number of times as the third color number is repeated while changing the weighting factor. Calculate the number of the second small area images It can be made to perform processing.

  The present invention is also a multispectral image information acquisition method, wherein a first image having a first color number of three or more colors and a first number of pixels corresponding to a broadband spectral sensitivity characteristic with respect to a subject image. The image information of the image and the subject image have a second number of pixels and a second number of colors corresponding to different spectral sensitivity characteristics for each pixel. An imaging step for acquiring image information of a second image having a third spectral color number that is larger than the first color number when overlapping all the pixels with a broadband spectral sensitivity characteristic. And an image of the third image having the third color number and the first pixel number based on the image information of the first image and the image information of the second image obtained in the imaging step. A multispectral image calculation step for generating information, and And generating the image information of the third image as a multi-spectral image information for the image.

  In the multispectral image information acquisition method according to the present invention, in the imaging step, the same number of subject images as the total number of the first color number and the second color number are duplicated by a branching optical system, and the subject The same number of image sensors as the first number of colors arranged on the number of image planes equal to the first number of colors on which the image is duplicated, and each image sensor has a first number of pixels. The image sensor has a spectral sensitivity characteristic of a broadband spectral sensitivity characteristic. The first image sensor acquires image information of the first image, and the subject image is duplicated in the second color number. And the same number of image sensors as the second number of colors arranged on the same number of image planes, and each image sensor has a second number of pixels, and each pixel of the image sensor is Different The spectral sensitivity characteristic is narrow and overlaps with the broadband spectral sensitivity characteristic, and the number of colors for collecting all the pixels included in the image sensor of the same number as the second color number. The image information of the second image can be obtained by a second image sensor having a third color number larger than the first color number.

  In the multispectral image information acquisition method according to the present invention, in the imaging step, two subject images are duplicated by the branching optical system and arranged on the image plane of one subject image duplicated by the branching optical system. A first image sensor having a first number of pixels color-coded by the color filter having the first color number corresponding to the broadband spectral sensitivity characteristic, wherein a color filter is arranged for each pixel. The image information of the first image is acquired by one image sensor, and one image sensor arranged on the image plane of the other subject image duplicated by the branching optical system is provided for each pixel. A filter is provided, and has the third number of pixels color-coded by the third color filter corresponding to the narrow band spectral sensitivity characteristic. Image information of the second image obtained by the second image sensor, and image information of the first number of pixels corresponding to the broadband spectral sensitivity characteristic obtained by the first image sensor in the imaging step. To a demosaicing step of generating image information of an image having the first number of colors and the fourth number of pixels by a demosaicing process. In the multispectral image calculation step, the imaging step includes the first step. Image information of the second number of pixels corresponding to the spectral sensitivity characteristics of the narrow band obtained by the second image sensor, and the number of first colors and the number of fourth pixels demosaicing processed in the demosaicing step. Based on the image information of the image, the image information of the image having the third color number and the fourth pixel number can be calculated. That.

  In the multispectral image information acquisition method according to the present invention, in the imaging means step, one image sensor arranged on an image plane on which a subject image is formed, and a color filter is arranged for each pixel. The first number of pixels color-coded by the color filter of the first color number corresponding to the spectral sensitivity characteristic of the wide band and the color filter of the third color number corresponding to the spectral sensitivity characteristic of the narrow band Image information of the first number of pixels is acquired by the image sensor having the image information, and further, the image information obtained by the image sensor in the imaging means step is the pixel information corresponding to the broadband spectral sensitivity characteristic and the narrowband The image information separation step for separating the pixel information corresponding to the spectral sensitivity characteristics of the image and the wide band separated in the image information separation step A demosaicing step for generating image information of an image having the first number of colors and the fourth number of pixels by demosaicing processing from pixel information corresponding to the spectral sensitivity characteristics of the multispectral image calculation In the step, the pixel information corresponding to the spectral sensitivity characteristic of the narrow band separated in the image information separation step, and the first color number and the fourth pixel number demodulated in the demosaicing step are included. Based on the image information of the image, the image information of the image having the third color number and the fourth pixel number can be calculated.

  Moreover, in the multispectral image information acquisition method according to the present invention, in the multispectral image calculation step, all the first small regions are obtained for each first small region image obtained by dividing the first image. A second small area image having the same number of pixels as the first small area image is obtained by multiplying each color image of the first small area image by a weighting coefficient and adding them to the image. A process of calculating, wherein a value of a third small area image cut out from the second image as an area corresponding to the first small area image, and the value of the third small area image included in the second small area image The weighting factor is determined so as to minimize the difference between the small region image 3 and the corresponding value, and the same number of times as the third color number is repeated while changing the weighting factor. Calculate the number of second small area images It can be made to perform the process of.

  In the present invention, an image with a small number of channels and an image of a multispectral filter array are acquired as a set, and the spectral image is restored by signal processing, thereby reducing the number of imaging sensors and enabling one-shot imaging.

It is a block diagram which shows the structural example (1st Embodiment) of the multispectral image information acquisition apparatus to which this invention is applied. It is a block diagram which shows the other structural example of the multispectral image information acquisition apparatus to which this invention is applied. It is a block diagram which shows the structural example of the multispectral image calculation part in the said multispectral image information acquisition apparatus. It is a schematic diagram which shows the processing content with respect to one small area | region in the case of calculating independently in each band in the said multispectral image calculation part. It is a block diagram which shows the structural example (2nd Embodiment) of the multispectral image information acquisition apparatus to which this invention is applied. It is a block diagram which shows the other structural example of the multispectral image information acquisition apparatus to which this invention is applied. It is a block diagram which shows the structural example (3rd Embodiment) of the multispectral image information acquisition apparatus to which this invention is applied. It is a block diagram which shows the other structural example of the multispectral image information acquisition apparatus to which this invention is applied. It is a figure which shows the structure (minimum unit of the example of a pixel arrangement in the case of K = 16) of the color filter on the single-plate color image sensor with which the multispectral image information acquisition apparatus in 3rd Embodiment is equipped. It is a figure which shows the spectral sensitivity characteristic of the RGB imaging sensor set when simulating about 1st Embodiment (K = 16). It is a figure which shows 16 types of spectral sensitivity characteristics of MSFA set when simulating about 1st Embodiment (K = 16). It is a figure which shows the pixel arrangement | sequence of MSFA set when simulating about 1st Embodiment (K = 16). It is a figure which performs the simulation about 1st Embodiment (K = 16), and shows the result compared with the other method. It is a figure which performs the simulation about 1st Embodiment (K = 16), and shows the result compared with the other method. It is a figure which performs the simulation about 1st Embodiment (K = 16), and shows the result compared with the other method. It is a figure which shows the example of the pixel arrangement | sequence of the single-plate imaging sensor set when simulating about 3rd Embodiment (K = 16). It is a figure which shows the spectral sensitivity characteristic of the color filter set when simulating about 3rd Embodiment (K = 16). It is a figure which shows 16 types of spectral sensitivity characteristics of MSFA set when simulating about 3rd Embodiment (K = 16). It is a figure which shows the spectral image used when simulating about 3rd Embodiment (K = 16). It is a figure which performs the simulation about 3rd Embodiment (K = 16) and shows the result compared with the other method. It is a figure which performs the simulation about 3rd Embodiment (K = 16) and shows the result compared with the other method. It is a figure which shows the spectrum image comprised from a total of three dimensions of two dimensions in a space direction and one dimension in a wavelength direction. As a configuration example of a color filter arranged on an imaging sensor, (A) shows an RGB filter, and (B) shows a multispectral filter array.

  Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. Needless to say, the present invention is not limited to the following examples, and can be arbitrarily changed without departing from the gist of the present invention.

  In the following description, the multispectral image output from the method / apparatus has Nx × Ny elements (pixels) in the spatial direction and K elements (total Nx × Ny × K elements) in the wavelength direction. It shall consist of An Nx × Ny image corresponding to an element in the wavelength direction is referred to as a “band image”. Each band image is defined as an image observed through a sensor having an arbitrary narrow band spectral sensitivity characteristic.

<First Embodiment>
The present invention is applied to, for example, a multispectral image information acquisition apparatus 10 having a configuration as shown in FIG.

  The multispectral image information acquisition apparatus 10 includes a photographing lens 11 that forms a subject image, a branching optical system 12 that branches incident light incident through the photographing lens 11 into four optical paths, and the branching optical system. Image sensors 13R, 13G, 13B for imaging three primary colors (R, G, B) provided in the optical path of incident light branched into four by the system 12, and an image sensor 14 for imaging MSFA, Multispectral supplied with image information of RGB images obtained by image sensors 13R, 13G, and 13B for imaging primary color (R, G, B) images and MS mosaic images obtained by the image sensor 14 for imaging MSFA images. An image calculation unit 15 is provided.

  The imaging lens 11 forms a subject image on the four imaging sensors 13R, 13G, 13B, and 14 by the incident light branched into four by the branching optical system 12.

  The image sensors 13R, 13G, and 13B for imaging the three primary color (R, G, and B) images all have Nx × Ny pixel counts. The image sensor 14 for MSFA imaging is not limited to this, but the following description will be made assuming that the number of pixels is Nx × Ny.

  The image information of the R image, G image, B image, and MS mosaic image obtained by capturing the subject image with the four image sensors 13R, 13G, 13B, and 14 is sent to the multispectral image calculation unit 15. The multispectral image calculation unit 15 converts multispectral image information based on the image information of R image, G image, B image, and MS mosaic image sent from the four image sensors 13R, 13G, 13B, and 14. Calculate and output.

  The multispectral image information acquisition apparatus 10 can have a function of outputting an RGB image like the multispectral image information acquisition apparatus 10A shown in FIG.

  Here, the same components as those in the multispectral image information acquisition apparatus 10 are denoted by the same reference numerals, detailed description thereof is omitted, and different components are described.

  That is, the multispectral image information acquisition apparatus 10A stores RGB image information of R, G, and B images obtained by image sensors 13R, 13G, and 13B for imaging three primary color (R, G, and B) images. The image recording unit 16 and the MS mosaic image recording unit 17 that stores image information of the MS mosaic image obtained by the image sensor 14 for MSFA imaging, and the function of outputting the RGB image recorded in the RGB image recording unit 16 as it is Have If necessary, the image information of the RGB image recorded in the RGB image recording unit 16 and the image information of the MS mosaic image recorded in the MS mosaic image recording unit 17 are sent to the multispectral image calculation unit 15, The multispectral image calculation unit 15 calculates and outputs image information of the multispectral image. In the case of this configuration, it is possible to acquire information on a multispectral image as necessary while maintaining compatibility with an existing color system based on RGB. Furthermore, the present invention includes the case where the RGB image recording unit 16, the MS mosaic image recording unit 17, and the multispectral image calculation unit 15 are implemented as separate devices.

  The spectral sensitivities of the image sensors 13R, 13G, and 13B for imaging the three primary color (R, G, B) images with respect to incident light are the spectral sensitivity characteristics of RGB cameras used for general color image information collection, or characteristics similar thereto. To do. This spectral sensitivity may be realized by an optical member that divides an optical path, a color filter provided on an image sensor, or a combination of both. The image sensor 14 for MSFA imaging is a multispectral array type sensor in which different types of color filters are arranged for each pixel, and K types of narrow-band color filters are arranged. The spectral sensitivity characteristics of the K color filters correspond to the spectral sensitivity characteristics of each band image of the output multispectral image. In addition, it is desirable that any wavelength band in which the K color filters have spectral sensitivity overlap with the wavelength band in which the RGB camera has sensitivity.

  As described above, the image information of the R image, G image, and B image obtained by the image sensors 13R, 13G, and 13B for imaging the three primary colors (R, G, and B) is sampled at high density at all pixel positions. On the other hand, the information of each of the K types of color filters, that is, the image information of the MS mosaic image obtained by the image sensor 14 for MSFA imaging, is sampled at a low density with only one of the K pixels. Will be.

  The multispectral image calculation unit 15 in the multispectral image information acquisition apparatus 10 includes a small region cutout unit 151, a weight coefficient calculation unit 152, and a multispectral image generation unit 153, as shown in FIG.

  In the multispectral image calculation unit 15, the small region cutout unit 151 cuts out the designated small region from each image. Thereafter, the weighting coefficient calculation unit 152 calculates a weighting coefficient for multiplying each of the R, G, and B channel images. The weighting coefficient is calculated as a different value for each band of the K band image. At this time, the spectral coefficient characteristics of the RGB channels, the spectral sensitivity characteristics of the K types of MSFA filters, and the MSFA color filter array information are input to the weighting coefficient calculation unit 152. The spectral image generation unit 153 multiplies the R, G, and B small region images output from the small region cutout unit 151 by the weighting coefficient output from the weight calculation unit 152 and adds them together. A multispectral image corresponding to the small region is generated. A multispectral image is generated and output by repeating the above processing for all regions.

  The weighting coefficient calculation unit 152 calculates a different coefficient for each band. There are two methods for calculating the weighting factor: a method of calculating each band independently, and a method of calculating all band images simultaneously.

  Here, the processing content for one small region when each band is calculated independently is shown in the schematic diagram of FIG.

  That is, as shown in FIG. 4, pixel information corresponding to the k-th band color filter is selected from the MSFA mosaic image of the small area (hereinafter referred to as a k-band selected pixel), and the pixel at the same position is selected as the R area R An image, a G image, and a B image are selected (hereinafter referred to as RGB selection pixels). Calculate the weighting coefficient so as to minimize the square error between the result obtained by multiplying the RGB selection pixel by the weighting coefficient and the k-band selection pixel, and multiplying the RGB image by the calculated weighting coefficient Then, the image of the kth band is calculated.

  Hereinafter, details of the processing in the multispectral image calculation unit 15 will be described.

  The calculation of the multispectral image is performed for each small region cut out from the image output from the image sensor. In the following, processing relating to one small area will be described. In addition, the number of pixels included in the small region is N, and the original spectral spectrum corresponding to the i-th pixel in the small region is

(L-dimensional column vector). Here, L is the number of sampling points when discretized at sufficiently fine intervals in the wavelength band that handles spectrum information. For example, L = 81 dimensions in which the visible range 380 to 780 [nm] is sampled at intervals of 5 nm is used. .

  Also,

Single wavelength image corresponding to the first wavelength

Assuming (N-dimensional column vector), the spectral spectrum for N pixels can be expressed as the following equation (1).

  First, an ideal multispectral image is defined. Respective spectral sensitivity characteristics of K types of color filters of MSFA are expressed as column vectors, and an L × K dimensional matrix in which these vectors are arranged

And If the image sensor has a linear characteristic, information of an ideal multispectral image corresponding to the i-th pixel is expressed by the following equation (2).

It is expressed. In addition, the single band image of the kth band

Assuming (N-dimensional column vector), a multispectral image for N pixels can be expressed as the following equation (3).

Now, multispectral information has been roughly sampled by MSFA. An M _k × N-dimensional matrix that samples M _k pixels from N pixels by a color filter corresponding to the k-th band

Then, the sampled information can be expressed as the following equation (4).

  here,

The row vector is a vector in which only the element corresponding to the pixel to be sampled is 1 and all other elements are 0. On the other hand, the spectral sensitivity characteristics of the RGB channels are expressed as column vectors, and an L × 3D matrix in which these three vectors are arranged is

And If the image sensor has linear characteristics, the resulting RGB image

(N × 3D matrix) can be expressed as the following equation (5).

  In the multispectral image calculation unit 15, the observed information

And RGB image

Based on this information, an ideal multispectral image

Is estimated and output.

  In the present invention, each band image

Is estimated as shown in the following equation (6).

  here,

Is a weighting factor for estimating an image of the k-th band with a three-dimensional column vector. There are two methods for calculating the weighting factor: a method of calculating each band image independently and a method of calculating all bands simultaneously. Hereinafter, details of each method will be described.

Substituting the weight coefficient calculation method equation (6) for each band independently into equation (4) yields the following equation (7).

  Where the unknown is the 3D column vector

Other variables can be obtained from the observation data and the array information of the filter array. So to minimize the square error on both sides

It is possible to ask. When linear multiple regression analysis is used, the following equation (8) is obtained.

  here,

Is

Means the estimation of The method of minimizing the square error on both sides of Equation (7) is not limited to linear multiple regression analysis. By performing this operation for all band images, weighting coefficients corresponding to all bands are calculated.

Multispectral image estimated weighting factor calculation method for all bands simultaneously

RGB image observed for

It is also possible to estimate in consideration of consistency with In this case, it is necessary to select the same small region for all the bands and simultaneously estimate the weighting coefficients corresponding to all the bands. In the following, the total number of pixels sampled in each band

And However, when all pixels of the image sensor are assigned to one of the K color filters

It is.

  Now ideal multispectral image

RGB image estimated from

And here,

It is. Also,

Is

Pseudo inverse of

Is an arbitrary matrix that becomes a K × K unit matrix). If K is a large number,

Is

Value close enough to the RGB image

Can be estimated with high accuracy. Estimated multispectral image

The condition for satisfying the relationship of the expression (9) is given by the following expression (10).

here,

It is.

Then, equation (10) can be rewritten as the following equation (11).

Expression (11) becomes the following expression (12).

here,

Represents the Kronecker product. On the other hand, equation (7) also

When rewritten into a notation using the following expression (13):

On the right side of equation (13)

Matrix of

Then, the following equation (14) is obtained.

When Expression (12) and Expression (14) are combined into one expression, the following Expression (15) is obtained.

In Equation (15), the unknown is a 3K-dimensional column vector.

Other variables can be obtained from observation data, RGB channel spectral sensitivity characteristics, MSFA spectral sensitivity characteristics, and MSFA filter array information. Therefore, to minimize the square error on both sides,

Can be requested. However, in equation (15)

Is a coefficient that weights the errors of Equation (12) and Equation (14) in the square error minimization of both sides of Equation (12) and Equation (14). The vector on the left side of equation (15) is

, The matrix on the right side

Then, for example, when linear multiple regression analysis is used, the following equation (16) is obtained.

  The above is the detailed description regarding the weighting coefficient calculation.

  Here, a supplementary description will be given of a multispectral image calculation method in a case where the image sensor 14 for capturing MSFA images is different from the number of pixels Nx × Ny of the image sensors 13R, 13G, and 13B for capturing RGB images. However, it is assumed that the image sensor 14 for capturing MSFA images and the image sensors 13R, 13G, and 13B for capturing RGB images acquire images of the same subject area. First, resolution conversion is performed on an RGB image into an image having the same number of pixels as the MSFA mosaic image. A weighting factor is calculated using the resolution-converted RGB image. Thereafter, when a multispectral image is generated by adding the RGB images using the calculated weighting factor, the original RGB image before resolution conversion is used.

  Next, it supplements about the division | segmentation into a small area. In the case of the method of calculating the weighting coefficient independently for each band, the small area for each band may be the same or different. On the other hand, in the method of calculating the weighting coefficient simultaneously for all bands, it is necessary to unify the small areas of all bands. Further, in any case, the division into small regions may be performed stochastically. This means that one pixel belongs to a plurality of small areas, and the probability of belonging is defined. At this time, in the calculation of the multispectral image, the equation (4)

The

Replace with here,

Each row vector is a vector in which only the element corresponding to the pixel to be sampled has a positive value of 1 or less other than 0 and all other elements are 0. The value of an element other than 0 represents the probability that the pixel is included in the small area currently processed by a numerical value between 0 and 1.

From

There is no change from the above-described processing except for the replacement.

  Next, a method for extending the model of Equation (6) in multispectral image calculation will be described. The model of Equation (6) is for each band image

Are represented by the addition of RGB images, and the R, G, and B images serve as image bases. Therefore, it is possible to extend the model of Equation (6) by adding a general-purpose image base. For example, since a component with a low spatial frequency is included in many images, it is the image base with the lowest spatial frequency.

(N-dimensional column vector)

A method is considered. In this case, the weighting factor

Becomes a four-dimensional column vector, and the unknown is increased by one. Similarly, any image base that appears to be valid can be added.

  That is, in the multispectral image information acquisition apparatus 10 according to the first embodiment, the first number of colors = 3, the second number of colors = 1, and the third number of colors = 16. A branch optical system 12 that duplicates the subject image formed by the same number of colors as the sum of the first color number and the second second color number, that is, four images, and the branch optical system. 12, the same number as the first number of colors arranged on the number of image planes equal to the number of first colors reproduced by 12, that is, the three image sensors 13R, 13G, and 13B, and the second color. The same number of images as the second number of colors arranged on the same number of image planes, that is, one image sensor 14 is provided. The three image sensors 13R, 13G, and 13B each have the first number of pixels and have broadband spectral sensitivity characteristics. The image sensor 14 has a second number of pixels, each pixel of the image sensor has a different spectral sensitivity characteristic, and the spectral sensitivity characteristic is narrow and overlaps with the broadband spectral sensitivity characteristic. ing.

  In the multispectral image information acquisition apparatus 10, the image information of the first image having the first number of colors and the first number of pixels corresponding to the broadband spectral sensitivity characteristic is obtained from the image sensor 13R. , 13G, and 13B, and the subject image has a second number of pixels and a different spectral sensitivity characteristic for each pixel, and the spectral sensitivity characteristic is a narrow band and the broadband spectral sensitivity characteristic. The image sensor 14 acquires image information of the second image having a third color number larger than the first color number having the overlap, and the acquired image information of the first image and the second image information. The image information of the third image having the third color number and the first pixel number is generated as multispectral image information for the subject image based on the image information of the image.

  Here, in the multispectral image information acquisition apparatus 10 according to the first embodiment, the second number of colors = 1, and the third number of colors larger than the first number of colors by one image sensor 14. The image information of the second image is acquired. For example, the second color number = 2, and the branching optical system 12 adds the first color number and the second second color number. The same number as the number of colors, that is, five subject images may be duplicated, and the image sensor 14 may be constituted by two image sensors.

  In this case, the two image sensors constituting the image sensor 14 each have a second number of pixels, each pixel has a different spectral sensitivity characteristic, and the spectral sensitivity characteristic is a narrow band. The number of colors when all the pixels included in the image sensor having the same number as the second number of colors are overlapped with the broadband spectral sensitivity characteristic is a third number of colors larger than the first number of colors. It is supposed to be.

  In other words, the multispectral image information acquisition apparatus 10 according to the first embodiment has the above-described first color number and second second color number as a subject image formed by the photographing lens 11 as an imaging unit. A branch optical system 12 that replicates the same number of images as the number of colors, and the first number of colors arranged on each of the number of image planes equal to the first color number replicated by the branch optical system 12; The same number of image sensors, each image sensor having a first number of pixels, and the spectral sensitivity characteristics of the image sensors are broadband spectral sensitivity characteristics, and the second image sensor. The image sensors having the same number as the second number of colors arranged on the same number of image planes as the number of colors, each image sensor having a second number of pixels, and each pixel of the image sensor Each The spectral sensitivity characteristic is narrow and overlaps with the broadband spectral sensitivity characteristic, and the number of colors when all the pixels included in the image sensor of the same number as the second color number are collected. May comprise a second image sensor having a third number of colors greater than the first number of colors.

  In the multispectral image information acquisition apparatus 10, the image information of the first image having the first color number and the first pixel number of three or more colors corresponding to the broadband spectral sensitivity characteristic with respect to the subject image. Is acquired by the first image sensor, and the subject image has a second number of pixels and a second number of colors corresponding to different spectral sensitivity characteristics for each pixel. Is an image information of a second image having a narrow band and an overlap with the broadband spectral sensitivity characteristic, and the number of colors when all the pixels are collected is a third color number larger than the first color number. Is acquired by the second image sensor, and the third color is used as multispectral image information for the subject image based on the acquired image information of the first image and image information of the second image. And the first pixel number It is possible to generate the image information of the third image.

<Second Embodiment>
Further, the present invention is applied to a multispectral image information acquisition apparatus 20 having a configuration as shown in FIG. 5, for example.

  The multispectral image information acquisition device 20 is incident on a photographic lens 21 that forms a subject image, an optical low-pass filter 22 that is provided at the rear stage of the photographic lens 21, and the optical lens 22 that is incident on the photographic lens 21 via the optical low-pass filter 22. Branching optical system 23 for branching incident light into two optical paths, and for imaging three primary colors (R, G, B) provided in the optical path of incident light branched into two by this branching optical system 23 A single-plate RGB image sensor 24 and an image sensor 25 for imaging MSFA images, a demosaicing unit 26 to which image information of RGB mosaic images obtained by the RGB image sensor 24 is supplied, and an RGB signal from the demosaicing unit 26 An MS mosaic obtained by the image sensor 25 for imaging the MSFA image while supplying image information of the image. Comprising a multispectral image calculating unit 27 the image information of the image are supplied.

  The imaging lens 21 forms a subject image on the two imaging sensors 24 and 25 by the incident light branched into two by the branching optical system 23. At this time, the subject image from which the high-frequency component is cut by the optical low-pass filter 22 is formed on the two image sensors 24 and 25.

  The RGB image sensor 24 is a single-plate image sensor in which three primary color (R, G, B) color filters are arranged in a mosaic pattern on the imaging surface, and has a number of pixels of Nx × Ny. The image sensor 25 for MSFA imaging is not limited to this, but the following description will be made assuming that the number of pixels is Nx × Ny.

  The image information of the RGB mosaic image obtained by the RGB image sensor 24 is obtained by the demosaicing unit 26 with three images of R, G, and B each having Nx × Ny pixels, that is, an R image, a G image, and a B image. Is converted into image information. The image information of the R image, the G image, and the B image converted by the demosaicing unit 26 is supplied to the multispectral image calculation unit 27 together with the image information of the MS mosaic image obtained by the image sensor 25 for imaging the MSFA image. Then, the multispectral image calculation unit 26 calculates and outputs multispectral image information based on the image information of the R image, the G image, the B image, and the MS mosaic image.

  The multispectral image information acquisition apparatus 20 can have a function of outputting RGB images, like the multispectral image information acquisition apparatus 20A shown in FIG.

  Here, the same components as those in the multispectral image information acquisition apparatus 20 are denoted by the same reference numerals, detailed description thereof will be omitted, and different components will be described.

  That is, the multispectral image information acquisition apparatus 20A is obtained by the RGB image recording unit 28 that stores image information of the R image, G image, and B image obtained by the demosaicing unit 26, and the image sensor 25 for MSFA imaging. An MS mosaic image recording unit 29 that stores image information of the MS mosaic image is provided, and the RGB image recorded in the RGB image recording unit 28 is output as it is. If necessary, the image information of the RGB image recorded in the RGB image recording unit 28 and the image information of the MS mosaic image recorded in the MS mosaic image recording unit 29 are sent to the multispectral image calculation unit 27, The multispectral image calculation unit 27 calculates and outputs image information of the multispectral image. In the case of this configuration, it is possible to acquire information on a multispectral image as necessary while maintaining compatibility with an existing color system based on RGB. Furthermore, the present invention includes a case where the RGB image recording unit 28, the MS mosaic image recording unit 29, and the multispectral image calculation unit 27 are implemented as separate devices.

  In the multispectral image information acquisition apparatus 20, the RGB image sensor 24 is a single-plate image sensor in which color filters corresponding to R, G, and B are arranged for each pixel. The spectral sensitivity characteristics of R, G, and B are the same as those described in the first embodiment. The arrangement of the R, G, and B color filters is a general arrangement represented by the Bayer arrangement, or an arrangement similar thereto.

  The image sensor 25 for MSFA imaging is the same as that described in the first embodiment.

  The optical low-pass filter 22 is intended to prevent aliasing distortion in the RGB image output from the demosaicing unit 26. On the RGB image sensor 24, each of the R, G, and B pixels can be arranged in at least one pixel in at least two pixels in the vertical and horizontal directions. Therefore, the optical low-pass filter 22 has a high-frequency component up to about half the Nyquist frequency of the image sensor. It is desirable that the characteristics be cut. The place where the optical low-pass filter 22 is inserted may not be between the imaging lens 21 and the branch optical system 23. An arbitrary method can be used for the demosaicing process in the demosaicing unit 26.

  The function of the multispectral image calculation unit 27 is the same as that of the multispectral image calculation unit 15 in the first embodiment.

  In other words, the second embodiment is a multispectral image information acquisition device 20 in which the first number of colors = 3, the second number of colors = 1, and the third number of colors = 16, A branching optical system 23 that duplicates the subject image formed by the photographing lens 21 into two images, and one image sensor arranged on the image plane of one subject image duplicated by the branching optical system 23. Image information of the first image having a first number of pixels color-coded by the color filter of the first number of colors corresponding to the broadband spectral sensitivity characteristic, wherein a color filter is arranged for each pixel. The first image sensor 24 for acquiring the image and one image sensor arranged on the image plane of the other subject image duplicated by the branching optical system 23, and a color filter is arranged for each pixel. , A second image sensor for acquiring image information of the second image having the second number of pixels color-coded by the color filter having the third color number corresponding to the spectral sensitivity characteristic of the narrow band. 25.

  Then, from the image information of the first number of pixels corresponding to the broadband spectral sensitivity characteristic obtained by the first image sensor 24, the demosaicing processing in the demosaicing unit 26 performs the first color number and the fourth color number. Image information of an image having the number of pixels is generated, and image information of the second number of pixels corresponding to the narrow band spectral sensitivity characteristic obtained by the second image sensor 25 and demosaicing processing by the demosaicing unit 26. Based on the image information of the image having the first color number and the fourth pixel number, the multispectral image calculation unit 27 uses the third color number as multispectral image information for the subject image, Image information of the image having the fourth number of pixels is calculated.

  Here, also in the multispectral image information acquisition apparatus 20 according to the second embodiment, the second color number = 1, and the third color larger than the first color number by one image sensor 25. The image information of the second image that is a number is acquired. For example, a plurality of second colors are used, and the same number of subject images as the second colors are duplicated by the branching optical system 23. The second image sensor 25 may be composed of the same number of image sensors as the second number of colors.

In this case, the same number of image sensors as the second number of colors constituting the second image sensor 25, each image sensor has the second number of pixels, and each pixel has a different spectral sensitivity characteristic. The spectral sensitivity characteristic is narrow and overlaps with the broadband spectral sensitivity characteristic, and the number of colors when all the pixels included in each image sensor are collected is a third color number larger than the first color number. It is supposed to be.
<Third Embodiment>
Further, the present invention is applied to a multispectral image information acquisition apparatus 30 having a configuration as shown in FIG. 7, for example.

  The multispectral image information acquisition device 30 includes a photographic lens 31 that forms a subject image, an optical low-pass filter 32 provided at the rear stage of the photographic lens 31, and incident light from the photographic lens 31 passes through the optical low-pass filter 32. A single-plate color image sensor 33 incident thereon, an image information separation unit 34 to which image information of a color image obtained by the color image sensor 33 is supplied, and image information of RGB mosaic images from the image information separation unit 34 And a multispectral image calculation unit 36 to which image information of RGB images is supplied from the demosaicing unit 35 and image information of MS mosaic images is supplied from the image information separation unit 34. Prepare.

  The imaging lens 31 forms a subject image on a single-plate color imaging sensor 33. At this time, the subject image from which the high-frequency component is cut by the optical low-pass filter 32 is formed on the single-plate color image sensor 233.

  In this multispectral image information acquisition apparatus 30, the color imaging sensor 33 is a single-plate image sensor in which a color coding filter combining RGB and MS is provided on the imaging surface, and has a number of pixels of Nx × Ny.

  From the image information of the color image obtained by the color image sensor 33, the image information separation unit 34 separates the image information of the RGB mosaic image and the image information of the MS mosaic image.

  The image information of the RGB mosaic image separated in the image information separation unit 34 is converted by the demosaicing unit 35 into three images of R, G, and B each having Nx × Ny pixels, that is, an R image, a G image, It is converted into image information of a B image. The image information of the R image, the G image, and the B image converted by the demosaicing unit 35 is supplied to the multispectral image calculation unit 36 together with the image information of the MS mosaic image separated by the image information separation unit 34. And the multispectral image calculation part 36 calculates and outputs multispectral image information based on each image information of R image, G image, B image, and MS mosaic image.

  The multispectral image information acquisition device 30 can have a function of outputting RGB images, like the multispectral image information acquisition device 30A shown in FIG.

  Here, the same components as those of the multispectral image information acquisition apparatus 30 are denoted by the same reference numerals, and detailed description thereof will be omitted, and different components will be described.

  That is, the multispectral image information acquisition apparatus 30A includes an RGB image recording unit 37 that stores R image, G image, and B image information obtained by the demosaicing unit 35, and an MS mosaic obtained by the image information separation unit 34. An MS mosaic image recording unit 38 that stores image information of an image is provided, and the RGB image recorded in the RGB image recording unit 37 is output as it is. If necessary, the image information of the RGB image recorded in the RGB image recording unit 37 and the image information of the MS mosaic image recorded in the MS mosaic image recording unit 38 are sent to the multispectral image calculation unit 36, The multispectral image calculation unit 36 calculates and outputs image information of the multispectral image. In the case of this configuration, it is possible to acquire information on a multispectral image as necessary while maintaining compatibility with an existing color system based on RGB. Furthermore, the present invention includes a case where the RGB image recording unit 37, the MS mosaic image recording unit 38, and the multispectral image calculation unit 36 are implemented as separate devices.

  The spectral sensitivity characteristics of the pixels corresponding to RGB in the single-plate color image sensor 33 provided in the multispectral image information acquisition apparatus 30 are the same as those described in the first embodiment. Further, K types of spectral sensitivity characteristics of pixels corresponding to MS are the same as those described in the first embodiment.

  The optical low-pass filter 32 is intended to prevent aliasing distortion in the RGB image output from the demosaicing unit 35, and its characteristics are the same as those described in the second embodiment.

  An arbitrary technique can be used for the demosaicing process of the demosaicing unit 35.

  The operation of the multispectral image calculation unit 36 is the same as that described in the first embodiment.

  Here, the arrangement of the color filters on the single-plate color image sensor 33 provided in the multispectral image information acquisition apparatus 30 will be described.

  If 2 × 2 4 pixels are defined as one pixel group, R, G, and B color filters and one type of color filter among K types of color filters are arranged in this pixel group. Therefore, one pixel is provided for four pixels where R, G, and B color filters are arranged, whereas one pixel is provided for each of 4 × L pixels. Will exist.

  For example, as shown in FIG. 9, the minimum unit in the pixel arrangement example when K = 16, as shown in FIG. 9, when K = 16, 16 = 4 × 4 pixels are required in order to arrange K types of color filters one pixel at a time. Therefore, the minimum unit of the color filter on the single-plate color image sensor 33 is 8 × 8 pixels. The arrangement of the K types of narrow band color filters is arbitrary.

  That is, the third embodiment is a multispectral image information acquisition apparatus 30 in which the first number of colors = 3, the second number of colors = 1, and the third number of colors = 16. A first image sensor arranged on an image plane on which a subject image is formed by the imaging lens 31, wherein a color filter is arranged for each pixel, and the first color corresponding to the broadband spectral sensitivity characteristic. And an image sensor 33 having a first number of pixels color-coded by the number of color filters and the third color filter corresponding to the narrow-band spectral sensitivity characteristic.

  Then, the image information obtained by the image sensor 33 is separated by the image information separation unit 34 into pixel information corresponding to the broadband spectral sensitivity characteristic and pixel information corresponding to the narrow band spectral sensitivity characteristic. An image having the first number of colors and the fourth number of pixels is obtained by demosaicing processing in the demosaicing unit 35 from pixel information corresponding to the broadband spectral sensitivity characteristic separated by the image information separating unit 34. Image information, pixel information corresponding to the spectral sensitivity characteristics of the narrow band separated by the image information separation unit 34, the first color number demodulated by the demosaicing unit 35, and the first color number. On the basis of the image information of the image having the number of pixels of 4, the multispectral image calculation unit 36 performs multispectral image information on the subject image. As calculates the third number of colors, the image information of the image having the number fourth pixel.

<Embodiment in which output format is changed>
In any case of the first embodiment, the second embodiment, and the third embodiment, instead of outputting a multispectral image, other types of image data calculated from the multispectral image are used. It can be converted and output.

In the above embodiment, image sensors corresponding to RGB channels (R image sensor 13R, G image sensor 13G and B image sensor 13B in the first embodiment, RGB image sensor 24 in the second embodiment, In the third embodiment, assuming that the number of pixels of a single-plate color image sensor 33) is N _RGB and the number of types of MSFA color filters is K, the number of pixels N from the multispectral image calculation units 15, 27, 36 is N. _{An RGB} K-band image is output. On the other hand, from multispectral image calculation units 15, 27, and 36, (1) a multispectral image of a band larger than the K band, (2) a multispectral image of a band smaller than the K band, and (3) a spectral radiance Image information can be output after being converted into an image (4) spectral reflectance image (5) XYZ image (6) RGB image or the like. At this time, the number of bands or color channels in the converted image information is J, and the j-th channel image is q _j . However, for simplicity, a case will be described in which the number of pixels of the image information after conversion is equal to N _RGB . q _j is the original spectral image

(L is 1 to L), or K-band multispectral image

That can be calculated by linear calculation, that is, weighting factor

Or

Using the following equation (17)

When converting to image information as defined, it is possible to implement by simpler calculation. Substituting equation (6) into equation (17),

To convert the converted image information

It can be expressed as. That is, by using Expression (18), it is possible to directly calculate the image information after the conversions (1) to (6) from the RGB image without obtaining the K-band multispectral image.

  The present embodiment is applied to a case where a color when illuminated with light different from that at the time of photographing is calculated using a spectral spectrum of illumination light at the time of photographing and a spectral spectrum of illumination light for observation.

<Other embodiments>
Furthermore, in the first to third embodiments, the spectral sensitivity characteristics of the broadband image sensor have been described as spectral sensitivity characteristics corresponding to RGB. However, the spectral sensitivity characteristics are replaced with arbitrary broadband spectral sensitivity characteristics in the target wavelength band. It is also possible to do.

  As Example 1, simulation is performed with the following settings for the first embodiment (K = 16), and the results compared with other methods are shown in FIGS.

Setting In the first embodiment, K = 16 is set. In other words, from data acquired by three imaging sensors corresponding to R, G, and B and one imaging sensor of a multispectral filter array system in which filters of 16 colors are arranged (a total of four imaging sensors), 16 A 16-band image corresponding to the color filter is generated. FIG. 10 shows the spectral sensitivity characteristics of the RGB imaging sensor, and FIG. 11 shows the 16 types of spectral sensitivity characteristics of the MSFA. Further, FIG. 12 shows a pixel array of the MSFA. In FIG. 12, numerical values 1 to 16 represent the types of narrow band color filters, and are numbered sequentially from the short wavelength side. Area division at the time of multispectral image calculation is a rectangular area of 16 × 16 pixels, and is common to all band images. Further, the calculation of the weighting factor is performed independently for each band.

  A partial region was cut out from three types of spectral images in a database of spectral images of natural scenes (Hyperspectral Images of Natural Scenes by D. H. Foster, The University of Manchester) and used as a subject to be photographed. Table 1 shows the specifications of the cut-out spectrum image.

Comparison method
4-band: 4 pieces of spectral sensitivity characteristics of 4 channels that combine the spectral sensitivity characteristics of G in FIG. 10 into two parts, the long wavelength side and the short wavelength side, and the R and B spectral sensitivity characteristics. From the data acquired by the imaging sensor, a 16-dimensional spectrum is estimated by Wiener estimation for each pixel, and a spectrum image is restored.

  4-band FA x4: Assuming that four types of filters are arranged for each pixel and imaging with four filter array type imaging sensors, four different types of color filters are arranged for each imaging sensor. As a result, data corresponding to 16 types of color filters can be acquired in total. The spectral sensitivity characteristics corresponding to the 16 types of color filters are the same as those in FIG. One color filter is arranged for every four pixels. From the data acquired in this way, an image of 16 wavelengths is restored by linear interpolation.

Results FIG. 13 to FIG. 15 show the normalized root mean squared error (RMMSE) of the root mean square normalized for each band image. “RGB + 16FA” is according to the present invention. It can be seen that the NRMSE of the present invention is minimized.

  Next, as Example 2, simulations are performed for the third embodiment (K = 16) with the following settings, and the results compared with other methods are shown in FIGS.

Setting In the first embodiment, K = 16 is set. The number of pixels of the image sensor is 512 × 512 pixels. FIG. 16 shows a pixel array of the single plate imaging sensor. The R, G, and B pixels are sampled for every 2 × 2 = 4 pixels, whereas the pixels corresponding to one type of color filter out of 16 types are 1 in 8 × 8 = 64 pixels. Only the pixels are sampled. The spectral sensitivities of the R, G, and B color filters and the 16 types of spectral sensitivities of the MSFA are shown in FIGS.

  It is assumed that an optical high-frequency cut that halves the frequency band is mounted.

  As an original, as shown in FIGS. 19A and 19B, spectral images of 512 × 512 pixels in the spatial direction and 81 dimensions in the wavelength direction (sampled from 380 to 780 [nm] at 5 nm intervals) were used ( However, this is an image estimated from a 16-band multispectral image).

  Area division at the time of multispectral image calculation was a rectangular area of 32 × 32 pixels. In addition, in the calculation of the weighting factor, both the case of calculating independently for each band and the case of calculating all the bands simultaneously are implemented. Furthermore, both the case where the direct current base in which all elements are 1 are added and the case where it is not added are implemented as the image base. As a method according to the present invention, the results of four patterns of these combinations were compared.

Demosaicing processing Data of R, G, and B pixels of high-density sampling are sampled every other pixel vertically and horizontally. Since it is assumed that the high frequency component is cut in advance so that the frequency band is approximately halved, it is considered that the aliasing distortion hardly occurs in the frequency domain. Therefore, an image (256 × 256 pixels) obtained by extracting each pixel of R, G, and B is projected onto the frequency domain by discrete Fourier transform, and after multiplying the coefficient corresponding to each phase difference by the frequency domain, After filling with zeros to form a 512 × 512 image, the image is returned to a 512 × 512 image by discrete inverse Fourier transform. As a result, an image in which a 512 × 512 high-frequency component is cut is restored from information on R, G, and B pixels sampled every other pixel, and can be regarded as a result of independent demosaicing with a band image. .

Comparison Method Assuming that color filters corresponding to 16 wavelengths are arranged uniformly on an imaging sensor of 512 × 512 pixels, an arrangement equivalent to the pixel arrangement shown in FIG. 12 was repeated. A pixel corresponding to one type of color filter is sampled by 1 × 4 × 4 = 16 pixels.

Results 256 × 256 pixels at the sampling positions of the MS pixels were taken out and evaluated by NRMSE. 20 and 21 show the NRMSE of the image. First, it can be seen that the result of the present invention reduces NRMSE to 1/3 or less when compared with the result of the comparison method (equal sampling, linear interpolation). Furthermore, when the four results according to the present invention were compared, the estimation error of the 15th and 16th bands was reduced by adding a DC base. Furthermore, the method of calculating the weighting coefficient for all the bands simultaneously has a slight error reduction compared to the method of calculating each band independently.

10, 10A, 20, 20A, 30, 30A Multispectral image information acquisition device, 11, 21, 31 photographing lens, 12, 23 branch optical system, 13R, 13G, 13B three primary color image pickup image sensor, 14, 25 MSFA Image sensor for imaging, 15, 27, 36 Multispectral image calculation unit, 16, 28, 27 RGB image recording unit, 17, 29, 38 MS mosaic image recording unit, 22, 23 Optical low-pass filter, 24 RGB image sensor , 26, 35 demosaicing unit, 33 color image sensor, 34 image information separation unit 34, 151 small area segmentation unit, 152 weight coefficient calculation unit, 153 multispectral image generation unit,

Claims (12)

For the subject image formed by the photographic lens, image information of a first image having a first color number of three or more colors and a first pixel number corresponding to a broadband spectral sensitivity characteristic, and the subject image On the other hand, it has a second number of pixels and has a second number of colors corresponding to different spectral sensitivity characteristics for each pixel, and the spectral sensitivity characteristics are narrow and overlap with the above-mentioned broadband spectral sensitivity characteristics. Imaging means for outputting image information of the second image having a third color number that is larger than the first color number when collecting all the pixels;
Based on the image information of the first image and the image information of the second image obtained by the imaging means, the image information of the third image having the third color number and the first pixel number is obtained. A multispectral image calculation means for generating,
A multispectral image information acquisition apparatus that generates image information of the third image as multispectral image information for the subject image.   The imaging means includes a branching optical system that duplicates an object image formed by the photographing lens into an image having the same number of colors as the total number of the first and second colors, and the branching optical system. The same number of image sensors as the first number of colors arranged on the number of image planes equal to the number of duplicated first colors, and each image sensor has a first number of pixels. The spectral sensitivity characteristic of the image sensor is the same as the number of the first image sensor, which is a broadband spectral sensitivity characteristic, and the number of the second colors arranged on the same number of image planes as the second number of colors. Each image sensor has a second number of pixels, each pixel of the image sensor has a different spectral sensitivity characteristic, and the spectral sensitivity characteristic is a narrow band and the broadband spectral sensitivity described above. Have overlap with properties, A second image sensor having a third color number larger than the first color number is provided for collecting all pixels included in the same number of image sensors as the second color number. The multispectral image information acquisition apparatus according to claim 1. The first image sensor is three image sensors for capturing a red image, a green image, and a blue image,
The second image sensor is a single image sensor having the second number of pixels in which a color filter is arranged for each pixel and color-coded by the third color filter. The multispectral image information acquisition apparatus according to claim 2. The imaging means includes a branching optical system that duplicates the subject image formed by the photographing lens into two images, and one image arranged on the image plane of one subject image replicated by the branching optical system. A sensor having a color filter for each pixel, the first image having a first number of pixels color-coded by the color filter having the first color number corresponding to the broadband spectral sensitivity characteristic; A first image sensor for acquiring image information and one image sensor arranged on the image plane of the other subject image duplicated by the branching optical system, wherein a color filter is arranged for each pixel. The image information of the second image having the second number of pixels color-coded by the color filter having the third color number corresponding to the spectral sensitivity characteristic of the narrow band is acquired. A second image sensor of,
Further, from the image information of the first pixel number corresponding to the broadband spectral sensitivity characteristic obtained by the first image sensor, an image having the first color number and the fourth pixel number is obtained by demosaicing processing. It has a demosaicing unit that generates image information,
The multispectral image calculation means includes the second pixel number of image information corresponding to the narrow-band spectral sensitivity characteristic obtained by the second image sensor, and the first demosaicing process performed by the demosaicing unit. The image information of the image having the third color number and the fourth pixel number is calculated based on the image information of the image having the number of colors and the fourth pixel number. Multispectral image information acquisition apparatus. The imaging means is a single image sensor arranged on an image plane on which a subject image is formed by the imaging lens, and a color filter is arranged for each pixel, and corresponds to the broadband spectral sensitivity characteristic. An image sensor having a first number of pixels color-coded by a color filter having a first color number and a color filter having a third color number corresponding to the spectral sensitivity characteristic of the narrow band;
Further, the image information separation unit that separates the image information obtained by the imaging unit into pixel information corresponding to the broadband spectral sensitivity characteristic and pixel information corresponding to the narrow band spectral sensitivity characteristic; and A demosaicing unit that generates image information of an image having the first number of colors and the fourth number of pixels by demosaicing processing from pixel information corresponding to the spectral sensitivity characteristics in a wide band separated by the image information separation unit. With
The multispectral image calculation means includes pixel information corresponding to the narrow-band spectral sensitivity characteristic separated by the image information separation unit, the first color number demodulated by the demosaicing unit, and a fourth color. 2. The multispectral image information acquisition according to claim 1, wherein image information of the image having the third color number and the fourth pixel number is calculated based on image information of the image having the number of pixels. apparatus.   In the imaging means, the pixel on the image sensor includes a first group of pixels corresponding to a broadband spectral sensitivity characteristic in a pixel group in which a number of adjacent pixels including the first color number and the second color number are collected. Each of the color filters having the number of colors is arranged, and the second color filter of the third color number corresponding to the narrow-band spectral sensitivity characteristic that overlaps the wide-band spectral sensitivity characteristic. A color filter of the number of colors is arranged, and in the group of adjacent pixel groups corresponding to the number obtained by dividing the third color number by the second color number, the narrow-band first filter is provided. 6. The multispectral image information acquisition apparatus according to claim 5, wherein the pixel groups are arranged so that all the color filters having the number of colors of 3 are included.   The multispectral image calculation means is configured such that, for each first small region image obtained by dividing the first image, for each of the first small region images, each color of the first small region image. A process of calculating a second small area image having the same number of pixels as the first small area image by multiplying the image by a weighting factor and adding them together, the first small area image The difference between the value of the third small area image cut out from the second image as the area corresponding to the second small area image and the value corresponding to the third small area image included in the second small area image is minimized. A process of calculating the second small area image having the same number of times as the third number of colors by repeating the same number of times as the third number of colors while changing the weighting factor. The method according to any one of claims 1 to 5, wherein: Multispectral image data acquisition apparatus mounting. With respect to the subject image, image information of a first image having a first color number of three or more colors and a first number of pixels corresponding to a broadband spectral sensitivity characteristic, and a second It has a number of pixels, a second number of colors corresponding to different spectral sensitivity characteristics for each pixel, and the spectral sensitivity characteristics are narrow and overlap with the broadband spectral sensitivity characteristics, collecting all pixels. An imaging step of acquiring image information of a second image having a third color number greater than the first color number,
Based on the image information of the first image and the image information of the second image obtained in the imaging step, image information of the third image having the third color number and the first pixel number is obtained. A multi-spectral image calculation step to generate,
A method for acquiring multispectral image information, comprising generating image information of the third image as multispectral image information for the subject image. In the imaging step,
The same number of subject images as the total number of colors of the first color number and the second color number are duplicated by the branching optical system,
The same number of image sensors as the first number of colors arranged on the number of image planes equal to the first number of colors on which the subject image is duplicated, and each image sensor is a first The image information of the first image is obtained by the first image sensor having a number of pixels and the spectral sensitivity characteristic of the image sensor is a broadband spectral sensitivity characteristic,
The same number of image sensors as the second number of colors arranged on the same number of image planes as the second number of colors on which the subject image is duplicated, and each image sensor has a second number Each pixel of the image sensor has a different spectral sensitivity characteristic, and the spectral sensitivity characteristic is narrow and overlaps with the broadband spectral sensitivity characteristic, and the same number as the second number of colors. The image information of the second image is acquired by the second image sensor having a third color number that is larger than the first color number when collecting all the pixels included in the image sensor. The multispectral image information acquisition method according to claim 8. In the imaging step, two subject images are duplicated by a branching optical system, and one image sensor is arranged on the image plane of one subject image duplicated by the branching optical system. A first image sensor having a first number of pixels color-coded by the color filter having the first color number corresponding to the broadband spectral sensitivity characteristic is provided with image information of the first image. One image sensor arranged on the image plane of the other subject image duplicated by the branching optical system and having a color filter for each pixel, corresponding to the narrow band spectral sensitivity characteristic Image information of the second image is acquired by the second image sensor having the second number of pixels color-coded by the color filter having the third number of colors.
Further, the first number of colors and the fourth number of pixels are obtained by demosaicing processing from the image information of the first number of pixels corresponding to the broadband spectral sensitivity characteristic obtained by the first image sensor in the imaging step. Having a demosaicing step to generate image information of an image having a number;
In the multispectral image calculation step, the image information of the third number of pixels corresponding to the narrow-band spectral sensitivity characteristic obtained by the second image sensor in the imaging step and the demosaicing process in the demosaicing step are performed. The image information of the image having the third color number and the fourth pixel number is calculated based on the image information of the image having the first color number and the fourth pixel number. The multispectral image information acquisition method according to claim 8. In the imaging means step, a first image sensor arranged on an image plane on which a subject image is formed, a color filter is arranged for each pixel, and the first image sensor corresponding to the broadband spectral sensitivity characteristic is provided. Image information of the first number of pixels is acquired by an image sensor having a first number of pixels color-coded by the color filter of the number of colors and the color filter of the third number of colors corresponding to the spectral sensitivity characteristics of the narrow band. And
Further, image information obtained by separating the image information obtained by the image sensor in the imaging means step into pixel information corresponding to the broadband spectral sensitivity characteristic and pixel information corresponding to the narrow band spectral sensitivity characteristic. A separation step;
Demosaicing for generating image information of an image having the first number of colors and the fourth number of pixels by demosaicing processing from information on pixels corresponding to the spectral sensitivity characteristics of the broadband separated in the image information separation step. Has steps,
In the multispectral image calculation step, pixel information corresponding to the spectral sensitivity characteristics of the narrow band separated in the image information separation step, the first number of colors and the fourth number demodulated in the demosaicing step. 9. The multispectral image information acquisition according to claim 8, wherein the image information of the image having the third color number and the fourth pixel number is calculated based on the image information of the image having the number of pixels. Method.   In the multispectral image calculation step, for each first small region image obtained by dividing the first image, for each of the first small region images, each color of the first small region image A process of calculating a second small area image having the same number of pixels as the first small area image by multiplying the image by a weighting factor and adding them together, the first small area image The difference between the value of the third small area image cut out from the second image as the area corresponding to the second small area image and the value corresponding to the third small area image included in the second small area image is minimized. A process of calculating the second small area image having the same number of times as the third number of colors by repeating the same number of times as the third number of colors while changing the weighting factor. Any one of claims 8 to 11, wherein Multispectral image information acquisition method according to item 1.