JP6815628B2 - Multispectral imager - Google Patents

Description

The present invention relates to a multispectral technique capable of measuring the reflection spectral characteristics of a subject in pixel units from an image acquired by imaging the subject.

In recent years, the high functionality and high performance of digital cameras using image sensors such as CCD and CMOS have been remarkable. In particular, due to rapid progress in semiconductor manufacturing technology, the pixel structure of an image sensor is becoming finer. As a result, the number of pixels of the image sensor is increased and the drive circuit is highly integrated, and the image quality of the image sensor is greatly improved. In addition, the image sensor has also been miniaturized.

Recently, the improvement of color image quality has been remarkable. In addition to the usual color imaging using the three primary color filters of red (R), green (G), and blue (B), the quality of the color image of the subject is improved by imaging using a large number of color filters. Is being done. For example, the multispectral camera introduced in Patent Document 1 is equipped with a large number of optical filters having different spectral characteristics in front of the CCD, and images are taken while mechanically exchanging the optical filters. As a result, detailed color information and spectrum information of the subject can be obtained from each image acquired by imaging.

Further, Patent Document 2 introduces a multispectral camera capable of reducing the number of optical filters used by optimally designing an optical filter.

Patent Document 3 is a mechanism for replacing an optical filter by using a one-dimensional imaging element, a beam splitter that splits light incident through a lens, and a tunable filter that selects and transmits light in an arbitrary wavelength band. Introduces a technology that can obtain images in various wavelength bands without using.

Further, Patent Document 4 introduces a technique for obtaining a high-quality color image by using a single-plate color image sensor including a standard RGB color filter and a color filter other than the standard RGB color filter.

Further, Patent Document 5 has first to Nth (N is an integer of 2 or more) optical filters in which the total number of peaks including the number of peaks of transmittance and the number of underpeaks are different, and their spectral characteristics. Introduces the technology to calculate the reflection spectral characteristics from the subject using.

Japanese Unexamined Patent Publication No. 2005-181038 Japanese Unexamined Patent Publication No. 2010-12280 Japanese Unexamined Patent Publication No. 2012-138652 Japanese Unexamined Patent Publication No. 2008-136251 International Publication No. 2016/031922

The prior art disclosed in Patent Document 1 and Patent Document 2 requires a mechanical mechanism for exchanging a plurality of optical filters. Therefore, there is a problem that the size of the image pickup apparatus becomes large and the mechanical part must be regularly maintained. In the prior art disclosed in Patent Document 3, since a one-dimensional image sensor is used, the one-dimensional image sensor is moved (scanned) in the direction perpendicular to the pixel arrangement direction in order to obtain a two-dimensional image. A mechanical mechanism is needed. Further, if the subject is not stationary, there is a problem that the imaging result becomes a two-dimensional image having blur. Further, in the prior art disclosed in Patent Document 4, the standard RGB color filter used and the other color filters each transmit only light in a specific (narrow) wavelength band. However, there is a problem that further improvement in sensitivity cannot be expected. The conventional technique disclosed in Patent Document 5 solves the above-mentioned problems, but since a plurality of optical filters (first to Nth optical filters) must be arranged on a plurality of pixels in the image sensor, imaging is performed. There is a problem that the manufacturing cost of the element becomes high.

The present invention is an improvement of the conventional technique disclosed in Patent Document 5, and can improve the reflection spectroscopic characteristics of a subject without arranging the first to Nth optical filters directly on a plurality of pixels of the image sensor. It provides a low-cost color imaging system that can be calculated, is compatible with color image sensors, and does not require an infrared cut filter.

In the multispectral imaging apparatus according to one aspect of the present disclosure, the imaging surface is a pixel unit including a plurality of light sensing cells and a color filter that transmits light of a specific color facing at least a part of the plurality of light sensing cells. The image pickup elements are arranged one-dimensionally or two-dimensionally, and each light sensing cell in each pixel outputs a photoelectric conversion signal according to the amount of light received, and the first to N (N). Is an optical filter of 2 or more), each having one or more transmittance peaks or underpeaks in a predetermined wavelength band, and the total number of the peaks and the number of the underpeaks combined. A light sensing cell in which the color filter in the imaging element is arranged by sequentially imaging through the first to Nth optical filters having different peak numbers, the imaging light source, and the first to Nth optical filters. A signal processing circuit for processing the first to Nth photoelectric conversion signals output from each of the above is provided. The first to Nth optical filters are arranged at any position on the path from the light from the image pickup light source to being reflected by the subject and incident on the image pickup device. The predetermined wavelength band includes the transmission wavelength band of the color filter. The signal processing circuit calculates the spectral reflection characteristics of the subject in the transmission wavelength band based on the first to Nth photoelectric conversion signals.

The above-mentioned comprehensive or specific embodiment may be realized by a system, a method, an integrated circuit, a computer program, a recording medium, or any combination thereof.

According to the multispectral image pickup apparatus according to one aspect of the present invention, since the first to Nth optical filters are not directly arranged on a plurality of pixels of the image pickup element, a low-cost image pickup system can be realized. Further, since the predetermined wavelength band of the first to Nth optical filters is set to include the transmission wavelength band of a specific color filter among the plurality of color filters, the light is transmitted through the specific color filter. From the converted light signal, the spectral characteristics of the captured image in the transmission wavelength band of a specific color filter can be calculated.

It is a block diagram of the multispectral image pickup apparatus in Embodiment 1 of this invention. It is a top view of the illumination light control liquid crystal panel in Embodiment 1 of this invention. FIG. 5 is a spectral characteristic diagram of eight types of optical filters 1a to 1h and an ND filter according to the first embodiment of the present invention. It is a spectroscopic characteristic diagram of the R pixel, G pixel, and B pixel of the color image sensor in Embodiment 1 of this invention. FIG. 5 is a reflection spectroscopic characteristic diagram of a subject having a reflection peak at about 740 nm (0.74 μm) in the first embodiment of the present invention. It is a spectroscopic characteristic diagram which shows the result of having imaged the subject by simulation, and calculated the reflection spectrum thereof. It is a reflection spectroscopic characteristic diagram of a subject having a reflection peak at about 620 nm (0.62 μm) and about 900 nm (0.9 μm). It is a spectroscopic characteristic diagram which imaged the subject by simulation and calculated the reflection spectrum. It is a spectral transmission characteristic diagram when two of each of eight kinds of optical filters in Embodiment 1 of this invention are stacked, and the spectral transmission characteristic diagram of the DC filter in that case. It is a spectroscopic characteristic diagram of the near infrared transmission filter in Embodiment 2 of this invention. It is a figure which shows the characteristic which summed and averaged the spectral transmission characteristics of the optical filters 1a to 1h shown in FIG. It is a figure which shows an example of the spectroscopic property of an optical filter.

(Embodiment 1)
The multispectral image sensor according to the first embodiment of the present invention includes a color image sensor including a red (R) color filter (having a high transmittance for light of about 600 nm or more) and N + 1 (N is 2). It has a lighting device including an optical filter (the above integer). The multispectral image pickup apparatus sequentially switches N + 1 optical filters to irradiate the subject with light passing through the individual optical filters as illumination light, and images the subject each time. Of the N + 1 optical filters, each of the N optical filters has a transmittance peak or under peak of 1 in a predetermined wavelength band (for example, 550 nm to 1060 nm, including the light transmission band of the red color filter). Have one or more. The total number of peaks, including the number of transmittance peaks and the number of underpeaks, in the N optical filters is different from each other. The remaining one optical filter is an averaging optical filter (sometimes referred to as a "DC filter") having a transmittance approximately intermediate between the peak and the under peak of the transmittance. In the present embodiment, an ND (Neutral Density) filter is used as an example of the averaging optical filter. In this embodiment, N = 8, and eight types of optical filters and one ND filter are used.

In the present specification, the term "peak" means that, in the above-mentioned predetermined wavelength band, the transmittance is within one continuous partial band whose transmittance exceeds the average value in the wavelength band (in which the transmittance is equal to or less than the average value). Does not mean) means that the transmittance is maximized. On the other hand, "under-peak" means that the transmittance is minimized within one continuous partial band in which the transmittance is lower than the above average value (in which the transmittance never exceeds the average value). To do. Further, in the present specification, a pixel may be referred to as a light sensing cell.

The principle of calculating the reflection spectral characteristics of the subject from the captured image is the same as the principle disclosed in Patent Document 5. That is, although the configuration of this embodiment is different from the configuration disclosed in Patent Document 5, the characteristics of the N optical filters used and the signal processing itself are the same. Specifically, in Patent Document 5, N pixels in which N optical filters having a predetermined wavelength band having a relatively wide band (for example, 380 nm to 760 nm) are arranged and N pixels in which an optical filter is not arranged 1 A plurality of pixel units are arranged two-dimensionally adjacent to each other with one pixel as one pixel unit. The photoelectric conversion signals are set to E (i) (i = 1 to N, specifically N = 8) and E (0), and the reflection spectroscopic characteristics of the subject are calculated by calculation using these signals. .. On the other hand, in the present embodiment, N (specifically, N = 8 as described above) optical filters whose predetermined wavelength band is red or higher (near infrared wavelength band) are provided. It will be used. Then, only the pixels in which the red (R) color filter is arranged (hereinafter referred to as R pixels) are targeted, and the pixel signal when the same R pixel is imaged by switching N optical filters is E ( i) Let (i = 1 to N), and let E (0) be the pixel signal when the image is taken through the ND filter (or with the ND filter removed). The reflection spectral characteristics of the subject can be calculated using the pixel signals E (0) and E (i). Hereinafter, the basic principle of the process for calculating the reflection spectral characteristics in the present embodiment will be described.

The N optical filters have, for example, a peak at the lower limit wavelength of a predetermined wavelength band and a peak or underpeak at the upper limit wavelength of the predetermined wavelength band. For example, of the N optical filters, the first optical filter may be an optical filter having two peaks and one underpeak in a predetermined wavelength band. The second optical filter can be an optical filter having a total number of peaks having two peaks and one underpeak in a predetermined wavelength band. The third optical filter can be an optical filter having two peaks and two underpeaks in a predetermined wavelength band and having a total number of peaks of four. Similarly, the total number of peaks of the i-th (i is an integer of 1 or more and N or less) is i + 1. As described above, the N optical filters are characterized in that the total number of peaks increases by one as the number of code numbers increases. The light transmission characteristics of each optical filter have variability with respect to wavelength.

The above N optical filters are arranged in front of the light source, and N + 1 imaging is performed while switching the optical filters. The N + 1 imaging includes imaging performed in a state where the ND filter is arranged or in a state where neither optical filter is arranged. The image pickup device can typically be a general color image pickup device in which red, green, and blue color filters are arranged so as to face a plurality of pixels. In the signal processing in the present embodiment, only the signals from the plurality of pixels in which the red color filter is arranged are used. Therefore, the image sensor may have a plurality of pixels in which a color filter that transmits light in at least a red or near infrared wavelength range is arranged. In other words, the image sensor does not have to have a plurality of pixels in which green and blue color filters are arranged.

When an image is taken in a state where none of the optical filters is arranged or when an ND filter is arranged, each R pixel generates a signal obtained by photoelectrically converting light in a predetermined wavelength band. When an image is taken with the first optical filter arranged, each R pixel is a light containing a particularly large amount of a specific (relatively narrow) wavelength band of one of all the light components of a predetermined wavelength band. Is photoelectrically converted to generate a signal. When an image is taken with the second optical filter arranged, each R pixel is a light containing a particularly large amount of light components in two specific (relatively narrow) wavelength bands among all the light components in a predetermined wavelength band. Is photoelectrically converted to generate a signal. Similarly, when an image is taken with the i-th optical filter arranged, each R pixel has i light components in a specific (relatively narrow) wavelength band among all the light components in a predetermined wavelength band. A signal obtained by photoelectrically converting light containing a particularly large amount of light is generated.

Considering a function in which the wavelength is the independent variable and the light reflectance of the subject is the dependent variable, the signal acquired in a predetermined wavelength band without an optical filter is the DC component of the light reflectance of the subject. It is considered to have. On the other hand, it is considered that the signal acquired with the first to Nth optical filters arranged has a part of the DC component and N AC components in the light reflectance distribution of the subject. Therefore, using these signals, it is possible to approximate the light reflectance of the subject in the form of a finite Fourier series. This approximation method is the approach of this embodiment.

但し、式１におけるａ（ｉ）は下記の式２で表され、ｉは０から予め設定された最大自然数Ｍまでの整数である。また、式２における積分範囲はＸ＝０〜Ｗである。

Assuming that a wavelength function representing the reflected light energy from one point of the subject exists in a predetermined wavelength band λ1 to λ2 (however, λ2-λ1 = W), the function is a shift wavelength X (0 ≦) from the wavelength λ1. It is represented by F (X) using X ≦ W). F (X) is a function defined only in the range of 0 ≦ X ≦ W, but in the following explanation, it is assumed that F (X) is an even function, and it is expanded by a Fourier series. Think. Then, F (X) can be approximated by the finite Fourier cosine series represented by Equation 1. The shift wavelength X may be simply referred to as the wavelength X below.

However, a (i) in Equation 1 is represented by Equation 2 below, and i is an integer from 0 to the preset maximum natural number M. The integration range in Equation 2 is X = 0 to W.

If a (i) represented by the equation 2 can be created from the pixel signal of the image sensor, the reflection spectral characteristics of the subject can be approximately calculated. However, for that purpose, each optical filter needs to have a spectral characteristic that changes in a chordal manner with respect to a change in wavelength. Such an optical filter can be said to be a fairly special optical filter. The present embodiment also relaxes the peculiarity of such an optical filter, and the spectral characteristics of the optical filter used do not necessarily have to change in a chordal manner with respect to a change in wavelength.

For example, suppose that one of the optical filters has the spectral characteristics shown by the solid line in FIG. 10, and the characteristics are represented by G (X). In G (X), the transmittance (hereinafter, also referred to as “transmittance”) when the maximum value of the light transmittance is 1 is Ka on average, and the transmittance fluctuates depending on the wavelength, and the fluctuating component thereof. It is assumed that (also referred to as an AC component) is represented by (PP / 2) × AC (X) using the difference PP between the maximum value and the minimum value. That is, it is expressed by G (X) = Ka + (PP / 2) AC (X). Further, in the state where the optical filter is not arranged, the amount of received light is slightly reduced due to light reflection on the surface of the image sensor, and the transmission ratio is Kd. For reference, in FIG. 10, the characteristic is represented by a broken line.

Assuming that the light from the subject is 100% photoelectrically converted, the signal Sd acquired in the state where the optical filter is not arranged is represented by the following equation 3, and the signal Sa of the pixel in which the optical filter is arranged is expressed by the equation 4. expressed.

Equation 5 is derived from Equations 3 and 4. It can be seen that the intensity of the light when the reflected light from the subject is attenuated by the AC component of the optical filter can be obtained by calculating the pixel signal.

If the AC (X) of any of the optical filters is a cosine function, the Fourier coefficient shown in Equation 2 can be obtained, and the reflection spectral characteristic F (X) of the subject can be calculated from Equation 1 using the result. However, when the spectral characteristics of each optical filter have variability as shown in FIG. 10 but have an AC component that is not a cosine function, Equations 2 and 1 cannot be used.

と近似できる。Ｚ'＝ｄＺ／ｄＸとすると、式５は式７で表される。

This embodiment solves this problem, and if certain conditions are satisfied, the reflection spectral characteristics of the subject can be calculated even if the fluctuation in the spectral characteristics of the optical filter is not represented by the cosine function. A certain condition is that in any optical filter, the alternating current component AC (X) of the spectral characteristics becomes close to the cosine function with respect to the change of the variable Z associated with the wavelength X. If this condition is met, the left side of Equation 5 will be

Can be approximated to. Assuming that Z'= dZ / dX, Equation 5 is represented by Equation 7.

Equation 7 means that the Fourier coefficient of (F (Z) / Z') can be obtained by an operation using the pixel signals Sa and Sd. As a result, the result of expanding (F (Z) / Z') to a finite Fourier series is obtained, and by multiplying the expanded result by Z'and further converting F (Z) to F (X). , The reflection spectral characteristics of the subject can be obtained.

The above is the basic principle of this embodiment. Hereinafter, a specific example of the present embodiment will be described based on the basic principle.

First, the configuration of the multispectral image pickup apparatus in this embodiment will be described. FIG. 1 is a configuration diagram of a multispectral imaging device according to the present embodiment. This multispectral image pickup device includes an imaging lens 1, a wideband optical filter 9, a color image sensor 3, a signal generation / reception unit 4, an image processing unit 6, an image memory 7, a signal output unit 8, and the like. It includes a light source 10 for illuminating a subject, an illumination light control liquid crystal panel 2, and a liquid crystal controller 5.

Although the imaging lens 1 is depicted as a single lens in FIG. 1, it may be a combination of a plurality of lenses. The broadband optical filter 9 is designed to transmit only light having a wavelength of 400 nm to 1060 nm in this embodiment. The color image sensor 3 is a Bayer-arranged CMOS image sensor using a red, green, and blue color filter in the present embodiment, but may be another type of color image sensor such as a CCD having a red color filter. The signal generation / reception unit 4 receives the image signal from the color image sensor 3 and sends a signal for driving the color image sensor 3 to the color image sensor 3. In addition, a signal for controlling the illumination light is sent to the liquid crystal controller 5.

The signal generation / reception unit 4 may be composed of an LSI such as a CMOS driver, for example. The image processing unit 6 transmits the image signal from the signal generation / reception unit 4 to the image memory 7, and reads and processes the image signal from the image memory 7. The image processing unit 6 can be realized by a combination of a signal processing circuit such as a known digital signal processing processor (DSP) and software that executes image processing in the present embodiment. Alternatively, the image processing unit 6 may be composed of dedicated hardware. The signal processing in the present embodiment is executed by the signal processing circuit in the image processing unit 6. The image memory 7 may be composed of a known semiconductor memory such as DRAM or SRAM. The signal output unit 8 outputs the signal from the image processing unit 6 to the outside. Specifically, the captured image, the calculated reflection spectral characteristic data, and the like are output.

The light source 10 is a light source for illuminating a subject, and is a halogen light source having a color temperature of 4000 K in the present embodiment. The light source 10 may be another type of light source, for example, a fluorescent lamp, an LED, or an LED array, as long as the spectral characteristics of the emission are known. The illumination light control liquid crystal panel 2 receives the light from the light source 10 and switches the optical characteristics as illumination to the subject, and is a composite in which eight optical filters and one ND filter are arranged adjacent to each other on the same plane. It includes an optical filter 2a and a liquid crystal panel 2b for adjusting the amount of transmitted light bonded to the optical filter 2a.

FIG. 2 is a plan view showing the configuration of the composite optical filter 2a in the illumination light control liquid crystal panel 2. Eight optical filters numbered 1a, 1b, 1c, 1d, 1e, 1f, 1g, and 1h in ascending order of the number of transmittance peaks and underpeaks and the ND filter are arranged on the same plane. There is. A liquid crystal panel 2b is bonded to the back surface of the composite optical filter 2a. However, the arrangement of each filter is not limited to the arrangement shown in the figure, and there is no problem even if the arrangement is different.

FIG. 3 is a diagram showing the spectral characteristics (wavelength dependence of transmittance) of the eight optical filters 1a to 1h and the ND filter. In the optical filters 1a to 1h, the number of peaks and underpeaks in the transmittance within a predetermined wavelength band increases in that order. In the figure, the vertical axis represents the relative transmission ratio, the horizontal axis represents the wavelength, and the unit of the horizontal axis is micrometer (μm). The "predetermined wavelength band" in this embodiment is 550 nm to 1060 nm. The eight optical filters 1a to 1h can be produced by, for example, using a titanium oxide film having a refractive index of 2.3 and depositing the eight optical filters on a transparent glass substrate with different thicknesses. In the present embodiment, in the wavelength band of 550 nm to 1060 nm, the eight optical filters 1a to 1h have a peak at 550 nm and a peak or underpeak at 1060 nm. The total number of the total transmittance peaks and underpeaks in the wavelength band 550 nm to 1060 nm is 2 for the optical filter 1a, 3 for the optical filter 1b, 4 for the optical filter 1c, and 5 for the optical filter 1d. The optical filter 1e has six, the optical filter 1f has seven, the optical filter 1g has eight, and the optical filter 1h has nine.

The transmittance of each optical filter repeatedly fluctuates with increasing wavelength, but their period is not constant. However, it can be a constant period for other variables associated with wavelength. In the present embodiment, if the wavelength is X, the transmittance of each optical filter is Y, and Z represented by the quadratic function of X represented by the following equations 8 and 9 is an independent variable, the variable component of the transmittance Y is It can be approximated by the cosine function.

In addition, in this specification, changing approximately like a periodic function is expressed as "periodically changing". Periodic changes in the present specification do not necessarily mean periodic changes in the strict sense.

FIG. 4 shows the spectral characteristics (wavelength dependence of the transmittance of the filter) of the R pixel, G pixel (pixel in which the green color filter is arranged), and B pixel (pixel in which the blue color filter is arranged) of the color image pickup element 3. It is a figure which shows sex). In FIG. 4, the vertical axis represents the relative transmission ratio, the horizontal axis represents the wavelength, and the unit of the horizontal axis is nanometer (nm). In this embodiment, the reflection spectroscopic characteristics of the subject are calculated using the photoelectric conversion signal of the R pixel. Therefore, the wavelength band (wavelength band of about 600 nm or more) at which the transmittance of the R pixel shown by the solid line in FIG. 4 is high is the detection target. One of the features of this embodiment is that spectrum analysis can be performed particularly in the near infrared wavelength band.

Next, the imaging operation of the multispectral imaging apparatus in this embodiment will be described. After turning on the light source 10, the signal generating / receiving unit 4 sends the optical filter 1a to 1h and the signal generating / receiving unit 4 to the liquid crystal controller 5 so as to increase the light transmittance of the nine liquid crystal regions behind the ND filter in order to transmit light. Send instructions (control signals). The liquid crystal controller 5 controls the liquid crystal panel 2b according to the instruction. For example, the liquid crystal region behind the optical filter 1a is in a state of transmitting light, and the other liquid crystal region is in a state of blocking light. Then, only the light transmitted through the optical filter 1a becomes the illumination light. Imaging is performed every time the optical filter that transmits light is switched, and one imaging is completed after nine imagings. In the following description, the photoelectric conversion signal (pixel signal) obtained from the R pixel when the liquid crystal region facing the ND filter is in a transmissive state is represented as E (0), and the first to eighth optical filters 1a to 1h The photoelectric conversion signals obtained from the R pixels when the liquid crystal region facing the above is in a transmissive state are represented by E (1) to E (8), respectively.

Next, the signal processing of the multispectral image pickup apparatus in this embodiment will be described. The image processing unit 6 in the present embodiment calculates the Fourier coefficients a (i) (i = 0 to 8) from the pixel signals E (i) (i = 0 to 8) and uses the Fourier series expansion to perform reflection spectroscopy of the subject. Calculate the characteristics. More specifically, as disclosed in Patent Document 5, the image processing unit 6 performs the following processing.

但し、ＡＣ（ｉ，Ｘ）の入力変数について、ｉは色フィルタ１ａ〜１ｈの順に付番した数値（１〜８）で、Ｘは上記の波長である。 In the following description, the symbols shown in FIG. 10 are used as they are with respect to the characteristics of the optical filters 1a to 1h and the ND filter. However, in the following description, the actual wavelength, not the shift wavelength, is represented as X. When the pixel signals E (0) and E (1) to E (8) are represented by using the wavelength X and the function F (X) indicating the reflection spectral characteristics of the subject in the wavelength band of 550 nm to 1060 nm, the following equation is expressed. 10. It is represented by the formula 11.

However, regarding the input variables of AC (i, X), i is a numerical value (1 to 8) numbered in the order of the color filters 1a to 1h, and X is the above wavelength.

From Equation 10 and Equation 11, the following Equation 12 is obtained.

AC (i, X) can be approximated by the cosine function. Therefore, the equation 12 can be expressed by the following equation 13.

Assuming that Z'= dZ / dX, Equation 13 is expressed by the following Equation 14.

Regarding the Fourier coefficient of F (Z) / Z', a (0) is (2 / W) ∫ (F (Z) / Z') dZ, which is (2 / W) ∫F (X) dX. It is the same. Therefore, a (0) is represented by the following equation 15 using the equation 10.

On the other hand, a (i) in i ≧ 1 is expressed by the following equation 16 using equation 14 from equation 2.

Using the Fourier coefficients obtained from Equations 15 and 16, F (Z) / Z'can be approximately expanded by the Fourier series shown in Equation 17 below.

In Equation 17, the value corresponding to the shift wavelength X in Equation 1 is (X-550). By multiplying this result by the value of Z', F (Z (X)), that is, F (X) can be obtained.

As described above, the Fourier coefficient can be calculated using the pixel signal of the image sensor, and the reflection spectral characteristics of the subject can be approximately calculated by multiplying the Fourier series expansion of Equation 1 and Z'using them.

In the case of the present embodiment, in the above formula, W is 510 nm, which is the wavelength width of a predetermined wavelength band (550 to 560 nm), but in formulas 15 and 16, the wavelength width is set to a fixed value π. Kd is the average transmission ratio of the ND filter, for example 0.797, Ka is the average transmission ratio of the entire optical filters 1a to 1h, for example 0.797, and PP is the average transmission ratio peak and underpeak of the entire optical filters 1a to 1h. The difference is, for example, 0.255. Further, X in the formula 17 is a wavelength in a predetermined wavelength band, and the range thereof is 550 to 1060 nm. Z in Equation 17 is a dependent wavelength value associated with wavelength X, and those relational expressions are represented by Equations 8 and 9. Z'is the result of differentiating Z with respect to X and is determined by the wavelength X. In Equation 17, the reflection spectral characteristic of the subject is represented by F (Z), but since Z is represented by the wavelength X, it may be represented by F (X).

As described above, one imaging is completed by nine imagings while changing the spectrum of the illumination light. Each image is photoelectrically converted by the color image sensor 3 into an electric signal, which is recorded in the image memory 7 via the signal generation / reception unit 4 and the image processing unit 6. The image processing unit 6 reads out nine image information (R pixel signal only) recorded in the image memory 7 one by one, determines signal values E (0) to E (8) at one R pixel position, and determines the signal values E (0) to E (8). The signal processing of the formulas 15 to 17 is performed, and the processing is further performed over the entire screen.

The results of the simulation for calculating the reflection spectral characteristics of the subject will be described below. However, as simulation conditions, the light source is a halogen lamp with a color temperature of 4000 K, the optical filter has the spectral characteristics shown in FIG. 3, and the R pixel of the color image sensor has the spectral characteristics (sensitivity) shown by the solid line in FIG. And. Further, as the adjustment of the pixel signal, the signals of E (1) to E (8) are multiplied by 1.025 and the subsequent processing is performed.

FIG. 5A shows the reflection spectroscopic characteristics of a subject having a reflectance peak at about 740 nm (0.74 μm). FIG. 5B shows the result of a simulation in which the subject is imaged and its reflection spectral characteristics are calculated. Further, FIG. 6A shows the reflection spectroscopic characteristics of a subject having reflectance peaks at about 620 nm (0.62 μm) and about 900 nm (0.9 μm). FIG. 6B shows the result of a simulation in which the subject is imaged and its reflection spectral characteristics are calculated. In all the results, the actual spectroscopic characteristics and the simulated spectroscopic characteristics do not completely match, but it can be seen that they are similar in the big picture.

As described above, the multispectral image pickup device of the present embodiment includes a color image pickup device including a red color filter and an illumination device including nine optical filters. One of the nine optical filters is an ND filter, and each of the remaining eight optical filters has a light transmission band including the light transmission band of the red color filter and is transmitted in a predetermined wavelength band. It has one or more rate peaks or underpeaks, and the total number of peaks, which is the sum of the number of transmittance peaks and the number of underpeaks, is different. The image pickup apparatus sequentially switches the nine optical filters, irradiates the subject with light passing through each optical filter as illumination light, and takes an image each time. Only the image formed by the R pixel signal can be extracted from the captured image group, and the reflection spectral characteristics of the subject in the red to near infrared wavelength region can be calculated using the above equations 8 to 10. This means that by devising lighting, it is possible to realize spectrum analysis in the near-infrared band that cannot be captured by human vision, even if a normal color camera is used instead of a camera that uses a special image sensor. ing. Its practical effect is great.

Although the number of optical filters is nine in the present embodiment, the number is not limited to this, and the accuracy of spectrum analysis is improved by increasing the number of optical filters. Further, although the ND filter is used in the present embodiment, the same effect can be obtained by using another type of optical filter (DC filter) having a transmittance intermediate between the peak and the underpeak in other optical filters. .. Further, in the eight optical filters of the composite optical filter, one transparent glass substrate on which a titanium oxide film is vapor-deposited is used, but there is no problem even if a plurality of optical filters of the same type are stacked. In such a configuration, the light transmittance is slightly lowered, but the difference between the peak and the underpeak of the transmittance is widened, so that the spectrum analysis accuracy is improved. For reference, FIG. 7 shows the spectral transmission characteristics when two of each of the eight optical filters 1a to 1h are stacked, and the spectral transmission characteristics of the DC filter in that case.

Further, in the present embodiment, the illumination light control liquid crystal panel 2 is arranged in front of the light source 10, but the present invention is not limited to this. For example, although miniaturization is required, there is no problem even if the illumination light control liquid crystal panel 2 is arranged in the pupil portion of the camera lens. That is, if N + 1 optical filters are arranged somewhere between the light source and the subject or between the subject and the image pickup unit of the image pickup apparatus and the image can be taken through each of these optical filters, the same result can be obtained by the above processing. Can be obtained. Further, in the present embodiment, a wideband optical filter 9 that transmits only light having a wavelength of 400 nm to 1060 nm is used, but since the sensitivity characteristics of a normal CCD or CMOS image sensor are extremely low in a wavelength band other than the relevant region, a wide band is used. There is no problem even if the optical filter 9 is not used. Further, although the spectral characteristics of N + 1 optical filters were designed by targeting the spectral sensitivity characteristics of the R pixels in the pixel unit of the color image sensor 3, the pixels having the yellow color filter also have red and near infrared wavelengths. In order to transmit the light, N + 1 optical filters may be designed with the pixel as a target. Furthermore, N + 1 optical filters may be designed by selecting a color filter so that not only spectrum analysis targeting the red and near infrared wavelength bands but also spectrum analysis targeting the blue and ultraviolet wavelength bands can be performed, for example. ..

(Embodiment 2)
Next, a second embodiment of the present invention will be described. In this embodiment, the device configuration and signal processing are the same as those in the first embodiment, but the spectral characteristic data of another optical filter (hereinafter, referred to as “special optical filter”) registered in advance in the image processing unit. , The spectral sensitivity data of the image pickup device, the spectral transmission characteristic data of the optical system of the entire imaging apparatus, and the like, and signal processing using the spectral characteristic data is added. Specifically, the special optical filter is a filter that transmits near-infrared light (hereinafter, also referred to as a near-infrared transmission filter).

An object of the present embodiment is to obtain a normal color image even if the infrared cut filter, which is indispensable for a normal color camera, is removed. For that purpose, it is necessary to investigate how much the near-infrared light component is removed as a signal by mounting an infrared cut filter in the color camera. In the present embodiment, in order to estimate the signal amount of the near-infrared light component, the spectral characteristic data of the near-infrared transmission filter is registered in advance as a special optical filter in the image processing unit. By calculating the spectral characteristics in the wavelength band equal to or higher than the red wavelength band from the captured image, the signal amount of the near-infrared light component can be calculated from the spectral characteristics of the near-infrared transmission filter and various spectral characteristics of the entire imaging device.

Hereinafter, the calculation of the above-mentioned near-infrared light component will be described. However, assuming that the same signal processing as in the first embodiment is performed to calculate the spectral characteristics (in the wavelength band equal to or higher than the red wavelength band) of the subject image, the subsequent processing will be described.

The spectral characteristics of the subject image are calculated, and the transmission ratio is expressed as O (X). Here, X is a wavelength, and the spectral characteristics of the subject image are calculated from the light transmission wavelength band of the R pixel. Therefore, it is considered that the value of O (X) is almost 0 in the wavelength band shorter than red.

In the present embodiment, the same Bayer array color image sensor as in the first embodiment is used, and when an image is taken using the ND filter among N + 1 optical filters, the accurate RGB signals of the subject image (hereinafter, respectively) are used. (Denoted as Ro, Go, Bo) is obtained. On the other hand, the red (R) signal can be calculated by the following equation 13 using the calculated spectral characteristics of the subject image and the spectral characteristics of all the imaging devices including the color image sensor. However, the spectral characteristics of all the image pickup devices except the color image sensor are S (X), the spectral sensitivity characteristics of the R pixels of the color image sensor are R (X), and the calculated R signal is referred to as Rc. Although the signal Ro obtained by imaging and the calculated signal Rc are different in numerical value, they are considered to be originally the same value. Therefore, the ratio Ro / Rc is calculated.

Next, in order to estimate the signal amount of the near-infrared light component in the red signal Ro, the signal amount when the near-infrared transmission filter is used is calculated by the following equation 19. However, the spectral characteristics of the near-infrared transmission filter are referred to as IR (X), and the calculated near-infrared light component signal is referred to as Rir. Here, an example of the spectral characteristics of the near-infrared transmission filter is shown in FIG. In the figure, the vertical axis is the relative transmission ratio, the horizontal axis is the wavelength, and the unit is nanometer (nm).

The signal of the near-infrared light component in the red signal Ro by multiplying the calculated near-infrared light component signal Rir and the ratio Ro / Rc, that is, by calculating Rir × (Ro / Rc). The quantity Ro_ir can be calculated. Once Ro_ir is obtained, a red signal from which near-infrared light is removed can be generated by subtracting Ro_ir from the red signal Ro.

Next, the calculation process of the near-infrared light component in the green (G) signal and the blue (B) signal will be described. In this embodiment, not only the R pixel of the color image sensor but also the G pixel and the B pixel have spectral sensitivity characteristics (specifically, shown in FIG. 4), and from these characteristics, the red wavelength band or higher The G signal and B signal in the wavelength band can be calculated. Further, the signal amount when the near infrared transmission filter is used can also be calculated by the following equations 20 and 21. In both equations, G (X) and B (X) represent the spectral sensitivity characteristics of the G pixel and the B pixel of the color image sensor, respectively, and the calculated near-infrared light component signals are Gir and Bir, respectively. Notated as.

Here, the near-infrared light component in the R pixel was calculated in the red wavelength band and a wavelength band longer than that, but if the wavelength band is the same, the calculated signal and the actual signal even if the spectral characteristics in that band change. The signals output to are considered to be the same. That is, their ratio is considered to be constant, and the ratio Ro / Rc can be applied to the calculation of the near-infrared light component in the G pixel and the B pixel.

Therefore, the signal amount Go_ir of the near-infrared light component in the green signal Go is calculated by the calculation of Gir × (Ro / Rc), and the signal amount Bo_ir of the near-infrared light component in the green signal Bo is Bir × (Ro). / Rc) is calculated. If Go_ir and Bo_ir are obtained, the green signal and the green signal from which the near infrared light is removed can be generated by subtracting Go_ir and Bo_ir from the green signal Go and the green signal Bo, respectively.

As described above, the multi-spectral imager of the present embodiment has an RGB image signal when imaged by using an ND filter and a calculated subject image (in the red wavelength band and a longer wavelength band). Near-infrared transmission using the spectral characteristics, the spectral characteristic data of the near-infrared transmission filter registered as a special optical filter in advance, the spectral sensitivity data of the imaging element, and the spectral transmission characteristic data of the optical system of the entire imaging device. The pixel signal when the filter is used is calculated. By subtracting the signal from the RGB signal obtained by imaging, a normal color image can be obtained without using an infrared cut filter.

In the present embodiment, the spectral characteristics of the near-infrared transmission filter are shown in FIG. 8, but this is only an example, and other spectral characteristics may be used as long as they transmit near-infrared light. .. In addition, the spectral characteristics of the near-infrared transmission filter are registered in advance in the image processing unit, but the spectral characteristics of the infrared cut filter and the transparent filter are also registered in advance, and the spectral characteristics of the near-infrared transmission filter are processed by their difference processing. May be calculated.

In the present embodiment, a color image sensor having all the pixels of R, G, and B is used, but it is not always necessary to use the color image sensor. For example, an image sensor having no G pixel can be used. Hereinafter, a method of acquiring information on the green wavelength of the subject using such an image sensor will be described.

The spectral characteristics obtained by summing and averaging the spectral transmission characteristics of the eight optical filters 1a to 1h in the above embodiment are equivalent to the spectral transmission characteristics of the specific bandpass filter. For example, when the spectral transmission characteristics of the optical filters 1a to 1h shown in FIG. 3 are summed and averaged, the characteristics shown in FIG. 9 are obtained. Therefore, the peak wavelength is approximately 0 by subtracting the pixel signal E (0) obtained by imaging through the ND filter (averaging filter) from the total mean value of the pixel signals E (1) to E (8). A signal similar to the image signal obtained through a .55 μm (550 nm) green (G) filter is obtained. That is, there is an advantage that a green signal can be created even if there is no G pixel in which the green filter is arranged.

If the spectral sensitivity of the image sensor is extremely low for blue light, a green signal can be created, which means that even without a red (R) filter, if the created green signal is subtracted from each pixel signal, each pixel can be created. Means that a signal corresponding to the signal of the R pixel can be produced. From this, it is possible to omit the red filter, that is, the color filter becomes unnecessary.

Further, when the design value of the above "predetermined wavelength band" is set to, for example, 470 nm to a little less than 900 nm with respect to the optical filters 1a to 1h, the blue (B) filter having a peak wavelength of about 470 nm from the total average value of each optical filter. It is possible to produce a signal (called a B signal) similar to the image signal obtained through the above. By subtracting this signal from each pixel signal, a signal having a component higher than the green wavelength is obtained. That is, the optical filters 1a to 1h can be used for spectral analysis from the green wavelength to a wavelength of less than 900 nm. In other words, a G signal and an R signal can be produced, and when combined with the above B signal, an RGB color signal can also be produced. Depending on the image sensor, the spectral sensitivity is considerably low at about 850 nm or more, and the spectral sensitivity of light below the blue wavelength is low, so that multispectral can acquire color information without using a wideband filter or an infrared cut filter. A camera can be realized.

As described above, the signal processing circuit captures the average value of the first to Nth photoelectric conversion signals obtained by imaging through the first to Nth optical filters through the averaging optical filter. The signal obtained by subtracting the (N + 1) th photoelectric conversion signal obtained by the above may be generated. As a result, even if there is no pixel corresponding to a specific color, the information of that color can be acquired.

As described above, the present disclosure includes the multispectral imaging apparatus described in the following items.

[Item 1]
Imaging in which pixel units including a plurality of light sensing cells and a color filter that transmits light of a specific color facing at least a part of the plurality of light sensing cells are arranged one-dimensionally or two-dimensionally on an imaging surface. An image sensor, which is an element, and each light sensing cell in each pixel outputs a photoelectric conversion signal according to the amount of received light.
First to Nth (N is an integer of 2 or more) optical filters, each of which has one or more transmittance peaks or underpeaks in a predetermined wavelength band, and the number of the peaks and the underpeaks. The first to Nth optical filters, which differ in the total number of peaks including the number of
Light source for imaging and
Signal processing that processes the first to Nth photoelectric conversion signals output from the light sensing cell in which the color filter is arranged in the image sensor by sequentially imaging through the first to Nth optical filters. Circuit and
With
The first to Nth optical filters are arranged at any position on the path from the light from the image pickup light source to being reflected by the subject and incident on the image pickup device.
The predetermined wavelength band includes the transmission wavelength band of the color filter.
The signal processing circuit calculates the spectral reflection characteristics of the subject in the transmission wavelength band based on the first to Nth photoelectric conversion signals.
Multispectral imaging device.

[Item 2]
The transmittance Y of each of the first to Nth optical filters changes periodically with respect to the change of the variable Z associated with the wavelength λ in the predetermined wavelength band.
The signal processing circuit uses at least the first to Nth photoelectric conversion signals, N kinds of periodic functions having an independent variable as the variable Z, and a change value of the variable Z with respect to a minute change of the wavelength λ. To calculate the spectral characteristics of the captured image in the spectral characteristic band.
Item 1 multispectral imaging device.

[Item 3]
An averaging optical filter having a transmittance intermediate between the peak and the underpeak in the first to Nth optical filters is provided.
The spectral reflection characteristics of the subject in the transmission wavelength band are further used by using the (N + 1) th photoelectric conversion signal obtained from the light sensing cell in which the color filter is arranged by imaging through the averaging optical filter. The multispectral imaging apparatus according to item 1 or 2, which is calculated.

[Item 4]
The signal processing circuit further uses the (N + 1) th photoelectric conversion signal obtained from the light sensing cell in which the color filter is arranged by imaging without passing through the first to Nth optical filters. The multispectral imaging device according to item 1 or 2, which calculates the spectral reflection characteristics of a subject in the transmission wavelength band.

[Item 5]
The multispectral imaging device according to item 3, wherein the signal processing circuit generates a signal obtained by subtracting the (N + 1) th (N + 1) photoelectric conversion signal from the average value of the first to Nth photoelectric conversion signals.

[Item 6]
The signal processing circuit has data showing the spectral sensitivity characteristics of the entire multispectral image pickup device including the image pickup device and data showing the spectral transmission characteristics of another optical filter.
Using the calculated signal indicating the spectral reflection characteristics of the subject in the transmitted wavelength band, the signal amount of the light transmitted through the other optical filter and photoelectrically converted is calculated, and the photoelectric conversion signal of the pixel unit is used as described above. Subtract the signal amount to generate a new image signal,
The multispectral imaging apparatus according to any one of items 1 to 5.

[Item 7]
The multispectral image pickup apparatus according to any one of items 1 to 6, wherein the color filter transmits light in the red wavelength band.

[Item 8]
The multispectral imaging device according to item 7, wherein the other optical filter has a property of transmitting near-infrared light.

The multispectral image pickup device of the present invention is effective mainly for all solid-state cameras using a solid-state image sensor. For example, it can be used for consumer cameras such as digital still cameras and digital video cameras, and industrial surveillance cameras.

1 Imaging lens 2 Illumination light control liquid crystal panel 2a Composite optical filter 2b Liquid crystal panel 3 Color image sensor 4 Signal generation / reception unit 5 Liquid crystal controller 6 Image processing unit 7 Image memory 8 Signal output unit 9 Broadband optical filter 10 Light source 1a, 1b 1c, 1d, 1e, 1f, 1g, 1h, ND, DC optical filter

Claims (8)

Imaging in which pixel units including a plurality of light sensing cells and a color filter that transmits light of a specific color facing at least a part of the plurality of light sensing cells are arranged one-dimensionally or two-dimensionally on an imaging surface. An image sensor, which is an element, and each light sensing cell in each pixel outputs a photoelectric conversion signal according to the amount of received light.
First to Nth (N is an integer of 2 or more) optical filters, each of which has one or more transmittance peaks or underpeaks in a predetermined wavelength band, and the number of the peaks and the underpeaks. The first to Nth optical filters, which differ in the total number of peaks including the number of
Light source for imaging and
Signal processing that processes the first to Nth photoelectric conversion signals output from the light sensing cell in which the color filter is arranged in the image sensor by sequentially imaging through the first to Nth optical filters. Circuit and
With
The first to Nth optical filters are arranged at any position on the path from the light from the image pickup light source to being reflected by the subject and incident on the image pickup device.
The predetermined wavelength band includes the transmission wavelength band of the color filter.
The transmittance Y of each of the first to Nth optical filters changes periodically with respect to the change of the variable Z associated with the wavelength λ in the predetermined wavelength band.
The signal processing circuit uses at least the first to Nth photoelectric conversion signals, N kinds of periodic functions having an independent variable as the variable Z, and a change value of the variable Z with respect to a minute change of the wavelength λ. Te, and calculates the spectral reflectance characteristic of the subject in the transmission wavelength band,
Multispectral imaging device. Each pixel unit of the image pickup device further includes a color filter that transmits light of a color different from the specific color.
The multispectral imaging apparatus according to claim 1. An averaging optical filter having a transmittance intermediate between the peak and the underpeak in the first to Nth optical filters is provided.
The spectral reflection characteristics of the subject in the transmission wavelength band are further used by using the (N + 1) th photoelectric conversion signal obtained from the light sensing cell in which the color filter is arranged by imaging through the averaging optical filter. The multispectral imaging apparatus according to claim 1 or 2, which is calculated. The signal processing circuit further uses the (N + 1) th photoelectric conversion signal obtained from the light sensing cell in which the color filter is arranged by imaging without passing through the first to Nth optical filters. The multispectral imaging device according to claim 1 or 2, which calculates the spectral reflection characteristics of a subject in the transmission wavelength band. The multispectral imaging apparatus according to claim 3, wherein the signal processing circuit generates a signal obtained by subtracting the (N + 1) th (N + 1) photoelectric conversion signal from the average value of the first to Nth photoelectric conversion signals. .. The signal processing circuit has data showing the spectral sensitivity characteristics of the entire multispectral image pickup device including the image pickup device and data showing the spectral transmission characteristics of another optical filter.
Using the calculated signal indicating the spectral reflection characteristics of the subject in the transmitted wavelength band, the signal amount of the light transmitted through the other optical filter and photoelectrically converted is calculated, and the photoelectric conversion signal of the pixel unit is used as described above. Subtract the signal amount to generate a new image signal,
The multispectral imaging apparatus according to any one of claims 1 to 5. The multispectral image pickup apparatus according to any one of claims 1 to 6, wherein the color filter transmits light in the red wavelength band. The multispectral imaging device according to claim 6 , wherein the other optical filter has a property of transmitting near-infrared light.

Images