JP3368012B2 - Image data processing device - Google Patents

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to an image data processing device having a wide dynamic range.

[0002]

2. Description of the Related Art In recent years, solid-state image pickup devices such as CCD image pickup devices have been widely used as an image capturing means for television cameras, electronic still cameras and the like. However, the dynamic range of this solid-state image sensor is narrower than that of silver salt and the like, and the image quality is significantly deteriorated depending on the shooting conditions.

Therefore, as a method of expanding the dynamic range of a solid-state image pickup device, a plurality of images having different exposures in the same scene are photographed, and the plurality of image data are combined by some operation to expand the dynamic range. Has been proposed, and it is considered to output the image data thus obtained by an output device such as a monitor.

[0004]

However, when the image data whose dynamic range has been expanded as described above is output by an output device such as a monitor or a printer, the dynamic range width of the output device is larger than the dynamic range width of the composite image. Sometimes it is narrow. In such a case, it is necessary to perform an appropriate compression operation according to the dynamic range width of the composite image and the output device dynamic range width and output characteristics. However, if this compression operation is appropriately performed, the contrast of the image may decrease. However, the balance of brightness is lost, and the image becomes very unnatural compared to the standard image taken with proper exposure before performing the combining process. Further, there is a problem in that it is necessary to perform the compression operation many times to evaluate the image until a good image is obtained as a result of the compression.

The present invention has been made in view of the above problems, and an object of the present invention is to provide an image data processing apparatus which can easily obtain a good image having a wide dynamic range.

[0006]

Means for Solving the Problems] was to achieve the object
Therefore, the present invention does not include the following (1) image data processing device.
(9) Configure as in (9). (1) Dynamic recording of image data output to the output device
To match the dynamic range of the output device
Processing means and the dynamic range of the image data
Image data processing device having operation means for changing
Place (2) The image data is obtained by the dynamic range expanding means.
Image data with an expanded dynamic range
(1) The image data processing device described above. (3) Before including image pickup means for obtaining image data
The image data processing device according to (1). (4) Dynamic range expansion means
The image data of a plurality of sheets with different exposure amounts are combined.
(2) The image data processing device described above. (5) The dynamic range expansion means uses the first exposure
Image data imaged in an amount and the first exposure amount
Of the second image data captured with different second exposure amounts
From both image data, both image data both have a predetermined brightness level
Common area in the
Brightness level in the first image data after matching
Is a signal outside the predetermined range, the corresponding second image
1 by substituting the signal of the brightness level of the data
(2) above for composing a wide dynamic range image
Image data processing device. (6) The image pickup means includes a plurality of image pickup elements and an optical path of image pickup light.
And a spectroscopic means for dividing an image into each of the image pickup devices,
And an exposure control means for changing the exposure amount of each image sensor
The image data processing device according to (3) above. (7) Based on the information synthesized by the dynamic range expansion means
The dynamic range width of the image data is
According to the dynamic range width or output characteristics
(4) The image data processing device described above. (8) Based on the information synthesized by the dynamic range expansion means
The dynamic range width of the image data is
The fit the Namikkurenji width or output characteristics
(5) The image data processing device as described above. (9) The dynamic range of the image data is color space
The image data processing device according to (1) above, which is changed above.
Place

[0007]

[0008]

[0009]

[0010]

[0011]

[0012]

[0013]

[0014]

[0015]

[0016]

[0017]

1 is a block diagram showing the configuration of a first embodiment of the present invention. The image data processing apparatus of FIG. 1 is composed of a block of an image pickup means a and a block of an image synthesizing means b, and is configured so that the output device c can output image data having a wide dynamic range.

In FIG. 1, 1 is a lens on which image pickup light from a subject is incident, 2 is an optical low-pass filter, and 3 is CC.
An image sensor such as D, 4 is an image sensor control circuit for controlling the operation of the image sensor 3, 5 is an A / D converter for converting an image signal from the image sensor 3 into a digital signal, and 6 is a computer for image synthesis Thus, the dynamic range expanding means for expanding the dynamic range by the operating means d constitutes.
Reference numeral 7 is a memory (storing means) for storing synthesis information when the wide dynamic range image is synthesized, and 8 is a data processing circuit for matching the dynamic range width of the output image data with the dynamic range width or output characteristic of the output device c. It constitutes the control means. 20 is a color filter.

Next, the operation will be described. A subject image (not shown) by the image pickup means a is projected onto the image pickup element 3 through the lens 1, the optical low pass filter 2 and the color filter 20. Here, the image pickup device control circuit 4 is configured to sequentially obtain a plurality of image signals having different exposure amounts (aperture value, storage time, etc.) in the same scene by one image pickup operation, and further obtain these image signals. Stored in the memory 7. After that, the computer 6 reads out from the memory 7 and synthesizes one wide dynamic range image, and at the same time, the memory 7 stores the synthesis information such as the pixel value required at the time of synthesis. Further, when the user requests the compression of the dynamic range width of the composite image data in accordance with the characteristics of the output device c, the data processing circuit 8 performs the compression operation based on the composite information stored at the time of the composite. And outputs the data, and when there is no request from the user, adds the composite information to the composite image data and outputs it, or outputs it as it is.

Next, the synthesis algorithm in the computer 6 of FIG. 1 will be described with reference to the conceptual diagram of FIG. In FIG. 2, 9
Is an image data obtained by converting the image signal obtained from the image sensor 3 with an appropriate (standard) exposure amount into 8-bit data by the A / D converter 5 and represented by a bar graph (referred to as normal image data).
Similarly, 10, 10 'are image data obtained by increasing the exposure (called bright image data), 11, 11'
Is image data obtained by reducing the exposure (called dark image data), 12 is image data whose dynamic range is expanded by combining, and 13, 13 ', 14 and 14' are points representing values of arbitrary pixels. Is.

The synthesis algorithm will be described below with reference to FIG. Here, it is assumed that each image data has been converted into 8-bit data by the A / D converter 5 in FIG. 1, and thus the pixel value has a value of 0 to 255. First, consider synthesis of the normal image 9 and the bright image 10. When synthesizing images having different brightness levels, that is, different outputs of the image sensor 3 with respect to changes in the light amount of the subject, it is necessary to align the brightness levels of the image data of each other. For this reason, if the reference point 13 is a pixel that is not crushed in both the normal image 9 and the bright image 10, the bright image 1
Since 0 has a larger exposure amount than the normal image 9, the value at this point is larger than that of the normal image 9. Therefore, 1 in FIG.
There is a positional relationship such as 3, 13 '. Therefore, in order to make the brightness levels of both image data the same, this reference point 13,
Make the value of 13 'the same. Then bright image 1
0 is slid as a whole, and a bright image 1 is displayed on the coordinates.
The position is 0 '.

Similarly, the normal image 9 and the dark image 11
In the synthesizing process, the pixels having no crushing are used as the reference point 14 in order to match the brightness levels of both image data.
Then, since the dark image 11 has a smaller exposure amount than the normal image 9, the value at this point is smaller than that of the normal image. Therefore, there is a relationship such as 14 and 14 'in the figure, and in order to make the brightness levels of both image data the same, this reference point 1
If the values of 4 and 14 'are made the same, the dark image 11 is slid as a whole, and the dark image 11' is at the position on the coordinates.

That is, a common area in which both image data brightness levels are within a predetermined range is extracted from both image data of standard exposure standard image data and non-standard image data of different exposure, and the brightness levels of this common area are matched. After that, one wide dynamic range image is synthesized by replacing the non-standard image corresponding to the area where the brightness level of the standard image is outside the predetermined range with the area where the brightness level is outside the predetermined range.

By the above operation, the brightness levels of the respective images become the same. After that, 0-B of normal image 9
Up to (reference point 13) are replaced with A to B (reference point 13) of the bright image 10 ', and C (reference point 1) of the normal image 9 is replaced.
4) to 255, C (reference point 1) of the dark image 11 '
4) Replace with D. At the same time, as a guideline for compressing the image data in accordance with the output device c, the pixel values (A, 0, B, C, 255, D) that became the reference at the time of composition.
Are stored in the memory 7. By combining the three pieces of image data by such an operation, the image data 12 having an expanded dynamic range can be obtained.

Next, the calculation method on the computer 6 for the operation of FIG. 2 will be described with reference to FIG. In FIG. 3, reference numeral 15 represents a pixel value in an arbitrary area of the normal image data, and 16
Represents the pixel value of the same area as the area 15 of the bright image data, and 17 represents the combined value of the combined areas 15 and 16.

First, in order to make the brightness levels of the normal image and the bright image to be the same, a pixel which is not crushed in both image data, the reference point 13 in FIG. 2, is set. here,
If the normal image is 100 as the value of the reference point 13,
The reference point 13 'of the bright image is 200. Then, in order to match the brightness levels, 100, which is 200-100, is subtracted from all the pixel values of the bright image, and the result is slid. Then, the blackened area of the normal image (in the example of FIG. 3, the area where the pixel value of the normal image 15 is 0) is supplemented with the value of the bright image to perform the synthesis. Although the synthesis of the normal image and the bright image has been described above, the same applies to the synthesis of the normal image and the dark image.

The image data that has been subjected to the dynamic range expansion processing as described above is output by the output device c. However, the dynamic range width of the output device c may be narrower than that of the composite image data. In that case, the dynamic range width of the composite image data must be compressed before output. FIG. 4 is a diagram showing an example of compression of image data having a wide dynamic range, which represents the image data in a perceptual color space. In FIG. 4, reference numeral 30 indicates composite image data whose dynamic range has been expanded by the main image composition processing, and reference numeral 31 indicates image data 30 compressed in an arbitrary range. Definition of each coordinate is shown in FIG. Indicates.
In addition, FIG. 4A shows the synthesized image data 30 linearly compressed, FIG. 4B shows the bright portion emphasized and compressed, and FIG. 4C shows the dark portion emphasized and compressed. d)
Indicates that the bright part is cut and compressed, and it is desirable that the dynamic range width can be freely changed and output in this way. Therefore, the operation of the data processing circuit 8 that compresses the image data whose dynamic range has been expanded according to the characteristics of the output device c will be described below with reference to FIG.

In FIG. 5, reference numeral 24 is image data in which the dynamic range is expanded using the present synthesis algorithm, 2
5 and 29 are image data obtained by compressing the image data 24 into 8-bit data, and 26 to 28 are asymptotes. Here, as an example, the dynamic range of the composite image data is 8 bits.
A description will be given of a case where the output device c has a value equal to or larger than the width and the dynamic range width of the output device c is 8 bits.

The image data 24 in FIG. 5A is the same as the composite image data 12 in FIG. 2, and the pixel value is 8 bi.
It has a width of t or more. In the image data 24, the bright image is replenished between A and B, and the dark image is replenished between C and D. Here, it is assumed that the dynamic range width of the output device c is 8 bit width, and in order to perform compression in accordance with this, the original standard image is between 0 and 255 on the graph of FIG. Worth, so AB
If the image data between C and D is compressed between 0 and B, and the image data between C and D is compressed between C and 255, the brightness level is almost the same as that of the image data of proper exposure, and the normal image 9 of FIG. An image with an expanded dynamic range can be obtained. Therefore, the value of the composition information (A, B, C, D, 0, 255) stored in the memory 7 at the time of composition is used. Hereinafter, an example of a calculation formula when the compression operation described in FIG. 5A is linearly performed on a computer will be shown.

(1) (A to B) → (0 to B) NewPix = (B-0) ÷ (BA) × (OldPix-A) + 0 (2) (B to C) → (B to C) As it is (3) (C to D) → (C to 255) NewPix = (255-C) ÷ (DC) × (OldPix-C) + C As above, compression was performed using the linear format. As shown in (b), a curve 29 having straight lines 26 to 28 as asymptotes
By creating the image and performing the compression in a non-linear manner, the output value with respect to the light intensity becomes smooth, so that a more natural image can be obtained. Further, when it is difficult to create a curved line, compression may be performed by combining short straight lines to approximate the curved line 29. Further, it is desirable that these compression operations can be performed by a device other than the main imaging device. In such a case, as shown in FIG.
By adding the compression information (composite information) of the memory 7,
The compression operation as described above can be performed by a computer or the like.

In this way, by combining the brightness levels of the respective image data and then combining them, a natural composite image with a wide dynamic range in which the brightness balance is obtained without pseudo contours, and complicated calculation is required. I will not
The composition time is short and can be applied to movies.

Further, by storing the values of A, 0, B (reference point), C (reference point), 255, D, etc., which were used as a guideline for expanding the dynamic range, the coordinates thereof can be stored. As a guide, it is possible to easily perform compression in accordance with the dynamic range width of the output device c, and to output from one wide dynamic range composite image with the dynamic range width freely changed by the user's intention.

The number of image data to be combined is not limited to three, and any number of image data may be used as long as it is two or more.

FIG. 7 is a block diagram showing the configuration of the second embodiment of the present invention, and the same symbols as those in FIG. 1 indicate the same components. In FIG. 7, 18 is a first image sensor, 3 is a second image sensor, 19 is a third image sensor, 20 is a color filter, 21 is a color difference signal generation circuit for dividing a color signal into a luminance signal and a color difference signal, Reference numeral 8 is a signal processing circuit for generating an RGB signal from the color difference signal and the luminance signal, and 23 is a prism (spectral means) for dividing the image from the lens 1 into a plurality of image pickup devices 3, 18, and 19.

The operation will be described with reference to FIG. 7. A subject image (not shown) passes through the lens 1 and the prism 23, and further passes through the optical low pass filter 2 and the color filter 20 and is projected onto the image pickup devices 18, 3, and 19. Here, the first image pickup device 1 is operated by the image pickup device control circuit 4 in one shooting operation.
8 is a color signal with an appropriate exposure amount, the second image pickup device 3 is a luminance signal with increased exposure (hereinafter referred to as bright luminance signal), and the third image pickup device 19 is a luminance signal with reduced exposure (hereinafter, dark signal). (Called the luminance signal). As a method for this, ND filters having different transmissivities are provided on the front surface of each image sensor or the accumulation time of each image sensor is made different.
Each image signal thus obtained is converted into a digital signal by the A / D converter 5.

Next, the block of the image data synthesizing means will be described. The synthesizing algorithm is the same as that of the first embodiment. That is, the brightness levels of a plurality of pieces of image data having different exposure amounts are combined according to the brightness levels of the image data taken with an appropriate exposure amount.

The color signal from the image pickup device 18 enters a color difference signal generation circuit 21, where a luminance signal Yn (hereinafter referred to as a normal luminance signal) and color difference signals R-Yn and B-Yn are generated. The difference between the luminance levels of the normal luminance signal Yn and the bright luminance signal Yb from the image sensor 3 (pixel value difference between the reference point 13 of the normal image 9 and the reference point 13 'of the bright image 10 in FIG. 2) is calculated by the computer 6 Is calculated and stored in the memory 7. Similarly, the difference in brightness level between the normal brightness signal Yn and the dark brightness signal Yd from the image sensor 19 is also stored.

By temporarily storing the difference in the brightness level as described above, it is possible to easily subtract the difference from the bright brightness signal from the next time and to add the difference to the dark brightness signal to make the normal operation simple. It is possible to match the brightness level of the brightness signal (bright image 10 'and dark image 11' in FIG. 2). Then, the signal in which the brightness levels are matched is combined by the computer 6 as the data 12 of FIG. 2, and the combined image is displayed in the bit width of the normal image (8 bits in this example) by the data processing circuit 8. The color difference signal generation circuit 2 performs compression so as not to make a difference.
When combined with the color difference signal from 1, it is possible to synthesize an image with an expanded dynamic range. that time,
The data processing circuit 8 operates in the same manner as in the first embodiment.

By performing the composition with the above-mentioned configuration, it is possible to prevent the pseudo contour from being created by the composition. In addition, if multiple image data with different exposures are simultaneously captured using multiple image sensors and the difference in the brightness level of each brightness signal is stored once, it is possible to combine them by a simple calculation from the next time. Therefore, it can be applied to moving subjects and movies.

In each of the above-described embodiments, the image data processing device has been described in which the image pickup block and the image data composition block are integrated, but even if both blocks are separated, it is in accordance with the gist of the present invention. Is obvious.

[0041]

As described above, according to the present invention, for example, when a plurality of image data having different exposure amounts are combined to obtain one image data, the brightness level of the standard image data taken with proper exposure is obtained. By combining the brightness levels of the non-standard image data taken with other exposure amounts, and then combining,
Since it is possible to eliminate the pseudo contour of the combined image data and prevent an unnatural image in which the brightness balance is lost, it is possible to combine a natural image with a wide dynamic range. Further, since the composition algorithm is simple and complicated calculation such as region discrimination in composition is not required unlike the conventional art, the composition time is fast and it can be applied to a moving image.

Further, by storing in the memory the synthesis information such as the value of the pixel used as the reference at the time of synthesis, one image data having an expanded dynamic range is converted into the dynamic range width and output characteristics of the output device. Can be easily compressed according to the user's will, and the width of the dynamic range can be freely changed according to the user's intention to output, so that various output devices output good images with a wider dynamic range than standard images. be able to.

[Brief description of drawings]

FIG. 1 is a block diagram showing a configuration of a first exemplary embodiment of the present invention.

FIG. 2 is a conceptual diagram of a synthesis algorithm.

FIG. 3 is an explanatory diagram showing a calculation method of a synthesis algorithm.

FIG. 4 is an explanatory diagram showing an example of compression of image data.

FIG. 5 is a diagram showing a compression example of dynamic range expansion data.

FIG. 6 is a diagram showing an image format.

FIG. 7 is a block diagram showing a configuration of a second exemplary embodiment of the present invention.

[Explanation of symbols]

3 image sensor 4 Image sensor control circuit (exposure control means) 6 calculator (dynamic range expansion means) 8 Data processing circuit (control means) 18 Image sensor 19 Image sensor 23 Prism (Spectroscopic means) a Imaging means c Output device d Operating means

Claims (9)

(57) [Claims] 1. A dyna of image data to be output to an output device.
Mick range width is the dynamic range width of the output device
An image data processing apparatus comprising: a processing unit for adjusting the dynamic range of the image data; and an operating unit for changing the dynamic range of the image data. Wherein the image data, the image data processing apparatus according to claim 1 Symbol mounting, characterized in that image data with enlarged dynamic range by the dynamic range expansion means. 3. A process according to claim 1 Symbol, characterized in that an imaging means for obtaining image data placing the image data processing apparatus. 4. The image data processing apparatus according to claim 2 , wherein the dynamic range expansion means synthesizes a plurality of pieces of image data having different exposure amounts in the same scene. 5. The dynamic range expansion means comprises a first
First image data and the first captured by the first exposure amount
From both image data of the second image data imaged with the second exposure amount different from the exposure amount, a common area in which both image data brightness levels are within a predetermined range is extracted, and the brightness levels of this common area are matched. After that, the signal of the area in which the brightness level in the first image data is outside the predetermined range is converted into the corresponding first signal.
By replacing the luminance level of the signal of the second image data, the image data processing apparatus according to claim 2, wherein the benzalkonium to synthesize one wide dynamic range image. 6. The image pickup means includes a plurality of image pickup elements, a spectroscopic means for dividing an image into each of the image pickup elements in an optical path of image pickup light, and an exposure control means for changing an exposure amount of each of the image pickup elements. The image data processing apparatus according to claim 3, further comprising: 7. The image data processing apparatus according to claim 4, wherein the dynamic range width of the image data is matched with the dynamic range width or output characteristic of the output means based on the information synthesized by the dynamic range expansion means. 8. The image data processing apparatus according to claim 5, wherein the dynamic range width of the image data is matched with the dynamic range width or output characteristic of the output means based on the information synthesized by the dynamic range expansion means. 9. A dynamic range of the image data
Is variable in the color space.
1. The image data processing device according to 1.