JP6132610B2 - Image processing apparatus and image processing method - Google Patents

Description

The present invention relates to an image processing apparatus and an image processing method.

  Multi-layer processing may be performed to accurately remove noise components of image data read from the image sensor. Similarly, multi-layer processing may be performed in order to efficiently perform feature point tracking from image data.

  In general, in multi-hierarchy processing, image data read from an image sensor is subjected to low-pass filter processing and down-sampling processing in a plurality of hierarchical processing steps to separate an input image into a plurality of frequency components. ing. In each separated hierarchy, various filter processes such as noise removal are performed. Thereafter, the lower-level image is enlarged and the images of the respective layers are combined to obtain final output image data.

  In the multi-layer processing, multi-tap processing is performed in filter processing such as a low-pass filter in each layer. When multi-tap processing is performed at the image edge of each layer, it is necessary to secure extra pixels in the invalid pixel region for each layer. Since the low-pass filter and downsampling process are performed even in the invalid pixel area, when securing extra pixels in the lower layer, approximately twice as many pixels are required in the upper layer, and many extra pixels are required in the upper layer. turn into. For example, when the low-pass filter of each layer is set to three horizontal taps in the five-layer process, the number of extra pixels required in each layer will be described with reference to FIG.

  FIG. 7 is a diagram illustrating a conventional method for processing the image edge of each layer. In FIG. 7, the right side is an effective pixel region and the left side is an invalid pixel region. Focusing on the invalid pixel area, one pixel is required in the fourth layer. Furthermore, 3 pixels are required in the third layer, 7 pixels in the second layer, and 15 pixels in the first layer.

  In order to improve filter accuracy and image quality, it is better that the number of filter taps is larger. However, if the number of filter taps is increased, the number of extra pixels required in the input image data tends to increase.

  When the number of extra pixels in the invalid pixel area increases, there is a problem that the number of pixels that can be secured in the effective pixel area with respect to the entire input image decreases, and the angle of view of the output image becomes narrow. Further, since the low-pass filter and down-sampling processing are performed for the invalid pixel area of each layer in the same manner as the effective pixel area, it takes extra processing time compared to processing only the effective pixel area. there were.

  With respect to the above problem, Patent Document 1 secures an extra pixel in each layer by copying an extra pixel in an invalid pixel region from a neighboring effective pixel region in each layer during multi-layer processing.

  In Patent Document 2, when the feature point to be tracked is in the invalid pixel region at the end of the image, the upper layer image is affine transformed, the feature point is extracted by moving in the center direction of the image, and then the inverse affine transformation is performed. ing. With the above processing, feature points can be extracted in the invalid pixel region even when the field of view rate is reduced in multi-layer processing.

JP 05-304621 A Japanese Patent Application No. 2010-79375

  However, in the conventional technique disclosed in Patent Document 1 described above, the pixels in the invalid pixel area are copied from the pixels in the effective area in the same hierarchy, so that the center of gravity of the pixel at the image edge is shifted from the center in the lower hierarchy. There was a problem that would decrease.

  Moreover, in the prior art disclosed in Patent Document 2 described above, an affine conversion circuit is required, and thus the circuit scale increases. Further, since the image quality deteriorates due to the conversion process, it is difficult to adapt to the development process such as noise removal.

  Accordingly, the present invention provides an image processing technique that minimizes the number of pixels in the invalid pixel area with a simple circuit and reduces the center-of-gravity shift of pixels at the image edge in each layer in multi-layer processing.

In order to achieve the above object, the present invention provides:
An image processing apparatus that repeatedly performs downsampling processing on original image data to generate image data of a plurality of layers having different numbers of pixels,
Extra pixel generation means for performing a process of generating extra pixels in the invalid pixel area outside the effective pixel area in the input image data;
Sampling processing means for performing filter processing and downsampling processing according to a predetermined number of taps for the image data including the extra pixels,
With
The extra pixel generation means includes
In the process of generating the image data of the second hierarchy, the extra pixels used in the process of generating the image data of the first hierarchy that is higher than the second hierarchy are duplicated to generate the extra pixels. It is characterized by that.

  According to the present invention, it is possible to provide an image processing technique in which the number of invalid pixel areas is minimized with a simple circuit in multi-layer processing, and the center-of-gravity shift of pixels at the image edge in each layer is reduced.

Functional block diagram showing a configuration example of an image processing apparatus corresponding to an embodiment of the invention The block diagram which shows the structural example of the multi-hierarchy image generation part 104 FIG. 6 is a diagram illustrating a method for processing an image edge of each layer according to the first embodiment. The figure explaining the conventional method of processing the image edge of each layer by copying the effective pixel area FIG. 6 is a diagram illustrating a method for processing an image edge of each layer according to the second embodiment. FIG. 10 is a diagram illustrating a method for processing an image edge of each layer according to the third embodiment. The figure explaining the processing method of the image edge of the conventional each hierarchy

  Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings.

[Embodiment 1]
Hereinafter, Embodiment 1 of the present invention will be described with reference to FIGS. FIG. 1 is a functional block diagram of Embodiment 1 of the image processing apparatus of the present invention. The image processing apparatus can be realized as an imaging apparatus such as a camera, for example. In addition to a camera, a video camera, a mobile phone, a smartphone, a personal computer, or the like may be used as long as the device has a shooting function. Further, even when there is no shooting function, it can be realized as a device for processing RAW data stored in a memory.

  In FIG. 1, the imaging optical unit 101 includes a lens, a diaphragm, and the like, and performs focus adjustment and exposure adjustment. The image sensor 102 is a CCD, CMOS, or the like that converts an optical image into an electrical signal. The A / D conversion circuit 103 converts an analog image signal from the image sensor 102 into digital image data. The multi-hierarchy image generation unit 104 generates digital image data of a plurality of different bands from the original image data output from the A / D conversion circuit 103, and outputs a multi-hierarchy image. The signal processing unit 105 generates video data by performing noise removal, gamma processing, interpolation processing, matrix conversion, composition processing, and the like on the multi-layer image input from the multi-layer image generation unit 104. A memory interface (I / F) 106 writes / reads video data and various control data to / from a memory (DRAM) 108. The CPU 107 functions as a control unit that performs various controls. The display unit 109 is composed of a liquid crystal monitor or the like, and displays the video data created by the signal processing unit 105 at any time so that the user can observe the state of the subject in real time.

  In the image processing apparatus of FIG. 1, the predetermined block may be configured by hardware using a dedicated logic circuit or a memory. Alternatively, the processing program stored in the memory may be configured by software by a computer such as a CPU executing the processing program.

  In the above configuration, the light incident on the imaging optical unit 101 is imaged on the light receiving surface of the image sensor 102 and is output from the image sensor 102 as an analog image signal. The analog image signal is converted into digital image data (RAW data) by the A / D conversion circuit 103 and input to the multi-layer image generation unit 104.

  FIG. 2 is a block diagram illustrating a configuration of the multi-layer image generation unit 104. In FIG. 2, the multi-layer image generation unit 104 includes a plurality of stages of processing circuits 201 to 204 including an extra pixel generation unit, a delay line (DL) block, a low-pass filter (LPF) block, and a downsampling (DS) block in series. Connected and configured. In the present embodiment, the LPF can be composed of, for example, a horizontal and vertical 3-tap filter.

  With reference to FIG. 2, processing in the multilevel image generation unit 104 will be described. Input image data Sin is input to the multi-layer image generation unit 104. Sin is original image data output from the A / D conversion circuit. Sin branches and one of them is output as it is to become a first layer image Sout1 having the same size as the input image. The other of Sin is input to the extra pixel generator 1. The extra pixel generation unit 1 generates extra pixels in the invalid pixel area at the end of the image and outputs them to DL1. DL1 stores image data of different lines in order to apply a low-pass filter in the vertical direction in the subsequent stage, and waits for output. The LPF 1 performs low-pass filtering on the image data output from the DL 1 in the horizontal and vertical directions. Then, it is down-sampled to half the size in the horizontal and vertical directions by DS1, is generated as output image data Sout2 of the second hierarchy, and is output to the subsequent stage and the outside. By executing the same process a plurality of times, the third layer output image data Sout3, the fourth layer output image data Sout4, and the fifth layer output image using the image data input from Sout2, Sout3, Sout4 Data Sout5 is generated.

  Through the above processing, in the present embodiment, the multi-layer image generation unit 104 divides the input image into five layers, and each of the layer images Sout1 to Sout5 is equal to the first layer, which is the highest layer, and is divided into two parts. 1/4, 1/8, and 1/16 of the image data are output. Further, since the low-pass filter process is performed in stages, the high frequency components are removed in stages, and the band of the generated digital image data is gradually shifted toward the low band. The number of layers and the downsampling magnification are not limited to this, and the input image may be divided into other frequency bands by changing the applied filter.

  With reference to FIG. 3, the processing method of the image edge of each hierarchy in Embodiment 1 is demonstrated. In FIG. 3, the pixels at the left end of the input image data are denoted by A1 to M1. Therefore, A1 to M1 are the first hierarchy (Sout1), A2 to G2 are the second hierarchy (Sout2), A3 to D3 are the third hierarchy (Sout3), A4 to C4 are the fourth hierarchy (Sout4), and A5 is the fifth hierarchy. (Sout5). Image data from the first layer to the fourth layer is image data to be processed by the processing circuits 201 to 204. At the same time, the second to fifth layer image data is output image data from the processing circuits 201 to 204.

  The invalid pixel area is on the left side of the dotted line, and the effective pixel area is on the right side. The coordinate of the leftmost pixel in the effective pixel area is 0, and the horizontal axis is taken. The number of extra pixels in the invalid pixel area is one pixel in the horizontal direction. In order to simplify the description, the number of horizontal taps of the low-pass filter is 3 taps, the filter coefficient is [1, 2, 1], and a filter is applied to the same position in the upper layer and the left and right 3 pixels. Here, since a 3-tap filter is used, one extra pixel is sufficient, but the number of extra pixels increases according to the number of taps. For example, if it is 5 taps, it becomes 2 pixels, and if it is 7 taps, it becomes 3 pixels.

  In downsampling, the output of the low-pass filter is thinned out for each pixel and output. The line connecting the pixels represents the relationship between the upper layer and lower layer pixels in the DS. For example, the pixel value of B2 can be represented by (A1 + B1 + C1) / 4. A hatching block in the invalid pixel area indicates an invalid pixel that is not used in the present embodiment. If all the extra pixels in each layer are generated by a low-pass filter and downsampling from the pixels in the invalid pixel region, these hatched pixels are also required. In this embodiment, the number of invalid pixels in the first hierarchy is 15 pixels.

  In the present embodiment, the pixels in the invalid pixel area necessary for the LPF processing in each lower layer are generated by copying the extra pixels in the first layer. That is, the pixel values of the extra pixel A1 are copied and used for the extra pixel A2, the extra pixel A3 of the third hierarchy, and the extra pixel A4 of the fourth hierarchy.

  However, it is not always necessary to use extra pixels in the first layer. For example, it is possible to copy and use extra pixels in a hierarchy higher than the hierarchy in which LPF processing is performed. In this case, the pixel value of the extra pixel A3 in the third layer is used by copying the pixel value of the extra pixel A2, and the pixel value of the extra pixel A4 in the fourth layer is used by copying the pixel value of the extra pixel A3. be able to. The extra pixels in the upper layer can be acquired from the upper stage processing circuit or the extra pixel generation unit of the processing circuit.

  Next, referring to FIG. 3 and FIG. 4, what is the deviation of the center of gravity between the extra pixel generation method in the first embodiment and the method of copying the pixel in the invalid pixel region from the neighboring effective pixel region in Patent Document 1? Explain how it will be different. In the filtering process, the center of gravity of the extra pixel is shifted at the image end of each layer in the effective pixel region direction, particularly in the third and lower layers due to the spatial arrangement of the pixels. The shift of the center of gravity of each pixel at the image end according to the present embodiment will be described with reference to FIG.

  First, the center of gravity of the pixel B2 is (−1 + 0 × 2 + 1) / 4 = 0. Since the pixel A2 is copied from A1 in the highest hierarchy, the coordinates of the pixel are the same as A1. Therefore, the center of gravity of the pixel B3 is (−1 + 0 × 2 + 2) /4=1/4=0.25. Similarly, when calculated by the above method, the center of gravity of B4 is (−1 + 1/4 × 2 + 4) /4=7/8≈0.9, and the center of gravity of A5 is (−1 + 7/8 × 2 + 8) / 4 = 35 / 16≈2.2. Next, the shift amount of the center of gravity of each pixel at the image end when the pixels in the invalid pixel area are copied from the neighboring effective pixel areas will be described with reference to FIG. FIG. 4 differs from the method of FIG. 3 in that the method of generating extra pixels in the second to fourth layers is a method of copying from pixels in the neighboring effective pixel region in the same layer. That is, the pixel value of the extra pixel A2 is a copy of the pixel value of the pixel B2, and similarly, the pixel value of the extra pixel A3 and A4 is a copy of the pixel value of the pixels B3 and B4, respectively.

  In this case, the center of gravity of the pixel B2 is (−1 + 0 × 2 + 1) / 4 = 0. The pixel A2 is copied from the pixel B2, and the center of gravity of the pixel B3 is (0 + 0 × 2 + 2) /4=1/2=0.5. The center of gravity of B4 is (1/2 + 1/2 × 2 + 4) /4=11/8≈1.4, and the center of gravity of A5 is (11/8 + 11/8 × 2 + 8) /4=97/32≈3.0. The shift of the center of gravity increases as the level increases.

  Comparing the above calculation results, in the case of this embodiment, in the order of the second, third, fourth, and fifth layers, 0: 0.25: 0, 9: 2.2 0: 0.5: 1.4: 3.0. Note that the closer the value is to 0, the smaller the deviation. As described above, according to the extra pixel generation method according to the present embodiment, the shift of the center of gravity can be reduced as compared with the case where the pixels in the invalid pixel area are copied from the neighboring effective pixel areas.

  In the present embodiment described above, by reducing the number of extra pixels in the invalid pixel area in the multi-layer process, the processing efficiency is improved as compared with the method in which the extra pixels in the invalid pixel area are generated by the low-pass filter and the downsampling process. The visual field ratio of the output image with respect to the input image can be improved. Further, the center-of-gravity shift of the pixel at the image end can be reduced as compared with the method of copying the extra pixel from the effective pixel region. Furthermore, since the above processing can be realized with a simple circuit, an effect of reducing the circuit scale can be expected.

[Embodiment 2]
Next, Embodiment 2 will be described. In the first embodiment, the extra pixels in the lower layer are generated by copying the extra pixels in the first layer or the upper layer. On the other hand, in the second embodiment, a method for generating an extra pixel in the lower layer by reducing the pixels in the invalid area in the upper layer, and an extra pixel in the lower layer by copying the extra pixel in the upper layer having a different band are generated. Used in combination with the above method.
In the first embodiment, the pixels in the invalid pixel area in the first hierarchy that is the highest hierarchy are copied and used as extra pixels in the lower hierarchy below the second hierarchy. However, when there is an edge or noise near the boundary between the invalid pixel region and the effective pixel region and the high frequency component is large, there is a possibility that the influence of the high frequency component is greatly influenced at the image edge in the lower layer having many low frequency components. Therefore, in the present embodiment, a larger number of extra pixels in the first hierarchy is secured than in the first embodiment. Specifically, in the first embodiment, there is only one pixel, but in the example of FIG. 5 described later, seven pixels are secured. In addition, extra pixels in the upper layer are generated by pixel downsampling, and extra pixels in the lower layer copy the same number of pixels from the extra pixels in the upper layer. The number of pixels to be secured can be determined based on the number of filter taps and the number of processing circuit stages.

  A method of generating extra pixels in the second embodiment will be described with reference to FIG. In FIG. 5, the number of horizontal pixels in the invalid pixel area is 7 pixels from A1 to G1. Extra pixels in the second to fourth layers are generated by the following procedure. The extra pixels in the second and third layer image data are generated by down-sampling the pixels of the upper layer of one layer, that is, the first and second layer image data. The extra pixels in the fourth layer are generated by copying the extra pixels in the third layer. Note that low-pass filter processing may be performed during downsampling. For example, when determining the extra pixel value of A2, low pass filter processing may be performed with 3 taps for the pixel values of A1, B1, and C1, and a filter coefficient of [1, 2, 1]. As a result of the above processing, the centers of gravity of the pixels D2, B3, and B4 in the second, third, and fourth hierarchies become 0, and there is no deviation. Also, since the extra pixel A3 is copied for the pixel A5, (−4 + 0 × 2 + 8) / 4 = 1 is obtained, and the deviation of the center of gravity is greatly reduced as compared with 2.2 in the first embodiment. I understand that.

  In the second embodiment described above, the number of necessary invalid pixel regions is about half that in the case where all the extra pixels in all layers are generated by downsampling from the pixels in the upper layer, and the deviation of the center of gravity is further reduced. Therefore, the extra pixel generation method in the second embodiment is an effective method when a larger number of pixels in the ineffective pixel region can be secured than in the first embodiment.

[Embodiment 3]
Next, Embodiment 3 will be described. In the first embodiment, the extra pixels in the lower layer are generated by copying the extra pixels in the first layer or the upper layer. On the other hand, in the third embodiment, the extra pixels in the lower layer are generated by weighting and adding the extra pixels in the invalid pixel region in the upper layer having different bands and the extra pixels in the effective pixel region in the same layer with a certain coefficient ratio. To do. An extra pixel generation method according to the third embodiment will be described with reference to FIG. In FIG. 6, the number of horizontal pixels in the invalid pixel area is one pixel A1. Extra pixels in the second to fourth layers are generated by the following procedure. The pixel A2 generates a value obtained by adding α1 times the extra pixel A1 in the invalid pixel area of the first hierarchy and (1−α1) times of the neighboring pixel B2 in the effective pixel area of the second hierarchy. Similarly, extra pixels in each layer are generated by calculating A3 and A4 pixels. The coefficients α2 and α3 used in the lower layer may be the same value as α1, or may be different values.

  The closer the value of the coefficient α is to 1, the larger the ratio of the pixel value of A1, and the closer the value is to 0, the larger the ratio of the pixels in the effective pixel area of the same hierarchy. Therefore, when there is an edge or noise near the boundary between the invalid pixel area and the effective pixel area and the high frequency component is large, the ratio of the pixel value of A1 including the high frequency is decreased, and the ratio of the pixels in the effective pixel area of the same layer is increased. This can reduce the influence of high-frequency components at the image edge. On the other hand, when the high frequency component is not particularly large near the boundary between the invalid pixel region and the effective pixel region, the ratio of the pixel value of A1 can be increased to reduce the shift of the center of gravity. This tendency becomes stronger as the hierarchy goes down. For example, in the fourth hierarchy, since the effective pixels are considerably lacking in high-frequency components, many high-frequency components are included near the boundary between the invalid pixel area and the effective pixel area. If this is the case, it is desirable to lower the ratio of A1 further than the upper hierarchy. For example, α1> α2> α3. On the other hand, when many high-frequency components are not included near the boundary, for example, α1 = α2 = α3.

  In the present embodiment, the necessary number of pixels in the invalid pixel area is the same as that in the first embodiment, which is one pixel. Similarly to the second embodiment, this is effective when there is an edge or noise near the boundary between the invalid pixel area and the effective pixel area and the high frequency component is large and the number of pixels in the invalid pixel area cannot be secured. That is, by controlling the value of the coefficient α while minimizing the number of pixels in the invalid pixel region, it is possible to cope with a case where a high frequency component is large near the boundary.

  The entire original image data can also be used as an effective pixel area. In this case, the extra pixel generation unit 1 of the uppermost processing circuit 201 generates the A1 pixel in the invalid pixel area of the first hierarchy by copying the pixel B1 at the end of the effective pixel area of the original image data. The extra pixel generation units in the second and subsequent stages can calculate extra pixels in the target hierarchy using extra pixels in the upper hierarchy. Therefore, it is possible to reduce the center-of-gravity shift of the extra pixel in the lower layer compared to the case of copying from the effective pixel region in all layers.

  As mentioned above, although preferable embodiment of this invention was described, this invention is not limited to these embodiment, A various deformation | transformation and change are possible within the range of the summary.

(Other examples)
The present invention can also be realized by executing the following processing. That is, software (program) that realizes the functions of the above-described embodiments is supplied to a system or apparatus via a network or various storage media, and a computer (or CPU, MPU, or the like) of the system or apparatus reads the program. It is a process to be executed.

Claims (8)

An image processing apparatus that repeatedly performs downsampling processing on original image data to generate image data of a plurality of layers having different numbers of pixels,
Extra pixel generation means for performing a process of generating extra pixels in the invalid pixel area outside the effective pixel area in the input image data;
Sampling processing means for performing filter processing and downsampling processing according to a predetermined number of taps for the image data including the extra pixels,
With
The extra pixel generation means includes
In the process of generating the image data of the second hierarchy, the extra pixels used in the process of generating the image data of the first hierarchy that is higher than the second hierarchy are duplicated to generate the extra pixels. An image processing apparatus. The extra pixel generation means includes
The image processing apparatus according to claim 1, wherein an extra pixel having the same number of pixels as an extra pixel used in a process of generating image data of an upper layer is generated. The extra pixel generation means includes
The image processing apparatus according to claim 1, wherein an extra pixel is generated by duplicating an extra pixel used in a process of generating image data of an upper hierarchy only at a predetermined hierarchy or lower. 4. The image processing apparatus according to claim 1, wherein an initial downsampling process is performed using pixels at an end of the effective pixel area of the original image data as extra pixels. 5. An image processing apparatus that repeatedly performs downsampling processing on original image data to generate image data of a plurality of layers having different numbers of pixels,
Extra pixel generation means for performing a process of generating extra pixels in the invalid pixel area outside the effective pixel area in the input image data;
Sampling processing means for performing filter processing and downsampling processing according to a predetermined number of taps for the image data including the extra pixels,
With
The extra pixel generation means includes
In the process of generating the image data of the second hierarchy, the extra pixels used in the process of generating the image data of the first hierarchy that is higher than the second hierarchy, and the input image data An image processing apparatus, wherein extra pixels are generated by weighted addition of pixels at an end of an effective pixel region. 6. The image according to claim 5, wherein the weighting is such that the weight of the extra pixels used in the process of generating the image data of the first hierarchy is reduced in the process of generating the image data of the lower hierarchy. Processing equipment. An image processing method for repeatedly performing downsampling processing on original image data to generate image data of a plurality of layers each having a different number of pixels,
A process of generating extra pixels in the invalid pixel area outside the effective pixel area in the input image data;
A sampling processing step for performing filtering processing and downsampling processing according to a predetermined number of taps for the image data including the extra pixels,
Including
In the process of generating the image data of the second hierarchy, the extra pixels used in the process of generating the image data of the first hierarchy that is higher than the second hierarchy are duplicated to generate the extra pixels. An image processing method. An image processing method for repeatedly performing downsampling processing on original image data to generate image data of a plurality of layers each having a different number of pixels,
Filtering and downsampling processing corresponding to a predetermined number of taps is performed on the input image data using pixels included in the effective pixel area of the image data and extra pixels not included in the effective pixel area. Including a sampling process ,
The pixel value of the extra pixel used in the process of generating the image data of the second hierarchy is the value of the extra pixel used in the process of generating the image data of the first hierarchy that is higher than the second hierarchy. An image processing method having the same value as a pixel value.

Images