JP2006245677A - Image processor, image processing method, and image processing program - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide an image processor that can stably generate a proper superresolution image. <P>SOLUTION: The image processor is provided with: an image storage section 101 for sequentially storing images photographed by an imaging section 101; a flow detection section 120 for detecting an optical flow with respect to a reference image being an image at an optional point of time among the images stored in the image storage section 110; a moving time calculation section 131 for obtaining a time until a moving distance exceeds a predetermined threshold value on the basis of the optical flow; a moving speed calculation section 132 for calculating a moving speed of pixels on the basis of the moving time and the moving distance; a moving direction calculation section 133 for calculating an angle of the moving direction on the basis of the moving direction information included in the optical flow; a speed information output section 134 for outputting the speed information including the angle of the moving direction and the moving speed; and a superresolution image generating section 142 for calculating a positional deviation between images consecutive in time series on the basis of the speed information and generating the superresolution image on the basis of the positional deviation. <P>COPYRIGHT: (C)2006,JPO&NCIPI

Description

  The present invention relates to an image processing apparatus, an image processing method, and an image processing program for detecting a region having motion from a captured time-series image and executing super-resolution processing for improving the resolution of the region.

  In recent years, image processing apparatuses that can acquire an image sequence and perform image processing at a high frame rate of 100 frames / second or more are widely used. In such an image processing apparatus, it is possible to perform image processing such as tracking of an object moving at high speed, which is difficult to realize with an image sequence photographed at a low frame rate.

  In the image processing apparatus as described above, an image sequence with a high frame rate is acquired by acquiring images at a short interval of 1/100 second or less from an imaging apparatus such as a CMOS camera. When images are acquired at short intervals of 1/100 seconds or less using an imaging device such as a CMOS camera, the exposure time of the imaging surface is shortened. Therefore, in order to acquire an image having a sufficient dynamic range, imaging is performed. It is necessary to secure a sufficient area of the pixels constituting the imaging surface of the apparatus.

  However, since the area of the imaging surface generally has limitations due to manufacturing technology, cost, etc., increasing the area of the pixels that make up the imaging surface reduces the total number of pixels on the imaging surface, and the resolution of the image obtained Becomes lower. For this reason, in an image processing apparatus using an image sequence with a high frame rate, when acquiring an image having a dynamic range necessary for processing, the resolution of the image to be acquired must be lowered, The types of image processing that can be realized have been limited.

  On the other hand, a technique called super-resolution has been proposed as image processing for generating a high-resolution image that exceeds the physical resolution of the imaging surface of the imaging apparatus. Super-resolution refers to extracting the velocity information of a moving area, detecting the amount of positional deviation per image of the area from the extracted velocity information with an accuracy less than the size of one pixel, and detecting the detected positional deviation. This means that a high-resolution image exceeding the physical resolution is generated by interpolating luminance values between pixels from a plurality of images in the image sequence based on the amount. Hereinafter, a high-resolution image generated by super-resolution is referred to as “super-resolution image”.

  For example, in Non-Patent Document 1, an optical flow that is a vector representing the flow of each pixel is obtained from two images before and after an image at an arbitrary time point in a captured image sequence, and an image is obtained from the obtained optical flow. The face image is super-resolved by obtaining the amount of misalignment between them. In general, in order to generate an appropriate super-resolution image, it is important to accurately determine the amount of positional deviation used for interpolation between pixels.

S. Baker, T. Kanade, “Super-Resolution Optical Flow”, CMU-RI-TR-99-36, 1999

  However, when the method for calculating the amount of misalignment performed in Non-Patent Document 1 is applied to an image sequence with a high frame rate, the optical flow between images is very small for an object moving at a low speed. There is a problem that the amount of misalignment cannot be detected with high accuracy, and an inappropriate super resolution image that is significantly different from the actual image is generated because super resolution is performed based on such an erroneous amount of misalignment. there were.

  The present invention has been made in view of the above, and stably generates an appropriate super-resolution image for an image sequence captured at a high frame rate regardless of the moving speed of the captured object. An object is to provide an image processing apparatus, an image processing method, and an image processing program.

  In order to solve the above-described problems and achieve the object, the present invention provides an image processing apparatus, wherein an image storage unit that sequentially stores images taken by an imaging unit, and the image stored by the image storage unit, From a flow detection unit that detects an optical flow that is a vector including information on a movement direction and a movement distance of a pixel on the image with respect to a reference image that is the image at an arbitrary time point, and the optical flow detected by the flow detection unit A movement time calculation unit that obtains a time required for the movement distance to exceed a predetermined threshold, and a movement that calculates the movement speed of the pixel based on the time obtained by the movement time calculation unit and the movement distance An angle of the moving direction is calculated based on information on the moving direction included in the optical flow detected by the velocity calculating unit and the flow detecting unit. A moving direction calculation unit, a speed information output unit that outputs speed information including the angle of the moving direction calculated by the moving direction calculation unit and the moving speed calculated by the moving speed calculation unit, and the speed information output unit. A super-resolution image generation unit that calculates a positional deviation amount between the images that are continuous in time series from the output velocity information, and generates a super-resolution image based on the calculated positional deviation amount; It is characterized by that.

  The present invention also provides an image processing method and an image processing program capable of executing the above apparatus.

  According to the present invention, in a time-series image with a high frame rate, regardless of the moving speed of a pixel included in an area including a pixel having motion, the speed information of the pixel in the area is calculated with high accuracy. Super-resolution processing can be executed based on the amount of positional deviation calculated from the information. For this reason, there is an effect that an appropriate super-resolution image can be stably generated while ensuring a dynamic range of an image shot at a high frame rate.

  Exemplary embodiments of an image processing apparatus, an image processing method, and an image processing program according to the present invention are explained in detail below with reference to the accompanying drawings.

(First embodiment)
The image processing apparatus according to the first embodiment detects an optical flow of pixels between images, and obtains and calculates a time required for the pixel moving distance to exceed a predetermined threshold from the detected optical flow. The moving speed of the pixel is calculated from the measured time and the moving distance of the pixel, the amount of positional deviation between successive images in time series is calculated from the calculated moving speed and the moving direction of the pixel, and the calculated positional deviation amount is calculated. Based on this, a super-resolution image is generated.

  Here, the optical flow refers to a vector including information on the moving direction and moving distance of the pixel on the currently processed image with respect to the reference image. The reference image is an image serving as a reference for detecting an optical flow, and an image at an arbitrary time can be used as a reference image. The reference image is updated to the image currently being processed, for example, when the pixel moving distance exceeds a predetermined threshold. Further, the positional deviation amount refers to a difference in position of corresponding pixels in time-sequential images.

  FIG. 1 is a block diagram illustrating a configuration of an image processing apparatus 100 according to the first embodiment. As shown in the figure, the image processing apparatus 100 includes an image storage unit 110 and a high-resolution image storage unit 150 as main hardware configurations. Further, the image processing apparatus 100 according to the first embodiment includes a flow detection unit 120, a speed information generation unit 130, and a super-resolution processing unit 140 as main software configurations.

  The image storage unit 110 is a storage unit that sequentially stores images captured by the imaging unit 101 that captures time-series images. The high resolution image storage unit 150 is a storage unit that stores the high resolution image generated by the super resolution image generation unit 142 in the super resolution processing described later.

  The flow detection unit 120 sets a reference image at an arbitrary time with respect to a time-series image captured by the imaging unit 101 and stored in the image storage unit 110, and the reference image and an image later in time are set. The optical flow, which is a vector including information on the moving direction and moving distance of each pixel, is detected from the luminance pattern between.

  As a flow detection method, any conventionally used method can be applied. As an example, the flow detection unit 120 in the image processing apparatus 100 according to the first embodiment includes a local mask setting unit 121, a correlation value calculation unit 122, and a correspondence determination unit 123 as detailed configurations.

  The local mask setting unit 121 sets an image at an arbitrary time point in the time-series image as a reference image, and sets a local mask including pixels for which an optical flow is obtained for each of the reference image and the current image from which the optical flow is detected. To do. As the size of the local mask, for example, 5 × 5 pixels and 7 × 7 pixels can be set.

  The correlation value calculation unit 122 calculates a correlation value indicating the degree of association between the local mask region of the reference image and the local mask region of the current image from the luminance information in the local mask set by the local mask setting unit 121. To do. The correlation value is calculated by comparing the luminance of each pixel between the local mask area in the reference image and the local mask area in the current image.

  For each pixel in the local mask region of the current image, the correspondence determination unit 123 determines a pixel whose correlation value calculated by the correlation value calculation unit 122 is equal to or larger than a predetermined threshold and is the maximum as the local mask of the reference image. A process of associating with each pixel in the region is performed. As a result, a vector from each pixel in the reference image to each pixel in the subsequent image is defined, and the optical flow is represented by this vector.

  The speed information generation unit 130 calculates a movement time when the movement distance of each pixel reaches a certain amount from the optical flow of each pixel detected by the flow detection unit 120, and calculates the movement calculated from the movement direction, the movement distance, and the movement time. Speed information of each pixel including the speed is generated. The speed information generation unit 130 includes a movement time calculation unit 131, a movement speed calculation unit 132, a movement direction calculation unit 133, and a speed information output unit 134.

  The movement time calculation unit 131 calculates the movement time of each pixel by calculating the number of images until the size of the optical flow detected by the flow detection unit 120 reaches the index distance. Here, the index distance is a threshold of the movement distance of each pixel. The movement time to reach the index distance is calculated, and the index distance is divided by the calculated movement time, so that each pixel can be obtained even in an image sequence at a high frame rate. It is used to make it possible to stably calculate the moving speed of. The index distance is set in advance by obtaining a value with which the above-described flow detection unit 120 can detect an optical flow with high accuracy.

  The movement speed calculation unit 132 calculates the movement speed of each pixel by dividing the index distance by the movement time calculated by the movement time calculation unit 131. The movement direction calculation unit 133 calculates the movement direction of each pixel from the optical flow used when the movement time calculation unit 131 calculates the movement time. The moving direction is represented by the angle of the optical flow. The speed information output unit 134 outputs speed information whose elements are the angle of the moving speed calculated by the moving speed calculating unit 132 and the moving direction calculated by the moving direction calculating unit 133.

  The super-resolution processing unit 140 sets a processing target area to be subjected to super-resolution processing from the speed information generated by the speed information generating unit 130 and the position information of each pixel, and averages the speed information for each processing target area. By calculating a value, obtaining a positional deviation amount between images from the average value of velocity information, and interpolating between pixels using the obtained positional deviation amount and a past image stored in the image storage unit 110. A super-resolution image is generated. The super-resolution processing unit 140 includes a processing target area setting unit 141 and a super-resolution image generation unit 142.

  The processing target area setting unit 141 refers to the speed information output from the speed information output unit 134 and sets a target area for super-resolution. That is, the processing target region setting unit 141 has a small variation in the moving speed and moving direction of pixels included in an area having an area larger than a predetermined value in the image, and the moving speed is less than zero. A region that is large and less than one pixel is detected and set as a processing target region.

  The super-resolution image generation unit 142 generates a high-resolution image of the entire image region by interpolation from the current image, and then performs a high-resolution image for each region set as the processing target region by the processing target region setting unit 141. Super-resolution processing is performed using a high-resolution image for the image of the previous frame stored in the storage unit 150, and a super-resolution image is generated by combining with the high-resolution image generated by interpolation.

  Next, image processing by the image processing apparatus 100 according to the first embodiment configured as described above will be described. FIG. 2 is a flowchart showing the overall flow of the image processing in the first embodiment.

  First, after an image is captured by the imaging unit 101 and a time-series image is stored in the image storage unit 110, the flow detection unit 120 executes a flow detection process (step S201). Next, the speed information generation unit 130 executes speed information generation processing (step S202). Finally, the super-resolution processing unit 140 executes super-resolution processing (step S203), and the image processing ends. Details of the flow detection process, the speed information generation process, and the super-resolution process will be described later.

  Next, details of the flow detection process shown in step S201 by the image processing apparatus 100 according to the first embodiment configured as described above will be described. FIG. 3 is a flowchart illustrating an overall flow of the flow detection process in the image processing apparatus 100 according to the first embodiment.

  First, the local mask setting unit 121 sets an image at an arbitrary time point in the time-series image as a reference image, and sets a local mask including pixels for which an optical flow is obtained in the reference image and the subsequent current image (step S301). ). Next, the correlation value calculation unit 122 compares the luminance of each pixel between the local mask area in the reference image and the local mask area in the current image, and calculates a correlation value (step S302).

  As a method for calculating the correlation value, any conventionally used method can be applied. For example, use the method of calculating the absolute value difference sum or the sum of squared differences of the luminance of the corresponding pixels in the two areas to be compared, or the method of calculating the value normalized by the average value and variance in the local mask Can be configured to. Note that these are merely examples, and the present invention is not limited to these as long as the correlation value indicating the degree of association between the areas in which the local masks are set is calculated.

  In order to improve the reliability of the optical flow to be finally detected, a local mask setting condition may be provided. For example, for a pixel that does not have a contrast or a pixel that has a contrast but has only a single-directional edge, it is difficult to determine the corresponding pixel correctly, so that a setting condition is set so that a local mask is not set. Can be configured to impose. These are merely examples, and the present invention is not limited to these as long as the setting conditions are for improving the reliability of the optical flow.

  Next, the correspondence determination unit 123 associates each pixel in the local mask region where the correlation value calculated by the correlation value calculation unit 122 is greater than or equal to a predetermined threshold value and the maximum with each pixel in the local mask region of the reference image. Thus, an optical flow that is a vector from each pixel in the reference image to each pixel in the current image is detected (step S303).

  Next, details of the speed information generation process shown in step S202 by the image processing apparatus 100 according to the first embodiment configured as described above will be described. FIG. 4 is a flowchart illustrating an overall flow of the speed information generation process in the image processing apparatus 100 according to the first embodiment.

First, the movement time calculation unit 131 calculates the movement time until the size of the optical flow detected by the flow detection unit 3 reaches the index distance D (step S401). Specifically, in each pixel (x, y), the optical flow detected between the subsequent n-th image is sequentially set as f (x, y; n), and the following equation (1) is used. Calculate travel time.
t (x, y) = {n | f (x, y; n) = D} (1)

  That is, when the optical flow between the n-th image and the subsequent n-th image is equal to the index distance D, the magnitude of the optical flow at the pixel (x, y) is n of the pixel (x, y). Calculated as the required time t (x, y).

  Next, the moving speed calculating unit 132 calculates the moving speed v (x, y) of each pixel by dividing the index distance D by the moving time calculated by the moving time calculating unit 131 (step S402). .

  Next, the movement direction calculation unit 133 uses the optical flow f (x, y; n) | n = t (x, y) used when the movement time calculation unit 131 calculates the movement time t (x, y). From this, the movement direction d (x, y) of each pixel is calculated (step S403). The moving direction is represented by the angle of the optical flow f (x, y; n).

  Next, the speed information output unit 134 uses the movement speed v (x, y) calculated by the movement speed calculation unit 132 and the movement direction d (x, y) calculated by the movement direction calculation unit 133 as elements. The speed information V (x, y) to be output is output (step S404), and the speed information generation process ends.

  Next, details of the super-resolution processing shown in step S203 by the image processing apparatus 100 according to the first embodiment configured as described above will be described. FIG. 5 is a flowchart illustrating an overall flow of super-resolution processing in the image processing apparatus 100 according to the first embodiment.

  First, the processing target area setting unit 141 sets a processing target area that is a target area for super-resolution (step S501). As a method of setting the processing target area, for example, (i) the entire image is divided into fixed areas of arbitrary units, and processing is performed when the variation in speed information in the area is equal to or less than a predetermined threshold value. A method of setting as a target region, and (ii) a method of searching surrounding pixels having similar speed information for each pixel in an image and setting a region including these as a processing target region can be applied.

Details of the method mentioned in the above example (i) will be described below. First, the processing target area setting unit 141 divides an image into areas of, for example, 8 × 8 pixel units, and calculates variations in speed information of pixels included in the respective areas. The variation in the moving speed in the speed information is, for example, using the total number N of pixels in which the optical flow is detected in the region and the average v ₀ of the moving speed v (x, y) in these pixels as follows: It can be calculated by equation (2).

The variation in the moving direction in the speed information is, for example, the total number N of pixels in which optical flow is detected in the region, and the average value d of vector angles d (x, y) representing these optical flows. It can be calculated by the following equation (3) using ₀ .

  The processing target region setting unit 141 sets a region where the variation in the moving speed and the variation in the moving direction calculated by these equations are each equal to or less than a predetermined threshold value as one processing target region.

  Next, details of the method mentioned in the above example (ii) will be described. First, the processing target area setting unit 141 divides an image into areas of, for example, 8 × 8 pixel units, and classifies the pixels in which the optical flow is detected according to the angle of the optical flow. Next, the processing target area setting unit 141 calculates the difference between the pixel coordinates and the moving speed between the classified pixels, groups the pixels whose calculated differences are smaller than a predetermined threshold, and sets these pixels. And a predetermined range of pixels around it is set as one processing target area.

  Note that these are merely examples, and any method can be applied as long as the target area for super-resolution processing is set using speed information.

  After the processing target area is set in step S501, the super-resolution image generation unit 142 executes a super-resolution image generation process. Details of the super-resolution image generation processing will be described below. In the following, a case where a super-resolution image having a resolution M times that of the original image is generated will be described.

  First, the super-resolution image generation unit 142 generates a high-resolution image by interpolating luminance values between pixels for an input image that is a target of super-resolution image generation (step S502). Any commonly used method such as bilinear interpolation can be applied as the interpolation method.

  Next, the super-resolution image generation unit 142 cuts out a region corresponding to the processing target region from the image of the previous frame stored in the high resolution image storage unit 150, and for each pixel included in this region, the image of the previous frame. An estimated high-resolution image is generated by generating an image that has been moved with an accuracy of less than one pixel by the amount of positional deviation with respect to (step S503). As the positional deviation amount, a value calculated by calculating an average value of moving speeds of pixels in which an optical flow included in the processing target area is detected and multiplying this by M is used.

  Next, the super-resolution image generation unit 142 executes the following processing to correct the estimated high-resolution image and generate an appropriate super-resolution image (steps S504 to S506).

First, the super-resolution image generation unit 142 generates an estimated low-resolution image having the same resolution as the input image for the estimated high-resolution image (step S504). The luminance value P _L (x, y) of each pixel of the estimated low resolution image is obtained by calculating the average of the corresponding luminance values on the corresponding estimated high resolution image by the following equation (4). Here, P _E (x, y) is the luminance value of the pixel at the coordinates (x, y) in the estimated high resolution image.

Next, the super-resolution image generation unit 142 compares the estimated low-resolution image and the input image for each pixel to obtain a difference, and when the difference is larger than a predetermined threshold R, the absolute value of the difference is R The amount of correction of the estimated high-resolution image such that If the luminance value of the pixel at the coordinates (x, y) in the input image is P _I (x, y), the correction amount dp (x, y) can be calculated by the following equation (5).

Next, the super-resolution image generation unit 142 calculates the corresponding pixel {P _H (i, j) | Mx ≦ i <M (x + 1), My in the estimated high-resolution image from the calculated correction amount dp (x, y). For each of ≦ j <M (y + 1)}, the pixel P _S of the super-resolution image is generated by correcting the luminance value of the pixel according to the following equation (6) (step S506).

  After correcting the estimated high-resolution image in this way (steps S504 to S506), the super-resolution image generating unit 142 determines the region corresponding to the processing target region of the high-resolution image generated by interpolation as the estimated high-resolution image. Overwriting is performed to generate a super-resolution image (step S507).

  Next, the super-resolution image generation unit 142 determines whether or not all the processing target areas have been processed (step S508). When all the processing target areas have not been processed (step S508: NO), The processing is repeated for the next processing target area (step S503).

  When all the processing target areas have been processed (step S508: YES), the super-resolution image generation unit 142 stores the generated super-resolution image in the high-resolution image storage unit 150 (step S509), and super-resolution. The image generation process ends.

  Thus, in the image processing apparatus 100 according to the first embodiment, the movement speed is calculated by obtaining the movement time until the movement distance of each pixel reaches a predetermined index distance. Even when an object moving at a low speed at a high frame rate is photographed and the optical flow between images becomes very small, the moving speed can be calculated with high accuracy. Therefore, the super-resolution processing performed with reference to the moving speed can be executed with high accuracy.

(Second Embodiment)
In the image processing apparatus 100 according to the first embodiment, the optical flow can be calculated with high accuracy, and the super-resolution processing can be performed with high accuracy by performing the super-resolution processing using the value. Since the super-resolution processing according to the reliability of the optical flow is not performed, an inappropriate super-resolution image may be generated.

  Therefore, the image processing apparatus according to the second embodiment stores the detected optical flow in the flow storage unit, calculates the reliability of the currently detected optical flow using the stored optical flow, A super-resolution image is generated with higher accuracy by weighting with the calculated reliability.

  FIG. 6 is a block diagram illustrating a configuration of an image processing apparatus 600 according to the second embodiment. As shown in the figure, in the second embodiment, the reliability calculation unit 635 and the flow storage unit 660 are added, and the functions of the speed information output unit 634 and the super-resolution image generation unit 642 are changed. This is different from the first embodiment. Other configurations and functions are the same as those in FIG. 1, which is a block diagram showing the configuration according to the first embodiment, and thus are denoted by the same reference numerals and description thereof is omitted here.

  The flow storage unit 660 stores an optical flow from the reference image of each pixel to the current image. The stored optical flow is deleted at the same time when the reference image is updated, and storage of the optical flow for the updated reference image is newly started.

  The reliability calculation unit 635 calculates the reliability of the optical flow for the currently processed image. Here, the reliability of the optical flow refers to a value indicating the degree of accuracy of the optical flow detected by the flow detection unit 120. For example, the reliability calculation unit 635 uses the movement direction included in the optical flow detected in the image having the largest correlation value with respect to the reference image among the optical flows stored in the flow storage unit 660, and the currently processed image. An optical flow having a smaller difference from the moving direction included in the optical flow is calculated with a higher reliability. Details of the reliability calculation method will be described later.

  Next, speed information generation processing by the image processing apparatus 600 according to the second embodiment configured as described above will be described. FIG. 7 is a flowchart showing an overall flow of the speed information generation process in the second embodiment.

  The movement time calculation process, the movement speed calculation process, and the movement direction calculation process from step S701 to step S703 are the same as the process from step S401 to step S403 in the image processing apparatus 100 according to the first embodiment. Description is omitted.

  After the movement direction calculation unit 133 calculates the movement direction (step S703), the reliability calculation unit 635 detects the optical flow angle detected in the image having the largest correlation value with respect to the reference image and the image currently being processed. The reliability of the optical flow for the currently processed image is calculated from the angle of the optical flow for (step S704).

As a reliability calculation method, for example, the following method can be applied. That is, the reliability calculation unit 635 stores the currently detected optical flow angle θ [rad] in the flow storage unit 660 between the reference image and the current image, and the size is larger than 0 and the index distance D Among the smaller optical flows, the angle of the optical flow having the largest correlation value is θ _r [rad], and the reliability r (x, y; n) is calculated by the following equation (7).

  The reliability calculation method is an example, and the method is not limited to this as long as it is a method for calculating a value indicating the degree of probability of optical flow.

  Next, the speed information output unit 634 adds the movement speed v (x, y) calculated by the movement speed calculation unit 132 and the movement direction d (x, y) calculated by the movement direction calculation unit 133, The speed information V (x, y) having the reliability r calculated by the reliability calculation unit 635 as an element is output (step S705), and the speed information generation process ends.

  In addition, you may comprise so that a movement time and a movement direction may be calculated with higher precision by using the information of the optical flow memorize | stored in the flow memory | storage part 660. FIG. For example, instead of using the first image that reaches the index distance D as the number of images until the index distance D is reached, the movement time calculation unit 131 stores the optical flow that is the index distance D in the flow storage unit 660. Of these, an image having the maximum optical flow correlation value may be used.

  In addition, instead of using the angle of the optical flow that has reached the index distance D as the movement direction, the movement direction calculation unit 133 calculates an average value of all the optical flows having a size of 1 or more stored in the flow storage unit 660. You may comprise so that it may be used.

  Next, super-resolution processing by the image processing apparatus 600 according to the second embodiment configured as described above will be described. FIG. 8 is a flowchart showing an overall flow of the super-resolution processing in the second embodiment.

  The processing target area setting process, the interpolated high resolution image generation process, and the estimated high resolution image generation process from step S801 to step S806 are the same as those from step S501 to step S506 in the image processing apparatus 100 according to the first embodiment. Therefore, the description thereof is omitted.

  The super-resolution image generation unit 642 corrects the estimated high-resolution image (step S806), and then weights and adds the corrected luminance value of the estimated high-resolution image and the luminance value of the high-resolution image by interpolation using the reliability as a weighting coefficient. Thus, a super-resolution image is generated (step S807).

For example, the super-resolution image generation unit 642 calculates an average value r _A of the reliability of the optical flow for the pixels in which the optical flow in the processing target area is detected, and calculates r _A and (1−r _A ). As a weighting coefficient, the luminance value P _S (x, y) of the corrected estimated high-resolution image and the luminance value P _H (x, y) of the pixel at the coordinates (x, y) in the high-resolution image generated by interpolation between the pixels ) And the luminance value P _R (x, y) of each pixel of the super-resolution image is calculated by the following equation (8).
P _R (x, y) = r _A P _S (x, y) + (1−r _A ) P _H (x, y) (8)

  After the super-resolution image generation unit 642 generates a super-resolution image, the completion determination process and the image storage process from step S808 to step S809 are performed from step S508 in the image processing apparatus 100 according to the first embodiment. Since the processing is the same as that up to S509, description thereof is omitted.

  As described above, in the image processing apparatus 600 according to the second embodiment, the reliability of the optical flow is used as a weighting coefficient, and the luminance value of the high-resolution image generated by interpolation and the positional deviation amount calculated from the optical flow are used. Since the luminance value of the generated high-resolution image is weighted and added, generation of an inappropriate super-resolution image can be suppressed, and generation of the super-resolution image can be executed with higher accuracy.

(Third embodiment)
In general, in super-resolution processing, the correlation between a super-resolution image and a real image increases as the number of processed images increases. Therefore, the correlation between the super-resolution image generated immediately after being set as the processing target region and the actual image is low. Therefore, the image processing apparatus according to the third embodiment goes back to the past image with the upper limit of the value obtained by dividing the index distance by the moving speed for the region immediately after being set as the processing target region, and from there, The correlation between the super-resolution image and the actual image is improved by repeatedly performing the super-resolution processing using a plurality of images up to the image.

  FIG. 9 is a block diagram illustrating a configuration of an image processing apparatus 900 according to the third embodiment. As shown in the figure, the third embodiment differs from the second embodiment in that the function of the super-resolution image generation unit 942 is changed. Since other configurations and functions are the same as those in FIG. 6 which is a block diagram showing the configuration according to the second embodiment, the same reference numerals are given, and description thereof is omitted here.

  The super-resolution image generation unit 942 in the image processing apparatus 900 according to the third embodiment goes back to the past image stored in the image storage unit 110 for the processing target area to be processed for the first time, and starts from the past image. A plurality of images up to the current image are sequentially subjected to super-resolution processing.

  Next, super-resolution processing by the image processing apparatus 900 according to the third embodiment configured as described above will be described. FIG. 10 is a flowchart showing an overall flow of the super-resolution processing in the third embodiment.

  Since the processing target region setting process and the interpolated high-resolution image generation process from step S1001 to step S1002 are the same as those from step S501 to step S502 in the image processing apparatus 100 according to the first embodiment, the description thereof will be given. Omitted.

  The super-resolution image generation unit 942 generates a high-resolution image by interpolating luminance values between pixels (step S1002), and then the current processing target region is included in the processing target region processed in the image of the previous frame. Whether or not (step S1003).

  When the current processing target area is included in the processing target area processed in the image of the previous frame (step S1003: YES), the super-resolution image generation unit 942 is the same as the image processing apparatus 600 according to the second embodiment. In addition, a super-resolution image generation process based on only the image of the previous frame is executed (steps S1004 to S1008).

  The estimated high-resolution image generation processing from step S1004 to step S1008 is the same as the processing from step S803 to step S807 in the image processing apparatus 600 according to the second embodiment, and thus description thereof is omitted.

  In step S1003, when the current processing target area is not included in the processing target area processed in the image of the previous frame (step S1003: NO), the super-resolution image generation unit 942 divides the index distance D by the moving speed. The super-resolution image generation processing is repeatedly executed up to the current image by going back to the image of the past frame with the above value as the upper limit (steps S1009 to S1015). As a result, the number of images used for the super-resolution image increases even in the region immediately after being set as the processing target region, and it is possible to generate a super-resolution image highly correlated with the actual image. Become.

  First, the super-resolution image generation unit 942 cuts out a region corresponding to the processing target region in the past frame image (step S1009). Next, the super-resolution image generation unit 942 generates an image in which each pixel included in the cut-out area is moved with an accuracy of less than one pixel by the amount of positional deviation calculated in the currently processed image. Thus, an estimated high resolution image is generated (step S1010).

  Next, the super-resolution image generation unit 942 generates an estimated low resolution image having the same resolution as the input image for the estimated high resolution image (step S1011). The luminance value of the estimated low-resolution image is calculated by the expression (4) as in the image processing apparatus 600 according to the second embodiment.

Next, the super-resolution image generation unit 942 compares the estimated low-resolution image and the low-resolution partial image obtained by cutting out the region corresponding to the processing target region in the past frame image for each pixel, and obtains a difference. Is larger than a predetermined threshold value R, the correction amount of the estimated high-resolution image is calculated such that the absolute value of the difference is R (step S1012). Similar to the image processing apparatus 600 according to the second embodiment, the correction amount is set by setting the luminance value of the pixel at the coordinates (x, y) in the low-resolution partial image as P _I (x, y). It can be calculated by equation (5).

Next, the super-resolution image generation unit 942 calculates the luminance value of the pixel from the calculated correction amount dp (x, y) according to the equation (6), similarly to the image processing apparatus 600 according to the second embodiment. By correcting, the pixel P _S of the super-resolution image is generated (step S1013).

  Next, the super-resolution image generation unit 942 generates a super-resolution image by weighting and adding the corrected luminance value of the estimated high-resolution image and the luminance value of the high-resolution image by interpolation using the reliability as a weighting coefficient. (Step S1014). At this time, unlike the image processing apparatus 600 according to the second embodiment, the super-resolution image generation unit 942 uses a region corresponding to the processing target region in the past frame image as the luminance value of the high-resolution image by interpolation. The luminance value of the partial image that has been increased in resolution from the cut out low-resolution partial image by interpolation is used. That is, in the image processing apparatus 900 according to the third embodiment, since it is sufficient to use only the partial image of the processing target area of the image of the past frame, the high-resolution image obtained by interpolation is compared with the partial image. This is used only for weighted addition.

  Next, the super-resolution image generation unit 942 determines whether or not all the past frame images to be processed have been processed, that is, whether or not the super-resolution processing has been executed up to the current frame image ( Step S1015). If all the past frame images to be processed have not been processed (step S1015: NO), the processing is repeated for the next past frame image (step S1009). When all the past frame images to be processed have been processed (step S1015: YES), a completion determination process (step S1016) is executed.

  Since the completion determination process and the image storage process from step S1016 to step S1017 are the same as the process from step S508 to step S509 in the image processing apparatus 100 according to the first embodiment, the description thereof is omitted.

  As described above, in the image processing apparatus 900 according to the third embodiment, the super-resolution processing is performed using a plurality of images from the past image to the current image, going back to the past image. Correlation between the super-resolution image and the actual image due to the increase in the number of images can be improved.

  The image processing apparatus according to the first to third embodiments includes a control device such as a CPU, a storage device such as a ROM (Read Only Memory) and a RAM, an external storage device such as an HDD and a CD drive device, and a display. It has a display device such as a device and an input device such as a keyboard and a mouse, and has a hardware configuration using a normal computer.

  The image processing program executed by the image processing apparatus according to the first to third embodiments is a file in an installable format or an executable format, and is a CD-ROM, flexible disk (FD), CD-R, DVD. (Digital Versatile Disk) or the like recorded on a computer-readable recording medium.

  Further, the image processing program executed by the image processing apparatus according to the first to third embodiments is stored on a computer connected to a network such as the Internet, and is provided by being downloaded via the network. It may be configured. The image processing program executed by the image processing apparatus according to the first to third embodiments may be provided or distributed via a network such as the Internet.

  Further, the image processing programs of the first to third embodiments may be provided by being incorporated in advance in a ROM or the like.

  The image processing program executed by the image processing apparatuses according to the first to third embodiments has a module configuration including the above-described units (flow detection unit, speed information generation unit, super-resolution processing unit). As actual hardware, a CPU (processor) reads out and executes an image processing program from the storage medium, whereby each unit is loaded on the main storage device, and each unit is generated on the main storage device. ing.

  As described above, the image processing device, the image processing method, and the image processing program according to the present invention include a time-series image including a region having a motion imaged at a high frame rate, such as an obstacle detection device equipped with an in-vehicle camera. It is suitable for an image processing apparatus that executes super-resolution processing for a target.

1 is a block diagram illustrating a configuration of an image processing apparatus according to a first embodiment. 3 is a flowchart illustrating an overall flow of image processing in the image processing apparatus according to the first embodiment. It is a flowchart which shows the flow of the whole flow detection process in the image processing apparatus concerning 1st Embodiment. It is a flowchart which shows the whole flow of the speed information generation process in the image processing apparatus concerning 1st Embodiment. It is a flowchart which shows the flow of the whole super-resolution process in the image processing apparatus concerning 1st Embodiment. It is a block diagram which shows the structure of the image processing apparatus concerning 2nd Embodiment. It is a flowchart which shows the flow of the whole speed information generation process in the image processing apparatus concerning 2nd Embodiment. It is a flowchart which shows the flow of the whole super-resolution process in the image processing apparatus concerning 2nd Embodiment. It is a block diagram which shows the structure of the image processing apparatus concerning 3rd Embodiment. 14 is a flowchart illustrating an overall flow of super-resolution processing in the image processing apparatus according to the third embodiment.

Explanation of symbols

DESCRIPTION OF SYMBOLS 100 Image processing apparatus 101 Imaging part 110 Image memory | storage part 120 Flow detection part 121 Local mask setting part 122 Correlation value calculation part 123 Correspondence determination part 130 Speed information generation part 131 Travel time calculation part 132 Movement speed calculation part 133 Movement direction calculation part 134 Speed information output unit 140 Super-resolution processing unit 141 Processing target area setting unit 142 Super-resolution image generation unit 150 High-resolution image storage unit 600 Image processing device 634 Speed information output unit 635 Reliability calculation unit 642 Super-resolution image generation unit 660 Flow storage unit 900 Image processing device 942 Super-resolution image generation unit

Claims (8)

An image storage unit for sequentially storing images taken by the imaging unit;
A flow detection unit that detects an optical flow that is a vector including information on a moving direction and a moving distance of a pixel on the image with respect to a reference image that is the image at an arbitrary time among the images stored in the image storage unit; ,
A movement time calculation unit for obtaining a time required for the movement distance to exceed a predetermined threshold from the optical flow detected by the flow detection unit;
A moving speed calculating unit that calculates a moving speed of the pixel based on the time obtained by the moving time calculating unit and the moving distance;
A moving direction calculating unit that calculates an angle of the moving direction based on information of the moving direction included in the optical flow detected by the flow detecting unit;
A speed information output unit that outputs speed information including the angle of the movement direction calculated by the movement direction calculation unit and the movement speed calculated by the movement speed calculation unit;
A super-resolution image that calculates a positional shift amount between the images that are continuous in time series from the speed information output by the speed information output unit, and generates a super-resolution image based on the calculated positional shift amount. A generator,
An image processing apparatus comprising: A flow storage unit for sequentially storing the optical flows;
A reliability calculation unit that calculates a reliability indicating the degree of certainty of the optical flow detected by the flow detection unit, and
The flow detection unit sets a detection target region of the optical flow, calculates a correlation value indicating a degree of association between the reference image in the region and the image for detecting the optical flow, and the correlation value is Detecting the optical flow for pixels included in the region exceeding a predetermined threshold;
The reliability calculation unit detects the movement direction included in the optical flow detected in the image having the largest correlation value among the optical flows stored in the flow storage unit, and the flow detection unit detects Calculating the reliability based on a difference from the moving direction included in the optical flow;
The super-resolution image generation unit uses the reliability calculated by the reliability calculation unit as a weighting coefficient, and the luminance value calculated from the positional deviation amount and the pixel of the input image that is the image for performing the super-resolution processing 2. The image processing apparatus according to claim 1, wherein a super-resolution image is generated by calculating a luminance value of the super-resolution image by weighting and adding the luminance value calculated by interpolating the luminance value of . The super-resolution image generating unit uses the reliability average value r _A and (1-r _A ) calculated by the reliability calculation unit for the optical flow detected in the region as a weighting coefficient, The luminance value calculated from the positional deviation amount is P _S (x, y), and the luminance value calculated by interpolating the luminance value of the pixel of the input image is P _H (x, y). The image processing apparatus according to claim 2, wherein the luminance value P _R (x, y) of the super-resolution image is calculated by an addition formula.

  The super-resolution image generation unit acquires a past image that is the past image with respect to the input image from the image storage unit, and continues in time series from the acquired past image to the input image. The image processing apparatus according to claim 1, wherein a super-resolution image generation process is sequentially performed between images to generate a super-resolution image for the input image.   The super-resolution image generation unit generates a luminance value calculated by interpolating a luminance value of a pixel of the input image by moving a pixel by a distance corresponding to the positional deviation amount. The image processing apparatus according to claim 1, wherein a super-resolution image is generated by replacing with a value.   The super-resolution image generation unit, when a luminance value difference that is a difference between a luminance value of the high-resolution image and a luminance value of the input image exceeds a luminance value difference threshold that is a predetermined threshold, The brightness value calculated by correcting the brightness value of the high-resolution image and interpolating the brightness value of the pixel of the input image is corrected by the difference between the value difference and the brightness value difference threshold. The image processing apparatus according to claim 5, wherein a super-resolution image is generated by substituting with a luminance value. Among the images stored in the image storage unit that sequentially stores the images captured by the imaging unit, the vector includes information on the moving direction and moving distance of the pixels on the image with respect to the reference image that is the image at an arbitrary time. A flow detection step for detecting an optical flow;
A moving time calculating step for obtaining a time required for the moving distance to exceed a predetermined threshold from the optical flow detected by the flow detecting step;
A moving speed calculating step for calculating a moving speed of the pixel based on the time obtained by the moving time calculating step and the moving distance;
A moving direction calculating step of calculating an angle of the moving direction based on the moving direction information included in the optical flow detected by the flow detecting step;
A speed information output step for outputting speed information including the angle of the moving direction calculated by the moving direction calculating step and the moving speed calculated by the moving speed calculating step;
A super-resolution image that calculates a positional deviation amount between the images that are continuous in time series from the velocity information output by the velocity information output step, and generates a super-resolution image based on the calculated positional deviation amount. Generation step;
An image processing method comprising: Among the images stored in the image storage unit that sequentially stores the images captured by the imaging unit, the vector includes information on the moving direction and the moving distance of the pixel on the image with respect to the reference image that is the image at an arbitrary time. A flow detection procedure for detecting an optical flow;
A travel time calculation procedure for obtaining a time required for the travel distance to exceed a predetermined threshold from the optical flow detected by the flow detection procedure;
A moving speed calculating procedure for calculating a moving speed of the pixel based on the time obtained by the moving time calculating procedure and the moving distance;
A moving direction calculation procedure for calculating an angle of the moving direction based on information of the moving direction included in the optical flow detected by the flow detection procedure;
A speed information output procedure for outputting speed information including the angle of the moving direction calculated by the moving direction calculation procedure and the moving speed calculated by the moving speed calculation procedure;
A super-resolution image that calculates a positional deviation amount between the images that are continuous in time series from the velocity information output by the velocity information output procedure, and generates a super-resolution image based on the calculated positional deviation amount. Generation procedure,
An image processing program for causing a computer to execute.

Images