JP2007179354A - Optical flow calculation device, optical flow calculation method, optical flow calculation program and recording medium - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To calculate a high-accuracy optical flow to acquire a high-quality image. <P>SOLUTION: An image 101 of an object inside a frame image is inputted by an imaging part or the like, and is recorded with blurring included in a direction of an arrow 102. Image restoration processing is performed about the recorded image 101 of the object to obtain a restoration image 103 and in-frame motion blurring function (104). Here, a PSF (Point Spread Function) 105 is already known on the basis of a characteristic or the like of the imaging part, and the in-frame motion blurring function (104) is obtained by convoluting the PSF 105 and an in-frame motion vector 106, so that the in-frame motion vector 106 can be obtained by deconvoluting the in-frame motion blurring function (104) by the PSF 105. The in-frame motion vector 106 is increased (a frame interval time/an integral time) times to calculate an inter-frame motion vector, and the optical flow is determined. <P>COPYRIGHT: (C)2007,JPO&INPIT

Description

  The present invention relates to an optical flow calculation device, an optical flow calculation method, an optical flow calculation program, and a recording medium that calculate an optical flow in a series of frame images input from a camera with high accuracy.

  Conventionally, there is a method of calculating an optical flow (motion vector) in order to estimate a three-dimensional structure of an object existing in a certain frame image from a plurality of temporally continuous frame images. The optical flow (motion vector) is calculated based on a base point in the first frame image and a corresponding point corresponding to the base point in the second frame image.

  More specifically, for example, it corresponds to the pixel value of a predetermined area including the base point in the first frame image and the base point in the search area centering on the base point in the first frame image in the second frame image. A correlation value of pixel values in a predetermined area including corresponding points is calculated. Then, the corresponding point of the predetermined area having the highest calculated correlation value is determined as an official corresponding point, and a motion vector is calculated from the determined corresponding point and the base point in the first frame image. In this manner, a motion vector is calculated for each pixel in the first frame image, and an optical flow (motion vector) for estimating the three-dimensional structure of an object existing in a certain frame image is calculated (for example, (See Patent Document 1).

  Here, using FIGS. 9-1 and 9-2, optical images of two temporally continuous frame images (here, a first frame image and a second frame image) are obtained by a conventional optical flow calculation method. A case where a flow is calculated will be described. FIG. 9A is an explanatory diagram of a first frame image showing a conventional optical flow calculation method. In FIG. 9A, the first frame image 910 is composed of an attention point 911 and a template 912. The attention point 911 may be located at an arbitrary coordinate (x1, y1), and may be a base point of a motion vector in the first frame image 910 and a second frame image 920 described later. Further, the template 912 is obtained by cutting out a minute region centered on the coordinates (x1, y1) of the point of interest 911, and a smaller size is required in order to obtain a fine optical flow.

  Next, FIG. 9-2 will be described. FIG. 9-2 is an explanatory diagram relating to a second frame image showing a conventional optical flow calculation method. 9-2, the second frame image 920 includes a corresponding point 921, a minute area 922 at the corresponding point 921, and a search area 923. The corresponding point 921 is located at the coordinates (x2, y2) corresponding to the attention point 911 in the search area 923. The search area 923 is an area centered on the coordinates (x1, y1) of the point of interest 911 and includes a micro area 922 centered on the coordinates (x2, y2) of the corresponding point 921.

  The optical flow is calculated by first calculating a correlation value between the template 912 in the first frame image 910 and the minute region 922 centered on the coordinates (x2, y2) in the second frame image 920. More specifically, for example, the RMS (Root Mean Square) of the difference between the pixel values of the template 912 of the first frame image 910 and the minute region 922 around the coordinates (x2, y2) of the second frame image 920 is used. Calculate the root mean square)). Then, a correlation map is created while performing a raster scan in the search area 923, and the coordinate (x2max, y2max) where the correlation value is the largest is set as the corresponding point 921 of the coordinate (x1, y1). From these two corresponding coordinates, a motion vector at the coordinates (x1, y1) is calculated as (x2max−x1, y2max−y1), and this is calculated for each pixel in the first frame image 910 to obtain an optical flow. be able to.

JP-A-10-91793

  However, in the above-described prior art, when the movement of the object in the image is fast, the moving distance of the object between the first frame image and the second frame image becomes long, so the coordinates that are corresponding points of the coordinates (x1, y1) (X2max, y2max) is a position away from the coordinates (x1, y1). Therefore, when the object moves quickly, the narrow search region does not include the coordinates (x2max, y2max) corresponding to the coordinates (x1, y1), and thus there is a problem that an appropriate optical flow cannot be obtained.

  On the other hand, in order to obtain a fine optical flow, the corresponding points in the second frame image must be determined from local features around the point of interest in the first frame image, so the template size is set small. There is a need. Therefore, if the object movement is fast and the search area is wide, the feature amount included in the small template is relatively small, and multiple corresponding point candidates with the same correlation value are detected. There is a problem that a corresponding point cannot be determined and an appropriate optical flow cannot be obtained. In other words, there is a problem that the faster the movement of the object, the more accurate the optical flow cannot be obtained and the three-dimensional structure of the object cannot be estimated accurately.

  In order to eliminate the above-described problems caused by the conventional technology, the present invention calculates an optical flow easily and with high accuracy using a motion blur function obtained by a frame image restoration process to estimate the three-dimensional structure of an object. It is an object of the present invention to provide an optical flow calculation device, an optical flow calculation method, an optical flow calculation program, and a recording medium that can achieve high quality.

  In order to solve the above-described problems and achieve the object, an optical flow calculation device, an optical flow calculation method, an optical flow calculation program, and a recording medium according to the present invention are provided in an arbitrary sequence of frame images input from a camera. An arbitrary attention point is extracted from the first frame image, and a motion blur function related to the extracted attention point is calculated by image restoration processing related to the first frame image. Based on a blur function, an intra-frame motion vector of the point of interest in the first frame image is calculated, and based on the calculated intra-frame motion vector and the integration time and frame interval time of the camera, Between the first frame image and the second frame image next to the first frame image. The inter-frame motion vector of the target point is calculated, the motion vector between the calculated frame and determining the optical flow of the point of interest that.

  According to the present invention, the inter-frame motion vector between the second frame images at the point of interest can be calculated and the optical flow can be determined only by using the first frame image.

  Further, in the above invention, corresponding point coordinates corresponding to the attention point are extracted from the second frame image by extracting a corresponding point candidate corresponding to the attention point and based on the calculated inter-frame motion vector. The corresponding point corresponding to the attention point is determined based on the extracted corresponding point candidate and the estimated corresponding point coordinates, and the optical flow of the attention point is determined based on the determined corresponding point. May be determined.

  According to the present invention, the corresponding point corresponding to the target point is determined based on the corresponding point candidate extracted from the second frame image and the corresponding point coordinates estimated from the first frame image. It is possible to determine the optical flow of the attention point with respect to accuracy.

  According to the optical flow calculation device, the optical flow calculation method, the optical flow calculation program, and the recording medium according to the present invention, the optical flow at the point of interest can be determined by the inter-frame motion vector, so that the optical flow can be calculated easily and with high accuracy. Thus, there is an effect that the quality of the three-dimensional structure estimation of the object can be improved.

  Exemplary embodiments of an optical flow calculation device, an optical flow calculation method, an optical flow calculation program, and a recording medium according to the present invention will be described below in detail with reference to the accompanying drawings.

(Embodiment 1)
(Outline of optical flow calculation)
First, the outline of optical flow calculation according to the embodiment of the present invention will be described with reference to FIGS. FIG. 1 is an explanatory diagram showing an outline of an intra-frame motion vector according to an embodiment of the present invention.

  In FIG. 1, an object image 101 in a frame image input by an imaging unit (not shown) is recorded including a blur in the direction of an arrow 102. More specifically, for example, the image 101 of the object in the frame image input by the imaging unit has the luminance value of each pixel integrated for a predetermined time, and thus blurs even in the single frame image. Will be recorded.

  The intra-frame motion blur function 104 is a convolution (convolution integration) of the PSF 105 (Point Spread Function) and the intra-frame motion vector 106. Here, the PSF 105 is provided in advance by the imaging unit or the like, and may be configured based on the characteristics of the imaging unit. The intra-frame motion vector 106 is a motion vector of the object at a predetermined time when the luminance value of each pixel in the frame image is integrated by the imaging unit.

  The restored image 103 is a true object image or the like obtained when the object in the frame image is stationary. The restored image 103 is obtained by performing image restoration processing on the image 101 of the object in the frame image including blur. 103 and an intra-frame motion blur function 104 can be obtained.

  Next, an outline of calculation of the intra-frame motion vector 106 will be described. Assuming that the coordinates in the image 101 of the object in the frame image input by the imaging unit or the like is g (x, y), g (x, y) is f (x, y) and the object in the frame image is stationary. The true object image (restored image 103) obtained in the case of the above, h (x, y) can be represented by the following equation (1), with the intra-frame motion blur function 104 and n (x, y) as noise. .

    g (x, y) = f (x, y) * h (x, y) + n (x, y) (1)

  Here, “*” represents convolution (convolution integration), and f (x, y) and h (x, y) can be obtained from g (x, y) by a known image restoration process. it can. The outline of the procedure of image restoration processing by the AD (Ayer's Dainty) method will be described below.

Image restoration processing by the AD method is performed by the following procedure (1) to procedure (11).
Procedure (1): G (u, v) = FFT [g (x, y)]
Procedure (2): F ′ (u, v) = FFT [f ′ (x, y)] (initial f ′ (x, y) is a random image)
Procedure (3): H (u, v) = G (u, v) / (F ′ (u, v) + sn) (where sn is S / N in the imaging unit)
Procedure (4): h (x, y) = IFFT [H (u, v)]
Procedure (5): If h (x, y)> 0, h ′ (x, y) = h (x, y), otherwise h ′ (x, y) = 0
Procedure (6): H ′ (u, v) = FFT [h ′ (x, y)]
Procedure (7): F (u, v) = G (u, v) / (H ′ (u, v) + sn)
Procedure (8): f (x, y) = IFFT [F (u, v)]
Procedure (9): If f (x, y)> 0, f ′ (x, y) = f (x, y), otherwise f ′ (x, y) = 0
Procedure (10): Procedure (2) to procedure (9) are repeated a predetermined number of times.
Procedure (11): In each loop, f ′ (x, y) and h ′ (x, y) that minimize the Fourier error shown in the following equation (2) are respectively restored image 103 and intra-frame motion blur function 104. And

    Error = Σ | G (u, v) −F ′ (u, v) × H ′ (u, v) | (2)

  The intra-frame motion blur function h ′ (x, y) obtained by the above-described procedure (1) to procedure (11) is obtained by convolving the PSF 105 and the intra-frame motion vector 106 that the imaging unit or the like has in advance. Assuming that PSF 105 is psf (x, y) and intra-frame motion vector 106 is m (x, y), it can be expressed by the following equation (3).

    h ′ (x, y) = psf (x, y) * m (x, y) (3)

  From this equation (3), by deconvoluting h ′ (x, y) using known psf (x, y), the intra-frame motion vector 106 within the integration time in the image 101 of the object in the frame image. Can be calculated.

  Next, an outline of calculation of the inter-frame motion vector according to the embodiment of the present invention will be described with reference to FIGS. FIGS. 2-1 is explanatory drawing which shows the 1st frame image concerning embodiment of this invention. FIGS. FIG. 2-2 is an explanatory diagram showing a second frame image according to the embodiment of the present invention.

  2A, a first frame image 210 input to the optical flow calculation device is a frame image obtained by capturing a plurality of relatively moving rectangular objects, and includes a plurality of object images 214 to 216. Yes. More specifically, the relative movement is a movement of at least one of an imaging device and an object to be imaged. 2A, object images 214 to 216 have a PSF and an intra-frame motion vector in the frame image, for example, as described above with reference to FIG. Accordingly, the first frame image 210 has a shape similar to a hexagon instead of a rectangle due to the motion vector in the frame from the upper right to the lower left as illustrated. Further, the object images 214 to 216 have their respective centers as attention points 211 to 213. In FIG. 2B, the second frame image 220 input to the optical flow calculation device is a frame image continuous with the first frame image 210 described above with reference to FIG. 2-1, and corresponds to the object images 214 to 216. Image 224 to 226 of the object to be performed. In FIG. 2B, the object images 224 to 226 have a PSF and an intraframe motion vector in the frame image, for example, similar to the object images 224 to 226 described above. It has a shape. In addition, the object images 224 to 226 have the respective centers as corresponding point candidates 221 to 223.

  Next, an inter-frame motion vector between the first frame image 210 and the second frame image 220 in the optical flow calculation apparatus according to the embodiment of the present invention will be described with reference to FIG. FIGS. 2-3 is explanatory drawing which shows the inter-frame motion vector concerning Embodiment of this invention.

  FIG. 2-3 shows the first frame image 210 and the second frame image 220 described above on the same drawing, and the attention points 211 to 213 are inter-frame motions based on the attention points 211 to 213, respectively. Vectors 231 to 233 are included. These inter-frame motion vectors 231 to 233 are calculated based on the intra-frame motion vectors of the attention points 211 to 213 in the first frame image 210 calculated by the intra-frame motion vector calculation described above with reference to FIG. Can do.

  More specifically, for example, since the frame interval time between the first frame image 210 and the second frame image 220 is very short (for example, 33 milliseconds), the motion of the object is assumed to be constant velocity motion. be able to. The inter-frame motion vectors 231 to 233 that are the movements of the attention points 211 to 213 from the first frame image 210 to the second frame image 220 are the intra-frame motion vectors of the attention points 211 to 213 in the first frame image 210. Multiply by (frame interval time / integration time).

  Here, the optical flow calculation by the optical flow calculation apparatus according to the embodiment of the present invention will be described with reference to FIGS. 3A and 3B. The optical flow according to the embodiment of the present invention is calculated using the inter-frame motion vector and the correlation map described with reference to FIGS. FIG. 3A is an explanatory diagram of extraction of corresponding point candidates according to the embodiment of the present invention. 3A will be described using the attention point 211 and the corresponding point candidates 221 to 223 described above with reference to FIGS. 2-1 to 2-3.

  3A, the correlation map 300 indicates correlation peaks 301 to 303 of correlation values regarding the corresponding point candidates 221 to 223 in the vertical direction in a search area 310 centered on the coordinates of the point of interest 211. For example, the correlation map 300 is created by calculating a correlation value that is equal to or greater than a predetermined value with a pixel value of a template obtained by cutting out a minute region including the point of interest 211 while raster scanning the search region 310. The structure which extracts the correlation peaks 301-303 regarding H.223 may be sufficient. For example, although not shown, the template is a rectangular area centered on the point of interest 211.

  More specifically, the correlation values at the correlation peaks 301 to 303 include the pixel value of the template including the attention point 211 in the first frame image 210 and the corresponding point candidate 221 in the search region 310 in the second frame image 220. It can be obtained by calculating the RMS (root mean square) of the difference between the pixel values with the correlation value of the pixel values in the minute region centered at ˜223.

  Here, the calculation of the correlation peaks 301 to 303 may not reflect the exact correlation with the template at the point of interest 211 due to the influence of noise or the like. In FIG. 3A, the maximum peak among the correlation peaks 301 to 303 is the correlation peak 302 in the corresponding point candidate 222. If the corresponding point candidate 222 having the maximum correlation value among the correlation peaks 301 to 303 with respect to the template at the point of interest 211 is the corresponding point corresponding to the point of interest 211, the dotted line 304 becomes the optical flow of the point of interest 211.

  On the other hand, the end point of the inter-frame motion vector 231 based on the attention point 211 described above with reference to FIGS. 2-1 to 2-3 is the corresponding point candidate 221. According to the embodiment of the present invention, the configuration may be such that the corresponding point candidate 221 close to the inter-frame motion vector 231 is determined as the corresponding point corresponding to the attention point 211 from among the correlation peaks 301 to 303. Therefore, in FIG. 3A, the corresponding point candidate 221 is determined as the corresponding point corresponding to the attention point 211, and the inter-frame motion vector 231 becomes the optical flow of the attention point 211. In the description of FIG. 3A, the end point of the inter-frame motion vector 231 and the corresponding point candidate 221 are described as the same position. However, the corresponding point candidates 221 to 223 near the end point of the inter-frame motion vector 231 are noted. It is good also as a structure determined to the corresponding point corresponding to the point 211. FIG.

  Next, determination of corresponding points by the optical flow calculation apparatus according to the embodiment of the present invention will be described with reference to FIG. 3-2 is explanatory drawing shown about determination of the corresponding point concerning embodiment of this invention.

  3-2, attention points 211 to 213 are base points of the inter-frame motion vectors 231 to 233 described above with reference to FIGS. 2-1 to 2-3. In FIG. 3B, the end point of the inter-frame motion vector with the attention point 211 as the base point is a corresponding point candidate 221. The end point of the inter-frame motion vector with the attention point 212 as a base point is a corresponding point candidate 222. Similarly, the end point of the inter-frame motion vector with the attention point 213 as a base point is a corresponding point candidate 223. In FIG. 3B, the end points of the inter-frame motion vectors 231 to 233 and the corresponding point candidates 221 to 223 are described as the same position, but the corresponding point candidates 221 to 223 are extracted by creating a correlation map. It may be a thing, and it is good also as being in the end point vicinity of the inter-frame motion vectors 231-233.

  Here, the search area 310 centered on the coordinates of the point of interest 211 includes the corresponding point candidates 221 to 223, and the corresponding point candidates 221 to 223 having the highest correlation value with the point of interest 211 are shown in FIG. As described above, the corresponding point candidate 222 is located in the direction of the dotted line 304 from the attention point 211. On the other hand, the point of interest 211 in the first frame image 210 is assumed to have moved in the direction of the interframe motion vector 231 in the second frame image 220. Therefore, in FIG. 3B, although the corresponding point candidate 222 has a high correlation value with respect to the attention point 211, the corresponding point candidate 222 shows the maximum peak due to noise or the like. decide.

  Similarly, the corresponding point corresponding to the attention point 212 is determined as the corresponding point candidate 222, and the corresponding point corresponding to the attention point 213 is determined as the corresponding point candidate 223. In this embodiment, the optical flow can be calculated based on the relationship between the attention points 211 to 213 and the corresponding points corresponding to the attention points 211 to 213. Note that the attention points 211 to 213 are not limited to three points, and an optical flow may be calculated for each arbitrary pixel.

(Hardware configuration of optical flow calculation device)
Next, the hardware configuration of the optical flow calculation apparatus according to the embodiment of the present invention will be described with reference to FIG. FIG. 4 is a block diagram showing a hardware configuration of the optical flow calculation apparatus according to the embodiment of the present invention.

  In FIG. 4, the optical flow calculation apparatus includes an example of a CPU 401, a ROM 402, a RAM 403, an HDD (hard disk drive) 404, an HD (hard disk) 405, an FDD (flexible disk drive) 406, and a removable recording medium. FD (flexible disk) 407, a display 408, a communication I / F (interface) 409, a keyboard 410, a mouse 411, a video I / F 412, and a camera 413. Each component is connected by a bus 400.

  Here, the CPU 401 controls the entire optical flow calculation apparatus. The ROM 402 stores programs such as a boot program. The RAM 403 is used as a work area for the CPU 401. The HDD 404 controls reading / writing of data with respect to the HD 405 according to the control of the CPU 401. The HD 405 stores data written under the control of the HDD 404.

  The FDD 406 controls reading / writing of data with respect to the FD 407 according to the control of the CPU 401. The FD 407 stores data written under the control of the FDD 406, or causes the optical flow calculation device to read data stored in the FD 407.

  In addition to the FD 407, the removable recording medium may be a CD-ROM (CD-R, CD-RW), MO, DVD (Digital Versatile Disk), memory card, or the like. The display 408 displays data such as a document, an image, and function information as well as a cursor, an icon, or a tool box. As the display 408, for example, a CRT, a TFT liquid crystal display, a plasma display, or the like can be adopted.

  The communication I / F 409 is connected to a network 414 such as the Internet through a communication line, and is connected to other devices via the network 414. The communication I / F 409 controls an internal interface with the network 414 and controls input / output of data from an external device. As the communication I / F 409, for example, a modem or a LAN adapter can be employed.

  The keyboard 410 includes keys for inputting characters, numbers, various instructions, and the like, and inputs data corresponding to the pressed key into the apparatus. Moreover, a touch panel type input pad or a numeric keypad may be used. Also, by operating the mouse 411, the output of the sensor for detecting the movement of the lower body of the mouse 411 and the ON / OFF of each button on the upper body are inputted into the apparatus as needed to move the cursor, select a range, or You may move the window or change its size. Note that a trackball or a joystick may be used as long as the pointing device has the same function.

  The video I / F 412 is connected to the camera 413. The video I / F 412 includes a control IC that controls the entire camera 413 and a buffer memory such as a VRAM (Video RAM) that temporarily records a frame image captured by the camera 413. Further, the buffer memory may have a configuration having a plurality of frame buffers. For example, two frame buffers may alternately store frame images taken by a camera 413 described later. And the structure which controls the input / output to each structure of an optical flow calculation apparatus about the frame image image | photographed with the camera 413 may be sufficient.

  The camera 413 captures a subject as a series of continuous frame images. The camera 413 may be either monochrome or color, and is configured by a video camera or the like, and includes a photoelectric conversion element such as a C-MOS (Complementary Metal Oxide Semiconductor) or a CCD (Charge Coupled Device).

  The camera 413 is connected to the video I / F 412 via USB (Universal Serial Bus) or IEEE 1394 (Institut of Electrical and Electronic Engineers 1394), and the captured frame images are sequentially transmitted via the video I / F 412. It is good also as a structure output to a flow calculation apparatus.

(Functional configuration of optical flow calculation device)
Next, the functional configuration of the optical flow calculation apparatus according to the embodiment of the present invention will be described with reference to FIG. FIG. 5 is a block diagram showing a functional configuration of the optical flow calculation apparatus according to the embodiment of the present invention. In FIG. 5, an optical flow calculation apparatus 500 includes a frame image acquisition unit 501, a first frame image holding unit 502, a template cutout unit 503, a frame image restoration unit 504, an intra-frame motion vector calculation unit 505, a frame Inter-motion vector calculation unit 506, corresponding point coordinate estimation unit 507, optical flow determination unit 508, second frame image holding unit 509, correlation map generation unit 510, corresponding point candidate extraction unit 511, and extraction threshold The value designation unit 512 and the corresponding point determination unit 513 are configured.

  In FIG. 5, the frame image acquisition unit 501 includes an imaging unit 514, an amplification unit 515, and an A / D (analog / digital) conversion unit 516. Image signals of a series of frame images captured by the imaging unit 514 are output to the amplification unit 515. The amplifying unit 515 amplifies the image signal of the input frame image and outputs it to the A / D conversion unit 516. The A / D conversion unit 516 converts the image signal of the input frame image and outputs it as a digital image signal to the first frame image holding unit 502 and the second frame image holding unit 509. Note that the functions of the imaging unit 514, amplification unit 515, and A / D conversion unit 516 in the frame image acquisition unit 501 are specifically realized by, for example, the video I / F 412 and the camera 413 shown in FIG. To do.

  The first frame image holding unit 502 and the second frame image holding unit 509 record frame images of two frames that are continuously input from the frame image acquisition unit 501 as needed. In the description of FIG. 5, the first frame image holding unit 502 holds the first frame image 210, and the second frame image holding unit 509 holds the second frame image 220 that is continuous with the first frame image 210. It is a configuration. The first frame image holding unit 502 and the second frame image holding unit 509 are specifically configured by, for example, a buffer memory in the video I / F 412 shown in FIG. 4, HD 405, RAM 403, or other memory. Realize its function.

  The template cutout unit 503 cuts out a template centered on the coordinates of the point of interest 211 in the first frame image 210 held by the first frame image holding unit 502. The template is, for example, a rectangular region including the object image 214 with the attention point 211 as the center. In the description of FIG. 5, the attention point 211 is described. However, the attention point 211 is not limited thereto. For example, the template cutout unit 503 extracts an arbitrary point from the first frame image 210 as the attention point 211. A template may be cut out.

  The frame image restoration unit 504 performs image restoration processing on the template cut out by the template cutout unit 503, and separates it into a restored image and an intra-frame motion blur function. The image restoration processing may be, for example, a method based on the above-described AD method, and may be configured to perform image restoration processing a predetermined number of times and separate the restored image and the intra-frame motion blur function that minimize the Fourier error.

  The intra-frame motion vector calculation unit 505 performs the frame of the attention point 211 in the first frame image 210 based on the intra-frame motion blur function separated by the frame image restoration unit 504 and the blur function of the frame image acquisition unit 501. An internal motion vector is calculated. More specifically, referring to FIG. 1, the intra-frame motion blur function 104 separated by the frame image restoration unit 504 includes the PSF 105 (Point Spread Function) of the frame image acquisition unit 501 and the intra-frame motion vector. 106 is a convolution. Therefore, the intra-frame motion vector calculation unit 505 deconvolves the intra-frame motion blur function 104 using the PSF 105 of the frame image acquisition unit 501 to calculate the intra-frame motion vector 106.

  The inter-frame motion vector calculation unit 506 calculates an inter-frame motion vector based on the intra-frame motion vector calculated by the intra-frame motion vector calculation unit 505, the integration time, and the frame interval time. The integration time is a predetermined time for integrating the luminance values in the frame image. More specifically, referring to FIGS. 2-1 to 2-3, the luminance value of each pixel in the first frame image 210 is integrated. It is time to do. The frame interval time is the time between two consecutive frame images. More specifically, referring to FIGS. 2-1 to 2-3, the first frame image 210 and the second frame image 220 It is a time interval. The integration time and the frame interval time depend on the characteristics of the imaging unit 514, and the calculation of the inter-frame motion vector may be performed by multiplying the intra-frame motion vector by (frame interval time / integration time).

  The corresponding point coordinate estimation unit 507 estimates the corresponding point coordinates that are the corresponding points corresponding to the point of interest 211 based on the interframe motion vector calculated by the interframe motion vector calculation unit 506. More specifically, for example, the inter-frame motion vector is added to the coordinates of the point of interest 211 used when the template cutout unit 503 cuts out the template, and the corresponding point coordinates are estimated.

  The optical flow determination unit 508 determines the inter-frame motion vector calculated by the inter-frame motion vector calculation unit 406 as the optical flow of the point of interest 211. Further, the optical flow determination unit 508 may be configured to determine the optical flow of the attention point 211 based on the corresponding points determined by the corresponding point determination unit 513. More specifically, for example, the inter-frame motion vector of the target point 211 is calculated from the target point 211 and the corresponding point, and the optical flow of the target point 211 is determined. In the description of FIG. 5, the attention point 211 has been described. However, by determining the optical flow with the other coordinates of the first frame image 210 as the attention point, the optical for each pixel in the first frame image 210 is determined. The structure which acquires a flow may be sufficient.

  The correlation map generation unit 510 includes, in the second frame image 220 held by the second frame image holding unit 509, each region in the search region 310 centered on the template attention point 211 cut out by the template cutout unit 503. The correlation map 300 is generated by performing the correlation process. More specifically, for example, while performing a raster scan in the search area 310, a correlation value with a pixel value of a template including the attention point 211 is calculated, and a correlation value distribution map is generated. The correlation value can be obtained, for example, by calculating the RMS (root mean square) of the difference between the pixel value of the template and each pixel value in the search area 310.

  The corresponding point candidate extraction unit 511 extracts the corresponding point candidates 221 to 223 from the correlation values in the correlation map 300 using the threshold value specified by the extraction threshold value specifying unit 512 described later. More specifically, for example, in the correlation map 300, corresponding point candidates 221 to 223 of the correlation peaks 301 to 303 having a correlation value equal to or greater than a threshold value are extracted.

  The extraction threshold value designation unit 512 designates a correlation value threshold value for extracting corresponding point candidates 221 to 223 for the point of interest 211 in the correlation map 300. The threshold value may be any value, for example, and may be configured to be set by the user.

  The corresponding point determination unit 513 selects the corresponding point candidate 221 closest to the corresponding point coordinates estimated by the corresponding point coordinate estimation unit 507 from the corresponding point candidates 221 to 223 extracted by the corresponding point candidate extraction unit 511. The corresponding point corresponding to 211 is determined.

  Note that the template cutout unit 503, the frame image restoration unit 504, the intraframe motion vector calculation unit 505, the interframe motion vector calculation unit 506, the corresponding point coordinate estimation unit 507, the optical flow determination unit 508, and the correlation map generation described above with reference to FIG. Specifically, the unit 510, the corresponding point candidate extracting unit 511, the extraction threshold value specifying unit 512, and the corresponding point determining unit 513 are recorded in a recording medium such as the ROM 402, the RAM 403, and the HD 405 shown in FIG. When the CPU 401 executes the program, the function is realized by the communication I / F 409, the keyboard 410, and the mouse 411.

(Contents of processing of optical flow calculation apparatus 500)
Next, the contents of the process of the optical flow calculation apparatus 500 according to the embodiment of the present invention will be described with reference to FIG. FIG. 6 is a flowchart showing the contents of the process of the optical flow calculation apparatus according to the embodiment of the present invention. In the flowchart of FIG. 6, first, the optical flow calculation apparatus 500 determines whether or not an input of a frame image acquired by the frame image acquisition unit 501 has been received (step S601).

  In step S601, waiting for the input of the frame image, and if received (step S601: Yes), the first frame image holding unit 502 and the second frame image holding unit 509 receive two consecutive frame images. Is recorded (step S602). For recording the frame image, for example, the first frame image holding unit 502 holds the first frame image 210 and the second frame image holding unit 509 holds the second frame image 220. In addition, the frame image may be retained by repeatedly overwriting and erasing the frame image that is successively input for the necessary capacity.

  Next, the optical flow calculation device 500 calculates optical flow calculation coordinates based on the frame image recorded in step S602 (step S603). The optical flow calculation coordinates may be calculated by calculating the coordinates of an arbitrary point in the first frame image 210 held in the first frame image holding unit 502. Specifically, in the present embodiment, the attention point 211 is calculated. The coordinates of are calculated.

  Subsequently, the optical flow calculation device 500 executes a corresponding point coordinate estimation process corresponding to the optical flow calculation coordinates calculated in step S603 (step S604). Corresponding point coordinate estimation processing is, for example, a configuration that is performed based on the inter-frame motion vector related to the point of interest 211 in the first frame image 210 held in the first frame image holding unit 502. Details of the corresponding point coordinate estimation processing Will be described later with reference to FIG.

  Then, the optical flow calculation apparatus 500 executes a corresponding point candidate extraction process corresponding to the optical flow calculation coordinates calculated in step S603 (step S605). For example, the corresponding point candidate extraction processing may be performed based on the correlation map 300 related to the point of interest 211 in the second frame image 220 held in the second frame image holding unit 509. Details of the corresponding point candidate extraction processing Will be described later with reference to FIG.

  In the flowchart of FIG. 6, the corresponding point coordinate estimation process is executed in step S604 and the corresponding point candidate extraction process in step S605 is executed. However, the order of the processes is not limited to this, for example, The corresponding point candidate extraction process may be executed first, and then the corresponding point coordinate estimation process may be executed, or the corresponding point coordinate estimation process and the corresponding point candidate extraction process may be executed simultaneously.

  Next, a corresponding point is determined by the corresponding point determination unit 513 based on the corresponding point coordinates estimated in step S604 and the corresponding point candidate extracted in step S605 (step S606). More specifically, the corresponding point is determined by, for example, selecting the corresponding point candidate 221 closest to the corresponding point coordinates estimated in step S604 from the corresponding point candidates 221 to 223 extracted in step S605 as the attention point 211. The corresponding corresponding point is determined.

  Subsequently, the optical flow calculation device 500 calculates an optical flow (step S607). The optical flow is calculated based on the corresponding points corresponding to the attention point 211 determined in step S606, for example. More specifically, the optical flow determination unit 508 calculates the inter-frame motion vector of the attention point 211 from the attention point 211 and the corresponding point determined in step S606, and determines the optical flow of the attention point 211. .

  In the flowchart of FIG. 6, in step S607, the inter-frame motion vector of the target point 211 is calculated from the target point 211 and the corresponding point determined in step S606, and the optical flow of the target point 211 is determined. However, the optical flow may be determined based on the inter-frame motion vector calculated in the corresponding point coordinate estimation process in step S604 described later in FIG. In that case, the optical flow can be determined only by the first frame image without executing the corresponding point candidate extraction process using the second frame image in step S605.

  Then, the optical flow calculation apparatus 500 determines whether or not calculation of all optical flows has been completed (step S608). More specifically, for example, the optical flow determination unit 508 ends the determination of the optical flow for each pixel in the first frame image 210 with the other coordinates of the attention point 211 in the first frame image 210 as the attention point. Determine whether or not.

  In step S608, when calculation of all the optical flows has not been completed (step S608: No), the process returns to step S603 and is repeated. In step S608, when calculation of all optical flows has been completed (step S608: Yes), the series of processing ends.

  Next, the contents of the corresponding point coordinate estimation process (step S604 in FIG. 6) in the optical flow calculation apparatus 500 according to the embodiment of the present invention will be described with reference to FIG. FIG. 7 is a flowchart showing the contents of the corresponding point coordinate estimation process (step S604 in FIG. 6) according to the embodiment of the present invention.

  In the flowchart of FIG. 7, first, the template cutout unit 503 cuts out a template based on the optical flow calculation coordinates calculated in step S603 in FIG. 6 (step S701). For example, as described above, the template may be cut out as a template in a predetermined area centered on the coordinates of the attention point 211 calculated in step S603.

  Next, the frame image restoration unit 504 performs image restoration processing on the template cut out in step S701 (step S702). The image restoration processing may be, for example, a method based on the above-described AD method, and may be configured to perform image restoration processing a predetermined number of times and separate the restored image and the intra-frame motion blur function that minimize the Fourier error.

  Subsequently, the intra-frame motion vector calculation unit 505 calculates an intra-frame motion vector based on the intra-frame motion blur function separated in step S702 and the blur function of the frame image acquisition unit 501 (step S703). More specifically, for example, the intra-frame motion blur function is deconvoluted by the blur function of the frame image acquisition unit 501 to calculate the intra-frame motion vector of the point of interest 211 in the first frame image 210.

  Then, the inter-frame motion vector calculation unit 506 calculates an inter-frame motion vector based on the intra-frame motion vector calculated in step S703, the integration time, and the frame interval time (step S704). More specifically, the inter-frame motion vector may be calculated by multiplying the intra-frame motion vector by (frame interval time / integration time).

  Next, the corresponding point coordinate estimation unit 507 estimates the corresponding point coordinates as the corresponding point corresponding to the point of interest 211 based on the inter-frame motion vector calculated in step S704 (step S705). Specifically, for example, as described above, the corresponding point coordinates are estimated by adding the inter-frame motion vector to the coordinates of the point of interest 211 cut out of the template in step S701. And the corresponding point coordinate estimation process of step S604 in FIG. 6 is complete | finished, and it transfers to step S605.

  Furthermore, the contents of the corresponding point candidate extraction process (step S605 in FIG. 6) in the optical flow calculation apparatus 500 according to the embodiment of the present invention will be described with reference to FIG. FIG. 8 is a flowchart showing the contents of the corresponding point candidate extraction process (step S605 in FIG. 6) according to the embodiment of the present invention.

  In the flowchart of FIG. 8, first, correlation processing coordinates are calculated by the correlation map generation unit 510 (step S801). For example, as described above, the correlation processing coordinates may be calculated by calculating the coordinates for performing the correlation processing with the attention point 211 in the second frame image 220, and within the search region 310 centered on the coordinates of the attention point 211. It is good also as arbitrary coordinates.

  Subsequently, the correlation map generation unit 510 extracts a correlation region based on the correlation processing coordinates calculated in step S801 (step S802) and calculates a correlation value in the correlation region (step S803). For example, as described above, the correlation region may be cut out by cutting out a minute region centered on the correlation processing coordinates. The RMS (square) of the difference between the pixel value of the template at the target point 211 and the pixel value in the correlation region may be used. Average square root) may be calculated as the correlation value.

  Then, the correlation map generation unit 510 determines whether or not the generation of the correlation map 300 has been completed (step S804). More specifically, for example, it is determined whether or not the calculation of the correlation value is completed while performing a raster scan in the search area 310. Here, when the generation of the correlation map 300 is not completed (step S804: No), the process returns to step S801 and the process is repeated. When the generation of the correlation map 300 is completed (step S804: Yes), the corresponding threshold value candidate extraction unit 511 sets the extraction threshold value specified by the extraction threshold value specification unit 512 for the correlation map 300. Read (step S805).

  Next, the corresponding point candidate extraction unit 511 extracts the corresponding point candidates 221 to 223 from the correlation values in the correlation map 300 using the extraction threshold value read in step S805 (step S806). More specifically, for example, in the correlation map 300, corresponding point candidates 221 to 223 of the correlation peaks 301 to 303 having a correlation value equal to or greater than a threshold value are extracted. Then, the corresponding point candidate extraction process in step S605 in FIG. 6 is terminated, and the process proceeds to step S606.

  As described above, according to the embodiment of the present invention, the inter-frame motion vector of the target point can be calculated, and the optical flow can be determined based on the calculated inter-frame motion vector. Therefore, an optical flow can be obtained simply by using a single frame image, and even if the object moves at high speed, the optical flow is calculated without depending on the next frame image, and the three-dimensional structure of the object is estimated. The quality can be improved.

  Corresponding point coordinates are estimated by corresponding point coordinate estimation processing. Then, from the corresponding point candidates extracted by the corresponding point candidate extraction process, the one close to the estimated corresponding point coordinates is determined as the corresponding point, and the optical flow can be calculated. Therefore, even if there are multiple regions with similar luminance distributions in the search region, the exact corresponding points are determined, the optical flow is calculated easily and with high accuracy, and the quality of the three-dimensional structure estimation of the object is improved. Can be planned.

  As described above, according to the present invention, since the optical flow at the point of interest can be determined by the inter-frame motion vector, the optical flow can be calculated easily and with high accuracy to improve the quality of the three-dimensional structure estimation of the object. Can be planned.

  The optical flow calculation method described in this embodiment can be realized by executing a program prepared in advance on a computer such as a personal computer or a workstation. This program is recorded on a computer-readable recording medium such as a hard disk, a flexible disk, a CD-ROM, an MO, and a DVD, and is executed by being read from the recording medium by the computer. The program may be a transmission medium that can be distributed via a network such as the Internet.

(Supplementary note 1) Attention point extraction means for extracting an arbitrary attention point from an arbitrary first frame image in a series of frame images input from the camera;
A motion blur function calculating unit that calculates a motion blur function related to the target point extracted by the target point extracting unit by the image restoration processing related to the first frame image;
Intraframe motion vector calculation means for calculating an intraframe motion vector of the point of interest in the first frame image based on the motion blur function calculated by the motion blur function calculation means and the blur function of the camera. When,
Based on the intra-frame motion vector calculated by the intra-frame motion vector calculating means, and the integration time and frame interval time of the camera, the first frame image and the second second after the first frame image. An inter-frame motion vector calculating means for calculating an inter-frame motion vector of the point of interest between the frame image of
Optical flow determination means for determining the inter-frame motion vector calculated by the inter-frame motion vector calculation means as the optical flow of the target point;
An optical flow calculation device comprising:

(Supplementary Note 2) Corresponding point candidate extracting means for extracting a corresponding point candidate corresponding to the attention point from the second frame image;
Corresponding point coordinate estimating means for estimating corresponding point coordinates that become corresponding points corresponding to the point of interest based on the inter-frame motion vector calculated by the inter-frame motion vector calculating means;
A corresponding point determining unit that determines a corresponding point corresponding to the point of interest based on the corresponding point candidate extracted by the corresponding point candidate extracting unit and the corresponding point coordinate estimated by the corresponding point coordinate estimating unit; With
The optical flow determining means includes
The optical flow calculation apparatus according to appendix 1, wherein the optical flow of the attention point is determined based on the corresponding points determined by the corresponding point determination means.

(Supplementary Note 3) An attention point extraction step of extracting an arbitrary attention point from an arbitrary first frame image in a series of frame images input from the camera;
A motion blur function calculating step of calculating a motion blur function related to the attention point extracted by the attention point extraction step by the image restoration processing related to the first frame image;
An intra-frame motion vector calculation step of calculating an intra-frame motion vector of the point of interest in the first frame image based on the motion blur function calculated by the motion blur function calculation step and the blur function of the camera When,
Based on the intra-frame motion vector calculated by the intra-frame motion vector calculation step and the integration time and frame interval time of the camera, the first frame image and the second second after the first frame image. An inter-frame motion vector calculation step of calculating an inter-frame motion vector of the point of interest between the frame image of
An optical flow determination step of determining an inter-frame motion vector calculated by the inter-frame motion vector calculation step as an optical flow of the attention point;
An optical flow calculation method comprising:

(Supplementary Note 4) A corresponding point candidate extracting step of extracting a corresponding point candidate corresponding to the target point from the second frame image;
A corresponding point coordinate estimating step for estimating a corresponding point coordinate to be a corresponding point corresponding to the target point based on the inter-frame motion vector calculated by the inter-frame motion vector calculating step;
A corresponding point determining step for determining a corresponding point corresponding to the point of interest based on the corresponding point candidate extracted by the corresponding point candidate extracting step and the corresponding point coordinate estimated by the corresponding point coordinate estimating step; Including
The optical flow determination step includes
4. The optical flow calculation method according to appendix 3, wherein the optical flow of the attention point is determined based on the corresponding points determined in the corresponding point determination step.

(Supplementary Note 5) An attention point extraction step of extracting an arbitrary attention point from an arbitrary first frame image in a series of frame images input from the camera;
A motion blur function calculating step of calculating a motion blur function related to the attention point extracted by the attention point extraction step by the image restoration processing related to the first frame image;
An intra-frame motion vector calculation step of calculating an intra-frame motion vector of the point of interest in the first frame image based on the motion blur function calculated by the motion blur function calculation step and the blur function of the camera When,
Based on the intra-frame motion vector calculated by the intra-frame motion vector calculation step and the integration time and frame interval time of the camera, the first frame image and the second second after the first frame image. An inter-frame motion vector calculation step of calculating an inter-frame motion vector of the point of interest between the frame image of
An optical flow determination step of determining the inter-frame motion vector calculated by the inter-frame motion vector calculation step as the optical flow of the attention point;
An optical flow calculation program for causing a computer to execute.

(Supplementary Note 6) A corresponding point candidate extracting step of extracting a corresponding point candidate corresponding to the target point from the second frame image;
A corresponding point coordinate estimating step for estimating a corresponding point coordinate to be a corresponding point corresponding to the target point based on the inter-frame motion vector calculated by the inter-frame motion vector calculating step;
A corresponding point determining step for determining a corresponding point corresponding to the point of interest based on the corresponding point candidate extracted by the corresponding point candidate extracting step and the corresponding point coordinate estimated by the corresponding point coordinate estimating step; To the computer,
The optical flow determination step includes
Furthermore, the optical flow calculation program according to appendix 5, wherein the optical flow of the attention point is determined based on the corresponding points determined in the corresponding point determination step.

(Supplementary note 7) A computer-readable recording medium in which the optical flow calculation program according to any one of supplementary notes 4 to 6 is recorded.

  As described above, the optical flow calculation device, the optical flow calculation method, the optical flow calculation program, and the recording medium according to the present invention are used for estimating a three-dimensional structure of an object in an image from a two-dimensionally projected image. It is useful for calculating optical flow, especially when calculating optical flow related to moving images of fast moving objects such as vehicles and airplanes, and for images taken by imaging units installed on high-speed moving platforms such as aircraft. This is suitable for calculating an optical flow related to a moving image.

It is explanatory drawing which shows the outline | summary of the motion vector in a frame concerning embodiment of this invention. It is explanatory drawing which shows the 1st frame image concerning embodiment of this invention. It is explanatory drawing which shows the 2nd frame image concerning embodiment of this invention. It is explanatory drawing which shows the inter-frame motion vector concerning embodiment of this invention. It is explanatory drawing shown about extraction of the corresponding point candidate concerning embodiment of this invention. It is explanatory drawing shown about the determination of the corresponding point concerning embodiment of this invention. It is a block diagram which shows the hardware constitutions of the optical flow calculation apparatus concerning embodiment of this invention. It is a block diagram which shows the functional structure of the optical flow calculation apparatus concerning embodiment of this invention. It is a flowchart which shows the content of the process of the optical flow calculation apparatus concerning embodiment of this invention. It is a flowchart which shows the content of the corresponding point coordinate estimation process (step S604 of FIG. 6) concerning embodiment of this invention. It is a flowchart which shows the content of the corresponding point candidate extraction process (step S605 of FIG. 6) concerning embodiment of this invention. It is explanatory drawing regarding the 1st frame image which shows the conventional optical flow calculation system. It is explanatory drawing regarding the 2nd frame image which shows the conventional optical flow calculation system.

Explanation of symbols

400 bus 401 CPU
402 ROM
403 RAM
404 HDD
405 HD
406 FDD
407 FD
408 Display 409 Communication I / F
410 Keyboard 411 Mouse 412 Video I / F
413 Camera 500 Optical flow calculation device 501 Frame image acquisition unit 502 First frame image holding unit 503 Template cutout unit 504 Frame image restoration unit 505 Intraframe motion vector calculation unit 506 Interframe motion vector calculation unit 507 Corresponding point coordinate estimation unit 508 Optical Flow determining unit 509 Second frame image holding unit 510 Correlation map generating unit 511 Corresponding point candidate extracting unit 512 Extracting threshold value specifying unit 513 Corresponding point determining unit 514 Imaging unit 515 Amplifying unit 516 A / D converting unit

Claims (5)

Attention point extracting means for extracting an arbitrary attention point from an arbitrary first frame image in a series of frame images input from the camera;
A motion blur function calculating unit that calculates a motion blur function related to the target point extracted by the target point extracting unit by the image restoration processing related to the first frame image;
Intraframe motion vector calculation means for calculating an intraframe motion vector of the point of interest in the first frame image based on the motion blur function calculated by the motion blur function calculation means and the blur function of the camera. When,
Based on the intra-frame motion vector calculated by the intra-frame motion vector calculating means, and the integration time and frame interval time of the camera, the first frame image and the second second after the first frame image. An inter-frame motion vector calculating means for calculating an inter-frame motion vector of the point of interest between the frame image of
Optical flow determination means for determining the inter-frame motion vector calculated by the inter-frame motion vector calculation means as the optical flow of the target point;
An optical flow calculation device comprising: Corresponding point candidate extracting means for extracting corresponding point candidates corresponding to the point of interest from the second frame image;
Corresponding point coordinate estimating means for estimating corresponding point coordinates that become corresponding points corresponding to the point of interest based on the inter-frame motion vector calculated by the inter-frame motion vector calculating means;
A corresponding point determining unit that determines a corresponding point corresponding to the point of interest based on the corresponding point candidate extracted by the corresponding point candidate extracting unit and the corresponding point coordinate estimated by the corresponding point coordinate estimating unit; With
The optical flow determining means includes
The optical flow calculation apparatus according to claim 1, wherein the optical flow of the attention point is determined based on the corresponding point determined by the corresponding point determination unit. An attention point extraction step of extracting an arbitrary attention point from an arbitrary first frame image in a series of frame images input from the camera;
A motion blur function calculating step of calculating a motion blur function related to the attention point extracted by the attention point extraction step by the image restoration processing related to the first frame image;
An intra-frame motion vector calculation step of calculating an intra-frame motion vector of the point of interest in the first frame image based on the motion blur function calculated by the motion blur function calculation step and the blur function of the camera When,
Based on the intra-frame motion vector calculated by the intra-frame motion vector calculation step and the integration time and frame interval time of the camera, the first frame image and the second second after the first frame image. An inter-frame motion vector calculation step of calculating an inter-frame motion vector of the point of interest between the frame image of
An optical flow determination step of determining an inter-frame motion vector calculated by the inter-frame motion vector calculation step as an optical flow of the attention point;
An optical flow calculation method comprising: An attention point extraction step of extracting an arbitrary attention point from an arbitrary first frame image in a series of frame images input from the camera;
A motion blur function calculating step of calculating a motion blur function related to the attention point extracted by the attention point extraction step by the image restoration processing related to the first frame image;
An intra-frame motion vector calculation step of calculating an intra-frame motion vector of the point of interest in the first frame image based on the motion blur function calculated by the motion blur function calculation step and the blur function of the camera When,
Based on the intra-frame motion vector calculated by the intra-frame motion vector calculation step and the integration time and frame interval time of the camera, the first frame image and the second second after the first frame image. An inter-frame motion vector calculation step of calculating an inter-frame motion vector of the point of interest between the frame image of
An optical flow determination step of determining the inter-frame motion vector calculated by the inter-frame motion vector calculation step as the optical flow of the attention point;
An optical flow calculation program for causing a computer to execute. A computer-readable recording medium on which the optical flow calculation program according to claim 4 is recorded.

Images