JP4826315B2 - Image processing apparatus and method, and program - Google Patents

Description

  The present invention relates to an image processing apparatus, method, and program, and more particularly, to an image processing apparatus, method, and program that can estimate a current motion from past motion history information in tracking an object.

  Conventionally, there is a method of tracking a designated object by tracking or analyzing an optical flow detected for each area by dividing the screen into a plurality of areas. In the correlation value calculation, for example, a block matching method and a gradient method are well known.

  Regardless of the block matching method, in other motion vector detection methods, for example, a sum of absolute differences between a reference block and a check block is often used as an evaluation method for detecting a motion vector.

  However, in this method, since only the correlation between the reference block and the inspection block is evaluated, a motion vector unrelated to the motion of the object in the actual image is selected, the motion vector in the image is disturbed, Alternatively, motion vector deterioration such as the presence of an isolated motion vector that is uncorrelated with the nearby motion vector occurs.

Therefore, there are indicators using the magnitude of the evaluation value of the correlation value calculation, the roughness of the image, the spatial correlation of the motion vector, or the like as an index representing the certainty of the motion vector. For example, in Patent Document 1, it is possible to determine the likelihood of a motion vector based on whether or not the motion accuracy is equal to or greater than a threshold, and to detect the motion with high accuracy using the accuracy of the motion vector.
Japanese Patent Laid-Open No. 2005-301984

  However, in Patent Document 1, for example, when a tracking point disappears due to an intersection with another object, the foreground object is erroneously tracked when the motion accuracy of the other object that has become the foreground exceeds a threshold value. There is.

  Here, an example in which tracking fails due to an intersection with another object will be described with reference to FIG.

  In the example of FIG. 1, it is assumed that there are an object 301 and an object 302 that move in different directions in the frame n−1, and a tracking point 311 is set for the object 301 as a tracking target.

  In frame n-1, the motion vector of the object 301 is detected as a result of matching between frame n-1 and frame n. In frame n, the tracking point 311 moves to the object 302 that is the foreground of the object 301 due to the intersection of the object 301 and the object 302, and as a result of matching between the frame n and the frame n + 1, the motion vector of the object 302 becomes It will be detected. Furthermore, in the frame n + 1 after the intersection, the tracking point 311 completely moves to the object 302, and the tracking of the object 301 fails.

  As described above, there is a case where the motion vector obtained for tracking the object cannot be evaluated only by the evaluation value of the correlation value calculation or the spatial correlation of the motion vector.

  The present invention has been made in view of such a situation, and makes it possible to estimate a current motion from past motion history information in tracking an object.

An image processing apparatus according to one aspect of the present invention is an image processing apparatus that tracks a moving object, and stores a past motion vector of the object in the image as history information, and is stored in the storage means. A first maximum value and a first minimum value, and a difference between the first maximum value and the first minimum value from among the plurality of past motion vectors based on the history information. A dynamic range is obtained, a second maximum value obtained by adding the dynamic range to the first maximum value, a second minimum value obtained by subtracting the dynamic range from the first minimum value, and the second maximum value obtained. Range setting means for setting the maximum value and the second minimum value as the range of the predicted motion vector of the current frame, and the motion vector estimated in the current frame of the object Is outside the range of the prediction vector, it is determined that the current motion of the object cannot be estimated, and if it is within the range of the prediction vector, the current motion of the object can be estimated. Judgment means for judging, and extrapolation means for reading out and extrapolating the past motion vector stored in the storage means when the judgment means judges that estimation is impossible .

The range setting means sets a value obtained by adding a threshold value to the second maximum value to prevent the predicted motion vector range from becoming 0 when the dynamic range becomes 0. A value obtained by subtracting the threshold value from the second maximum value may be set as the second minimum value .

When the past motion vector is extrapolated continuously by the extrapolation means , the current motion vector of the object can be set to 0 vector.

The extrapolation means further includes count means for counting the number of times the past motion vector has been extrapolated, and when the judgment means judges that the estimation is possible, or the value of the counter by the counter means is a threshold value When it is determined that the counter value is exceeded, the counter value by the counter means can be initialized .

The image processing apparatus further includes detection means for detecting a pause of the image, and when the detection means detects the pause of the image, the range setting means sets the range of the prediction vector to a large value. Can do.

An image processing method according to an aspect of the present invention is an image processing method of an image processing apparatus that tracks a moving object, and stores past motion vectors of the object in an image as history information, and the stored Based on the history information, among the plurality of past motion vectors, a first maximum value and a first minimum value, and a dynamic range that is a difference between the first maximum value and the first minimum value. And obtaining a second maximum value obtained by adding the dynamic range to the first maximum value and a second minimum value obtained by subtracting the dynamic range from the first minimum value, and obtaining the second maximum value. And the second minimum value as a range of the predicted motion vector of the current frame, and the motion vector estimated in the current frame of the object is If it is out of bounds, it is determined that the current motion of the object cannot be estimated, and if it is within the range of the prediction vector, it is determined that the current motion of the object can be estimated and cannot be estimated. If it is determined that the stored motion vector is read and extrapolated .

A program according to one aspect of the present invention is a program that causes a computer to execute image processing of an image processing apparatus that tracks a moving object, and stores past motion vectors of the object in an image as history information. Based on the history information, a first maximum value and a first minimum value, and a difference between the first maximum value and the first minimum value from among the plurality of past motion vectors. A certain dynamic range is obtained, a second maximum value obtained by adding the dynamic range to the first maximum value, a second minimum value obtained by subtracting the dynamic range from the first minimum value, and the second maximum value obtained. Set the maximum value and the second minimum value as the range of the predicted motion vector of the current frame, and the motion estimated in the current frame of the object If the vector is outside the range of the prediction vector, it is determined that the current motion of the object cannot be estimated. If the vector is within the range of the prediction vector, the current motion of the object can be estimated. If it is determined that the estimation is impossible, the stored past motion vector is read and extrapolated .

In one aspect of the present invention, a past motion vector of an object in an image is stored as history information, and a first maximum value is selected from a plurality of past motion vectors based on the stored history information. And a first minimum value, and a dynamic range that is a difference between the first maximum value and the first minimum value, a second maximum value obtained by adding the dynamic range to the first maximum value, A second minimum value obtained by subtracting the dynamic range from the minimum value of the second frame is determined, and the second maximum value and the second minimum value are set as the range of the predicted motion vector of the current frame, and the current frame of the object If the estimated motion vector is outside the range of the prediction vector, it is determined that the current motion of the object cannot be estimated, and if it is within the range of the prediction vector, the object It is determined that the current motion can be estimated, when it is determined to be unable to estimate the past motion vectors stored is extrapolated is read.

  According to the present invention, during the tracking of an object, the current motion can be estimated from the past motion history information during the tracking of the object.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings.

  FIG. 2 shows a configuration example when the present invention is applied to a surveillance camera system.

  The imaging unit 21 includes, for example, a CCD (Charge Coupled Device) video camera or the like, and displays the captured image on the image display 24. The tracking target detection unit 22 detects the tracking target from the image input from the imaging unit 21, and outputs the detection result to the object tracking unit 23.

  The object tracking unit 23 operates to track the tracking point designated by the tracking target detection unit 22 in the image supplied from the imaging unit 21. The object tracking unit 23 outputs the tracking result to the image display 24 and controls the camera driving unit 26 so that the moved object can be imaged based on the tracking result. Based on the control from the object tracking unit 23, the camera driving unit 26 drives the imaging unit 21 so that the imaging unit 21 captures an image centered on the tracking point.

  The control unit 27 is configured by, for example, a microcomputer and controls each unit. The control unit 27 is connected to a removable medium 28 formed of a semiconductor memory, a magnetic disk, an optical disk, a magneto-optical disk, or the like as needed, and a program and other various data are supplied as needed. The instruction input unit 29 includes various buttons, switches, a remote controller using infrared rays or radio waves, and outputs a signal corresponding to an instruction from the user to the control unit 27.

  Next, the monitoring process executed by the monitoring camera system shown in FIG. 2 will be described with reference to the flowchart of FIG. In this processing, when the power of the monitoring system 1 is turned on, an area to be monitored is imaged by the imaging unit 21, and an image obtained by the imaging is imaged via the tracking target detection unit 22 and the object tracking unit 23. It is output to the display 24 and started.

  In step S <b> 1, the tracking target detection unit 22 executes processing for detecting a tracking target from the image input from the imaging unit 21. For example, when there is a moving object in the input image, the tracking target detection unit 22 detects the moving object as a tracking target, and selects a point with the highest luminance or a center point of the tracking target from the tracking target. Detection is performed as a tracking point, and the detection result is output to the object tracking unit 23.

  In step S2, the object tracking unit 23 executes a tracking process for tracking the tracking point detected in the process of step S1. The details of the tracking process will be described later with reference to FIG. 16. By this process, the tracking point in an object (for example, a person, an animal, or the like) to be tracked in the image captured by the imaging unit 21. (For example, the center of the eyes and head) is tracked, and the tracking result is output to the control unit 27.

  In step S3, the control unit 27 causes the image display 24 to superimpose and display a mark representing the tracking position on the image captured by the image capturing unit 21, based on the tracking result obtained in step S2.

  In step S4, the object tracking unit 23 detects the movement of the object based on the tracking result in the process of step S2, generates a camera driving signal for driving the camera so that the moved object can be imaged, and the camera driving unit. 26. In step S <b> 5, the camera drive unit 26 drives the imaging unit 21 based on the camera drive signal from the object tracking unit 23. Thereby, the imaging unit 21 pans or tilts the camera so that the tracking point does not deviate from the screen.

  In step S6, the control unit 27 determines whether or not to end the monitoring process based on an instruction from the user via the instruction input unit 29. If the end is not instructed by the user, the control unit 27 proceeds to step S1. Return and repeat the subsequent processing. If it is determined in step S6 that the user has instructed the end of the monitoring process, the control unit 27 ends the monitoring process.

  4A to 4C are diagrams showing examples of images displayed on the image display 24 at this time in time series. FIG. 4A is an example of an image in which an object 41 to be tracked is imaged by the imaging unit 21. In this example, a person who moves in the left direction in the figure is imaged as the object 41. In FIG. 4B, the object 41 has moved to the left in the figure from the position in FIG. 4A, and in FIG. 4C, the object 41 has moved further to the left from the position in FIG. 4B.

  The tracking target detection unit 22 detects the object 41 in step S1 of FIG. 3, and outputs the human eye that is the object 41 to the object tracking unit 23 as the tracking point 41A. In step S2, the object tracking unit 23 performs a tracking process.

  Next, a detailed configuration example and operation of the object tracking unit 23 in FIG. 2 will be described.

  FIG. 5 is a block diagram illustrating a functional configuration example of the object tracking unit 23. As shown in FIG. 5, the object tracking unit 23 includes a motion estimation unit 52, a scene change detection unit 53, a background motion estimation unit 54, a region estimation related processing unit 55, a transfer candidate holding unit 56, a tracking point determination unit 57, The control unit 59 and the suspension detection unit 60 are configured.

  The motion estimation unit 52 estimates the motion of the input image, and obtains the motion vector obtained as a result of the estimation and the accuracy of the motion vector, the scene change detection unit 53, the background motion estimation unit 54, the region estimation related processing unit 55, Output to the tracking point determination unit 57 and the temporary stop detection unit 60. The scene change detection unit 53 detects a scene change based on the motion vector supplied from the motion estimation unit 52 and the accuracy of the motion vector.

  The background motion estimation unit 54 executes processing for estimating the background motion based on the motion vector supplied from the motion estimation unit 52 and the accuracy of the motion vector, and supplies the estimation result to the region estimation related processing unit 55. The area estimation related processing unit 55 includes the motion vector supplied from the motion estimation unit 52 and the accuracy of the motion vector, the background motion supplied from the background motion estimation unit 54, and the tracking point information supplied from the tracking point determination unit 57. Based on the above, region estimation processing is performed. The region estimation related processing unit 55 generates a transfer candidate based on the input information, supplies the transfer candidate to the transfer candidate holding unit 56, and holds it.

  The tracking point determination unit 57 includes the motion vector supplied from the motion estimation unit 52 and the accuracy of the motion vector, the transfer candidate supplied from the transfer candidate holding unit 56, and the temporary stop detection result supplied from the temporary stop detection unit 60. Based on, the tracking point is determined, and information regarding the determined tracking point is output to the region estimation related processing unit 55.

  Based on the tracking point information output from the tracking target detection unit 22, the control unit 59 controls each of the motion estimation unit 52 to the tracking point determination unit 57 and the pause detection unit 60 to detect the detected tracking. While tracking the target, a control signal is output to the camera driving unit 26 so that the tracking point is displayed on the screen displayed on the image display 24, and the driving of the imaging unit 21 is controlled. Thereby, the tracking point is controlled so as not to go out of the screen. Further, the control unit 59 outputs a tracking result such as information on the position of the tracking point on the screen to the control unit 27 or the like.

  The pause detection unit 60 performs a pause detection process based on the motion vector supplied from the motion estimation unit 52, the accuracy of the motion vector, and the background motion supplied from the background motion estimation unit 54. A detection result indicating whether or not tracking is started is supplied to the tracking point determination unit 57.

  The object tracking unit 23 shown in FIG. 5 performs processing for tracking the tracking point detected by the tracking target detection unit 22, and when the tracking point cannot be tracked, the current tracking point is obtained from the past motion history. By performing the estimation process, it is possible to continuously track. That is, in the present embodiment, for example, as shown in FIG. 6, the tracking point is temporarily invisible, for example, a scene change occurs in which the object 72 covers the entire object 71 as the tracking target. Even in such a case, tracking can be continuously performed.

  In the example of FIG. 6, there are an object 71 and an object 72 that move in different directions at time F (t−1), and a tracking point 81 is set for the object 71 as a tracking target. For example, a block matching method is used to detect the motion vector of each object.

At time F (t−1), as a result of matching between time F (t−1) and time F (t), the motion vector V _A (vx, vy) = (− 4, 0) of the object 71 is detected. In addition, the motion vector V _B (vx, vy) = (3, 0) of the object 72 is detected. If the object 71 and the object 72 moving in different directions intersect at the time F (t + 1), the result of matching between the time F (t) and the time F (t + 1) is incorrect at the time F (t). The motion vector (−4, −4) may be detected. Thus, at time F (t + 1), the tracking point 81 moves to the object 72 that is the foreground of the object 71 due to the intersection of the object 71 and the object 72, and at time F (t + 1) and time F (t + 2). As a result of the matching, the motion vector V _B (vx, vy) = (3, 0) of the object 72 is detected. Further, at the time F (t + 2) after the intersection, the tracking point 81 completely moves to the object 72, and the tracking of the object 71 fails.

  Therefore, in the present embodiment, even if the scene change occurs and the tracking point 81 temporarily disappears from the image, the tracking process described later with reference to the flowchart of FIG. The current tracking point can be estimated, and the object 71 can be continuously tracked without the tracking point 81 moving to the object 72 that is not the tracking target and causing the tracking to fail.

  Next, a configuration example of the motion estimation unit 52 in FIG. 5 will be described with reference to FIG. As shown in FIG. 7, an input image is supplied to the evaluation value calculation unit 91, the activity calculation unit 92, and the motion vector detection unit 93.

  The evaluation value calculation unit 91 calculates an evaluation value related to the degree of coincidence between both objects associated by the motion vector, and supplies the evaluation value to the normalization processing unit 95. The activity calculation unit 92 calculates the activity of the input image and supplies it to the threshold value determination unit 94 and the normalization processing unit 95. The motion vector detection unit 93 detects a motion vector from the input image and supplies it to the evaluation value calculation unit 91 and the integration processing unit 96.

  The threshold determination unit 94 compares the activity supplied from the activity calculation unit 92 with a predetermined threshold, and supplies the determination result to the integration processing unit 96. The normalization processing unit 95 normalizes the evaluation value supplied from the evaluation value calculation unit 91 based on the activity supplied from the activity calculation unit 92, and supplies the obtained value to the integration processing unit 96.

  The integration processing unit 96 calculates the accuracy of the motion vector based on the normalization information supplied from the normalization processing unit 95 and the determination result supplied from the threshold value determination unit 94, and the obtained accuracy is detected as a motion vector. This is output together with the motion vector supplied from the unit 93.

  Next, the motion estimation process executed by the motion estimation unit 52 will be described with reference to the flowchart of FIG. Although the motion vector is obtained for a point, the accuracy is calculated using image data of a small block in the vicinity of two points associated by the motion vector, for example, centering on the point.

  In step S11, the motion vector detection unit 93 detects a motion vector from the input image. For this detection, for example, a block matching method or a gradient method is used. The detected motion vector is supplied to the evaluation value calculation unit 91 and the integration processing unit 96.

  In step S12, the evaluation value calculation unit 91 calculates an evaluation value related to the degree of coincidence between the two objects associated with the motion vector detected in the process of step S11. Specifically, for example, the sum of absolute differences of pixel values of two blocks centered on two points associated with motion vectors is calculated. That is, the motion vector V (vx, vy) detected by the motion vector detector 93 in the process of step S11, the point P (Xp, Yp) on the image Fi of the previous frame based on the motion vector V (vx, vy), and the temporal The relationship between the point Q (Xq, Yq) on the image Fj of the subsequent frame is expressed by the following equation (1).

  The evaluation value calculation unit 91 calculates an evaluation value Eval (P, Q, i, j) for the block centered on the point P and the block centered on the point Q based on the following equation (2).

  Each block is a square having 2L + 1 pixels on one side. The sum ΣΣ in the above equation (2) is performed between corresponding pixels when x is from −L to L and y is from −L to L. Therefore, for example, when L = 2, nine differences are obtained, and the sum of the absolute values is calculated. The evaluation value indicates that the two blocks match well as the value approaches zero.

  The evaluation value calculation unit 91 supplies the calculated evaluation value to the normalization processing unit 95.

  In step S13, the activity calculation unit 92 calculates an activity from the input image. The activity is a feature amount representing the complexity of the image, and as shown in FIG. 9, the difference between the target pixel Y (x, y) and the adjacent 8 pixels Y (x + i, y + j) for each pixel. The average value of the sum of absolute values is calculated based on the following equation (3) as activity Activity (x, y) at the target pixel position.

  In the case of the example in FIG. 9, the value of the pixel of interest Y (x, y) located in the center among 3 × 3 pixels is 110, and the values of eight pixels adjacent to it are 80, 70, and 75, respectively. , 100, 100, 100, 80, 80, the activity Activity (x, y) is expressed by the following equation.

  Activity (x, y) = {| 80-110 | + | 70-110 | + | 75-110 | + | 100-110 | + | 100-110 | + | 80-110 | + | 80−110 |} /8=24.375.

  Similar processing is performed for all pixels in the frame.

  In order to calculate the motion vector accuracy in units of blocks, the sum of the activities of all the pixels in the block expressed by the following equation (4) is defined as the activity (block activity) Blockactivity (i, j) of the block.

  In addition, the activity may be a variance value, a dynamic range, or the like.

  In step S14, the threshold determination unit 94 compares the block activity calculated in step S13 with a predetermined threshold set in advance. Then, a flag indicating whether or not the input block activity is larger than the threshold value is output to the integration processing unit 96.

  Specifically, as a result of the experiment, the block activity and the evaluation value have the relationship shown in FIG. 10 using the motion vector as a parameter. In FIG. 10, the horizontal axis represents the block activity Blockactivity (i, j), and the vertical axis represents the evaluation value Eval.

  When the motion is correctly detected (when the correct motion vector is given), the block activity and the value of the evaluation value are distributed in a region R1 below the curve 101 in the figure. On the other hand, when an erroneous motion (incorrect motion vector) is given, the block activity and the evaluation value are distributed from the curve 102 to the region R2 on the left side in the figure. Note that there is almost no distribution in the region other than the region R2 above the curve 102 and the region other than the region R1 below the curve 101. The curve 101 and the curve 102 intersect at a point P, and the value of the block activity at this point P is set as a threshold value THa. The threshold value THa means that if the value of the block activity is smaller than that, the corresponding motion vector may be incorrect (this will be described in detail later). The threshold determination unit 94 outputs a flag indicating whether the block activity value input from the activity calculation unit 92 is greater than the threshold THa to the integration processing unit 96.

  In step S15, the normalization processing unit 95 normalizes the evaluation value calculated in the process of step S12 based on the activity calculated in the process of step S13. Specifically, the normalization processing unit 95 calculates the motion vector accuracy VC according to the following equation (5).

  However, if the value of the motion vector accuracy VC is less than 0, the value is replaced with 0. Of the motion vector accuracy VC, the value obtained by dividing the evaluation value by the block activity is a straight line 103 whose position on the graph of FIG. Therefore, it indicates whether the region is in the lower region or the upper region in the figure. That is, the slope of the straight line 103 is 1, and if the value obtained by dividing the evaluation value by the block activity is greater than 1, the points corresponding to the value are points distributed in the area above the straight line 103. Means that. The motion vector accuracy VC obtained by subtracting this value from 1 means that the smaller the value, the higher the possibility that the corresponding points are distributed in the region R2.

  On the other hand, if the value obtained by dividing the evaluation value by the block activity is smaller than 1, it means that points corresponding to the value are distributed in the lower area of the straight line 103 in the figure. The motion vector accuracy VC at that time means that the larger the value (closer to 0), the corresponding points are distributed in the region R1. The normalization processing unit 95 outputs the motion vector accuracy VC obtained by the calculation in this way to the integration processing unit 96.

  In step S <b> 16, the integration processing unit 96 executes integration processing. Details of this integration processing are shown in the flowchart of FIG.

  In step S21, the integration processing unit 96 determines whether or not the block activity is equal to or less than a threshold value THa. This determination is performed based on the flag supplied from the threshold determination unit 94. When determining that the block activity is equal to or less than the threshold value THa, the integration processing unit 96 sets the value of the motion vector accuracy VC calculated by the normalization processing unit 95 to 0 in step S22. If it is determined in step S21 that the activity value is greater than the threshold value THa, the process of step S22 is skipped, and the value of the motion vector accuracy VC generated by the normalization processing unit 95 is output as it is together with the motion vector. Is done.

  This is because there is a possibility that a correct motion vector is not obtained if the block activity value is smaller than the threshold value THa even if the motion vector accuracy VC calculated by the normalization processing unit 95 is positive. Because there is. That is, as shown in FIG. 10, between the origin O and the point P, the curve 102 protrudes from the curve 101 to the lower side in the figure (below the straight line 133). In a region R3 in which the value of the block activity is smaller than the threshold Tha and is surrounded by the curves 101 and 102, the value obtained by dividing the evaluation value by the block activity is distributed in both the regions R1 and R2. There is a high possibility that a correct motion vector is not obtained.

  Therefore, in such a distribution state, processing is performed assuming that the accuracy of the motion vector is low. For this reason, in step S22, the motion vector accuracy VC is set to 0 when the value is smaller than the threshold Tha even if the value is positive. In this way, when the value of the motion vector accuracy VC is positive, it is possible to reliably represent that the correct motion vector is obtained. In addition, the larger the value of the motion vector accuracy VC, the higher the probability that a correct motion vector is obtained (the probability that the distribution is included in the region R1 increases).

  This coincides with an empirical rule that, in general, it is difficult to detect a motion vector with high reliability in a region where the luminance change is small (region where the activity is small).

  Next, a configuration example of the background motion estimation unit 54 in FIG. 5 will be described with reference to FIG. As illustrated in FIG. 12, the background motion estimation unit 54 includes a frequency distribution calculation unit 111 and a background motion determination unit 112.

  The frequency distribution calculation unit 111 calculates a frequency distribution of motion vectors. However, this frequency is weighted so that a likely motion is weighted by using the motion vector accuracy VC supplied from the motion estimation unit 52 of FIG. Based on the frequency distribution calculated by the frequency distribution calculation unit 111, the background motion determination unit 112 performs a process of determining a motion with the maximum frequency as a background motion, and outputs the background motion to the region estimation related processing unit 55 in FIG. .

  The background motion estimation process executed by the background motion estimation unit 54 will be described with reference to the flowchart of FIG.

  In step S31, the frequency distribution calculation unit 111 calculates a motion frequency distribution. Specifically, the frequency distribution calculation unit 111 assumes that the x coordinate and the y coordinate of the motion vector as a background motion candidate are each expressed in a range of ± 16 pixels from the reference point (= 16 × 2 + 1) × (16 × 2 + 1)), that is, a box for coordinates corresponding to possible values of a motion vector, and when a motion vector is generated, 1 is added to the coordinates corresponding to the motion vector. In this way, the motion vector frequency distribution can be calculated.

  However, when one motion vector is generated, if 1 is added, if the frequency of occurrence of a motion vector with low accuracy is high, a motion vector with low certainty may be determined as the background motion. . Therefore, when a motion vector occurs, the frequency distribution calculation unit 111 does not add the value 1 to the box (coordinates) corresponding to the motion vector, but instead multiplies the value 1 by the motion vector accuracy VC (= Motion vector accuracy VC). The value of the motion vector accuracy VC is normalized as a value between 0 and 1, and the closer the value is to 1, the higher the accuracy. Therefore, the frequency distribution obtained in this way is a frequency distribution obtained by weighting motion vectors based on their accuracy. This reduces the risk that a motion with low accuracy is determined as the background motion.

  Next, in step S32, the frequency distribution calculation unit 111 determines whether or not the process of calculating the motion frequency distribution has been completed for all blocks. When it is determined that there is an unprocessed block, the process returns to step S31, and the motion frequency distribution is calculated for the next block.

  As described above, the motion frequency distribution calculation process is performed on the entire screen, and when it is determined in step S32 that the processing of all blocks has been completed, the process proceeds to step S33, where the background motion determination unit 112 A process for searching for the maximum value of the distribution is executed. That is, the background motion determination unit 112 selects a frequency having the maximum frequency from the frequencies calculated by the frequency distribution calculation unit 111, and determines a motion vector corresponding to the frequency as a motion vector of the background motion. The motion vector of the background motion is supplied to the region estimation related processing unit 55 in FIG. 5, and is used for the determination processing whether or not the full screen motion and the background motion match.

  Next, a configuration example of the tracking point determination unit 57 in FIG. 5 will be described with reference to FIG. As shown in FIG. 14, the tracking point determination unit 57 includes a predicted motion vector memory 121, a motion history feature amount extraction unit 122, and a motion determination unit 123.

  The predicted motion vector memory 121 stores the latest past plural frames of data of the predicted motion vector of the object finally determined by the motion determination unit 123. The motion history feature quantity extraction unit 122 reads the latest predicted motion vector data of a plurality of past frames stored in the predicted motion vector memory 121 and extracts the motion feature quantity of the object. The motion determination unit 123 includes a motion vector of the current frame supplied from the motion estimation unit 52, a transfer candidate supplied from the transfer candidate holding unit 56, a pause detection result supplied from the pause detection unit 60, and a motion history feature. Based on the motion feature quantity supplied from the quantity extraction unit 122, the motion vector data of the current frame is evaluated, and a predicted motion vector as the evaluation result is output to the region estimation related processing unit 55, and the predicted motion vector memory 121 To store.

  Further, with reference to FIG. 15, details of the motion history feature amount extraction unit 122 and the motion determination unit 12 of the tracking point determination unit 57 of FIG. 14 will be described. As shown in FIG. 15, the motion history feature quantity extraction unit 122 includes a horizontal component minimum value calculation unit (MIN) 131, a horizontal component maximum value calculation unit (MAX) 132, a vertical component minimum value calculation unit (MIN) 133, And a vertical component maximum value calculation unit (MAX) 134.

  The horizontal component minimum value calculation unit 131 calculates the minimum value of the horizontal component of the past predicted motion vector from the latest predicted motion vector data of a plurality of past frames read from the predicted motion vector memory 121, and determines the calculated result as a motion determination. Output to the unit 123. The horizontal component maximum value calculation unit 132 calculates the maximum value of the horizontal component of the past predicted motion vector from the latest predicted motion vector data of a plurality of past frames read from the predicted motion vector memory 121, and determines the calculated result as a motion determination. Output to the unit 123.

  The vertical component minimum value calculation unit 133 calculates the minimum value of the vertical component of the past predicted motion vector from the latest predicted motion vector data of a plurality of past frames read from the predicted motion vector memory 121, and determines the calculated result as a motion determination. Output to the unit 123. The vertical component maximum value calculation unit 134 calculates the maximum value of the vertical component of the past predicted motion vector from the latest predicted motion vector data of a plurality of past frames read from the predicted motion vector memory 121, and determines the calculated result as a motion determination. Output to the unit 123.

  The motion history feature amount extraction unit 122 also calculates the dynamic range of the horizontal component of the motion vector from the minimum and maximum values of the horizontal component of the motion vector calculated by the horizontal component minimum value calculation unit 131 and the horizontal component maximum value calculation unit 132. And the dynamic range of the vertical component of the motion vector is calculated from the minimum value and the maximum value of the vertical component of the motion vector calculated by the vertical component minimum value calculation unit 133 and the vertical component maximum value calculation unit 134. The result is output to the motion determination unit 123.

  The motion determination unit 123 includes the maximum value MAX (x, y) and the minimum value MIN (x, y) of the past predicted motion vector supplied from the motion history feature amount extraction 122, and the difference between the maximum value and the minimum value. Based on the dynamic range DR (x, y) obtained from the above, the predicted motion vector ranges min (x, y) and max (x, y) of the current frame are calculated according to the following equations (6) and (7). .

  In Expression (6) and Expression (7), TH (x, y) is a threshold value. This threshold value is the dynamic range DR (x, y) = (0, 0) when the object has made a uniform linear motion between past frames, and the range of the predicted motion vector in the current frame is max (x , Y) −min (x, y) = (0, 0). For example, when the object is performing a constant velocity linear motion such as a telop, according to the equations (6) and (7), the range of the predicted motion vector is set to be small. When accelerating in units of frames, the range of the predicted motion vector is set to be large.

  Then, based on the range of the predicted motion vector calculated by Expression (6) and Expression (7), the motion determination unit 123 determines the predicted motion vector if the motion vector of the tracking point estimated in the current frame is within the range. If it is out of the range, it is determined that it is impossible to estimate the motion vector predictor.

  In addition, the motion determination unit 123 can estimate when the motion estimation result at the tracking point and the motion estimation result at a point in the vicinity of the tracking point match the motion occupying the majority, and cannot estimate when the motion estimation result does not match. It is also possible to make a determination. Thereby, it is possible to determine a motion vector having a high spatial correlation and a predicted motion vector having high temporal direction continuity.

  If the motion determination unit 123 determines that the motion vector cannot be estimated, the motion vector can be estimated near the original tracking point among points belonging to the same target area as the tracking point prepared in advance. Change the tracking point to the point that is and perform tracking. Details of this transfer processing are disclosed in Japanese Patent Laid-Open No. 2005-303983 previously proposed by the present applicant. When the tracking point is not displayed by this processing, a transfer candidate extracted and held in advance in a state where tracking is possible is selected, the transfer candidate is set as a new tracking point, and tracking is performed. .

  Furthermore, if the motion determination unit 123 determines that it is impossible to estimate the motion vector of the tracking point even if all the transfer candidates are set as new tracking points, the motion and acceleration of the object are just before the known values (past ) Is extrapolated (substitute) with the past predicted motion vector stored in the predicted motion vector memory 121, and the predicted motion vector of the tracking point of the current frame is determined.

  Here, when the movement of the object specified as the tracking target stops during the intersection with other objects, such as when the movement is greatly different from the past movement history, and continuously extrapolating with the past predicted motion vector, In the worst case, the tracking point moves regardless of the motion information in the image. Therefore, the motion determination unit 123 measures the number of times that the predicted motion vector is continuously extrapolated, determines whether to extrapolate with the past predicted motion vector by comparison with a preset threshold, and extrapolates. If it is determined that there is not, the zero vector is set and the extrapolation is terminated. That is, in this case, tracking is terminated.

  Next, a tracking process executed by the object tracking unit 23 will be described with reference to the flowchart of FIG.

  In step S41, the control unit 59 controls each unit so as to wait for an input of an image of the next frame. In step S42, the motion estimation unit 52 estimates the tracking point motion. That is, by capturing a frame (hereinafter referred to as a subsequent frame) that is temporally later than a frame including a tracking point designated by the user (here, referred to as a previous frame) in the process of step S41, two consecutive frames are eventually obtained. Thus, in step S42, the position of the tracking point in the subsequent frame corresponding to the tracking point in the previous frame is estimated, so that the motion vector of the tracking point is estimated.

  Note that “before or after in time” means the order of input or the order of processing. Usually, since the images of each frame are input in the order of imaging, in that case, the frame captured earlier in time becomes the previous frame, but the frame captured later in time is processed first. In this case, a frame that is imaged later in time becomes the previous frame.

  In step S <b> 43, the motion history feature amount extraction unit 122 reads the history data of predicted motion vectors over a plurality of past frames stored in the predicted motion vector memory 121. From the read history data, the horizontal component minimum value calculation unit 131 calculates the minimum horizontal component value of the past predicted motion vector, and the horizontal component maximum value calculation unit 132 calculates the maximum horizontal component value of the past predicted motion vector. Is calculated by the vertical component minimum value calculation unit 133, and the vertical component maximum value calculation unit 134 calculates the maximum vertical component vertical value of the past predicted motion vector. .

  In step S44, the motion history feature quantity extraction unit 122 calculates the dynamic range of the horizontal component of the motion vector from the minimum and maximum values of the horizontal component of the motion vector read and calculated in the process of step S43, The dynamic range of the vertical component of the motion vector is calculated from the minimum value and the maximum value of the vertical component of the motion vector.

  In step S45, the motion determination unit 123 determines the past predicted motion vector maximum and minimum values, and the maximum and minimum values based on the pause detection result (described later) output by the pause detection unit 60. From the obtained dynamic range, the predicted motion vector range of the current frame is estimated according to Equation (6) and Equation (7).

  In this process, when the tracking point is set after the moving image is temporarily stopped, the image of the same frame is continuously input to the object tracking unit 23 after the tracking point is set until the tracking is started. In this case, the dynamic range DR (x, y) = (0, 0), and the range of the predicted motion vector at the tracking point at the start of tracking becomes small, so it is impossible to track an object that moves quickly. Therefore, based on the suspension detection result output by the suspension detection unit 60, the range of the predicted motion vector is exceptionally the same value as the search area of the motion estimation unit 52 only when tracking is started from the suspension. Or set it to a very large value.

  In step S46, the motion determination unit 123 determines whether or not the tracking point motion can be estimated. Whether or not the tracking point motion can be estimated is determined by determining whether or not the tracking point motion vector estimated by the process of step S42 is within the range of the predicted motion vector estimated by the process of step S45. If it is within the range of the predicted motion vector, the estimation is possible, and if it is outside the range, it is determined that the estimation is impossible.

  If it is determined in step S46 that the movement of the tracking point can be estimated, the process proceeds to step S47, and the motion determination unit 123 initializes the continuous extrapolation counter. In step S48, the motion determination unit 123 shifts the tracking point by the amount of the motion vector estimated in the process of step S42. As a result, the tracking position in the subsequent frame after the tracking of the tracking point in the previous frame is determined.

  On the other hand, when it is determined in step S46 that the movement of the tracking point cannot be estimated, the process proceeds to step S49, and the motion determination unit 123 determines whether or not the transfer candidate is held in the transfer candidate holding unit 56. If it is determined that the transfer candidate is held, the process proceeds to step S50. In step S50, the motion determination unit 123 selects a transfer candidate closest to the original tracking point from the transfer candidates held in the transfer candidate holding unit 56, and transfers (changes) the selected transfer candidate. . Thereby, the transfer candidate point is set as a new tracking point.

  After the process of step S50, the process returns to step S46, and it is determined again whether or not the movement of the newly set tracking point can be estimated. If it can be estimated, the continuous extrapolation counter is set in step S47. In step S48, the tracking point is shifted by the estimated motion.

  If it is determined in step S49 that no transfer candidate is held in the transfer candidate holding unit 56, the process proceeds to step S51, and the motion determination unit 123 determines whether or not the continuous extrapolation counter is equal to or less than a threshold value. In other words, the processing here is continuous with the predicted motion vector in the past when the motion of the object specified as the tracking target stops during the intersection with another object and the motion moves greatly different from the past motion history. In the worst case, the tracking point moves regardless of the motion information in the image. Therefore, the motion determination unit 123 counts the number of times that the predicted motion vector is continuously extrapolated and compares it with a preset threshold value.

  If it is determined in step S51 that the continuous extrapolation counter is equal to or smaller than the threshold value, the process proceeds to step S52, and the motion determination unit 123 counts up the continuous extrapolation counter and stores it in the predicted motion vector memory 121 in step S53. The previous predicted motion vector immediately before being read is read and extrapolated. Thereby, the position of the tracking point of the current frame is determined.

  If it is determined in step S51 that the continuous extrapolation counter is not less than or equal to the threshold value, the process proceeds to step S54, and the motion determination unit 123 initializes the continuous extrapolation counter. In step S55, the motion determination unit 123 outputs the predicted motion vector as a zero vector. That is, as described above, when the extrapolation is continuously performed with the past predicted motion vector, the tracking point moves regardless of the motion information in the image. , The predicted motion vector is set to 0 vector, and the tracking is finished.

  After the process of step S48, step S53, or step S55, the process proceeds to step S56, and the motion determination unit 123 updates the predicted motion vector memory 121 based on the finally determined predicted motion vector. As a result, the predicted motion vector that has just been determined is stored as new history data in the predicted motion vector 121.

  After the update process of the motion vector predictor memory 121 in step S56 is completed, the process returns to step S41 again, and the subsequent processes are repeatedly executed.

  In this way, by using the feature amount calculated from the historical information of the predicted motion vector of the object over a plurality of past frames, it is possible to estimate whether the predicted motion vector of the current tracking point can be estimated and extrapolate the past predicted motion vector. By combining them, it is possible to obtain a highly accurate predicted motion vector. If the tracking point disappears between multiple frames due to the intersection with another object, even if the motion vector is likely to be a correlation vector, it is uncertain if the predicted motion vector at the current tracking point can be estimated. It is possible to eliminate a large motion vector, and it is possible to perform tracking with high accuracy.

  Next, a pause detection process executed by the pause detection unit 60 will be described with reference to the flowchart of FIG. This process is started by setting a counter for detecting a pause state to 0.

  The pause detection unit 60 obtains the tracking point motion vector from the motion estimation unit 52 and also obtains the background motion vector from the background motion estimation unit 54.

  In step S61, the pause detection unit 60 determines whether the counter is equal to or greater than the threshold value, and the motion vector of either the background or the tracking point on the object to be tracked is not a zero vector. If it is determined that the motion vector is equal to or greater than the threshold value and one of the background and the tracking point on the object to be tracked is not the 0 vector, the process proceeds to step S62.

  In step S62, the suspension detection unit 60 initializes a counter for detecting suspension, and determines in step S63 that tracking starts from the suspension. That is, if the counter is greater than or equal to the threshold value, it means that the counter is in a paused state, and if the motion vector of either the background or the tracking point is not a zero vector, motion has been resumed in the current frame. Therefore, it is determined that the tracking is started from the pause.

  If it is determined in step S61 that the counter is not equal to or greater than the threshold value, the motion vector of either the background or the tracking point is a 0 vector, or the motion vectors of both the background and the tracking point are 0 vectors, In step S64, the pause detection unit 60 determines that the tracking is not started from the pause.

  In step S65, the pause detection unit 60 determines whether both the background motion and the tracking point motion are 0 vectors, and if it is determined that both the background motion and the tracking point motion are 0 vectors, Proceeding to S66, the counter for detecting the pause state is incremented. By this process, detection of the pause state is started.

  If it is determined in step S65 that both the background motion and the tracking point motion are not zero vectors, that is, one of the background and tracking point motion vectors is not a zero vector, the process proceeds to step S67 and is in a paused state. Initialize a counter for detecting.

  After the process of step S63, step S66, or step S67, the pause detection unit 60 outputs the counter value as a detection result to the motion determination unit 123 of the tracking point determination unit 57. This detection result is used in the process of step S45 in FIG.

  After the process of step S68, the process returns to step S61, and the above-described process is repeated.

  In this way, it is estimated from the tracking point motion vector and the background motion vector whether the tracking start from the temporary stop. That is, since it is determined whether or not the stop is made using both the statistically reliable background motion vector and the motion vector of the local tracking point of the object, a highly accurate estimation result can be obtained. In addition, if the movement determination part 123 can acquire the information which shows whether the tracking start from a temporary stop is possible from the control part 59 etc., it is more preferable to use the information.

  Next, with reference to FIG. 18, an example will be described in which tracking can be continuously performed even when the tracking point temporarily disappears by executing the tracking processing of the present embodiment.

  In the example of FIG. 18, the object 71 and the object 72 moving in different directions intersect at the time F (t + 1), and the predicted motion vector is correct until the time F (t−1) immediately before the object 71 and the object 72 intersect. Suppose you want. In addition, as shown in FIG. 19A, the predicted motion vector memory 121 has predicted motion vectors (time (t-4), time (t-3), time (t-2), and time (t-1). -2, 0), (-2, 0), (-4, 0), (-4, 0) are stored.

At time F (t−1), as a result of matching between time F (t−1) and time F (t), the motion vector V _A (vx, vy) = (− 4, 0) of the object 71 is detected. In addition, the motion vector V _B (vx, vy) = (3, 0) of the object 72 is detected.

  At time F (t), an incorrect motion vector (−4, −4) is detected as a result of matching between time F (t) and time F (t + 1). Therefore, the motion history feature amount extraction unit 122 reads a predicted motion vector as shown in FIG. 19A from the predicted motion vector memory 121 and uses the maximum value MAX (x, y) = (− 2, 0) as the motion history feature amount. The minimum value MIN (x, y) = (− 4, 0) and the dynamic range DR (x, y) = (2, 0) are calculated. For example, when the threshold is set to TH (x, y) = (2, 1), the motion determination unit 123 sets the range of the predicted motion vector of the current frame to max according to Equation (6) and Equation (7). It is estimated that (x, y) = (2, 1) and min (x, y) = (− 4, −1). As a result, the incorrect motion vector (−4, −4) of the tracking point 81 estimated by matching between the time F (t) and the time F (t + 1) falls outside the range of the predicted motion vector, and an uncertain motion vector. Can be rejected as. At this time, the motion determination unit 123 extrapolates the predicted motion vector of the tracking point 81 by the predicted motion vector (−4, 0) of the previous past time F (t−1).

  As a result of this extrapolation, the position of the tracking point 81 at time F (t + 1) is a position shifted by the predicted motion vector (−4, 0) from the position at time F (t). In addition, as shown in FIG. 19B, the motion vector (−4, 0) at time F (t) is newly stored in the predicted motion vector memory 121.

Here, at the time F (t + 1), the tracking point 81 exists in the object 72 that is the foreground of the object 71 due to the intersection of the object 71 and the object 72, and therefore between the time F (t + 1) and the time F (t + 2). As a result of the matching, the motion vector V _B (vx, vy) = (3, 0) of the object 72 is detected as the motion vector of the tracking point 81. Therefore, the motion history feature amount extraction unit 122 reads a predicted motion vector as shown in FIG. 19B from the predicted motion vector memory 121, and uses the maximum value MAX (x, y) = (− 2, 0) as the motion history feature amount. The minimum value MIN (x, y) = (− 4, 0) and the dynamic range DR (x, y) = (2, 0) are calculated. For example, when the threshold is set to TH (x, y) = (2, 1), the motion determination unit 123 sets the range of the predicted motion vector of the current frame to max according to Equation (6) and Equation (7). It is estimated that (x, y) = (2, 1) and min (x, y) = (− 4, −1). As a result, the incorrect motion vector (3, 0) of the tracking point 81 estimated by the matching between the time F (t + 1) and the time F (t + 2) falls outside the range of the predicted motion vector and is rejected as an uncertain motion vector. can do. At this time, the motion determination unit 123 extrapolates the predicted motion vector of the tracking point 81 by the predicted motion vector (−4, 0) at the previous past time F (t).

  As a result of this extrapolation, the position of the tracking point 81 at the time F (t + 2) is a position shifted by the predicted motion vector (−4, 0) from the position at the time F (t + 1) and exists on the object 71. Become.

  As described above, in the determination of whether or not the predicted motion vector of the current tracking point can be estimated, the motion range of the current tracking point is estimated from the past motion history information of the tracking point, and the current motion with the immediately preceding motion that is a known value is Since the motion is extrapolated, even if the tracking point 81 disappears between multiple frames due to crossing by other objects, it is a motion vector with high spatial correlation and continuity in the time direction. Therefore, it is possible to determine a motion vector with high prediction, and to perform tracking with high accuracy.

  Further, by applying the motion availability determination to the motion detection for each pixel, it becomes possible to detect a motion with higher accuracy.

  In the above, it is assumed that the frame of the image is used as a processing unit and all frames are used. However, processing is not performed in units of fields or using all frames or fields, but is extracted by thinning out at a predetermined interval. It is also possible to use a modified frame or field.

  In the above description, an example in which the present invention is applied to a surveillance camera system has been described. However, the present invention is not limited thereto, and can be applied to various image processing apparatuses such as a television receiver.

  Further, in the above, the image processing unit is a frame, but it is of course possible to use a field as a processing unit.

  The series of processes described above can be executed by hardware or can be executed by software. When a series of processing is executed by software, a program constituting the software executes various functions by installing a computer incorporated in dedicated hardware or various programs. For example, it is installed from a program recording medium in a general-purpose personal computer or the like.

  As shown in FIG. 2, a program recording medium that stores a program that is installed in a computer and can be executed by the computer includes a magnetic disk (including a flexible disk), an optical disk (CD-ROM (Compact Disc-Read Only). Memory), DVD (including Digital Versatile Disc), magneto-optical disc, or removable media 28, which is a package media composed of semiconductor memory, or a hard disk on which a program is temporarily or permanently stored . The program is stored in the program recording medium using a wired or wireless communication medium such as a local area network, the Internet, or digital satellite broadcasting via an interface such as a router or a modem as necessary.

  In the present specification, the steps for describing a program are not only processes performed in time series in the order described, but also processes that are executed in parallel or individually even if they are not necessarily processed in time series. Is also included.

  The embodiment of the present invention is not limited to the above-described embodiment, and various modifications can be made without departing from the gist of the present invention.

It is a figure explaining the example which fails in tracking by the intersection with another object. It is a block diagram which shows the structural example of the surveillance camera system to which this invention is applied. It is a flowchart explaining a monitoring process. It is a figure which shows the example of the image displayed by the surveillance camera system of FIG. It is a block diagram which shows the structural example of the object tracking part of FIG. It is a figure explaining the tracking when a scene change occurs. It is a block diagram explaining the structural example of a motion estimation part. It is a flowchart explaining a motion estimation process. It is a figure explaining calculation of activity. It is a figure explaining the relationship between an evaluation value and activity. It is a flowchart explaining an integration process. It is a block diagram which shows the structural example of a background motion estimation part. It is a flowchart explaining a background motion estimation process. It is a block diagram which shows the structural example of a tracking point determination part. It is a block diagram explaining the further detail of a tracking point determination part. It is a flowchart explaining a tracking process. It is a flowchart explaining a temporary stop detection process. It is a figure explaining the tracking of this Embodiment. It is a figure which shows the example of the motion vector stored in an estimated motion vector memory.

Explanation of symbols

  DESCRIPTION OF SYMBOLS 1 Surveillance camera system, 21 Image pick-up part, 22 Tracking object detection part, 23 Object tracking part, 52 Motion estimation part, 53 Scene change detection part, 54 Background motion estimation part, 55 Area estimation related process part, 56 Transfer candidate holding part, 57 tracking point determination unit, 60 pause detection unit, 121 predicted motion vector memory, 122 motion history feature amount extraction unit, 123 motion determination unit

Claims (7)

In an image processing apparatus that tracks a moving object,
Storage means for storing past motion vectors of the object in the image as history information;
Based on the history information stored in the storage means, a first maximum value, a first minimum value, a first maximum value, and a first one out of a plurality of the past motion vectors. A second maximum value obtained by adding the dynamic range to the first maximum value, and a second minimum value obtained by subtracting the dynamic range from the first minimum value. Range setting means for setting the second maximum value and the second minimum value as the range of the predicted motion vector of the current frame;
If the motion vector estimated in the current frame of the object is outside the range of the prediction vector, it is determined that the current motion of the object cannot be estimated, and if the motion vector is within the range of the prediction vector, Determining means for determining that the current movement of the object can be estimated;
An image processing apparatus comprising: an extrapolation unit that reads and extrapolates the past motion vector stored in the storage unit when the determination unit determines that the estimation is impossible . The range setting means sets a value obtained by adding a threshold value to the second maximum value to prevent the predicted motion vector range from becoming 0 when the dynamic range becomes 0. The image processing apparatus according to claim 1, wherein a value obtained by subtracting the threshold value from the second maximum value is set as the second minimum value . If the past motion vector is extrapolated continuously by said extrapolation means, the image processing apparatus according to claim 1 for setting the current motion vector of the object to 0 vector. A counter for counting the number of times the past motion vector has been extrapolated by the extrapolator;
2. The image according to claim 1 , wherein the counter value by the counter unit is initialized when it is determined by the determination unit that it can be estimated, or when it is determined that the counter value by the counter unit exceeds a threshold value. Processing equipment. Further comprising detection means for detecting a pause of the image,
If the temporary stop of the image is detected by said detecting means, said range setting means, the image processing apparatus according to claim 1 for setting a range of the predicted vector to a large value. In an image processing method of an image processing apparatus for tracking a moving object,
Storing past motion vectors of the object in the image as history information;
Based on the stored history information, a first maximum value and a first minimum value, and a first maximum value and a first minimum value from among the plurality of past motion vectors. A dynamic range that is a difference between the first maximum value, a second maximum value obtained by adding the dynamic range to the first maximum value, and a second minimum value obtained by subtracting the dynamic range from the first minimum value, Setting the second maximum value and the second minimum value as a range of a predicted motion vector of a current frame;
If the motion vector estimated in the current frame of the object is outside the range of the prediction vector, it is determined that the current motion of the object cannot be estimated, and if the motion vector is within the range of the prediction vector, Determining that the current movement of the object is estimable;
An image processing method including a step of reading and extrapolating the stored past motion vector when it is determined that estimation is impossible . In a program for causing a computer to execute image processing of an image processing apparatus that tracks a moving object,
Storing past motion vectors of the object in the image as history information;
Based on the stored history information, a first maximum value and a first minimum value, and a first maximum value and a first minimum value from among the plurality of past motion vectors. A dynamic range that is a difference between the first maximum value, a second maximum value obtained by adding the dynamic range to the first maximum value, and a second minimum value obtained by subtracting the dynamic range from the first minimum value, Setting the second maximum value and the second minimum value as a range of a predicted motion vector of a current frame;
If the motion vector estimated in the current frame of the object is outside the range of the prediction vector, it is determined that the current motion of the object cannot be estimated, and if the motion vector is within the range of the prediction vector, Determining that the current movement of the object is estimable;
A program including a step of reading and extrapolating the stored past motion vector when it is determined that estimation is impossible .

Images