JP4818430B2 - Moving object recognition method and apparatus - Google Patents

Description

  The present invention relates to a moving object recognition method and apparatus, and more particularly to a moving object recognition method and apparatus for dividing and tracking a non-separated moving object.

  FIG. 6 is a captured image of a camera installed above the highway.

  Since vehicles frequently overlap each other on an image, it becomes difficult to track each vehicle by image processing. In order to solve this problem, it is necessary to install a plurality of cameras along the road and comprehensively process their images.

JP-A-9-161071

  However, since it is necessary to provide a plurality of cameras and image processing apparatuses, the cost increases. In addition, since the captured images of the respective cameras must be related and processed comprehensively, the processing becomes complicated.

  An object of the present invention is to provide a moving object recognition method and apparatus capable of dividing and recognizing even a motion vector of moving objects overlapped on an image is almost the same with a simple configuration. It is in.

  For example, as shown in FIG. 3, the moving object near the camera is separated.

Therefore, one aspect of the moving object recognition apparatus according to the present invention includes an image storage unit that stores a plurality of time-series images, and an image processing unit that performs image processing by going back in time. The image processing unit
1) By block matching between the image at time t and the image at time (t−1) among the plurality of time-series images, time (t−) from time t of the moving object included in the image at time (t−1) A motion vector to 1) and a moving object identification code are determined in units of blocks including a plurality of pixels;
The non-separated moving object is divided by performing the process (1) once or by repeating the tracking operation a plurality of times, and tracking the moving object backward in time.

  According to this moving object recognition device, even if the motion vectors of the overlapping moving objects on the image are almost the same, it is possible to divide and recognize them, and in a wide range along the longitudinal direction of the road. It is possible to track a moving object. Therefore, it is not necessary to provide a plurality of sets of cameras and image processing devices along the road and to associate image data between the cameras to perform complicated processing, thereby reducing costs.

  Other objects, configurations and effects of the present invention will become apparent from the following description.

It is a flowchart which shows the process sequence of the moving object recognition method which concerns on one Example of this invention. It is explanatory drawing of the moving object recognition process with respect to a time series image. It is a figure which shows typically the time series image imaged with the camera installed above the road center line. It is a figure which shows typically the time series image imaged with the camera in the branched road. It is a figure which shows typically the time series image at the time of installing a camera in the side of a road and imaging a moving object in the wide range along the longitudinal direction of a road. It is the picked-up image of the camera installed above the highway. It is a schematic block diagram of the moving object recognition apparatus which concerns on Example 1 of this invention. FIG. 8 is an explanatory diagram of processing in an ID generation / annihilation unit in FIG. 7. FIGS. 7A to 7C are explanatory diagrams of processing in the moving object update unit 27 in FIG. 7, respectively, an object map at time t, a presence prediction range of a moving object at time (t−1), and time (t -1) is an object map. (A) And (B) is an explanatory view of the advantage of the processing of the moving object updating unit 27 in FIG. 7, and shows the object maps at time (t−1) and time t, respectively, together with the vehicle image. It is a figure which shows typically the input-output relationship of the unknown ID determination part in FIG. (A) And (B) is processing explanatory drawing of this unknown ID determination part. (A) And (B) is processing explanatory drawing of this unknown ID determination part. It is a flowchart which shows the process in the unknown ID determination part of the moving object recognition apparatus which concerns on Example 2 of this invention.

  Embodiments of the present invention will be described below with reference to the drawings.

  First, the moving object recognition method of the first embodiment will be outlined with reference to FIG. FIG. 3 schematically shows images taken at time t = 1 to 4 captured by the camera.

  In order to capture a wide range along the longitudinal direction of the road 10 with one camera and track a moving object, the camera is installed above the center line of the road 10. When the camera is installed in this manner, an occlusion in which moving objects overlap on the image frequently occurs. In FIG. 3, the images of the vehicles M1 and M2 overlap at time t = 1 to 3, and the two cannot be distinguished and tracked.

  However, when the vehicles M1 and M2 approach the camera as in the image at time t = 4, the images of the vehicles M1 and M2 are separated if they do not collide.

  Therefore, the time-series images of these four frames are temporarily stored, the vehicles M1 and M2 are identified starting from t = 4, and the motion vectors of the vehicles M1 and M2 are obtained. Assuming an image at t = 3 in which the vehicles M1 and M2 are moved and the vehicles M1 and M2 are identified, and the correlation between this and the actual image at t = 3, Vehicles M1 and M2 are identified.

  Next, the vehicles M1 and M2 in the t = 2 image are identified by the same processing for the t = 3 and t = 2 images. Next, the vehicles M1 and M2 in the image at t = 1 are identified by the same processing for the images at t = 2 and t = 1.

  Such processing makes it possible to track the vehicles M1 and M2.

  FIG. 1 is a flowchart showing a processing procedure of this moving object recognition method. FIG. 1 shows a case in which a moving object is tracked backward in time for an image of n frames. For example, n = 36 and 12 frames per second. Note that the background image in which the moving object is not captured is stored in the storage device in advance.

  (S1) Substitute an initial value n at time t.

  (S2) The image of t is compared with the background image, the moving object in the image of t is recognized, and an identification code (ID) is given to each recognized moving object.

  (S3) The motion vector of the moving object in the image of t to which the ID is assigned is obtained based on the image of t and the image before or after this. This temporally preceding or following image may be an image only for obtaining a motion vector between t = n and t = n + 1 or between t = n and t = n−1.

  (S4) The ID of the moving object in the image of (t-1) is determined from the above correlation between the images of t and (t-1).

  (S5) If t> 2, the process proceeds to step S6. If t = 2, the moving object recognition process for the time-series images from t = 1 to n is terminated.

  (S6) The value of t is decremented by 1, and the process returns to step S3.

  In FIG. 2, the process of FIG. 1 is performed on the time-series image 1 composed of F (1) to F (n) at t = 1 to n. The image F (n + 1) at the next time is acquired and stored in the storage area of the image F (1). Then, the time series image 2 composed of F (2) to F (n + 1) is set as an image of t = 1 to n, and the process of FIG. 1 is performed. Such a process is repeated.

  In the processing for the time-series image 2, for the images F (2) to F (n), processing for recognizing a moving object compared to the background image and processing for obtaining a motion vector in step S3 in FIG. Since this process has already been performed for 1, these processes can be omitted.

  According to the first embodiment, it is possible to track a moving object in a wide range along the longitudinal direction of the road by a pair of cameras and an image processing device. Therefore, a plurality of sets of cameras and image processing devices are provided along the road, and it is not necessary to associate image data between the cameras and perform complicated processing, and the cost can be greatly reduced.

  FIG. 4 schematically shows a time series image when the road 11 is branched into roads 11a and 11b on one side and branched into roads 11c and 11d on the other side. In this case, by the above-described tracking, it is possible to determine which of the roads 11a and 11b the vehicles M1 and M2 have come to, and to which of the roads 11c and 11d, and traffic information on the road usage situation can be obtained.

  FIG. 5 schematically shows a time-series image when a camera is installed on the side of the road 12 and a moving object is imaged in a wide range along the longitudinal direction of the road 12. Also in this case, it is possible to track a moving object by going backward in time as in FIGS.

  The tracking result can be used for determination of traffic violations and automatic determination of traffic accidents that could not be done conventionally due to overlapping of moving objects on the image. For example, it is possible to change the lane in the lane change prohibited area and determine that the vehicle M1 has overtaken M2 as shown in FIG.

    FIG. 1 is a schematic block diagram of a moving object recognition apparatus according to Embodiment 1 of the present invention.

  This apparatus includes an electronic camera 15 that captures a road through which a moving object passes and outputs an image signal, an image processing apparatus 20 that processes the image to track the moving object and determines a traffic violation, and a traffic violation detection time. And an external storage device 40 for storing images.

  The image signal from the electronic camera 15 is digitized, and n-frame time-series images are temporarily stored in the image memory 21. For example, an image of one frame is stored in the image memory 21 every 1/12 seconds. The storage area is an area in which the oldest image in the time series image in the image memory 21 is stored.

  The background image generation unit 22 accesses the image memory 21, for example, creates a histogram of pixel values of the corresponding pixels for all images for the past 10 minutes, and sets the mode (mode) as the pixel value of the pixel. An image is generated as a background image and stored in a predetermined area in the image memory 21. The background image is periodically updated by the background image generation unit 22.

  The tracking processing unit 23 creates an object map in the memory 24. For example, when an image of one frame is 480 × 640 pixels, the tracking processing unit 23 writes information of 60 × 80 blocks in which 8 × 8 pixels correspond to one block in the object map memory 24. The tracking processing unit 23 determines whether or not there is a moving object in the time-series image, and if it exists, assigns a moving object ID to this block. Whether there is a moving object in a block is whether the sum of absolute values of the difference between each pixel of the block set in the frame image corresponding to the object map and the corresponding pixel in the background image is greater than or equal to a predetermined value Judgment by By assigning the same ID to adjacent blocks determined to have a moving object, a cluster corresponding to the moving object is created. FIG. 8 is an object map corresponding to the image at t = 4 in FIG. 3, and ID = 1 and ID = 2 are assigned to the vehicles M1 and M2, respectively.

  The tracking processing unit 23 includes an ID generation / annihilation unit 25, a motion vector calculation unit 26, a moving object update unit 27, and an unknown ID determination unit 28. The tracking processing unit 23 obtains a trajectory of the moving object by going back in time, and stores this in the trajectory memory 29. Store.

  When the traffic violation determination unit 30 determines whether the trajectory obtained as described above does not violate the rule based on the rules stored in the traffic rule storage unit 31, The image memory 21 stores the time series image at the time of violation in the external storage device 40.

  The ID generation / annihilation unit 25 assigns an ID on the exit side of the road on the image, deletes the ID on the entrance side, and creates the above-described object map.

  The motion vector calculation unit 26 varies the position of each block to which an ID in the object map at time t is assigned in units of pixels in the corresponding block in the image at t and the image at time (t−1). By applying the block matching used in the MPEG encoding to the current block, the motion vector of the current block in the t image is obtained. The motion vector calculation unit 26 further estimates, for each cluster (moving object), a representative value of the obtained plurality of motion vectors, for example, the mode, median, or average value, as the motion vector of the moving object, and corresponds this to the ID. Let me remember. When the representative value is a median or an average value, a more accurate value may be obtained by obtaining the representative value after excluding a motion vector that is significantly away from the center distribution.

  The motion vector calculation unit 26 also estimates the motion vector of a moving object that is missing on the image.

  Note that the motion vector in MPEG is for data compression, and there is no concept of a motion vector in units of clusters. If the motion vector is small, optical flow estimation can be used as a useful technique in addition to block matching.

  When the color of each pixel in a block belonging to a certain moving object is almost the same, that is, when a block with less texture exists in the cluster, an accurate motion vector cannot be obtained by a technique such as block matching or optical flow.

Therefore, the motion vector calculation unit 26 has a predetermined distribution of such a block, for example, a dispersion degree of all pixel values in the block, for example, variance σ ² , standard deviation σ, average deviation MD, quadrature deviation QD, range R, or entropy. For blocks that are less than or equal to the value, a motion vector is not obtained and is not used for motion vector estimation of a moving object. Even in this case, there is a block group with a non-uniform color such as a tire portion, so the motion vector calculation unit 26 can estimate the motion vector of the moving object.

  An edge image can be used to determine whether a motion vector is to be obtained. That is, edge enhancement processing is performed on the original image, and then converted into a binary image. If the number of edge pixels of “1” in the block is equal to or less than a predetermined value, the block is determined to be unsuitable for obtaining a motion vector May be. An edge-enhanced image can be obtained by performing a simple filtering process by applying a secondary differential operator such as a primary differential operator or a Laplacian operator to the original image. For obtaining the edge image, a known method such as Sobel, Roberts, or Prewitt may be used.

  Next, a specific example of processing in the moving object update unit 27 will be described with reference to FIG.

  (A) The moving object update unit 27 reads the object map t from the object map memory 24.

  (B) The cluster is translated by -MV in the object map. Here, MV is a motion vector of this cluster. Further, the periphery of the moved cluster is expanded to the outside of one block.

  (C) The cluster of (t-1) is determined by determining, for each block, whether or not there is a moving object in the image of (t-1) within the enlarged range (area indicated by the bold line). .

  By doing in this way, the cars that were separated and captured by the camera at time t are adjacent at time (t−1), and the cluster depends on the distance and angle between the camera and the moving object at that time. Even if the number and shape of the blocks change, they can be recognized as different cars. For example, one cluster shown in FIG. 10A can be recognized as two cars. In FIG. 10, the dark portion is a block of a moving body.

  When the enlarged range overlaps between the moving objects, the moving object update unit 27 determines which moving object the overlapping block belongs to, and therefore determines that the overlapping block is an ID unknown block.

  The unknown ID determination unit 28 determines the moving object ID of the ID unknown block as follows.

  FIG. 11 schematically shows the input / output relationship of the unknown ID determination unit 28.

  In FIG. 11, F1 and F2 schematically show part of an image stored in the image memory 21 of FIG. 7 at times t and (t−1), respectively. One grid represents one pixel. BL1t and BL1 (t-1) are boundaries of the moving object M1 at times t and (t-1), respectively. BL2 (t-1) is a boundary line of the moving object M2 at time (t-1).

  Q1 and Q2 are diagrams showing an outline of the object map stored in the object map memory 24 of FIG. 7 corresponding to the images F1 and F2, respectively. One cell represents one block. The object map is composed of blocks having a mobile object ID and blocks not having a mobile object ID. In FIG. 11, for easy understanding, the boundary line of the moving object is also represented by a bold line in the object map. The hatching in the object maps Q1 and Q2 indicates that the block is assigned an ID. A block with a right-upward diagonal line has ID = 1, and a block with a right-down diagonal line has ID = 2.

  Since all of the blocks B1, B2, and B3 in the object map Q2 include part of the moving objects M1 and M2, the moving object update unit 27 determines to which moving object these blocks belong. Can not do it.

  The unknown ID determination unit 28 determines the IDs of the blocks B1, B2, and B3 based on the images F1, F2, the object maps Q1 and Q2, and the motion vectors of the moving objects M1 and M2, and obtains an object map Q3.

  Hereinafter, a method for determining which of the moving objects M1 and M2 the block B2 in the object map Q2 belongs to will be specifically described. This determination is made based on the magnitude relationship between the evaluation value when the block B2 is assumed to be the moving object M1 and the evaluation value when the block B2 is assumed to be the moving object M2. The evaluation value includes the following elements (1) to (3).

(1) Evaluation element UN related only to the object map at time (t-1)
Regarding the eight blocks adjacent to the block B2, it is considered that the greater the number N1 of blocks having ID = 1, the higher the probability that the block B2 is a block of the moving object M1. Therefore, an element UN (1) that evaluates that the ID of the block B2 is 1 is expressed by the following equation, for example.

UN (1) = α (N1-9) ²
Here, α is a positive constant. Consider that block B1 has the assumed ID = 1. N1 is the number of blocks having ID = 1 for a total of 9 blocks including the block B2 and 8 blocks adjacent thereto. It can be said that the smaller the value of UN (1), the higher the probability that the block B2 belongs to the moving object M1.

  Similarly, an element UN (2) that evaluates that the ID of the block B2 is 2 is expressed by the following equation.

UN (2) = α (N2-9) ²
Consider that block B2 has the assumed ID = 2. N2 is the number of blocks having ID = 2 with respect to a total of 9 blocks including the block B2 and 8 blocks adjacent to the block B2.

  In the case of FIG. 11, UN (1) = 9α and UN (2) = 49α. Therefore, judging from only the object map Q2, it can be said that the probability that the block B2 is ID = 1 is higher than the probability that the ID = 2.

(2) Evaluation element US related to the object map at time t and the motion vector of the moving object
In FIG. 12A, the object maps Q1 and Q2 in FIG. 11 are generally overlapped and hatching is deleted. MV1 is a motion vector of the moving object M1, and a dotted line is obtained by translating BL1t by -MV1.

  As shown in FIG. 12B, by moving the block B2 in parallel by MV1, a determination frame B21 estimated to correspond to the block B2 at time t as seen from the coordinate system fixed to the moving object M1 is obtained. It is done. Even if a part of the moving object M1 and a part of the moving object M2 are included in the block B2 at time (t-1), a part of the moving object M2 is not included in the determination frame B21 at time t. In this case, it is possible to more accurately estimate whether or not the block B2 has ID = 1 by looking inside the determination frame B21.

  It is considered that the probability that the block B2 has ID = 1 becomes higher as the area S1 of the upward-sloping hatched portion (ID = 1 portion) in the determination frame B21 is larger.

  Therefore, by creating a map in which each block of the object map is enlarged to the same 8 × 8 ID array, the number of pixels with ID = 1 is counted on this enlarged map as shown in FIG. The hatched area S1 is obtained. A hatched portion in FIG. 13A indicates an area with ID = 1. When S1 is 64, the probability that ID = 1 is maximized. Therefore, an element US (1) that evaluates that the ID of the block B2 is 1 is expressed by the following equation, for example.

US (1) = β (S1-64) ²
Here, β is a positive constant. It can be said that the smaller the value of US (1), the higher the probability that the block B2 belongs to the moving object M1.

  Similarly, an element US (2) that evaluates that the ID of the block B2 is 2 is represented by the following equation.

US (2) = β (S2-64) ²
Here, S2 is the number of ID = 2 in the determination frame B22 obtained by translating the block B2 by MV2 as shown in FIG. MV2 is a motion vector of the moving object M2.

(3) Evaluation element UD related to the image at time (t-1) and t and the motion vector of the moving object
It can be said that the higher the correlation between the local images in the images F2 and F1 in FIG. 11 corresponding to the block B2 and the determination frame B21 in FIG. 12B, the higher the probability that the block B2 is ID = 1. B2F and B21F shown in FIG. 13B indicate local images in the images F2 and F1 corresponding to the block B2 and the determination frame B21 in FIG. 12, respectively.

  An element UD (1) for evaluating that the ID of the block B2 is 1 is expressed by the following equation, for example.

UD (1) = γΣ | B21F (i, j) −B2F (i, j) |
Here, γ is a positive constant, B21F (i, j) and B2F (i, j) are the pixel values of the i-th row and j-th column in the local images B2F and B21F, respectively, and Σ is i = 1 -8 and j = 1 to 8 (sum of all pixels in the block).

  It can be said that the smaller the value of UD (1), the higher the probability that the block B2 belongs to the moving object M1.

  Similarly, an element UD (2) for evaluating that the ID of the block B2 is 2 is represented by the following expression.

UD (1) = γΣ | B22F (i, j) −B2F (i, j) |
Here, B22F (i, j) is the pixel value of the i-th row and j-th column in the local image B22F, and the local image B22F is the local image in the image F1 corresponding to the determination frame B22 in FIG. It is.

In order to estimate the ID of the unknown ID block more accurately, the above (1) to (3) are combined, and the value of the evaluation function U (1) = UN (1) + US (1) + UD (1),
Based on the magnitude relation of U (2) = UN (2) + US (2) + UD (2), the ID of the block B2 is determined. That is,
If the value of U12 = U (1) −U (2) is negative, ID = 1 is determined, and if it is positive, ID2 = 2 is determined. The minimum values of U (1) and U (2) are both 0. The smaller the value of U12, the higher the probability that block B2 is ID = 1.

  In the object map Q2 in FIG. 11, the IDs of the blocks B1 and B3 are not considered in the above specific example, but actually, the IDs of the blocks B1, B2, and B3 are assumed at the same time, and each of the blocks B1, B2, and B3 is assumed. The IDs of the blocks B1, B2, and B3 are determined so that the total sum of the evaluation functions is minimized.

  At this time, a minimum value is obtained by using a stochastic relaxation algorithm such as a Metropolis algorithm or a Gibbs Sampler algorithm. In addition, a known simulated annealing method is applied in order to prevent the minimum value of the evaluation value from being mistaken for the minimum value and to quickly escape from the minimum valley.

  Thereby, ID of a some ID unknown block can be determined more correctly and easily.

  The constants α, β, and γ are determined so that a moving object tracking success rate is increased by processing an actual image.

  The inventors have mathematically confirmed that it is appropriate to use the evaluation function as described above. In other words, the Markov Random Field model is extended and applied to the spatio-temporal image, and the energy distribution of the expressed image is optimized by the stochastic relaxation process. As a result, the result evaluation function is minimized. Became. This also applies to Example 2 described later.

  In the first embodiment, the unknown ID is determined based on the magnitude relationship of the evaluation function values on the assumption that the ID unknown block at time (t-1) belongs to the moving object M1 or M2 that approaches or overlaps each other. .

  However, there are cases where a part of three or more moving objects is included in one block. Also, for many moving objects, it is necessary to determine which moving objects are approaching each other.

  On the other hand, both the evaluation elements US and UD have a large value between blocks having a low degree of correlation at time t and time (t−1), and a small value between blocks having a high correlation. It is possible to determine the above only by looking at these values. Also, the evaluation value can be easily calculated.

  Therefore, in the second embodiment of the present invention, the unknown ID determination unit 28 in FIG. 7 considers that the ID unknown block has one of all IDs = 1 to m included in the object map at time t. The unknown ID is determined based on the magnitude relationship of the function values.

  The above-mentioned evaluation function of the ID unknown block BKi when the ID unknown block at time (t-1) is represented by BK1 to BKp and the ID of the ID unknown block BKi is assumed to be IDj (1 ≦ IDj ≦ m). If the value of U = UN + US + UD is represented by U (BKi, IDj), the IDs of the ID unknown blocks BK1 to BKp are determined as follows.

That is, assuming that IDs of ID unknown blocks BK1 to BKp are ID1 to IDp, respectively, the value of the evaluation function Ut = U (BK1, ID1) + U (BK2, ID2) +... + U (BKp, IDp) is calculated. . ID1 to IDp at which this value is minimized are obtained by repeated calculation, and the obtained one is determined as the ID of the ID unknown block BK1 to BKp. In this iterative calculation, the stochastic relaxation algorithm and the simulated annealing method are applied.

  Other points are the same as those of the first embodiment.

  Next, a case where a solution is obtained using the metropolis algorithm will be described with reference to FIG.

  (S1) Initial values ID10 to IDp0 are given to identification codes ID1 to IDp of ID unknown blocks BK1 to BKp and their solutions IID1 to IIDp. The initial value 1 is substituted for i and j (i ← 1, j ← 1). A large initial value UtBmax is given to the previous value UtB of Ut.

(S2) Value of Evaluation Function Ut = U (BK1, ID1) + U (BK2, ID2) +... + U (BKp, IDp)
Calculate

  (S3, S4) If Ut <UtB, then IDi ← IDi, UtB ← Ut.

(S5) j ← j + 1
(S6, S7) If j ≦ m, the value of IDi of the block BKi is updated and the process returns to step S2. Otherwise, the process proceeds to the next step S6.

(S8) IDi ← IIDi, i ← i + 1
(S9) If i ≦ p, the process returns to step S2, otherwise the process ends.

  In this way, the IDs of the ID unknown blocks BK1 to BKp are determined as IID1 to IIDp, respectively.

  Theoretically, an appropriate ID can be determined by the above calculation with all blocks at time (t-1) as unknown ID blocks.

  Note that the present invention includes various other modifications.

  For example, depending on the required processing speed, the entire processing in the image processing apparatus in FIG. 7 can be executed by a computer.

  Further, the evaluation function only needs to include at least one of the three elements. The above function of each element is an example, and various types that change in one direction (monotonic change) as the estimation accuracy increases can be used.

(Supplementary Note 1) An image storage unit that stores a plurality of time-series images and background images;
An image processing unit, the image processing unit,
(1) For the image at time t among the plurality of time-series images, a moving object is recognized as compared with the background image, and an identification code is assigned to the recognized moving object,
(2) An identification code is given to the moving object included in the image at the time (t−1) from the correlation between the image at the time t and the image at the time (t−1) prior to the time t.
Accordingly, the non-separated moving object included in the image at the time (t-1) is divided.

    (Supplementary note 2) The moving object recognition apparatus according to supplementary note 1, wherein the image processing unit tracks the moving object by repeating the process (2).

    (Supplementary note 3) The moving object recognition device according to supplementary note 2, wherein the image processing unit determines whether or not the trajectory obtained by the tracking violates a traffic rule.

(Appendix 4) The image processing unit
(3) For the image at time (t-1), the presence or absence of a moving object is determined by comparison with the background image,
In (2) above, it is assumed that each region determined to have a moving object in the image at time (t-1) has an identification code for any of the moving objects in the image at time t. Based on the magnitude relationship between the evaluation function values at the time, an identification code to be assigned to the area is determined.
The moving object recognizing device according to supplementary note 1, wherein:

(Supplementary Note 5) The image processing unit
(4) Obtain a motion vector of each moving object included in the image at time t,
In (2) above,
Assuming that the first area on the moving object included in the image at time (t-1) has the identification code X, the second area on the image at time t corresponding to the first area. Is estimated based on the motion vector,
A value of the evaluation function including a degree of correlation between the first region and the second region is obtained;
Determining an identification code to be assigned to the first region based on the magnitude relationship of the value of the evaluation function obtained for each of the plurality of identification codes X;
The moving object recognizing device according to supplementary note 4, characterized by:

(Appendix 6) The image processing unit
(5) The image at the time (t-1) is divided into blocks composed of a plurality of pixels,
(6) Temporarily assigning an identification code to each block including a moving object,
The evaluation function includes a function that increases the probability that the target block has the same identification code as the number of blocks having the same identification code as the target block increases with respect to the block adjacent to the target block.
The moving object recognizing device according to appendix 5, wherein

(Appendix 7) In the image processing unit (4),
Dividing the first image at the time t and the second image before or after in time into blocks composed of a plurality of pixels;
For each block including a part of a moving object, find a block image in the first image and a block image in the second image that are similar to each other, and obtain a motion vector of the block;
Determining a representative value of each motion vector of a plurality of consecutive blocks having a motion vector as a motion vector of a moving object corresponding to the plurality of consecutive blocks;
The moving object recognizing device according to appendix 5, wherein

(Appendix 8) an electronic camera that images a moving object;
An image storage unit for storing a plurality of time-series images and background images captured by the electronic camera;
An image processing unit, the image processing unit,
(1) For the image at time t among the plurality of time-series images, a moving object is recognized as compared with the background image, and an identification code is assigned to the recognized moving object,
(2) An identification code is given to the moving object included in the image at the time (t−1) from the correlation between the image at the time t and the image at the time (t−1) prior to the time t.
Accordingly, the non-separated moving object included in the image at the time (t-1) is divided.

(Supplementary Note 9) In the moving object recognition method for processing a plurality of time-series images,
For the image at time t, the moving object is recognized compared with the background image, and an identification code is given to the recognized moving object,
An identification code is assigned to a moving object included in the image at time (t−1) from the correlation between the image at time t and an image at time (t−1) prior to time t.
Accordingly, the non-separated moving object included in the image at the time (t-1) is divided.

(Supplementary Note 10) In a computer-readable recording medium in which a program for processing a plurality of time-series images is recorded, the program includes:
For the image at time t, the moving object is recognized compared with the background image, and an identification code is given to the recognized moving object,
An identification code is assigned to a moving object included in the image at time (t−1) from the correlation between the image at time t and an image at time (t−1) prior to time t.
Thus, a non-separated moving object included in the image at the time (t-1) is divided.

10-12, 11a-11d Road 15 Electronic camera 20 Image processing device 21 Image memory 22 Background image generation unit 23 Tracking processing unit 24 Object map memory 25 ID generation / deletion unit 26 Motion vector calculation unit 27 Moving object update unit 28 Unknown ID Determination unit 29 Trajectory memory 30 Traffic violation determination unit 31 Traffic rule storage unit 40 External storage device M1, M2 Vehicle

Claims (8)

A storage device for storing time-series images and programs;
A processor coupled to the storage device;
The program for the processor
(1) By block matching between the image at time t and the image at time (t−1) among the plurality of time series images, the time (from time t of the moving object included in the image at time (t−1) ( determining a motion vector to t-1) and a moving object identification code in units of blocks including a plurality of pixels;
The non-separating moving object is divided by performing the process (1) once or by repeating the moving object once and tracking the moving object backward in time.
In the moving object recognition apparatus, when the program causes the processor to determine a moving object identification code of one block included in the image at the time (t−1),
Evaluation of correlation with respect to the block in each region i obtained by moving the block onto the image at the time t based on the motion vectors of a plurality of moving objects on the image at the time (t−1) The value of the function is obtained with respect to the identification code i of the moving object corresponding to the motion vector to this region, and the moving object identification code corresponding to the relative optimum value among the values for each of the plurality of moving objects is obtained. Give to the block,
The moving object recognition apparatus characterized by the above-mentioned. A storage device for storing time-series images and programs;
A processor coupled to the storage device;
The program for the processor
(1) By block matching between the image at time t and the image at time (t−1) among the plurality of time series images, the time (from time t of the moving object included in the image at time (t−1) ( determining a motion vector to t-1) and a moving object identification code in units of blocks including a plurality of pixels;
The non-separating moving object is divided by performing the process (1) once or by repeating the moving object once and tracking the moving object backward in time.
A moving object recognition apparatus, wherein the program causes the processor to determine a moving object identification code of a plurality of blocks included in the image at the time (t-1), for each block of the plurality of blocks. ,
Evaluation of correlation with respect to the block in each region i obtained by moving the block onto the image at the time t based on the motion vectors of a plurality of moving objects on the image at the time (t−1) The value of the function is determined with respect to the identification code i of the moving object corresponding to the motion vector to this region;
Each of the plurality of blocks is given a moving object identification code in which the sum of the values of the evaluation functions of the plurality of blocks is a relative optimal value.
The moving object recognition apparatus characterized by the above-mentioned. The evaluation function is represented by the sum of a plurality of sub-evaluation functions, and a first sub-evaluation function that becomes a more appropriate value as the total area of the block pieces of the moving object identification code i included in the region i is larger is defined as As one of the evaluation functions,
The moving object recognition apparatus according to claim 1 or 2, wherein The evaluation function is represented by the sum of a plurality of sub-evaluation functions, and the one block of absolute values of differences between the pixel values included in the one block and the corresponding pixel values included in the region i Including, as one of the plurality of sub-evaluation functions, a second sub-evaluation function that becomes an appropriate value as the sum of the pixels included in
The moving object recognition apparatus according to claim 1 or 2, wherein The evaluation function becomes more appropriate as the number of blocks of the moving object identification code i among the moving object identification codes assigned to the blocks around the one block on the image at time (t−1) increases. A third sub-evaluation function as one of the plurality of sub-evaluation functions,
The moving object recognition apparatus according to claim 3 or 4, wherein The evaluation function is
A first sub-evaluation function that becomes an appropriate value as the total area of the block pieces of the moving object identification code i included in the region i increases;
The smaller the sum of the absolute value of the difference between the value of the pixel included in the one block and the value of the corresponding pixel included in the region i is, the more appropriate the value is. Two sub-evaluation functions;
Third sub-evaluation that is more appropriate as the number of blocks of the moving object identification code i is larger among the moving object identification codes assigned to the blocks around the one block on the image at the time (t-1). Functions and
The moving object recognition apparatus according to claim 1, wherein the moving object recognition apparatus includes the sum of In a computer-readable recording medium on which a program for processing a plurality of time-series images is recorded, the program is
(1) By block matching between the image at time t and the image at time (t−1) among the plurality of time-series images, time (t) from time t of the moving object included in the image at time (t−1) -1) to determine a motion vector and a moving object identification code for each block including a plurality of pixels,
The processing (1) is performed once or repeatedly, and the moving object is tracked back in time to divide the non-separating moving object,
When the program causes the computer to determine a moving object identification code of one block included in the image at the time (t-1),
Evaluation of correlation with respect to the block in each region i obtained by moving the block onto the image at the time t based on the motion vectors of a plurality of moving objects on the image at the time (t−1) The value of the function is obtained with respect to the identification code of the moving object corresponding to the motion vector to this region, and the moving object identification code corresponding to the relative optimum value among the values for each of the plurality of moving objects is calculated. To give to the block,
A computer-readable recording medium. In a computer-readable recording medium on which a program for processing a plurality of time-series images is recorded, the program is
(1) By block matching between the image at time t and the image at time (t−1) among the plurality of time series images, the time (from time t of the moving object included in the image at time (t−1) ( a motion vector to t-1) and a moving object identification code are determined in units of blocks including a plurality of pixels;
The process (1) is performed once or repeated a plurality of times to track the moving object backward in time, thereby dividing the non-separated moving object,
When the program causes the computer to determine moving object identification codes of a plurality of blocks included in the image at the time (t-1), for each block of the plurality of blocks,
Correlation evaluation function for each block of region i obtained by moving the block on the image at time t based on the motion vectors of a plurality of moving objects on the image at time (t-1) For the moving object identification code i corresponding to the motion vector to this region,
Each of the plurality of blocks is given a moving object identification code in which the sum of the values of the evaluation functions of the plurality of blocks is a relative optimal value.
A computer-readable recording medium.

Images