JPH09270014A - Extraction device for moving body - Google Patents

Abstract

PROBLEM TO BE SOLVED: To correctly extract moving bodies even when they cross each other by separating an outline model into plural outline models at its self-crossing part or integrating different outline models which cross each other into one. SOLUTION: An outline model self-crossing decision part 6 decides whether or not part of an outline model deformed by an outline model deformation part 5 crosses or touches another part of the outline model and an outline model separation part 7 cuts the outline model at the touching or crossing part and separates it into plural outline models. An outline model mutual- crossing decision part 8 decides whether or not plural different outline models stored in an outline model storage part 4 contact or cross each other. An outline model integration part 9 integrates different outline models, decided by the outline model mutul-crossing decision part 8, to contact or cross each other, into one outline model.

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a device for extracting a plurality of moving objects from a moving image obtained from a CCD, an infrared camera or the like.

[0002]

2. Description of the Related Art As a technique for extracting and tracking an object from an image, an information processing Vol. 30, No. 9, pp. 1047 to pp. 1057 is especially used as an extraction and tracking technique that is not easily affected by noise.
SNAKES described in US Pat. According to this technique, if the outline of the object to be extracted is given in advance as the initial outline, the initial outline is attracted to the image feature such as an edge and deformed, and the outline of the object can be extracted. The tracking of a moving object by this SNAKE only sets the contour of the extraction result in a frame of a moving image as an initial contour in the next frame, and the set contour is also attracted to an image feature such as an edge, It can extract moving objects. SNAKE
The tracking method by S has an advantage that it is not necessary to associate moving objects independently extracted in each frame of a moving image with each other, which is difficult in the tracking method before SNAKES. Contour tracking method for moving objects and
-242247 image processing device is known.

[0003]

However, in the above-mentioned SNAKES, there is a problem that a plurality of objects cannot be extracted and tracked unless the number of objects to be extracted and tracked and a rough area are specified in advance. . Therefore, when the number of objects or the number of images to be processed is large, the processing load on this setting increases, and it is desired to automatically know how many objects are included in the image. The number of people
There was a problem that it could not meet the demand for detecting the number of people in the air conditioning control. In addition, SNAKE
In the conventional tracking device based on S, when tracking multiple objects at the same time, if the objects intersect each other while moving, each SNAKES is attracted to other than the object that it is tracking itself and must be tracked correctly. I can't. Further, the appearance of the third object on the way was not considered, and a human had to intervene to add a new contour.

In view of such a point, the present invention automatically and accurately extracts and tracks even if they intersect during the tracking of a plurality of objects without giving the number of moving objects to be extracted and tracked and their outlines in advance. It is an object of the present invention to provide a moving object extraction / tracking device that can be used.

Further, according to the present invention, even if a new moving object appears in the middle of tracking, it is automatically extracted and extracted without human intervention.
An object is to provide a device for extracting and tracking a moving object that can be tracked.

Further, the present invention integrates heterogeneous image information such as a thermal image and a visible image even if one image cannot be accurately extracted / tracked due to information loss, noise, etc. It is an object of the present invention to provide a moving object extraction / tracking device capable of automatically extracting / tracking a moving object.

Further, according to the present invention, the number of times of intersection with another moving object is recorded for each moving object, and it is possible to detect an object which is inconsistent with another moving object according to its size.
For example, it is an object of the present invention to provide a moving object extraction / tracking device capable of extracting a person who is suspiciously moving at an airport, a hotel lobby, a convenience store, or the like.

Further, the present invention makes it possible to recognize the situation where a plurality of moving objects are placed by calculating the speed of the moving objects and the distance between the moving objects from the extraction / tracking results. An object of the present invention is to provide a device for extracting a moving object, which can monitor a movement of a vehicle by such as a vehicle and can recognize a dangerous driving such as an accident, a traffic jam, or an inability to keep a distance between vehicles.

[0009]

According to a first aspect of the present invention, there is provided an image storage unit for continuously storing at least two images input in time series, and a motion detection unit for detecting motion information from the images. And a contour model storage unit for storing a contour model surrounding at least one moving object included in the image,
A contour model deforming unit that deforms the contour model according to a predetermined rule, and a contour that determines whether or not a part of the contour model contracted and deformed by the contour model deforming unit contacts or intersects another portion of the contour model. A model self-intersection determining unit, and a contour model splitting unit that splits the contour model into a plurality of contour models by cutting the contour model at a contact or intersection when it is determined that there is contact or intersection by the contour model self-intersection determining unit, A contour model mutual intersection determination unit that determines presence or absence of mutual contact or intersection of a plurality of different contour models stored in the contour model storage unit, and at least 2 that contact or intersection is detected by the contour model mutual intersection determination unit. A contour model integration unit that integrates two different contour models into one contour model, and the contour model that satisfies a predetermined condition An extraction completion determination unit that determines that extraction of a moving object is completed by stopping deformation, and a feature amount of a region in an image surrounded by the contour model whose contraction deformation has been stopped by the extraction completion determination unit is a feature amount of a moving object. And a moving object collating unit that associates moving objects between images at different times, and a moving object extraction device.

According to the second aspect of the present invention, a contour model adding section is added to newly add one contour model having the same size as the frame of the image each time the image to be processed is updated in time series. It is a device for extracting a characteristic moving object.

According to a third aspect of the present invention, there is provided a heterogeneous image storage unit for continuously storing at least two visible images and thermal images input in the same field of view in a time series, and moving from the visible image and the thermal image. Heterogeneous image information integrated motion detection section for detecting information of at least one, and at least one included in the image
A contour model storage unit that stores a contour model surrounding one moving object, a contour model transformation unit that transforms the contour model according to a predetermined rule, and a portion of the contour model transformed by the previous contour model transformation unit A contour model self-intersection determination unit that determines whether or not another part of the model has touched or intersected, and the contour at the contact or intersection portion when the contour model self-intersection determination unit determines that there is contact or intersection. A contour model splitting unit that cuts the model and divides it into a plurality of contour models, and a contour model mutual intersection determination that determines whether or not a plurality of different contour models stored in the contour model storage unit contact or intersect each other Section and at least two different contour models determined to have contact or intersection by the contour model mutual intersection determination section as one ring. A contour model integration unit that integrates into a model, an extraction completion determination unit that determines that the extraction of a moving object is completed by stopping the deformation of the contour model that satisfies a predetermined condition, and the contour whose deformation is stopped by the extraction completion determination unit. A heterogeneous image feature amount extraction storage unit that extracts and stores the feature amount of each region of the visible image and the thermal image surrounded by the model, and a plurality of feature amounts from the heterogeneous images stored in the heterogeneous image feature extraction storage unit And a heterogeneous image feature amount integrated collating unit that associates moving objects between images at different times according to the above.

According to a fourth aspect of the present invention, a unit for counting the number of mutual intersections for each moving object is the unit for counting the number of mutual intersections for each contour model, and the number of mutual intersections counted by the unit for counting the number of mutual intersections for each contour model. A movement characterized by adding an unmatched moving object determination unit that determines an object that is extracted and tracked by a contour model that is a predetermined number of times or more per hour as an object that has a mismatched movement with respect to other moving objects. It is an object extraction device.

According to a fifth aspect of the present invention, a moving speed calculation unit for calculating the speed of a moving object from the position of the contour model in each image obtained in time series and a distance between contour models for calculating the distance between different contour models. A moving object extraction device characterized by adding a calculation unit.

[0014]

Embodiments of the present invention will be described below with reference to the drawings.

(First Embodiment) FIG. 1 is a configuration diagram showing a first embodiment. In FIG. 1, 1 is a camera for capturing a moving object, 2 is an image storage unit that digitizes images captured by the camera 1 in time series, and stores at least two consecutive images, and 3 is stored in the image storage unit 2. The motion detection unit 4 for detecting motion information from the temporally continuous images, and the contour candidate points for searching the contour position of the moving object in the image stored in the image storage unit 2 and the connecting order thereof. The stored contour model storage unit 5 is a contour for deforming the contour model by moving the contour candidate points stored in the contour model storage unit 4 in a direction that minimizes the position and the evaluation function defined from the image. The model deforming unit 6 is a contour model self-intersection judging unit for judging whether or not a part of the contour model deformed by the contour model deforming unit 5 is in contact with or intersecting another part of the contour model, and 7 is a ring. When the model self-intersection determination unit 6 determines that there is a contact or an intersection, the contour model dividing unit that cuts the contour model at the contact or intersection portion and divides it into a plurality of contour models, 8 is stored in the contour model storage unit 4. Contour model mutual intersection determination unit for determining the presence or absence of mutual contact or intersection of a plurality of different contour models, and 9 is at least two different contour models for which the contour model mutual intersection determination unit 8 determines that there is contact or intersection. Is integrated into one contour model, the contour model integration unit 10 is configured to store contours stored in the contour model storage unit 4 when the distance between two contour candidate points connected to each other is within a predetermined range. At least one new contour candidate point is added between the candidate points, and a series of three contour candidate points connected to each other by the connection order is added. A contour candidate point generating / erasing unit that deletes contour candidate points that are connecting points of the two line segments when the angle formed by the two line segments is within a predetermined range. When the evaluation function does not decrease due to the movement, the movement of the contour candidate points is stopped to stop the deformation of the contour model and it is determined that the extraction of the moving object is completed. Is a feature amount extraction storage unit that extracts and stores the feature amount of the region in the image surrounded by the stopped contour model as the feature amount of the moving object, and the moving object matching unit that associates the moving objects between images at different times. A control unit 15 is connected to each unit of 1 to 14 and controls the start and stop of the operation of each unit or the exchange of data between each unit, such as a memory such as a ROM or a RAM and a CP.
The computer is composed of U, and is connected to an input device 16 such as a keyboard and a mouse as shown in FIG. 1 and a display device 17 such as a CRT via various interfaces.

Next, the operation of this embodiment will be described. FIG. 2 is a flowchart showing the operation of the moving object extraction / tracking device according to the present embodiment. FIG. 3 is a part of an image of a moving object passing each other in order to explain the operation of the present embodiment more clearly. That is, since the present invention can extract and track a moving object from such an image, it can be applied to, for example, a person monitoring system in security inside and outside a building. The image storage unit 1 is usually composed of a RAM, and each pixel data of the digitized image is
For example, the brightness is stored by representing it with 8 bits. The operation procedure of this embodiment is shown below.

[STEP1a] Capture of Image A video obtained in time series from the camera 1 is digitized and stored in the image storage unit 2 as an image I _T , T = 0, 1, ....
Hereinafter, the image I _T is processed in order from T = 0.

[STEP2a] Installation of initial contour When T = 0, one polygon (contour model) including all moving objects in the image I ₀ (x, y) stored in the image storage unit 2 is set. The initial contour candidate points v _mi (x
_It is stored in the contour model storage unit 4 as _mi (0), y _mi (0)) (i = 1,2, ..., Nm; m = 1,2, ..., m_max). m represents a number given to a plurality of contour models, and initially m_max = 1. Further, i represents the connecting order of the contour candidate points forming the contour model. The contour model storage unit 4 is usually composed of a RAM. The operator may operate the input device 15 such as a keyboard or a mouse while watching the image displayed on the display device 16 to input an initial contour candidate point and store it in the contour model storage unit 4. Alternatively, a point obtained by equally dividing a side of a rectangle approximately equal to the size of an image into Nm may be stored in the contour model storage unit 4 in advance as an initial contour candidate point.

[STEP 3a] Motion Detection The motion detection unit 3 stores the 2 values stored in the image storage unit 2.
From each of the images I _T and I _{T + 1} , each contour candidate point v _mi (x
The movement vector at the position of _mi , y _mi ) is detected. The movement vector can be obtained by the method of least squares based on the gradient method, for example. In the gradient method, the intensity of the image is I _T (x,
y), the partial derivative with respect to the time and space, and the apparent velocity (u, v) = (dx / dt, dy of the object at the point (x, y) in the image.
Use the following constraint equation for / dt).

[0020]

[Equation 1]

Assuming now that the movement vector is uniform in the local small area near each contour candidate point, that is, the constraint equation of the equation 1 has the same solution in the local small area S, each contour candidate The motion vector at the point position minimizes (u,
v) is required.

[0022]

[Equation 2]

Therefore, (u, v) satisfying ∂E / ∂u = 0 and ∂E / ∂v = 0 may be obtained by the following equation. However, (i, j) is a point
Represents the coordinates of a point in the neighborhood area S of (x, y).

[0024]

(Equation 3)

However, i and j are the x and y coordinates of the pixels belonging to the small area S in the image, and Σ is calculated for (i, j) εS. Det is calculated by the following formula.

[0026]

(Equation 4)

[STEP4a] Deformation of Contour Model The contour model deformation section 5 defines the energy E _snake (v of the contour model defined by the sum of the energy terms represented by the following equations.
_The contour model is deformed by moving the contour candidate points stored in the contour model storage unit so as to minimize _mi ).

1) E _spline representing the smoothness of the contour model

[0029]

(Equation 5)

The moving direction of the contour candidate point that minimizes the equation 5 is such that the contour model contracts.

2) E _area corresponding to the area of the closed region surrounded by the contour model

[0032]

(Equation 6)

However, x _{mNm + 1} = x _m1 and y _{mNm + 1} = y _m1 .
The movement direction of the contour candidate point that minimizes the equation 6 is a direction perpendicular to the contour model. If you use E _area ,
The contour of the concave object can be extracted.

3) E _dist that averages the distances between contour candidate points

[0035]

(Equation 7)

However, d _av is the average distance between contour candidate points. 4) Gradient magnitude of the image brightness I _T (v _mi ) E _edge

[0037]

(Equation 8)

The moving direction of the contour candidate point that minimizes the equation (8) is the direction in which the brightness gradient increases, that is, the direction toward the contour portion of the object. Here, the number 5-9
_{w sp1, w sp 2, w} area, w dist, w edge> 0 are the respective weight coefficients of each energy terms.

A concrete deformation of the contour model is, for example, contour candidate points v _mi (x _mi (t), y _mi (t)) (i = 1,2, ..., Nm; m = 1,2 ,. ) energy when moving to a position in the vicinity in a pixel unit _{E s} nake (v _mi) is carried out by moving once smallest position. t (i = 0,1, ...) represents the number of movements of each point,
The position near the contour candidate point v _mi (x _mi , y _mi ) is, for example, v
_{m i} (x _mi-1 , y _mi-1 ), v _mi (x _mi-1 , y _mi ), v _mi (x _mi-1 ,
y _{mi + 1} ), v _mi (x _mi , y _mi-1 ), v _mi (x _{m i} , y _{mi + 1} ), v _mi (x _{mi + 1} ,
It suffices to consider 8 neighborhoods of y _mi-1 ), v _mi (x _{mi + 1} , y _mi ), and v _mi (x _{mi + 1} , y _{mi + 1} ). However, the energy E _snake (v
_{If mi} ) increases, do not move. At this time, the number of contour candidate points that have moved is stored in C _move .

Here, since a moving object is extracted, when moving each contour candidate point, when the absolute value of the motion vector at the position of each contour candidate point obtained in STEP 2a is less than a predetermined value, the image of the image is The weight coefficient w _edge of the energy E _edge representing the magnitude of the gradient of the brightness _IT (v _mi ) is set to 0. As a result, the contour model is drawn only to the moving edge, and the moving object can be extracted.

[STEP 5a] Judgment of Self-Intersection of Contour Model
2 segment v_mkv_{mk + 1}And line segment v_mhv_{mh + 1}(k = 1,2, ..., Nm; h = 1,
2, ..., Nm) for all combinations of contacts or intersections
Set. However, the two connected lines are excluded. Line segment v_mkv
_{mk + 1}And line segment v_mhv _{mh + 1}If touches or intersects, then
There are real numbers p (0 ≤ p ≤ 1) and q (0 ≤ q ≤ 1) that satisfy.

[0042]

[Equation 9]

Equation 9 is a simultaneous equation concerning p and q,
It has a solution when the determinant det calculated by the following equation is not 0. Note that when det = 0, the line segment v _mk v _{mk + 1} and the line segment v _mh v _{mh + 1} may _{match.Therefore} , in this case, the intersection is determined before the calculation of the determinant det. I'll do it.

[0044]

(Equation 10)

When det ≠ 0, p, q calculated by the following equation
Respectively satisfy 0 ≦ p ≦ 1,0 ≦ q ≦ 1, the contour model self-intersects at the intersection (p, q) of the line segment v _mk v _{mk + 1} and the line segment v _mh v _{mh + 1.} It is judged that the contact is made and STEP6a is executed.
Execute STEP7a.

[0046]

[Equation 11]

[STEP6a] Division of Contour Model With respect to the contour model judged by the contour model self-intersection judging section 6 to have self-intersection, the contour model dividing section 7 divides the contour model into a plurality of pieces and returns to STEP2a. The self-intersecting contour model is split at contour candidate points that form a line segment having an intersection. E _area as the energy of the contour model
An example of the self-intersection of a typical contour model that occurs when is used is shown in FIG. As shown in FIG. 4, the self-intersecting contour model is split into two by connecting contour candidate points v _mk and v _{mh + 1} , and v _mh and v _{mk + 1} , respectively. As a result, the concatenation order is {v _m1,, ..., v _mk , v _{mh + 1} , ..., v _mn } and {v _{mk + 1} , ...
, V _mh }, the set of contour candidate points is newly divided into independent contour models. These newly independent contour models are stored in the contour model storage unit 4. Also, since the number of contour models increases due to division,
Update the value of m_max.

[STEP 7a] Judgment of Mutual Intersection of Contour Model The contour model mutual intersection judgment unit 8 determines two different contour models v _fk and v _gh (f = 1,2, ..., m_max; k = 1,2, ...,). Nf; g = 1,2,
,, m_max; h = 1,2, ..., Ng; f ≠ g), the mutual contact or intersection is determined. Specifically, contact or intersection is determined for all combinations of the line segment v _fk v _{fk + 1} forming the contour model v _fk and the line segment v _gh v _{gh + 1} forming the contour model v _gh . Mutual contact or intersection of two different contour models v _fk and v _gh , ie line segment v _fk v _{fk + 1} and line segment v
_{If gh} v _{gh + 1} touches or intersects, there exist real numbers p (0 ≦ p ≦ 1) and q (0 ≦ q ≦ 1) that satisfy the following equation.

[0049]

(Equation 12)

Equation 12 is a simultaneous equation regarding p and q, and has a solution when the determinant det calculated by the following equation is not zero. When det = 0, the line segment v _fk v _{fk + 1} and the line segment v _gh v _{gh + 1}
May intersect, so the intersection in this case is the determinant
This match is judged before the calculation of det.

[0051]

(Equation 13)

When det ≠ 0, p, q calculated by the following equation
, Respectively, satisfying 0 ≦ p ≦ 1, 0 ≦ q ≦ 1, the two different contour models are the intersection points of the line segment v _fk v _{fk + 1} and the line segment v _gh v _{gh + 1} .
At (p, q), it is determined that they have touched or intersected with each other, and STEP8a is executed. Otherwise, STEP9a is executed.

[0053]

[Equation 14]

[STEP8a] Consolidation of Contour Models With respect to at least two contour models that are determined to be in contact with or intersecting each other by the contour model mutual intersection determination unit 8, the contour model integration unit 9 configures all of them. The convex polygon including the contour candidate points is obtained, and the set of contour candidate points forming the convex polygon is stored in the contour model storage unit 4 as an integrated contour model. Also, since the number of contour models decreases due to the integration, the value of the number m_max is updated. FIG. 5 shows typical mutual intersections that occur in two different contour models and a contour model obtained by integrating them.

[STEP 9a] Determining Generation / Elimination of Contour Candidate Points The contour candidate point generation / elimination unit 10 produces / eliminates new contour candidate points. For example, contour candidate points are generated in the following cases. When the distance between adjacent contour candidate points satisfies | v _{mi + 1} -v _mi |> D _Th , the contour candidate point generation / annihilation unit 10 newly separates between the two points v _mi and v _{m i + 1.} Generate points. However, D _Th is the maximum distance between predetermined contour candidate points. At this time, if even one contour candidate point is generated
G _flag = 1 and G _flag = 0 if not generated. By the additional generation of contour candidate points, it is possible to compensate for the number of contour candidate points per object, which decreases due to division, and to correctly extract the contours of a plurality of objects.

[0056] In addition, the contour candidate point creation and annihilation unit 10 is, for example, cosθ> θ _Th (θ is _vmi v _mi-1 and v _mi v angle of _{mi + 1)}
The contour candidate points v _mi satisfying are deleted. However, θ _Th is a threshold value for determining the sharpness of a predetermined contour model. At this time, if even one contour candidate point disappears
D _flag = 1, and D _flag = 0 if it does not disappear. By this procedure of disappearing contour candidate points, contour candidate points that are stopped due to noise at positions other than the contour of the object are deleted,
False contour extraction can be avoided.

[0057] [STEP10a] Extraction completion determination extraction completion determination section 11 determines completion of the extraction of the moving object in the image I _T. For all contour models, no contour candidate points have moved (C _move ≤ C _Th ; C _Th ≥ 0 is a threshold value for the number of contour candidate points that have been moved in advance) and new contour candidate points are generated. When there is no disappearance (G
_{If flag} = 0 and D _flag = 0), or t ≥ tmax (tmax is the maximum number of movements of the contour candidate points set in advance), it is determined that the extraction of the moving object in the image I _T is completed, and STEP
Go to 11a. Otherwise, set T = T + 1 and return to STEP3a. When the extraction is completed, if different contour models are displayed in different colors on the display unit 16, the visibility of different moving objects is improved, for example, in a surveillance system.

[STEP 11a] Extraction of feature quantity of moving object_mi(M = 1,2,
,, image surrounded by m_max) I _TThe feature amount of the region of
Extract as a trait. For example, the luminance histogram
Product distribution H_m(k) (k = 0,1, ..., 255 for 8bit) is the feature quantity
Available. When T = 0, the feature quantity H_m(k) is the feature extraction
H in memory 12_l(k) (l = 1,2, ..., l_max; l_max is the feature quantity
The number of moving objects remembered), and T = T + 1
Then go back to STEP 3a. Feature value H when T> 0_m(k) is extracted and STEP12
to a.

[STEP12a] Collation of Moving Object The moving object extracted by the moving object collating unit 13 at the current (T = T_present) image I _{T_presen t} (T_present> 0) to be processed is the feature amount extraction storage unit 12 Which moving object is extracted in the past (T = T_past; T_past <T_present) stored in is compared with the feature similarity (distance d between histograms). Specifically, the distance d between the histograms represented by the following equation is obtained for all combinations of m and l, and for those whose value is less than or equal to a predetermined threshold D _match , from the combination having the minimum value, In order, the moving object having the characteristic amount H _m (k) extracted by the contour model v _mi and the moving object having the characteristic amount H _l (k) stored in the characteristic amount extraction storage unit 12 are associated with each other.

[0060]

(Equation 15)

Here, since the frequency of the histogram differs depending on the size of the extracted area, it is normalized with the maximum frequency. Further, for the associated moving object, the feature amount H _l (k) stored in the feature amount extraction storage unit 12 is
The contour model v _mi is replaced with the feature amount H _m (k) extracted. Further, regarding the moving objects that are not associated with each other, the feature amount is newly stored in the feature amount extraction storage unit 12. When the matching of the moving object is completed, T = T + 1 is set and the process returns to STEP 3a.

The correspondence does not necessarily have to be performed for all contour models and all moving objects stored in the feature amount extraction storage unit 12. For example, for a contour model that has been fused once, it is performed when it is divided again. I'll do it. Further, in the feature amount extraction storage unit 12, if the color of the contour model from which the moving object is extracted is also stored together with the feature amount, the color of the contour model from which the corresponding moving object is extracted is stored. The same color can be displayed on the display unit 16, and the visibility of extraction / tracking of the same moving object is improved.

According to the above procedure, as shown in FIG.
One contour model (Fig. 6 (a)) that includes the whole set of moving objects (here, pedestrians) is divided into two and each of two pedestrians is extracted and tracked (Fig. 6 (b)). (D)) Two pedestrians overlap (Fig. 6 (e)), and at the same time, the two contour models are integrated into one (Fig. 6 (f)) and tracking continues (Fig. 6 (g). )), And eventually, when two pedestrians pass each other, one contour model is divided into two again, and two pedestrians can be independently extracted and tracked (Figs. 6 (h) to (i)). ).

As described above, according to the first embodiment, even when a plurality of moving objects pass each other by dividing the contour models by judging the self-intersection and the mutual intersection of the contour models. Can be extracted and tracked.

(Second Embodiment) Next, a second embodiment of the moving object extracting / tracking apparatus according to the present invention will be described. In the first embodiment, the self-intersection and the mutual intersection of the contour models are determined, and the contour models are divided and integrated to extract and extract even a plurality of moving objects.
Explained that you can trace. In the second embodiment, even when another moving object that was not initially within the view angle of the camera newly enters the view angle of the camera on the way, the moving object can be extracted and tracked. It has been taken into consideration. In the first embodiment, the case where there is always the same moving object within the field of view of the camera has been described as an example. However, in a monitoring system inside or outside a building or plant, movements that come into the field of view one after another. It is necessary to extract and track objects.

FIG. 7 is a block diagram of the second embodiment.
The same components as those of the first embodiment shown in FIG. 1 are designated by the same reference numerals, and detailed description thereof will be omitted. The difference between the second embodiment and the first embodiment is that in the first embodiment,
Every time an image at the next time is stored in the image storage unit 2, one contour model having the same size as the frame of the image is newly added and the contour model adding unit 17 is added. As a result, moving objects that come into the angle of view one after another are extracted.
It becomes possible to track.

The operation of the contour extracting apparatus of the second embodiment having the above configuration will be described with reference to the flowchart of FIG. However, STEP1b of the second embodiment
~ STEP2b, STEP4b, STEP6b ~ STEP7b and STEP9b ~ STEP
13b are STEP1a to STEP2a, S of the first embodiment, respectively.
Since the procedure is exactly the same as that of STEP3a, STEP5a to STEP6a and STEP8a to STEP12a, different STEP3b, STEP5b and STEP
Only 8b will be described.

[STEP3b] Addition of Contour Model The contour model addition unit 15 has a rectangular contour which is approximately equal to the size of the image I _T (x, y) stored in the image storage unit 2 when T> 0. The model is newly stored in the contour model storage unit 4. At this time, the number of contour models is updated to m_max = m_max + 1.

[STEP5b] Deformation of Contour Model When the contour model deforming unit 5 deforms the contour model, when the contour candidate points of the newly added contour model are in the existing contour model model, the brightness of the image I The weight coefficient w _edge of the energy E _edge representing the magnitude of the gradient of _T (v _mi ) is set to 0. This prevents the newly added contour model from extracting again the moving object already extracted by the existing contour model. Note that the same processing is performed for the contour model in which the newly added contour model is divided by self-intersection.

[STEP8b] Judgment of Mutual Intersection of Contour Model When the contour model mutual intersection judgment unit 7 judges mutual contact or intersection of two different contour models, the newly added contour model and the division of the contour model. Mutual intersection with the newly generated contour model is not determined. This prevents the moving object extracted by the existing contour model from being extracted again.

According to the above procedure, as shown in FIG.
When a new moving object appears, the newly added contour model is added (Fig. 9 (b)), and the moving object extracted by the existing contour model is not extracted again (Fig. 9 (c)). It is possible to extract only moving objects that appear in (Fig. 9
(d)).

As described above, according to the second embodiment, each time a new image is input, a new contour model corresponding to the size of the image is newly added. Extraction and tracking of a newly appearing moving object without re-extracting the existing moving object because the mutual intersection is not judged and the edge in the area surrounded by the existing contour model is not drawn. be able to.

(Third Embodiment) Next, a third embodiment of the contour extracting apparatus of the present invention will be described. In the first and second embodiments, it has been described that a plurality of moving objects can be extracted and tracked using one type of image. Furthermore, the third embodiment integrates different types of image information such as a thermal image or a visible image when accurate extraction / tracking cannot be performed with one type of image due to information loss, noise, etc. It considers the extraction and tracking of moving objects.

FIG. 10 is a block diagram of the present embodiment. The same components as those of the first embodiment shown in FIG. 1 are designated by the same reference numerals and detailed description thereof will be omitted. The difference between the third embodiment and the first embodiment is that in the first embodiment,
An infrared camera 18, a half mirror 19, and a mirror 20 for picking up a thermal image having the same field of view as the visible image are added, and further, the image storage unit 2 in the first embodiment is provided with different image information of the thermal image and the visible image. Is replaced with a heterogeneous image storage unit 21 for storing the same, and the same motion detection unit 3 is replaced with a heterogeneous image information integrated motion detection unit 22 that detects motion information from a thermal image of an infrared camera and a visible image of a CCD camera, and the same feature extraction is performed. The storage unit 12 is replaced with a different image feature extraction / storage unit 23 that extracts and stores the feature amount of the area surrounded by the contour model from each of the thermal image and the visible image, and the same moving object matching unit 13 is used to extract and store the different image feature. That is, the replacement is performed by the heterogeneous image feature amount integrated collating unit 24 which associates a moving object with a plurality of feature amounts from heterogeneous images stored in the unit 23.

The operation of the contour extracting apparatus according to the third embodiment having the above configuration will be described with reference to the flowchart of FIG. However, only STEP1c, STEP3c, STEP4c, STEP11c, and STEP12c different from the first embodiment will be described.

[STEP1c] Capture of image Digitize the images obtained in time series from the camera 1 and the infrared camera 18 and digitize them as visible images I _T ^CCD and thermal images I _T ^TH , T = 0,1, ... The image is stored in the image storage unit 21. Hereinafter, the image is processed by the I _T ^CCD and the thermal image I _T ^TH in order from T = 0.

[STEP3c] Motion Detection The heterogeneous image information integrated motion detector 22 detects the visible images I _T ^CCD , I _{T + 1} ^CCD and the thermal images I _T ^TH , I _{T +} stored in the heterogeneous image storage 21. The image information of four pieces of ₁ ^TH is integrated to detect the movement vector at the position of each contour candidate point v _mi (x _mi , y _mi ). The constraint method of the gradient method shown in (Equation 1) is the visible image I _T
Since it holds for each of the ^CCD and the thermal image I _T ^TH , generally the movement vector (u, v) at the point (x, y) in the image
Can be obtained by solving the following simultaneous equations.

[0078]

(Equation 16)

However,

[0080]

[Equation 17]

It is assumed that But here, each contour candidate point
v The movement vector is uniform in the local small area S ((x, y) ∈ S) in the vicinity of _mi (x _mi , y _mi ) for the visible image and the infrared image, that is, the same for the visible image and the infrared image. It is obtained by the least squares method, assuming that the constraint equation of (Equation 16) has the same solution in the local small area S of. As a result, it is possible to more accurately calculate the movement vector by reducing the influence of noise and supplementing the information that is not captured in each image alone.

[STEP4c] Deformation of Contour Model In the contour model deforming section 5, the energy E of the contour model is
_The contour model is transformed so as to minimize the _snake (v _mi ), but the brightness gradient of the thermal image given by the following equation is added as new energy in addition to the brightness gradient of the visible image.

[0083]

(Equation 18)

The moving direction of the contour candidate point that minimizes the equation 18 is the direction toward the edge in the thermal image.
Here, w _edge ^TH > 0 is a weighting coefficient of this energy term. By introducing this energy, for example, the contour of a person in white clothes in front of a white wall, which does not exist in the visible image, can be extracted because the contour appears in the thermal image.

[STEP 11c] Extraction of feature quantity of moving object The heterogeneous image feature quantity extraction storage unit 23 allows the contour model _vmi to be extracted.
Visible image I _T ^CCD and thermal image I surrounded by (m = 1,2, ..., m_max)
The feature amount of the area of _T ^TH is extracted as the feature amount of the moving object. As the feature amount, for example, a cumulative distribution of a histogram of brightness and temperature can be used.

[STEP12c] Collation of moving object The heterogeneous information-integrated moving object collating unit 24 determines, for example, the luminance.
Features such as cumulative distribution of temperature and temperature histograms
The quantities are integrated to associate moving objects. Multiple different
For the integration of features, see Technical Report PRU94-131 (1995-03).
Based on the Dempster-Shafer theory described in pp.9-16
It can be done by a method. In this embodiment, visible
Similarity of features extracted from each image and thermal image
From the degrees, the confidence level of which moving object
To associate moving objects with the highest certainty
Just do it. Here, the contour model v_miIs being extracted by
When the moving image is stored in the feature extraction storage unit 12,
L_max moving objects w _lAny of (l = 1,2, ..., l_max)
The procedure of checking whether or not is described.

Now, for the moving object extracted by the contour model v _mi , histograms of brightness and temperature extracted from the visible and thermal images as feature amounts are H _m ^CCD and H _m ^{T H} , respectively, and the feature amount extraction storage unit 12 Let H _l ^CCD and H _l ^TH be the features extracted from the thermal and visible images of the moving object w _l stored in, respectively, and the features of the moving object and the moving object w _l extracted by the contour model v _mi. Similarity of
S _CCD (w _l ) and S _TH (w _l ) are given by the following formulas.

[0088]

[Equation 19]

However, the similarity is α ₁ ≧ α ₂ ≧ ... ≧ α
_{The rearrangement is performed as l_max} , β ₁ ≧ β ₂ ≧ ... ≧ β _{l_max} . At this time, for each of the feature amounts extracted from the visible and thermal images, which moving object w _l (set A) is (is included in) which moving object is extracted by the contour model v _mi.
The certainty factor (basic probability) is expressed by the following equations 20 and 21.

[0090]

(Equation 20)

[0091]

(Equation 21)

The certainty factors (basic probabilities) obtained from these different types of feature quantities are integrated by the following equation.

[0093]

(Equation 22)

The integrated basic probability m of (Equation 22)
Probability of decision P (w _l ) calculated from _{CCD_TH} (A) by the following formula
Is determined to be the moving object extracted by the contour model v _mi .

[0095]

(Equation 23)

| A | represents the number of elements of A. As described above, according to the third embodiment, not only visible images but also thermal images are integrated to calculate a movement vector, a contour model is transformed, and a plurality of images extracted from those images are extracted. Since the moving object is collated using the feature amount, the moving object such as a person can be more accurately extracted and tracked.

(Fourth Embodiment) Next, a fourth embodiment of the contour extracting apparatus of the present invention will be described. First to third
In the above embodiment, extraction and tracking of a plurality of moving objects from an image obtained from a CCD camera or the like have been described. Furthermore, in the fourth embodiment, the number of times of intersection with another moving object is recorded for each moving object that is being extracted and tracked, and the movement that is inconsistent with the other moving object depending on its size. It enables the detection of objects, such as airports, hotel lobbies,
This is in consideration of extracting a person who is suspiciously moving at a convenience store or the like.

FIG. 12 is a block diagram of the fourth embodiment. The same components as those of the first embodiment shown in FIG. 1 are designated by the same reference numerals and detailed description thereof will be omitted. The difference between the fourth embodiment and the first embodiment is that the moving object-based mutual intersection number storage unit 25 records the number of mutual intersections for each moving object and the moving object-based mutual intersection number storage. An inconsistent moving object in which an object which is extracted and tracked by a contour model whose number of mutual intersections counted by the unit 25 is a predetermined number of times or more per unit time is determined as an object having an inconsistent motion with respect to other moving objects. That is, the determination unit 26 is added.

The operation of the contour extracting apparatus of the fourth embodiment configured as above will be described with reference to the flowchart of FIG. However, only STEP8d and STEP14d different from the first embodiment will be described.

[STEP8d] Mutual Intersection Determination of Contour Model and Storage of Number of Intersections The contour model mutual intersection determination unit 7 stores two different contour models v _fk and v _gh (f), which are stored in the contour model storage unit 4.
= 1,2, ..., m_max; k = 1,2, ..., Nf; g = 1,2, ..., m_max; h = 1,2, ...,
Ng; f ≠ g) mutual contact or intersection is determined for all combinations, and when there is contact or intersection, the number of times for each moving object stored in the feature amount extraction storage unit 12 is calculated for each moving object. It is stored in the intersection count storage unit 25.

[STEP14d] Judgment of Inconsistent Moving Object The inconsistent moving object determination unit 26 determines that the number of mutual intersections stored in the moving object-specific mutual intersection number storage unit 25 is a predetermined mutual intersection per unit time. A moving object that is equal to or greater than the threshold value of the number of times is determined as an object that is in a different motion that is inconsistent with other moving objects. In addition, for the contour model that is extracting the object with inconsistent movement,
If the color is displayed on the display unit 16 in a color different from that of the other contour models, the visibility of a suspicious person or the like is improved in, for example, a surveillance system.

As described above, according to the fourth embodiment, by recording the number of times of intersection with another moving object for each moving object that is being extracted and tracked, it is possible to determine whether the number of times of intersection is large or small. It is possible to detect a moving object that is inconsistent with the moving object.

(Fifth Embodiment) Next, a fifth embodiment of the contour extracting apparatus of the present invention will be described. First to fourth
In the above embodiment, it was explained that a moving object can be extracted and tracked from an image obtained from a CCD camera or the like, and rough information about the moving object such as suspicious movement can be obtained. Furthermore, the fifth embodiment can recognize the situation in which a plurality of moving objects are placed by calculating the speed of moving objects and the distance between moving objects from the results of extraction / tracking. Considering the realization of a system that can monitor the movement of the vehicle with the vehicle, recognize dangerous driving such as accidents, traffic jams, and inability to keep the distance between vehicles, and transmit the information to the driver and the control station. .

FIG. 14 is a block diagram of the fifth embodiment. The same components as those of the first embodiment shown in FIG. 1 are designated by the same reference numerals and detailed description thereof will be omitted. The difference between the fifth embodiment and the first embodiment is that there is an inter-contour model distance calculation unit 27 that calculates the distance between different contour models stored in the contour model storage unit 4, and for each moving object. In addition, a moving speed calculation unit 28 that calculates the speed of the moving object from the position of the contour model in each time-series image is added.

The operation of the moving object extracting / tracking apparatus of the fifth embodiment configured as described above will be described with reference to the flowchart of FIG. However, only STEP1e, STEP11e, and STEP14e different from the first embodiment will be described.

[STEP1e] Image acquisition For a vehicle on a highway or the like, the images obtained from the camera 1 in time series are digitized and stored in the image storage unit 2.

[STEP11e] Calculation of distance between contour models After the transformation is stopped by the extraction completion determination unit 11, the distance between different contour models stored in the contour model storage unit 4 is calculated as the distance between contour model calculation unit. Calculate by
The distance between the contour models is the minimum value of the distances between the contour candidate points forming the different contour models. The control unit 14
When this distance is smaller than a predetermined threshold value, it is determined that the inter-vehicle distance is short, and the road information display version 29
A message such as "Keep the distance between cars" is displayed on.

[STEP14e] Calculation of moving speed For each moving object associated between images obtained at different times by the moving object matching unit 13, for example, each movement is made from the change in distance per unit time of the position of the center of gravity of the contour model. Calculate the moving speed of an object. The control unit 14 continues for a predetermined time (for example, 10 minutes) when the moving speeds calculated from the images of a plurality of cameras installed at appropriate intervals are all below a predetermined value (for example, 5 km / h). If so, it is determined that the road is congested.

When the contour model mutual intersection determination unit 8 determines that there is a contact or an intersection between different contour models extracting different vehicles, the control unit 14 causes a contact between vehicles or a rear-end collision accident, or It can be determined that the vehicle distance is not maintained. For example, when the contour models intersect with each other, and when the cameras 1 capture the vehicles from directly above, it can be determined that the vehicles have come into contact with each other or have a rear-end collision. At this time, it is possible to prevent a further accident by displaying on the information display plate 29 that there is an accident for the following vehicle. Further, when the camera 1 captures a vehicle from an oblique direction, different contour models intersect each other because a plurality of vehicles appear to overlap each other from the camera 1 due to a short inter-vehicle distance. It is possible to judge that it is dangerous driving without keeping the distance, and it is possible to display a message similar to STEP 11e.

As described above, according to the fifth embodiment, a vehicle contact or a rear-end collision accident is determined by the intersection determination of different contour models, and the distance and speed between moving objects are determined from the extraction / tracking results. By calculating, it is possible to recognize a traffic situation on a highway, such as dangerous driving such as non-holding of inter-vehicle distance, traffic congestion, or the like.

[0111]

According to the first embodiment, by dividing a contour model into a plurality of contour models at a self-intersection portion or integrating them into one different contour model that intersects each other, a plurality of moving objects can be obtained. Even if they intersect, they can be correctly extracted.

According to the second embodiment, by adding one contour model corresponding to the frame of the image for each frame, the third moving object newly entering the angle of view can be obtained. Can also be extracted.

According to the third embodiment, different types of image information such as a thermal image and a visible image are integrated to perform motion estimation and matching of moving objects, so that a plurality of moving objects can be extracted more accurately. be able to.

According to the fourth embodiment, by counting the number of mutual intersections for each contour model, a moving object tracked by a large number of contour models behaves differently from surrounding moving objects. You can judge that Therefore, for example, it is possible to extract a person who is suspiciously moving at an airport, a hotel lobby, a convenience store, or the like.

According to the fifth embodiment, by extracting and tracking a vehicle on a highway or a tunnel, calculating the distance between contour models tracking different vehicles, and determining mutual intersections, Recognizing a dangerous state where the distance between vehicles is approaching, warning with an information display version and recognition of a collision accident,
Further, the speed of the moving object is calculated from the time change of the position of the contour model, and the traffic congestion can be recognized from the moving speed and its maintenance time.

[Brief description of drawings]

FIG. 1 is a block diagram of a first embodiment of the present invention.

FIG. 2 is a flowchart showing an operation procedure of the first embodiment of the present invention.

FIG. 3 is a diagram showing an example of an image in which moving objects pass each other.

FIG. 4 is a diagram showing an example of intersection / division of a contour model.

FIG. 5 is a diagram showing an example of mutual intersection and integration of contour models.

FIG. 6 is a diagram showing an extraction result according to the first embodiment.

FIG. 7 is a block diagram of a second embodiment of the present invention.

FIG. 8 is a flowchart showing an operation procedure of the second embodiment of the present invention.

FIG. 9 is a diagram showing extraction results for a new moving object invaded.

FIG. 10 is a block diagram of a third embodiment of the present invention.

FIG. 11 is a flowchart showing an operation procedure of the third embodiment of the present invention.

FIG. 12 is a block diagram of a fourth embodiment of the present invention.

FIG. 13 is a flowchart showing an operation procedure of the fourth embodiment of the present invention.

FIG. 14 is a block diagram of a fifth embodiment of the present invention.

FIG. 15 is a flowchart showing an operation procedure according to the fifth embodiment of the present invention.

[Explanation of symbols]

 1 camera 2 image storage unit 3 motion detection unit 4 contour model storage unit 5 contour model transformation unit 6 contour model self-intersection determination unit 7 contour model division unit 8 contour model mutual intersection determination unit 9 contour model integration unit 10 contour candidate point generation / Disappearing part 11 Extraction completion determination part 12 Feature amount extraction storage part 13 Moving object collation part 14 Control part 15 Input part 16 Display part 17 Contour model addition part 18 Infrared camera 19 Half mirror 20 Mirror 21 Different image storage part 22 Different image information integration Motion detection unit 23 Heterogeneous image feature amount extraction storage unit 24 Heterogeneous information integrated collation unit 25 Muting intersection count storage unit for each moving object 26 Mismatching moving object determination unit 27 Contour model distance calculation unit 28 Moving speed calculation unit 29 Road information display board

Claims (6)

[Claims] 1. An image storage unit for continuously storing at least two images input in time series, a motion detection unit for detecting motion information from the image, and at least one moving object included in the image. A contour model storage unit that stores a plurality of contour candidate points that form a contour model and a connection order of the contour models, and the contour candidate points that form the contour model have a minimum evaluation function defined from the position and the image. Whether or not a part of the contour model deformed by the contour model deforming part and the other part of the contour model contact or intersect with another part of the contour model by moving in the direction of And a contour model self-intersection determining unit that determines whether there is a contact or an intersection when the contour model self-intersection determining unit determines that there is a contact or an intersection. A contour model splitting unit that cuts a Dell and splits it into a plurality of contour models, and a contour model mutual intersection determination that determines whether or not a plurality of different contour models stored in the contour model storage unit contact or intersect each other Section, a contour model integration section that integrates at least two different contour models determined to have contact or intersection by the contour model mutual intersection determination section into one contour model, and a predetermined number of the contour candidate points. When the evaluation function does not decrease due to the movement, the movement of the contour candidate points is stopped to stop the deformation of the contour model and the extraction completion determination unit that determines that the extraction of the moving object is completed, and the extraction completion determination unit performs the deformation. A feature amount extraction storage unit that extracts and stores a feature amount of an area in an image surrounded by the stopped contour model as a feature amount of a moving object; Extraction apparatus for a mobile object, characterized in that a moving object collation section by between the images of different times the feature quantity stored in the output memory unit to associate the moving object. 2. Every time a new image is input to the image storage unit in time series, a new contour model having the same size as the frame of the image is newly added.
The moving object extracting apparatus according to claim 1, further comprising a contour model adding unit additionally stored in the contour model storage unit. 3. A heterogeneous image storage unit for continuously storing at least two visible images and thermal images input in the same visual field in time series, and a heterogeneous device for detecting motion information from the visible images and thermal images. An image information integrated motion detection unit,
A contour model storage unit that stores a plurality of contour candidate points that form a contour model that encloses at least one moving object included in the image with a closed curve and a connection order of the contour model, and a position of the contour candidate points that form the contour model. A contour model transformation unit that transforms the contour model by moving in a direction that minimizes the evaluation function defined from the image, and a part of the contour model transformed by the contour model transformation unit is different from the contour model. The contour model self-intersection determination unit that determines whether or not the portion has touched or intersected, and cuts the contour model at the contact or intersection portion when the contour model self-intersection determination unit determines that there is contact or intersection. The contour model dividing unit for dividing the contour model into a plurality of contour models and a plurality of different contour models stored in the contour model storage unit are mutually A contour model mutual intersection determination unit that determines whether or not a contact or an intersection has occurred, and a contour that integrates at least two different contour models determined to have a contact or an intersection by the contour model mutual intersection determination unit into a single contour model A model integration unit, stopping movement of the contour candidate points and stopping deformation of the contour model when movement of the evaluation function does not decrease for a predetermined number of the contour candidate points, and determines that extraction of a moving object is completed. An extraction completion determination unit, and a heterogeneous image feature extraction storage unit that extracts and stores a feature amount of each region of the visible image and the thermal image surrounded by the contour model whose deformation is stopped by the extraction completion determination unit, Different types of moving objects are associated with images at different times according to a plurality of feature amounts from different types of images stored in the different type image feature extraction storage unit. Extraction apparatus for a mobile object, characterized by comprising an image characteristic quantity integration verification unit. 4. A moving object that stores the number of mutual intersections determined by the contour model mutual intersection determination unit for each moving object whose characteristic amount is stored in the characteristic amount extraction storage unit or the heterogeneous image characteristic amount extraction storage unit. A moving object whose mutual intersection count stored in the separate mutual intersection count storage unit and the moving object-specific mutual intersection count storage unit is equal to or more than a predetermined number of times per unit time causes a movement inconsistent with other moving objects. 4. The moving object extraction device according to claim 1, further comprising an unmatched moving object determination unit for determining an object. 5. An inter-contour model distance calculating section for calculating the distance between different contour models by referring to the contour model storage section or the contour model mutual intersection determination section, the contour model storage section and the feature amount extraction storage section, or 5. A moving speed calculation unit for calculating the moving speed of each moving object based on the position of the contour model by referring to the different type image feature amount extraction storage unit is added. Moving object extraction device. 6. When at least a distance between two contour candidate points connected to each other in a connection order stored in a contour model storage unit is within a predetermined range, at least one new contour is provided between the contour candidate points. A candidate point is added, and when the angle formed by the two line segments connected by the three contour candidate points connected to each other in the connection order is within a predetermined range, the contour candidate point that is the connection point of the two line segments is selected. 6. The movement according to claim 1, further comprising a contour candidate point generation / disappearance unit that deletes the contour candidate points stored in the contour model storage unit and updates the connection order thereof. Object extraction device.