KR20220079183A - Device for sensing image motion, apparatus and method for tracking target using the same - Google Patents

Abstract

The present invention relates to an image fluctuation sensor and a target tracking apparatus and method using the same, and more particularly, to an image fluctuation sensor used in an electro-optical tracking system, and a target tracking apparatus and method using the same.
An image fluctuation detector according to an embodiment of the present invention is an image fluctuation sensor that detects whether fluctuation occurs in an image, and detects whether fluctuation occurs by calculating an amount of change in image entropy of sequentially input images.

Description

Image motion detector and target tracking device and method using the same

The present invention relates to an image fluctuation sensor and a target tracking apparatus and method using the same, and more particularly, to an image fluctuation sensor used in an electro-optical tracking system, and a target tracking apparatus and method using the same.

In general, a target tracking device using an Electro Optical Tracking System (EOTS) for tracking and detecting a target using image information acquired by an image sensor is mounted in a platform operating system.

The target tracking device automatically tracks a moving target even in the presence of disturbances such as vibration in tanks, ships, and aircraft, and maintains a stable line of sight for the observer, the position, speed, and angular velocity of the target. provide information such as

The target tracking device may generally include an image input device and a target detector, and the target detector includes all mechanical devices and electronic control devices that allow tracking to occur based on the calculated target position value. Therefore, in composing the target detector, it is accompanied by difficulties in simultaneously controlling not only the electronic components modeled and calculated relatively ideally but also the mechanical device including non-linear effects, and disturbances and images caused by mechanical vibrations generated during movement. A time delay due to processing greatly affects the performance of the entire target tracking device.

The target detection technology used in the target detector has been developed from NCC (normalized cross correlation) to DCF (Discriminative correlation filter) technology, and recently the MOSSE (Minimum output sum of squared error) method and the size of the target tracking area can be varied. The development of discriminative scale space tracking (DSST) showed significant improvement in target tracking speed and accuracy compared to other machine learning-based algorithms.

However, such a target detection technology exhibits better accuracy and performance than the conventional methods in a general target tracking scenario, but has a problem in that the tracking accuracy is remarkably deteriorated under actual rough driving conditions and rapid camera gaze movement conditions. In addition, the target detection technology using the learning process has a limitation in slowing the operation speed, so it is difficult to be applied as a real-time target tracking technology of an electro-optical tracking system for unmanned surveillance and reconnaissance and unmanned moving objects.

KR 10-2011-0122265 A

The present invention provides an image fluctuation detector capable of quickly and accurately detecting a sudden camera movement, and a target tracking apparatus and method using the same.

An image fluctuation detector according to an embodiment of the present invention is an image fluctuation sensor that detects whether fluctuation occurs in an image, and detects whether fluctuation occurs by calculating an amount of change in image entropy of sequentially input images.

In addition, the image fluctuation sensor according to an embodiment of the present invention is an image fluctuation detector for detecting whether fluctuation occurs in an image, comprising: an entropy calculator for calculating image entropy of sequentially input images; a change amount calculator for calculating a change amount of the calculated image entropy; and a signal generator for generating a shaking signal when the amount of change in image entropy calculated by the change amount calculator satisfies a logic condition.

The change amount calculator may calculate a change amount of image entropy per unit time.

The sequentially input image includes an infrared image and a visible ray image, and the change amount calculator may calculate an amount of change in an infrared image entropy calculated from the infrared image and a change amount of a visible ray image entropy calculated from the visible ray image, respectively. .

The entropy calculator may calculate the image entropy according to Equations 1 and 2 below.

[Equation 1]

[Equation 2]

(where I _i is the intensity value of the image, │I _i │ is the number of pixels in the image having the corresponding intensity value, n is the total number of pixels in the image, L is all the intensity values that the image can have, pdf(I _i ) is the probability density function of the image, and E _I is the image entropy.)

The change amount calculator may calculate the change amount of image entropy by Equation 3 below.

[Equation 3]

(Here, ΔE _I (t) is the amount of change in image entropy, E _I is the image entropy, t is the current image input time, and t-1 is the previous image input time.)

The signal generator satisfies all of a first logic condition for a change amount of infrared image entropy, a second logic condition for a change amount of visible ray image entropy, and a third logic condition for a correlation value between infrared image entropy and visible ray image entropy In this case, it may generate an oscillation signal.

The signal generator may generate the shaking signal according to Equations 4 and 5 below.

[Equation 4]

[Equation 5]

(here, GES(t) is the fluctuation signal, ΔE _IR (t) is the amount of change in the infrared image entropy, ΔE _RGB (t) is the change amount of the visible light image entropy, and R(t) is the tracking correlation for sequentially input images Response value, T _IR is the integration time of the infrared image, and T _RGB is the initialization frame rate of the visible light image.)

In addition, the target tracking apparatus according to an embodiment of the present invention, an image input device for receiving images sequentially; an image fluctuation detector for detecting whether or not fluctuation of the image occurs by calculating an amount of change in image entropy of the image input from the image input device; a movement tracker for calculating movement information of an image when a shaking signal is input from the image shaking sensor; and a target detector for detecting a target in an image according to the movement information calculated from the movement tracker.

The image input unit may include: a first image input unit for receiving an infrared image; and a second image input unit for receiving a visible ray image, wherein the image fluctuation detector calculates an amount of change in infrared image entropy calculated from the infrared image and a change amount of visible ray image entropy calculated from the visible ray image, respectively. Thus, fluctuation occurs when the first logical condition for the change amount of the infrared image entropy, the second logical condition for the change amount of the visible ray image entropy, and the third logical condition for the correlation value between the infrared image entropy and the visible ray image entropy are all satisfied signal can be generated.

The movement tracker may calculate movement information of the image in the form of a vector including the movement direction and movement distance of the image.

The movement tracker may calculate movement information of the image by calculating the movement direction and movement distance of the feature point in the image.

The movement tracker may calculate movement information of an image by adjusting the number of feature points according to image entropy.

The image input unit may include: a first image input unit for receiving an infrared image; and a second image input unit configured to receive a visible ray image, wherein the movement tracker may calculate movement information of the image by adjusting the number of feature points according to the infrared image entropy calculated from the infrared image.

The target detector may detect a target by moving a previously detected area within the image in a direction offsetting the moving direction and the moving distance of the image.

On the other hand, the target tracking method according to an embodiment of the present invention, the process of receiving images sequentially; Calculating the amount of change in the image entropy of the input image and detecting whether the fluctuation of the image occurs; calculating movement information of an image according to the shaking when the occurrence of shaking is detected; and detecting a target in the image according to the calculated movement information.

The process of sequentially receiving the images may include: sequentially receiving infrared images; and sequentially receiving visible ray images, wherein the process of sequentially receiving the infrared image and the process of sequentially receiving the visible ray image may be performed simultaneously.

The process of calculating the movement information of the image may include: determining the number of feature points according to an infrared image entropy calculated from the infrared image; and calculating the movement direction and movement distance of the determined number of feature points to calculate movement information of the image.

In the process of determining the number of feature points, the number of feature points may be determined by multiplying the number of all feature points in the image by an adaptive scale factor.

The scale factor may be determined by Equation 6 below.

[Equation 6]

(Where q _A is a constant, E _IR is infrared image entropy, and SF is a scale factor.)

The q _A may have a value selected from 0.01 to 0.02.

In the process of detecting the target, the target may be detected by moving a previously detected area within the image in a direction offsetting the moving direction and moving distance of the image calculated by Equation 7 below.

[Equation 7]

(here, u _t-1 ^x is the X-axis coordinate value of the feature point included in the image before movement, u _t ^x is the X-axis coordinate value of the feature point included in the image after movement, and u _t-1 ^y is included in the image before movement The Y-axis coordinate value of the acquired feature point, u _t ^y is the Y-axis coordinate value of the feature point included in the moved image, d ^x is the movement distance in the X-axis direction, d ^y is the movement distance in the Y-axis direction, and T is the frame number. it means.)

According to an embodiment of the present invention, when fluctuations occur in an image, it is possible to quickly and accurately detect whether fluctuations occur from the amount of change in image entropy with respect to the image.

In addition, when the occurrence of the shaking motion is detected, the target detection area can be corrected by calculating a motion vector according to the shaking motion, thereby continuously maintaining tracking without missing the target, and calculating the correct target coordinate value.

1 is a view schematically showing an image fluctuation sensor according to an embodiment of the present invention.
2 is a view showing a state of generating a shaking signal according to an embodiment of the present invention.
3 is a view showing a verification result of a shaking signal generated according to an embodiment of the present invention;
4 is a view schematically showing a target tracking apparatus according to an embodiment of the present invention.
5 is a diagram illustrating a state in which a scale factor is calculated according to an embodiment of the present invention.
6 is a diagram illustrating a state in which a feature point is adaptively extracted in a target tracking apparatus according to an embodiment of the present invention.
7 is a view showing an operation state of the target tracking apparatus according to an embodiment of the present invention.
8 is a view showing the performance of the target tracking apparatus according to an embodiment of the present invention.

Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings. However, the present invention is not limited to the embodiments disclosed below, but will be implemented in various different forms, and only the embodiments of the present invention allow the disclosure of the present invention to be complete, and the scope of the invention to those of ordinary skill in the art It is provided to fully inform the In order to describe the invention in detail, the drawings may be exaggerated, and like reference numerals refer to like elements in the drawings.

1 is a diagram schematically showing an image fluctuation sensor according to an embodiment of the present invention. Also, FIG. 2 is a diagram illustrating a state in which a shaking signal is generated according to an embodiment of the present invention, and FIG. 3 is a view showing a verification result of the shaking signal generated according to an embodiment of the present invention.

1 to 3 , the image fluctuation sensor 10 according to an embodiment of the present invention is an image fluctuation detector 10 that detects whether fluctuations occur in an image. It detects whether fluctuation occurs by calculating the amount of change.

More specifically, the image fluctuation sensor 10 according to an embodiment of the present invention includes an entropy calculator 110 for calculating an image entropy of sequentially input images, and a change amount calculation for calculating a change amount of the calculated image entropy. and a signal generator 130 for generating a shaking signal when the amount of change in image entropy calculated by the change amount calculator satisfies a logic condition.

In general, target tracking devices such as tanks, ships, and aircraft are used in static background image conditions due to their long surveillance distances. However, such a target tracking device must perform a target tracking task in disturbances caused by rapid camera movement in rough terrain as well as rapid camera movement. Here, the movement of the camera is driven by a mechanical device, and since the driving direction and distance information have already been presented, it does not significantly affect the target tracking. However, there is a problem in that it is not possible to accurately determine whether an image is shaken due to a disturbance caused by a sudden camera movement. Accordingly, for reliable target tracking, it is necessary to quickly and accurately detect whether a fluctuation has occurred in an image.

The image fluctuation detector 10 according to an embodiment of the present invention may be implemented as a sensor, and detects whether fluctuation occurs by calculating an amount of change in image entropy of sequentially input images.

To this end, the image fluctuation sensor 10 according to an embodiment of the present invention includes an entropy calculator 110 for calculating the image entropy of sequentially input images, and a change amount calculator for calculating the change amount of the calculated image entropy ( 120) and a signal generator 130 that generates a shaking signal when the amount of change in image entropy calculated by the change amount calculator satisfies a logic condition.

The image input to the image fluctuation sensor 10 may include an infrared image, a visible ray image, or an infrared image and a visible ray image. Electro Optical Tracking System (EOTS) generally receives an infrared image and a visible light image at the same time and tracks and detects a target using image information obtained. Hereinafter, the input image is an infrared image and a visible light image Although a case including all of the ray images will be described as an example, it goes without saying that the image input from the image fluctuation sensor 10 may be only an infrared image or only a visible ray image.

The entropy calculator 110 receives an image and calculates a feature of the image as entropy. In general, entropy is one of thermodynamic quantities indicating the state of a material system related to heat quantity and temperature. However, in an embodiment of the present invention, the concept of entropy is applied to an image to calculate the characteristics of the image.

Entropy of information theory was proposed by Claide Elwood Shannon in 1948 and used as an indicator for information chaos. It has also been used to calculate the minimum number of bits required for lossless compression of information in data analysis and communication systems. The basic formula for entropy proposed by Shannon is as Equation 1 below.

[Equation 1]

Here, p(a _j ) denotes a probability for a state a _j , and _A denotes a finite set of all possible states MA .

When the basic entropy equation proposed by Shannon is applied to an image, the probability density function of the image can be calculated as in Equation 2 below.

[Equation 2]

Here, I _i is the intensity value of the image, │I _i │ is the number of pixels of the image having the corresponding intensity value, n is the total number of pixels in the image, L is the maximum intensity value that the image can have, and pdf(I _i ) is It means the probability density function of the image.

The intensity value of the image may have a value of 1 to 256, for example, in the case of an 8-bit gray scale image. In addition, all intensity values L that the image may have may have a value of 256 in the case of an 8-bit grayscale image. Using this, the image entropy value can be calculated as in Equation 3 below.

[Equation 3]

Here, E _I means image entropy.

Of course, such image entropy can be used to detect blurring of an image due to camera movement. However, when images are sequentially input, since each image has different image entropy, it is possible to detect whether fluctuations have occurred in the image when the amount of change in image entropy is calculated.

The change amount calculator 120 calculates the amount of change in image entropy calculated as above, where the amount of change in image entropy may mean a change in image entropy per unit time, that is, the gradient of image entropy.

Here, the change amount calculator 120 may calculate the gradient of the image entropy by Equation 4 below.

[Equation 4]

Here, ΔE _I (t) is the amount of change in image entropy, E _I is the image entropy, t is the current image input time, and t-1 is the previous image input time.

That is, when images are sequentially input, the time at which the current image is input and the time at which the previous image is input may be expressed as t and t-1, respectively, and the image entropy at the current time and the previous time is E ^t , E , respectively. It can be expressed as ^t-1 . There is an example in which image entropy is used to detect motion blur, but in the present invention, the amount of change in image entropy is calculated using the amount of change in image entropy.

As described above, the sequentially input images may include an infrared image and a visible ray image. In this case, the change amount calculator 120 may calculate the change amount of the infrared image entropy calculated from the infrared image and the change amount of the visible ray image entropy calculated from the visible ray image, respectively. That is, the infrared image entropy may be calculated as E _IR , and the visible ray image entropy may be calculated as E _RGB . The process of calculating the gradient of the entropy of the infrared image from the infrared image and the process of calculating the gradient of the entropy of the visible ray image from the visible ray image can be calculated by applying Equations 2 to 4 described above for each image. A detailed description thereof will be omitted.

The signal generator 130 may generate a shaking signal when the amount of change in image entropy calculated by the change amount calculator 120 satisfies a logic condition. Such a signal generating unit 130 may generate a shaking signal as a digital signal, and the signal generating unit 130 generates a shaking signal by outputting a value of 1 when a logic condition is satisfied, and generates a shaking signal when the logic condition is not satisfied. In this case, a value of 0 is output and no shaking signal is generated.

Here, the signal generator 130 may generate the shaking signal according to Equations 5 and 6 below.

[Equation 5]

[Equation 6]

Here, GES(t) is the fluctuation signal, ΔE _IR (t) is the amount of change in the infrared image entropy, ΔE _RGB (t) is the change amount of the visible light image entropy, and R(t) is the tracking correlation for sequentially input images. The response value, T _IR , is the integration time of the infrared image, and T _RGB is the initialization frame rate of the visible light image.

The main factors for the signal generator 130 to generate the shaking signal are the image integration time and the image frame rate. Here, the integral time of the image is the time sensed by the image fluctuation sensor 10 , and the frame rate of the image refers to the frame rate of images sequentially input from the image fluctuation detector 10 . In general, in the case of an uncooled element such as a long-wavelength infrared (LWIR) detection sensor, the integration time is set to 600 s, and the frame value is set to 50 Hz, so that a _IR may be set to -0.06 and a _RGB may be set to 0.01.

In addition, R(t) denotes a correlation response value of a trace derived by inputting sequentially input images, that is, an infrared image and a visible ray image. The tracking correlation response value is a value indicating the strength or weakness of the correlation in a correlation diagram in which the values of two variables x and y are plotted on (x, y) on the coordinate plane, and the value can be expressed as R(t). . That is, a tracking correlation response value for sequentially input images may be determined by τ _R , and τ _R may have a value of 0.2.

In this case, the signal generating unit 130 generates a first logic condition for the amount of change in the entropy of the infrared image, a second logical condition for the change in the entropy of the visible ray image, and a third logic for the correlation value between the entropy of the infrared image and the entropy of the visible ray image. When all the conditions are satisfied, the shaking signal can be generated.

That is, the signal generating unit 130 generates a first logical condition for a change amount of infrared image entropy as shown in Equation 7 below, a second logic condition for a change amount of visible ray image entropy as shown in Equation 8 below, and When all of the third logical conditions for the correlation values of infrared image entropy and visible ray image entropy as in Equation 9 are satisfied, the shaking signal may be generated.

[Equation 7]

[Equation 8]

[Equation 9]

The signal generator 130 may generate a fluctuation signal as a digital signal when all of the above first logic condition, the second logic condition, and the third logic condition are satisfied, and the signal generator 130 generates the first logic condition, If any one of the second logic condition and the third logic condition is not satisfied, a value of 0 is output to not generate the shaking signal.

2 is a result of visually expressing the shaking signal derived by the equation, and shows that the timing at which the shaking occurs in the image can be detected as a digital signal such as the following GES(t).

Meanwhile, as described above, in the process of detecting the fluctuation, the parameters of the change amount of the infrared image entropy, the change amount of the visible ray image entropy, and the correlation value of the infrared image entropy and the visible ray image entropy are respectively used. 3 shows the results of the exclusion test showing that accurate fluctuations cannot be detected when even one of these parameters is not used in the detection of fluctuations. That is, when the parameter of the change amount of the infrared image entropy is not used or the parameter of the change amount of the visible ray image entropy is not used, the result that does not match the actual fluctuation (GT _FM ) of FIG. 3 is shown, and the three It can be seen that when all of the parameters of the branches are applied, the actually generated shaking (GT _FM ) and the shaking signal are almost identical.

Hereinafter, a target tracking apparatus according to an embodiment of the present invention using the above-described image fluctuation sensor 10 will be described in detail. Since the above-described contents with respect to the image fluctuation detector 10 may be directly applied to the image fluctuation detector 10 included in the target tracking apparatus according to an embodiment of the present invention, a redundant description will be omitted.

4 is a diagram schematically illustrating a target tracking apparatus according to an embodiment of the present invention. Also, FIG. 5 is a diagram illustrating a state in which a scale factor is calculated according to an embodiment of the present invention, and FIG. 6 is a diagram illustrating a state in which a feature point is adaptively extracted in the target tracking apparatus according to an embodiment of the present invention. Meanwhile, FIG. 7 is a diagram illustrating an operation of a target tracking apparatus according to an embodiment of the present invention, and FIG. 8 is a diagram illustrating performance of the target tracking apparatus according to an embodiment of the present invention.

4 to 8 , the target tracking apparatus according to an embodiment of the present invention calculates the amount of change in image entropy of an image input unit 30 for sequentially receiving images, and an image input from the image input unit 30 . an image fluctuation sensor 10 for detecting whether or not an image fluctuation occurs, and a movement tracker 20 and the movement tracker 20 for calculating movement information of an image when a fluctuation signal is input from the image fluctuation detector 10 ) includes a target detector 40 for detecting a target in the image according to the movement information calculated from.

The image input unit 30 sequentially receives images from the electro-optic tracking system. The image input unit 30 may include an optical unit and other driving units, and the optical unit may include a high-magnification telephoto lens. In addition, the other driving unit may be configured to change and drive the gaze direction of the optical unit.

Here, the image input unit 30 may include a first image input unit 30 for receiving an infrared image and a second image input unit 30 for receiving a visible light image. Electro Optical Tracking System (EOTS) generally receives an infrared image and a visible light image at the same time and tracks and detects a target using image information obtained. Hereinafter, the input image is an infrared image and a visible light image A case in which all ray images are included will be described as an example, but it goes without saying that the image input to the image input unit 30 may be only an infrared image or only a visible ray image.

The image fluctuation detector 10 calculates a change amount of image entropy of the image input from the image input unit 30 to detect whether the image fluctuation occurs. As for the image fluctuation sensor 10, the contents described above with reference to FIGS. 1 to 3 may be applied as it is. That is, the image fluctuation sensor 10 includes an entropy calculator 110 for calculating the image entropy of sequentially input images, a change amount calculator 120 for calculating a change amount of the calculated image entropy, and the change amount calculator. The calculated change in image entropy may include a signal generator 130 that generates a shaking signal when the logical condition is satisfied, and the change in infrared image entropy calculated from the infrared image and the visible ray calculated from the visible ray image By calculating the amount of change in image entropy, the first logical condition for the change amount of infrared image entropy, the second logical condition for the change amount of the visible light image entropy, and the third logic for the correlation value between the infrared image entropy and the visible light image entropy When all the conditions are satisfied, the shaking signal can be generated.

The movement tracker 20 calculates movement information of the image when the shaking signal is input from the image shaking sensor 10 . Here, the movement tracker 20 may calculate movement information of the image in the form of a vector including the movement direction and movement distance of the image.

That is, the movement tracker 20 receives the shaking signal from the image shaking sensor 10 and calculates the movement information of the image. Although there is a correlation tracking method as a representative technique for calculating the movement information of an image, the correlation output result of the image distorted by the motion blur phenomenon is very slow and the accuracy is not good. In an embodiment of the present invention, the background tracking performance is improved by adaptively controlling the number of key points based on the Shi-Tomasi key point detection method that can operate at the fastest speed for high-speed background tracking of a distorted image. That is, the scale factor value was determined according to the optimization process of the scale factor determined according to the resolution of the sensor.

Here, as described above, the image input unit 30 may include a first image input unit 30 for receiving an infrared image and a second image input unit 30 for receiving a visible light image, and the movement tracker 20 ) may calculate movement information of the image by adjusting the number of feature points according to the infrared image entropy calculated from the infrared image. In this case, the image input unit 30 calculates movement information of the image by calculating the movement direction and movement distance of the feature points in the image, and calculates movement information of the image by adjusting the number of feature points according to the image entropy.

The Shi-Tomasi feature point detection method calculates the movement direction and movement distance of the feature point for all feature points in the image to calculate movement information of the image. However, the image input device 30 according to an embodiment of the present invention determines the number of feature points according to the infrared image entropy calculated from the infrared image, and calculates the movement direction and movement distance of the determined number of feature points to obtain the movement information of the image. Calculate.

Here, the process of determining the number of feature points determines the number of feature points by multiplying the number of all feature points in the image by an adaptive scale factor. In this case, the scale factor may be calculated by Equation 10 below.

[Equation 10]

Here, q _A is a constant, E _IR is the infrared image entropy, and SF is a scale factor determined according to the resolution of the infrared image.

A scale factor optimization process between q _A =0.01 and q _A =0.02 to determine a reliable feature point at a level of 1% to 0.5% by confirming the response of the q _A value according to the entropy variation under the scale factor change condition. Determines the scale factor of the distributed level. FIG. 5( a ) shows a result in which the optimal scale factor value determined as described above is derived as SF=2×10 ⁴ . Meanwhile, FIG. 5(b) shows the precision and accuracy of the method (solid red line) for optimizing the scale factor according to the embodiment of the present invention compared to the conventional method (blue dotted line, green dotted line, black dotted line). rate) is significantly improved.

6 is a result showing the detection result of a background feature by representing an image having a high background complexity and an image having a low background complexity. The results of FIGS. 6(a) and 6(d) show that the accuracy of background tracking may be lowered in the case of FIG. 6(a) as a result of detecting excessive features, and FIGS. 6(b) and 6(e) In this case, the accuracy of background tracking may decrease as a result of detecting insufficient features. On the other hand, in the result of applying the adaptive feature boundary value to which the embodiment of the present invention is applied, excess or deficiency of feature detection is resolved and a stable background tracking operation is possible.

Hereinafter, a target tracking method according to an embodiment of the present invention will be described in detail. Since the target tracking method according to an embodiment of the present invention can be performed by the above-described target tracking apparatus, a description of the target tracking apparatus overlapping with the above description will be omitted.

The target tracking method according to an embodiment of the present invention includes a process of sequentially receiving images, a process of calculating an amount of change in image entropy of an input image to detect whether an image fluctuation occurs, and a process of detecting whether an image fluctuation occurs when the fluctuation occurs. It includes a process of calculating movement information and a process of detecting a target in an image according to the calculated movement information.

The process of receiving images sequentially is performed by the image input device 30 . Here, the image input unit 30 sequentially receives images from the electro-optic tracking system, the image input unit 30 may include an optical unit and other driving units, and the optical unit may include a telephoto lens of high magnification. In addition, the other driving unit may be configured to change and drive the gaze direction of the optical unit.

The process of receiving images sequentially includes a process of sequentially receiving an infrared image and a process of receiving a visible ray image sequentially, a process of sequentially receiving the infrared image and a process of receiving a visible ray image sequentially can be performed simultaneously.

That is, the image input unit 30 may include a first image input unit 30 for receiving an infrared image and a second image input unit 30 for receiving a visible light image. Electro Optical Tracking System (EOTS) generally receives an infrared image and a visible light image at the same time and tracks and detects a target using image information obtained. Hereinafter, the input image is an infrared image and a visible light image Although a case including all of the ray images will be described as an example, it goes without saying that the image input from the image fluctuation sensor 10 may be only an infrared image or only a visible ray image.

In the process of detecting whether the image fluctuation has occurred, the amount of change in image entropy of the input image is calculated to detect whether the image fluctuation has occurred. The process of detecting whether the image fluctuation occurs is performed by the image fluctuation detector 10 , and the image fluctuation detector 10 calculates the amount of change in the image entropy of the image input from the image input unit 30 to determine whether the image fluctuation occurs to detect As for the image fluctuation sensor 10, the contents described above with reference to FIGS. 1 to 3 may be applied as it is. That is, the image fluctuation sensor 10 includes an entropy calculator 110 for calculating the image entropy of sequentially input images, a change amount calculator 120 for calculating a change amount of the calculated image entropy, and the change amount calculator. The calculated change in image entropy may include a signal generator 130 that generates a shaking signal when the logical condition is satisfied, and the change in infrared image entropy calculated from the infrared image and the visible ray calculated from the visible ray image By calculating the amount of change in image entropy, the first logical condition for the change amount of infrared image entropy, the second logical condition for the change amount of the visible light image entropy, and the third logic for the correlation value between the infrared image entropy and the visible light image entropy When all the conditions are satisfied, the shaking signal can be generated.

The process of calculating the movement information of the image calculates the movement information of the image according to the shaking when the occurrence of the shaking is detected. Here, the process of calculating the movement information of the image includes the process of determining the number of feature points according to the infrared image entropy calculated from the infrared image, and calculating the movement direction and movement distance of the determined number of feature points to calculate the movement information of the image. process may be included. Also, in the process of determining the number of feature points, the number of feature points may be determined by multiplying the number of all feature points in the image by an adaptive scale factor.

That is, the movement tracker 20 calculates the movement information of the image when the shaking signal is input from the image shaking sensor 10 . Here, the movement tracker 20 may calculate movement information of the image in the form of a vector including the movement direction and movement distance of the image.

That is, the movement tracker 20 receives the shaking signal from the image shaking sensor 10 and calculates the movement information of the image. Although there is a correlation tracking method as a representative technique for calculating the movement information of an image, the correlation output result of the image distorted by the motion blur phenomenon is very slow and the accuracy is not good. In an embodiment of the present invention, the background tracking performance is improved by adaptively adjusting the number of key points based on the Shi-Tomasi key point detection method that can operate at the fastest speed for high-speed background tracking of a distorted image. That is, the scale factor value was determined according to the optimization process of the scale factor determined according to the resolution of the sensor.

Here, as described above, the image input unit 30 may include a first image input unit 30 for receiving an infrared image and a second image input unit 30 for receiving a visible light image, and the movement tracker 20 ) may calculate movement information of the image by adjusting the number of feature points according to the infrared image entropy calculated from the infrared image. In this case, the image input unit 30 calculates movement information of the image by calculating the movement direction and movement distance of the feature points in the image, and calculates movement information of the image by adjusting the number of feature points according to the image entropy.

The Shi-Tomasi feature point detection method calculates movement information of the image by calculating the movement direction and movement distance of the feature point for all feature points in the image. However, the image input device 30 according to an embodiment of the present invention determines the number of feature points according to the infrared image entropy calculated from the infrared image, and calculates the movement direction and movement distance of the determined number of feature points to obtain movement information of the image. Calculate.

Here, the process of determining the number of feature points determines the number of feature points by multiplying the number of all feature points in the image by an adaptive scale factor, and in this case, the scale factor may be calculated by Equation 10 above. At this time, as a scale factor optimization process, q _A =0.01 to q _A =0.02 to determine a reliable feature point at a level of 1% to 0.5% by confirming the response of the q _A value according to the entropy change under the scale factor change condition It is the same as described above that the scale factor of the level distributed between them can be determined.

In the process of detecting the target, the target is detected in the image according to the calculated movement information of the image. At this time, in the process of detecting the target, as shown in the GES and BT Tracking section of FIG. 7 , the target is detected by moving the previously detected area in the image in a direction that offsets the moving direction and moving distance of the image. can

That is, in the process of detecting the target, the target may be detected by moving it in a direction offsetting the moving direction and moving distance of the image calculated by Equation 11 below.

[Equation 11]

Here, u _t-1 ^x is the X-axis coordinate value of the feature point included in the image before movement, u _t ^x is the X-axis coordinate value of the feature point included in the image after movement, and u _t-1 ^y is the X-axis coordinate value of the feature point included in the image before movement. The Y-axis coordinate value of the feature point, u _t ^y is the Y-axis coordinate value of the feature point included in the image after movement, d ^x is the movement distance in the X-axis direction, d ^y is the movement distance in the Y-axis direction, and T is the frame number do.

To summarize, in the prior art, when a sudden camera motion occurs, such as frame #092 shown in FIG. A problem occurs, and there is a problem in that the target tracking function is lost like the #120 frame. However, according to the embodiment of the present invention, even when a sudden camera motion occurs, it is determined whether there is a target or global motion, predicts a vector of motion to correct the correlation of target tracking and an area where learning is performed, and accordingly, accurate A result of calculating the coordinate value of the target may be derived. That is, continuous tracking of a strong target such as frame #120 shown in FIG. 8(b) is possible.

As such, according to an embodiment of the present invention, when fluctuations occur in an image, it is possible to quickly and accurately detect whether fluctuations occur from the amount of change in image entropy with respect to the image.

In addition, when the occurrence of the shaking motion is detected, the target detection area can be corrected by calculating a motion vector according to the shaking motion, thereby continuously maintaining tracking without missing the target, and calculating the correct target coordinate value.

In the above, preferred embodiments of the present invention have been described and illustrated using specific terms, but such terms are only for clearly explaining the present invention, and the embodiments of the present invention and the described terms are the spirit of the following claims And it is obvious that various changes and changes can be made without departing from the scope. Such modified embodiments should not be individually understood from the spirit and scope of the present invention, but should be said to fall within the scope of the claims of the present invention.

10: video motion detector 20: movement tracker
30: video input device 40: target detector
110: entropy calculation unit 120: change amount calculation unit
130: signal generator

Claims (22)

An image fluctuation detector for detecting whether fluctuation occurs in an image, comprising:
An image fluctuation detector that detects whether fluctuation occurs by calculating the amount of change in image entropy of sequentially input images. An image fluctuation detector for detecting whether fluctuation occurs in an image, comprising:
an entropy calculator for calculating image entropy of sequentially input images;
a change amount calculator for calculating a change amount of the calculated image entropy; and
and a signal generator for generating a shaking signal when the amount of change in image entropy calculated by the change amount calculator satisfies a logic condition. 3. The method according to claim 2,
The change amount calculator calculates the amount of change in image entropy per unit time. 3. The method according to claim 2,
The sequentially input image includes an infrared image and a visible light image,
The change amount calculating unit is an image fluctuation sensor for calculating a change amount of the infrared image entropy calculated from the infrared image and the change amount of the visible ray image entropy calculated from the visible ray image, respectively. 3. The method according to claim 2,
The entropy calculator calculates the image entropy according to Equations 1 and 2 below.
[Equation 1]

[Equation 2]

(where I _i is the intensity value of the image, │I _i │ is the number of pixels in the image having the corresponding intensity value, n is the total number of pixels in the image, L is all the intensity values that the image can have, pdf(I _i ) is the probability density function of the image, and E _I is the image entropy.) 4. The method according to claim 3,
The change amount calculator calculates the change amount of image entropy according to Equation 3 below.
[Equation 3]

(Here, ΔE _I (t) is the amount of change in image entropy, E _I is the image entropy, t is the current image input time, and t-1 is the previous image input time.) 5. The method according to claim 4,
The signal generator,
If the first logical condition for the change amount of the infrared image entropy, the second logical condition for the change amount of the visible light image entropy, and the third logical condition for the correlation value of the infrared image entropy and the visible light image entropy are all satisfied, the shaking signal is Video shake sensor that generates. 5. The method according to claim 4,
The signal generator generates an image shaking signal according to Equations 4 and 5 below.
[Equation 4]

[Equation 5]

(here, GES(t) is the fluctuation signal, ΔE _IR (t) is the amount of change in the infrared image entropy, ΔE _RGB (t) is the change amount of the visible light image entropy, and R(t) is the tracking correlation for sequentially input images Response value, T _IR is the integration time of the infrared image, and T _RGB is the initialization frame rate of the visible light image.) an image input device for sequentially receiving images;
an image fluctuation detector for detecting whether or not fluctuation of the image occurs by calculating an amount of change in image entropy of the image input from the image input device;
a movement tracker for calculating movement information of an image when a shaking signal is input from the image shaking sensor; and
Target tracking device comprising a; target detector for detecting a target in the image according to the movement information calculated from the movement tracker. 10. The method of claim 9,
The video input device,
a first image input device for receiving an infrared image; and
Including; a second image input device for receiving a visible light image;
The video vibration detector,
By calculating the change amount of the infrared image entropy calculated from the infrared image and the change amount of the visible ray image entropy calculated from the visible light image, respectively,
If the first logical condition for the change amount of the infrared image entropy, the second logical condition for the change amount of the visible light image entropy, and the third logical condition for the correlation value of the infrared image entropy and the visible light image entropy are all satisfied, the shaking signal is generating target tracking device. 10. The method of claim 9,
The movement tracker, a target tracking device for calculating the movement information of the image in the form of a vector including the movement direction and the movement distance of the image. 10. The method of claim 9,
The movement tracker, a target tracking device for calculating the movement information of the image by calculating the movement direction and movement distance of the feature point in the image. 13. The method of claim 12,
The movement tracker is a target tracking device for calculating movement information of an image by adjusting the number of feature points according to image entropy. 13. The method of claim 12,
The video input device,
a first image input device for receiving an infrared image; and
Including; a second image input device for receiving a visible light image;
The movement tracker is a target tracking device for calculating movement information of the image by adjusting the number of feature points according to the infrared image entropy calculated from the infrared image. 10. The method of claim 9,
The target detector is a target tracking device for detecting a target by moving a previously detected area in the image in a direction offsetting the moving direction and the moving distance of the image. The process of receiving images sequentially;
Calculating the amount of change in the image entropy of the input image and detecting whether the fluctuation of the image occurs;
calculating movement information of an image according to the shaking when the occurrence of shaking is detected; and
A target tracking method comprising a; the process of detecting a target in the image according to the calculated movement information. 17. The method of claim 16,
The process of receiving the images sequentially,
a process of sequentially receiving infrared images; and
Including a process of sequentially receiving visible light images;
The process of receiving the infrared image sequentially and the process of receiving the visible light image sequentially are simultaneously performed. 18. The method of claim 17,
The process of calculating the movement information of the image is,
determining the number of feature points according to the infrared image entropy calculated from the infrared image; and
The process of calculating movement information of an image by calculating movement directions and movement distances of the determined number of feature points. 19. The method of claim 18,
The process of determining the number of the feature points is
A target tracking method for determining the number of feature points by multiplying the number of all feature points in an image by an adaptive scale factor. 20. The method of claim 19,
The scale factor is a target tracking method determined by Equation 6 below.
[Equation 6]

(Where q _A is a constant, E _IR is infrared image entropy, and SF is a scale factor.) 21. The method of claim 20,
wherein q _A is a target tracking method having a value selected from 0.01 to 0.02. 17. The method of claim 16,
The process of detecting the target is
A target tracking method for detecting a target by moving a previously detected area within an image in a direction offsetting the moving direction and moving distance of the image calculated by Equation 7 below.
[Equation 7]

(here, u _t-1 ^x is the X-axis coordinate value of the feature point included in the image before movement, u _t ^x is the X-axis coordinate value of the feature point included in the image after movement, and u _t-1 ^y is included in the image before movement The Y-axis coordinate value of the acquired feature point, u _t ^y is the Y-axis coordinate value of the feature point included in the moved image, d ^x is the movement distance in the X-axis direction, d ^y is the movement distance in the Y-axis direction, and T is the frame number. it means.)

Images