KR102629213B1 - Method and Apparatus for Detecting Moving Objects in Perspective Motion Imagery - Google Patents

Abstract

An obliquely captured video moving object detection method performed by a moving object detection device according to an embodiment includes determining an optical flow analysis frame interval for each frame area based on the captured geometric information for an inclined captured video for which photographed geometric information is provided. ; and detecting a moving object on an object basis based on the determined optical flow analysis frame interval.

Description

Oblique shooting video moving object detection method and apparatus {Method and Apparatus for Detecting Moving Objects in Perspective Motion Imagery}

The present invention relates to a method and device for detecting a moving object captured in an obliquely captured video.

In order to detect a moving object on the ground from a mobile platform such as an unmanned aerial vehicle, a procedure is performed to acquire obliquely captured video while moving and detect the moving object captured in the obliquely captured video. It is based on compensating for background movement caused by platform movement, matching adjacent frames as if taken from a stationary camera, and detecting differences due to moving objects. Moving object detection is accomplished by separating background elements that do not change between adjacent frames and foreground areas where changes are noticeable due to movement, and detecting objects on an object basis near the foreground area.

Foreground area separation is generally done through stochastic background modeling, which maintains a probability distribution to determine whether each pixel belongs to the background or foreground, and analyzes the optical flow (“optical flow”) expressed as the movement vector of each pixel to a certain level. An optical flow analysis method that filters out pixels that show optical flow (movement vector) larger than the size as foreground elements is widely used. Depending on the application purpose, there are previous studies that selectively apply one of the background modeling and optical flow analysis methods or use them in a complementary manner. Optical flow between two images of the same size searches for a pixel located at a certain image coordinate (i', j') in the other image that corresponds to a pixel at a random (i, j) image coordinate of one image, and determines the displacement of the image pixel coordinates. , In other words, (u, v)=(i'-i, j'-j) is expressed as a vector, so it is a method of finding corresponding points between two images. Additionally, unlike pixel-level background modeling, optical flow allows the size of pixel displacement to be known. Therefore, in applications where a separate image registration technique is not applied for computational efficiency or where quantitative information on movement amount is considered important, the optical flow analysis method is preferred.

All of these methods analyze pixel-unit differences between registered frames, and when real-world motion is projected into an image, if the displacement is less than a pixel unit, the change cannot be detected from the image. In the case of orthogonal (direct downward) shooting video, considering lens distortion, the pixel distance can be considered to be almost uniform across the entire video frame, so the comparison frame interval can be adjusted so that the degree of projection of movement is greater than the pixel unit. However, in the case of oblique shooting, the pixel displacement is large in the near field and small in the remote field, so if the frame interval is too large to detect a change in the distant field, the near field object may disappear from the screen or fall outside the search area of the optical flow algorithm and cannot be calculated correctly. may occur. In addition, if objects with large speed differences are mixed in both ortho and oblique shooting, it is difficult to detect the movement of all objects with uniform frame intervals.

Meanwhile, shooting geometric information for each video frame - camera position, posture, specifications, etc. - may be provided in the form of MISB (motion imagery standards board) standard video metadata or separate telemetry information, from which a frame camera sensor model is formed. The pixel distance for each image location can be calculated on a specific reference altitude. Using this information, it is possible to estimate the amount of displacement at each pixel location in a video frame depending on the speed of the moving object.

Republic of Korea Patent Publication No. 10-2121974, registration date: June 5, 2020.

Hyuk-Tae Kwon, Se-Hyung Park, and Hyun-Seung Seong, “Local background motion analysis for moving object detection in high-resolution oblique video footage,” 2019 Korean Society of Military Science and Technology General Conference, 2019. 6.

According to one embodiment, a method and apparatus for detecting a moving object in an obliquely captured video are provided for detecting a moving object by determining an optical flow analysis frame for each frame area in a video obliquely captured on a moving platform.

The problems to be solved by the present invention are not limited to those mentioned above, and other problems to be solved that are not mentioned will be clearly understood by those skilled in the art to which the present invention pertains from the following description.

A method of detecting an obliquely captured video moving object performed by a moving object detection device according to a first aspect includes determining an optical flow analysis frame interval for each frame area based on the captured geometric information for an obliquely captured video for which photographed geometric information is given. ; and detecting a moving object on an object basis based on the determined optical flow analysis frame interval.

A moving object detection device according to a second aspect includes a frame analysis interval calculation unit that determines an optical flow analysis frame interval for each frame area based on the shooting geometry information for an oblique shooting video for which shooting geometry information is given; and a moving object detection unit that detects the moving object on an object basis based on the determined optical flow analysis frame interval.

According to a third aspect, a computer-readable recording medium storing a computer program, wherein the computer program, when executed by a processor, causes the processor to perform an oblique shooting video moving object detection method performed by a moving object detection device. Includes commands for:

According to a fourth aspect, there is a computer program stored in a computer-readable recording medium, wherein the computer program, when executed by a processor, causes the processor to perform an oblique shooting video moving object detection method performed by a moving object detection device. Includes commands for:

According to one embodiment, a moving object is detected by determining an optical flow analysis frame for each frame area in a video captured obliquely on a moving platform, so that objects are detected throughout the frame without being affected by the non-homogeneous pixel distance of the video frame captured obliquely. By securing the moving displacement of more than a pixel unit, it is possible to detect both distant and near scenes, and both low-speed and high-speed moving objects.

Figure 1 shows the phenomenon in which the projected size of an object of the same size varies depending on the distance in an obliquely captured video in a frame camera shooting geometry.
Figure 2 shows three adjacent frames to explain that moving objects with the same speed are projected differently on the image in the near and far views in an oblique shooting video.
Figure 3 is a configuration diagram of a moving object detection device according to an embodiment of the present invention.
Figure 4 is an example of setting a segmented area in an obliquely captured video frame according to an embodiment of the present invention.
Figure 5 is a flowchart illustrating a method of detecting a moving object in an obliquely captured video performed by a moving object detection device according to an embodiment of the present invention.
Figure 6 is a flowchart illustrating a process for determining an optical flow analysis interval so that movement displacement appears in units of pixels or more in a divided area of a video frame according to an embodiment of the present invention.
FIG. 7 is a flowchart illustrating a process for calculating optical flow in which background motion is compensated in a divided region of a video frame according to an embodiment of the present invention.
Figure 8 is a flowchart for explaining the process of detecting a moving object by determining moving points in a divided area of a video frame and clustering the moving points in accordance with an embodiment of the present invention.

The advantages and features of the present invention and methods for achieving them will become clear with reference to the embodiments described below along with the accompanying drawings. However, the present invention is not limited to the embodiments disclosed below and may be implemented in various different forms. The present embodiments are merely provided to ensure that the disclosure of the present invention is complete and to be understood by those skilled in the art. It is provided to fully inform those who have the scope of the invention, and the present invention is only defined by the scope of the claims.

The terms used in this specification will be briefly explained, and the present invention will be described in detail.

The terms used in the present invention are general terms that are currently widely used as much as possible while considering the function in the present invention, but this may vary depending on the intention or precedent of a person working in the art, the emergence of new technology, etc. In addition, in certain cases, there are terms arbitrarily selected by the applicant, and in this case, the meaning will be described in detail in the description of the relevant invention. Therefore, the terms used in the present invention should be defined based on the meaning of the term and the overall content of the present invention, rather than simply the name of the term.

When it is said that a part ‘includes’ a certain element throughout the specification, this does not mean excluding other elements, but may further include other elements, unless specifically stated to the contrary.

Additionally, the term ‘unit’ used in the specification refers to software or hardware components such as FPGA or ASIC, and the ‘unit’ performs certain roles. However, ‘wealth’ is not limited to software or hardware. The 'part' may be configured to reside on an addressable storage medium and may be configured to reproduce on one or more processors. Therefore, as an example, 'part' refers to components such as software components, object-oriented software components, class components and task components, processes, functions, properties, procedures, Includes subroutines, segments of program code, drivers, firmware, microcode, circuits, data, databases, data structures, tables, arrays, and variables. The functionality provided within components and parts can be combined into a smaller number of components and parts or further separated into additional components and parts.

Below, with reference to the attached drawings, embodiments of the present invention will be described in detail so that those skilled in the art can easily implement the present invention. In order to clearly explain the present invention in the drawings, parts unrelated to the description are omitted.

Figure 1 shows the phenomenon in which the projected size of an object of the same size varies depending on the distance in an obliquely captured video in a frame camera shooting geometry.

When the camera position, shooting tilt angle, lens angle of view, or focal length are provided from frame sensor information, the pixel distance (ground distance indicated by one pixel) d for each image pixel location at a specific altitude can be easily calculated using the triangle similarity ratio as shown in Figure 1. there is. The frame rate (FPS) f can be obtained from the video standard. When the movement of the object is reflected in pixel units or more from the object's moving speed v (m/s), if the frame interval is n, the inequality in Equation 1 is established.

Figure 2 shows three adjacent frames to explain that moving objects with the same speed are projected differently on the image in the near and far views in an oblique shooting video.

The first surface 110 is an image plane of an image frame captured when the camera focus is located at point O1 at the first viewpoint. The second side 120 is an image plane captured when the camera focus is located at O2 at the second viewpoint, and is the frame immediately following the first side in the video. The third side 130 is the image plane captured when the camera focus is located at O3 in the third viewpoint, and is the frame immediately following the second side. Additionally, P1 is the point where the midpoint of the near moving object is located at the first viewpoint. P2 represents the position of the midpoint of this object at the second viewpoint, and P3 represents the position of the midpoint of this object at the third viewpoint. Likewise, Q1 is the point where the midpoint of the distant moving object is located at the first viewpoint, and Q2 and Q3 are points where the midpoint of the distant moving object is located at the second and third viewpoints, respectively.

p1 is the image point where point P1 is projected onto the first side 110, p2 is the image point where point P2 is projected onto the second side 120, and p3(131) is the image point where point P3 is projected onto the third side 130. This is the image point. Similarly, q1 is the image point where point Q1 is projected onto the first side 110, q2 is the image point where point Q2 is projected onto the second side 120, and q3(132) is the image point where point Q3 is projected onto the third side 130. It is a projected image point.

p'1 (151) is the image point where point P1 is projected, assuming that P1 at the first viewpoint is captured on the third side (130), and p'2 (141) is the second viewpoint on the third side (130). Assuming that P2 of is captured, point P2 is the projected image point. Likewise, q'1 (152) is the image point where point Q1 is projected, assuming that Q1 at the first viewpoint is captured on the third side (130), and q'2 (142) is the image point at which point Q1 is projected on the third side (130). Assuming that point Q2 is captured, point Q2 is the projected image point.

The process of projecting points P1, P2, Q1, and Q1 as p'1, p'2, q'1, and q'2 on the third surface 131, respectively, removes the camera movement and rotation effects to create the third viewpoint. It represents the process of registering images as if they were projected onto the camera plane.

An example of a phenomenon that can occur depending on the relationship between the speed of the moving object and the position and posture change of the camera is q'1 (152) and q'2 (142), q'2 (142) and q3 (132). Although the image coordinate distance differences are each less than 1, when the distance difference is accumulated in sub-pixel units, the distance difference between q'1 (152) and q3 (132) becomes more than 1 and when projected as an actual image, q'1 due to pixel-level quantization (152) and q'2 (142) may be located in the same pixel on the third side 130, and q'1 (152) and q3 (132) may be located in different neighboring pixels on the third side 130. there is. In this situation, the image points p'1 (151) and p'2 (141), onto which the near view is projected, are located in different pixels on the third side 130, and the image points p'2 (141) and p3 (131) are They are located in different pixels on the third side 130.

As described above, the moving object detection device and its oblique shooting video moving object detection method according to an embodiment of the present invention perform a series of procedures in a situation where the third side 130 is currently input in the video input and a moving object is detected. It can be done.

Figure 3 is a configuration diagram of a moving object detection device according to an embodiment of the present invention.

Referring to FIG. 3, the moving object detection device 300 according to one embodiment includes a frame analysis interval calculation unit 320 and a moving object detection unit 350, a frame queue management unit 310, and an optical flow calculation unit. It may further include at least one of the 330 and the background motion compensation unit 340. The elements constituting the moving object detection device 300 may be comprised of one or more processors. For example, one or more processors may be a general-purpose processor such as a central process unit (CPU), a digital signal processor (DSP), a graphics-specific processor such as a graphic process unit (GPU), a vision process unit (VPU), or a neural processor (NPU). It may be a processor dedicated to artificial intelligence, such as a process unit).

The frame analysis interval calculation unit 320 of the moving object detection device 300 determines the optical flow analysis frame interval for each frame area based on the shooting geometry information for an oblique shooting video for which shooting geometry information is provided.

The moving object detection unit 350 of the moving object detection device 300 detects the moving object on an object basis based on the determined optical flow analysis frame interval.

The frame queue management unit 310 of the moving object detection device 300 may specify the oblique shooting video as a frame queue in frame order, and in this case, the frame analysis interval calculation unit 320 may generate a plurality of frames for the latest frame in the frame queue. The divided areas are set, and the optical flow analysis frame interval is calculated by referring to the ground distance for each pixel calculated from the shooting geometric information at the center point of each set divided area. Here, in the plurality of divided regions, the center points of each divided region are arranged at equal intervals in the horizontal and vertical directions, and some areas between adjacent divided regions may overlap, and the plurality of divided regions may be imaged close to the center points of each divided region. It can be set as a borderline at a location closer to the border or neighboring center point.

Figure 4 is an example of setting a segmented area in an obliquely captured video frame according to an embodiment of the present invention. To limit the effect of the projection distance that varies depending on the pixel position of the image, a division area 410 of the image frame is set. The division area center points 420 are arranged at equal intervals in each direction horizontally and vertically, and the distance from the edge of the image to the first center point is half the distance to the two adjacent center points. The width of the segmented area is set to correspond to two-thirds of the distance from the center to the adjacent centers on both sides to allow overlap between segmented areas, and extends from the edge of the image to the image area.

Additionally, the optical flow analysis frame interval may be calculated and determined differently for each frame area, and may be determined so that the displacement of the moving object is projected in pixel units or more. When calculating the optical flow analysis frame interval, the moving distance of the moving object per unit frame interval is calculated from the time interval between frames obtained from the video standard and the speed information of the moving object, and the ground distance of the center point pixel of the divided area and the unit frame interval are calculated. Using the moving distance of each moving object, the frame interval at which the moving distance of the moving object is greater than or equal to the ground distance of the pixel can be calculated.

The optical flow calculation unit 330 of the moving object detection device 300 may acquire past frames for each determined optical flow analysis frame interval and then calculate the optical flow in the corresponding divided area among the plurality of divided areas. The background motion compensation unit 340 of the moving object detection apparatus 300 can estimate and compensate the background motion for each segment using the calculated optical flow. In this case, the moving object detection unit 350 can extract the moving object on an object basis by determining the moving point based on the standard score of the size of the moving vector for each segmented area.

The background motion compensation unit 340 configures dense equally spaced grid points from the optical flow for each analysis frame interval and adds the optical flow vector corresponding to the grid point to the grid point coordinates to form a set of corresponding pairs between images, and configures a set of corresponding pairs between images for each analysis frame interval. From the set of corresponding pairs between the constructed images, a transformation matrix that converts the image coordinates of the current frame to the image coordinates of the previous frame is estimated, and the displacement from the optical flow for each analysis frame interval to the image coordinates obtained by the transformation matrix estimated from the corresponding image coordinates for each pixel is estimated. By subtracting the vector from the optical flow, the optical flow that compensates for the background motion can be obtained.

The moving object detection unit 350 determines moving points whose size of the optical flow movement vector for each divided region satisfies the movement condition from the optical flow for each divided region in which the background motion is compensated, and extracts the moving object from the entire frame from the moving points. can do. In addition, the moving object detection unit 350 calculates the average and standard deviation of the size of the optical flow vectors of the pixels belonging to the divided area from the optical flow corresponding to the frame analysis interval for each divided area, and calculates the optical flow in each divided area. Pixels whose vector size is greater than the unit pixel length and whose standard score, which is the size of the optical flow vector minus the average and divided by the standard deviation, is greater than a certain standard, can be identified as moving points. Here, when determining the moving point, the optical flow size of the divided area is more than a unit pixel, the standard score of the optical flow size is more than a certain standard, and points that are a certain distance away from the edge of the divided area can be judged as moving points.

FIG. 5 is a flowchart illustrating a method for detecting a moving object in an obliquely captured video performed by a moving object detection device according to an embodiment of the present invention, and FIG. 6 is a flowchart showing a moving object in a divided area of a video frame according to an embodiment of the present invention. It is a flowchart for explaining the process of determining the optical flow analysis interval so that the displacement appears in pixel units or more, and Figure 7 is a flowchart for calculating optical flow with background motion compensated for in a divided area of a video frame according to an embodiment of the present invention. This is a flowchart for explaining the process, and Figure 8 is a flowchart for explaining the process of detecting a moving object by determining moving points in a segmented area of a video frame and clustering the moving points in accordance with an embodiment of the present invention.

Hereinafter, with reference to the attached drawings, we will take a closer look at the oblique shooting video moving object detection method performed by the moving object detection device 300 according to an embodiment of the present invention.

Referring to FIG. 5, the frame queue management unit 310 of the moving object detection device 300 can designate the oblique shooting video to the frame queue in frame order (510), and the frame analysis interval calculation unit 320 can specify the frame queue as A plurality of division areas are set for the latest frame (520), and the optical flow analysis frame interval is calculated by referring to the ground distance for each pixel calculated from the shooting geometric information at the center point of each set division zone (530).

Referring to FIG. 6, in step 610, a pixel corresponding to an arbitrary image point is obtained using the camera's position, shooting posture (tilt angle), focal length, and image plane size information from the frame camera shooting geometry as illustrated in FIG. 1. Ground distance (d) can be calculated by specifying the altitude of the subject surface.

In step 620, the moving distance of the moving object per unit frame interval (v) is calculated using the speed (v) of the moving object to be detected and the inter-frame time interval (1/f) information obtained from the number of frames per second (f) of the video standard. /f) can be calculated.

In step 630, the frame interval (n) required to divide the moving distance of the moving object to be detected into quantized pixel units can be calculated using Equation 1. For example, in the case of the example assumed above, an interval of 1 may be calculated in an area where the frame interval includes the distant view, and an interval of 2 may be calculated in an area near the near view.

Referring again to FIG. 5, the optical flow calculation unit 330 of the moving object detection device 300 may acquire past frames for each determined optical flow analysis frame interval and then calculate the optical flow in the corresponding divided area among the plurality of divided areas. (540). Additionally, the background motion compensation unit 340 of the moving object detection device 300 can estimate and compensate the background motion for each segment using the calculated optical flow (550).

Referring to FIG. 2, when inputting a video, it is impossible to project the object position at the previous time onto the third surface 130, so the transformation relationship for the image planes of the two locations is estimated and a point on one image plane is drawn. A method of moving to another image plane based on a transformation relationship is commonly used. Since this is a transformation in a reduced state from 3D to 2D, a homography or affine transformation relationship is usually applied by applying constraints. We assume that these constraints are practically valid because the video frame interval is a very short time. Estimation of the transformation relationship finds corresponding pairs between two images throughout the image, and minimizes the distance difference between the image points known from the corresponding pair relationship and the transformed image points when one image point is moved to another image point by the estimated transformation relationship. Apply optimization techniques and select a statistically robust hypothesis model using RANSAC (random sample consensus) techniques.

Referring to FIG. 7, in step 710, all optical flow analysis frame intervals calculated for each segment are collected, previous frames corresponding to each analysis frame interval are acquired, and the high-density optical flow between the current frame and each previous frame is calculated as a known optical flow. It is obtained using an algorithm (Farneback, 2003).

In step 720, a set of corresponding pairs is obtained by adding the optical flow displacement to the coordinates of each grid point of the image point of the previous frame corresponding to the equally dense grid points of the current frame image from the optical flow at each analysis frame interval. In an embodiment, the spacing between dense grid points may be 32 pixels.

In step 730, the transformation relationship between the two images is estimated from this corresponding pair set, and the distance difference between the transformation point coordinates when the grid point of the current frame is moved from the transformation relationship and the corresponding point coordinates obtained from the corresponding pair set is less than 1 pixel distance. The conversion is estimated using RANSAC. If this transformation is an affine transformation, it is calculated as a 2×3 matrix. In an embodiment, the analysis interval may be calculated as 1 or 2 for each segment, so this conversion estimate is between the first side 110 and the third side 130, and the second side 120 and the third side 130. Find each.

In step 740, from the optical flow obtained for each analysis frame interval and the estimated transformation relationship, pixel movement due to the transformation relationship is regarded as background motion, and optical flow in which the background motion is compensated is obtained. Background movement is caused by camera movement and rotation, which is projected onto the video frame tube transformation relationship. For each analysis frame interval, if the estimated transformation between the current frame and the previous frame of one analysis target interval is represented by a matrix M, background motion can be viewed as all points in the image moving by this transformation. When image coordinates (i,j) are converted to (i',j') by transformation M, the following conversion equation of Equation 2 is established.

Therefore, the displacement due to transformation M is calculated as (i'-i, j'-j), and the difference vector obtained by subtracting this displacement vector from the optical flow of the pixel is obtained as the optical flow in which the background motion is compensated.

Referring again to FIG. 5, the moving object detection unit 350 determines moving points for which the size of the optical flow movement vector for each divided region satisfies the movement condition from the optical flow for each divided region for which the background motion has been compensated, and determines moving points from the moving points. Moving objects can be extracted from the entire frame. In addition, the moving object detection unit 350 calculates the average and standard deviation of the size of the optical flow vectors of the pixels belonging to the divided area from the optical flow corresponding to the frame analysis interval for each divided area, and calculates the optical flow in each divided area. Pixels whose vector size is greater than the unit pixel length and whose standard score, which is the size of the optical flow vector minus the average and divided by the standard deviation, is greater than a certain standard, can be identified as moving points. Here, when determining the moving point, the optical flow size of the divided area is more than a unit pixel, the standard score of the optical flow size is more than a certain standard, and points that are a certain distance away from the edge of the divided area can be judged as moving points (810) .

Figure 8 is a flowchart for explaining the process of detecting a moving object by determining moving points in a divided area of a video frame and clustering the moving points in accordance with an embodiment of the present invention.

In step 810, under the assumption that the background modeling result includes an error, the average and standard deviation of the magnitude of the optical flow for all points in the divided area are calculated to determine a statistically significant moving point.

In step 820, in determining the moving point within the divided area, the standard score (Z-score) obtained by subtracting the average calculated from the optical flow size of each individual pixel and dividing it by the standard deviation is greater than or equal to a certain standard and the optical flow size is greater than or equal to the unit pixel length of 1. They are judged as moving points. Assuming a normal distribution, a standard score of 2 or higher is approximately 2.275% of the total, and a standard score of 3 or higher is approximately 0.135% of the total.

In step 830, the points determined as moving points for each segment are collected from the entire frame to construct an image in which the moving points and non-moving points are binarized, and blob analysis is performed on this image to cluster the moving points, respectively. Clusters of are detected as moving objects.

As described so far, according to one embodiment, in a video captured obliquely on a moving platform, an optical flow analysis frame is determined for each frame area to detect a moving object, thereby reducing the impact on the non-homogeneous pixel distance of the obliquely captured video frame. It is possible to detect both distant and near scenes, and low-speed and high-speed moving objects by securing the moving displacement of objects throughout the frame in units of pixels or more.

Meanwhile, each step included in the method for detecting moving objects in an oblique shooting video according to the above-described embodiment may be implemented in a computer-readable recording medium that records a computer program programmed to perform these steps.

Additionally, each step included in the method for detecting a moving object in an oblique shooting video according to the above-described embodiment may be implemented in the form of a computer program stored in a computer-readable recording medium that is programmed to perform these steps.

Combinations of each step in each flowchart attached to the present invention may be performed by computer program instructions. Since these computer program instructions can be mounted on the processor of a general-purpose computer, special-purpose computer, or other programmable data processing equipment, the instructions performed through the processor of the computer or other programmable data processing equipment perform the functions described in each step of the flowchart. It creates the means to carry out these tasks. These computer program instructions may also be stored on a computer-usable or computer-readable recording medium that can be directed to a computer or other programmable data processing equipment to implement a function in a particular way, so that the computer program instructions are computer-usable or computer-readable. The instructions stored in the recording medium can also produce manufactured items containing instruction means that perform the functions described in each step of the flowchart. Computer program instructions can also be mounted on a computer or other programmable data processing equipment, so that a series of operational steps are performed on the computer or other programmable data processing equipment to create a process that is executed by the computer, thereby generating a process that is executed by the computer or other programmable data processing equipment. Instructions that perform processing equipment may also provide steps for executing the functions described in each step of the flowchart.

Additionally, each step may represent a module, segment, or portion of code containing one or more executable instructions for executing specified logical function(s). Additionally, it should be noted that in some alternative embodiments it is possible for the functions mentioned in the steps to occur out of order. For example, two steps shown in succession may in fact be performed substantially simultaneously, or the steps may sometimes be performed in reverse order depending on the corresponding function.

The above description is merely an illustrative explanation of the technical idea of the present invention, and those skilled in the art will be able to make various modifications and variations without departing from the essential quality of the present invention. Accordingly, the embodiments disclosed in the present invention are not intended to limit the technical idea of the present invention, but are for illustrative purposes, and the scope of the technical idea of the present invention is not limited by these embodiments. The scope of protection of the present invention shall be interpreted in accordance with the claims below, and all technical ideas within the scope equivalent thereto shall be construed as being included in the scope of rights of the present invention.

300: Moving object detection device
310: Frame queue management unit
320: Frame analysis interval calculation unit
330: Optical flow calculation unit
340: Background motion compensation unit
350: Moving object detection unit

Claims (26)

A method of detecting an oblique video moving object performed by a moving object detection device, comprising:
For an oblique shooting video for which shooting geometry information is provided, determining an optical flow analysis frame interval for each frame area based on the shooting geometry information; and
Detecting a moving object on an object basis based on the determined optical flow analysis frame interval,
The step of determining the optical flow analysis frame interval is,
Designating the oblique shooting video to a frame queue in frame order; and
Setting a plurality of partition areas for the latest frame in the frame queue; and
Comprising: calculating the optical flow analysis frame interval with reference to the ground distance for each pixel calculated from the shooting geometry information at the center point of each set divided area,
The optical flow analysis frame interval is determined so that the displacement of the moving object is projected in pixel units or more,
The step of calculating the optical flow analysis frame interval is,
Calculate the moving distance of the moving object per unit frame interval from the time interval between frames obtained from the video standard and the speed information of the moving object, and calculate the ground distance of the center point pixel of the divided area and the moving distance of the moving object per unit frame interval. Calculate the frame interval at which the moving distance of the moving object is greater than or equal to the ground distance of the pixel.
Oblique shooting video moving object detection method. delete According to claim 1,
In the plurality of divided areas, the center points of each divided area are arranged at equal intervals in the horizontal and vertical directions, and some areas between adjacent divided areas overlap.
Oblique shooting video moving object detection method. According to claim 1,
The plurality of divided areas are set with a boundary line at an image boundary closer to the center points of each divided area or a position closer than a neighboring center point.
Oblique shooting video moving object detection method. According to claim 1,
The optical flow analysis frame interval is calculated differently for each frame area.
Oblique shooting video moving object detection method. delete delete According to claim 1,
The step of detecting the moving object is,
acquiring past frames at each determined optical flow analysis frame interval and then calculating optical flow in the corresponding divided area among the plurality of divided areas;
estimating and compensating background motion for each segment using the calculated optical flow; and
A step of extracting the moving object in units of the object by determining the movement point based on the standard score of the movement vector size for each divided area.
Oblique shooting video moving object detection method. According to claim 8,
In the above compensation step, dense equally spaced grid points are constructed from the optical flow for each analysis frame interval, and the optical flow vectors corresponding to the grid points are added to the grid point coordinates to form a set of corresponding pairs between images, and the images composed for each analysis frame interval are formed. From the set of corresponding pairs, a transformation matrix that converts the current frame image coordinates to the previous frame image coordinates is estimated, and a displacement vector leading to the image coordinates obtained by the transformation matrix estimated from the corresponding image coordinates for each pixel in the optical flow for each analysis frame interval is estimated. Obtaining an optical flow that compensates for the background motion by subtracting it from the optical flow
Oblique shooting video moving object detection method. According to claim 8,
The step of extracting on an object basis includes determining moving points whose size of the optical flow movement vector for each divided region satisfies the movement condition from the optical flow for each divided region for which the background motion has been compensated, and determining moving points from the moving points in the entire frame. extracting moving objects
Oblique shooting video moving object detection method. According to claim 10,
In the object-by-object extraction step, the average and standard deviation of the size of the optical flow vectors of the pixels belonging to the divided region are calculated from the optical flow corresponding to the frame analysis interval for each divided region, and the optical flow vector in each divided region is calculated. pixels whose size is equal to or greater than the unit pixel length and whose standard score, which is calculated by subtracting the average from the size of the optical flow vector and dividing it by the standard deviation, is greater than a certain standard, are identified as moving points.
Oblique shooting video moving object detection method. According to claim 11,
When determining the moving point, the optical flow size of the divided area is more than a unit pixel, the standard score of the optical flow size is more than a certain standard, and points that are a certain distance away from the edge of the divided area are judged as moving points.
Oblique shooting video moving object detection method. A frame analysis interval calculation unit that determines an optical flow analysis frame interval for each frame area based on the shooting geometry information for an oblique shooting video for which shooting geometry information is provided;
a moving object detection unit that detects a moving object on an object basis based on the determined optical flow analysis frame interval; and
It includes a frame queue management unit that specifies the oblique shooting video into a frame queue in frame order,
The frame analysis interval calculation unit sets a plurality of division areas for the latest frame in the frame queue, and determines the optical flow analysis frame interval by referring to the ground distance for each pixel calculated from the shooting geometry information at the center point of each division zone. Calculate,
The optical flow analysis frame interval is determined so that the displacement of the moving object is projected in pixel units or more,
When calculating the optical flow analysis frame interval, the moving distance of the moving object per unit frame interval is calculated from the time interval between frames obtained from the video standard and the speed information of the moving object, and the ground distance of the center point pixel of the divided area is calculated Calculate a frame interval at which the moving distance of the moving object is greater than or equal to the ground distance of the pixel using the moving distance of the moving object per unit frame interval.
Moving object detection device. delete According to claim 13,
In the plurality of divided areas, the center points of each divided area are arranged at equal intervals in the horizontal and vertical directions, and some areas between adjacent divided areas overlap.
Moving object detection device. According to claim 13,
The plurality of divided areas are set with a boundary line at an image boundary closer to the center points of each divided area or a position closer than a neighboring center point.
Moving object detection device. According to claim 13,
The optical flow analysis frame interval is calculated differently for each frame area.
Moving object detection device. delete delete According to claim 13,
an optical flow calculation unit that acquires past frames for each determined optical flow analysis frame interval and then calculates the optical flow in a corresponding divided region among a plurality of divided regions; and
It further includes a background motion compensation unit that estimates and compensates for background motion for each segment using the calculated optical flow,
The moving object detection unit determines the moving point based on the standard score of the movement vector size for each divided area and extracts the moving object in units of the object.
Moving object detection device. According to claim 20,
The background motion compensation unit configures dense equally spaced grid points from the optical flow for each analysis frame interval, adds optical flow vectors corresponding to the grid points to the grid point coordinates, and configures a set of corresponding pairs between images, and images composed for each analysis frame interval. From the set of corresponding pairs, a transformation matrix that converts the current frame image coordinates to the previous frame image coordinates is estimated, and a displacement vector leading to the image coordinates obtained by the transformation matrix estimated from the corresponding image coordinates for each pixel in the optical flow for each analysis frame interval is estimated. Obtaining an optical flow that compensates for the background motion by subtracting it from the optical flow
Moving object detection device. According to claim 20,
The moving object detection unit determines moving points whose size of the optical flow movement vector for each divided region satisfies the movement condition in the optical flow for each divided region for which the background motion has been compensated, and detects the moving object throughout the frame from the moving points. extracting
Moving object detection device. According to claim 22,
The moving object detection unit calculates the average and standard deviation of the size of the optical flow vector of the pixels belonging to the divided area from the optical flow corresponding to the frame analysis interval for each divided area, and the size of the optical flow vector in each divided area is calculated. Pixels that are longer than a unit pixel length and whose standard score is divided by subtracting the average from the size of the optical flow vector and dividing it by the standard deviation are more than a certain standard are identified as moving points.
Moving object detection device. According to claim 23,
When determining the moving point, the optical flow size of the divided area is more than a unit pixel, the standard score of the optical flow size is more than a certain standard, and points that are a certain distance away from the edge of the divided area are judged as moving points.
Moving object detection device. A computer-readable recording medium storing a computer program,
When the computer program is executed by a processor,
For an oblique shooting video for which shooting geometry information is provided, determining an optical flow analysis frame interval for each frame area based on the shooting geometry information; And detecting a moving object on an object basis based on the determined optical flow analysis frame interval,
Determining the optical flow analysis frame interval may include designating the obliquely captured video to a frame queue in frame order; Setting a plurality of partition areas for the latest frame in the frame queue; Comprising: calculating the optical flow analysis frame interval with reference to the ground distance for each pixel calculated from the shooting geometry information at the center point of each set divided area,
The optical flow analysis frame interval is determined so that the displacement of the moving object is projected in pixel units or more,
The step of calculating the optical flow analysis frame interval is,
Calculate the moving distance of the moving object per unit frame interval from the time interval between frames obtained from the video standard and the speed information of the moving object, and calculate the ground distance of the center point pixel of the divided area and the moving distance of the moving object per unit frame interval. Calculate the frame interval at which the moving distance of the moving object is greater than or equal to the ground distance of the pixel.
A computer-readable recording medium including instructions for causing the processor to perform an oblique shooting video moving object detection method. A computer program stored on a computer-readable recording medium,
When the computer program is executed by a processor,
For an oblique shooting video for which shooting geometry information is provided, determining an optical flow analysis frame interval for each frame area based on the shooting geometry information; And detecting a moving object on an object basis based on the determined optical flow analysis frame interval,
Determining the optical flow analysis frame interval may include designating the obliquely captured video to a frame queue in frame order; Setting a plurality of partition areas for the latest frame in the frame queue; Comprising: calculating the optical flow analysis frame interval with reference to the ground distance for each pixel calculated from the shooting geometry information at the center point of each set divided area,
The optical flow analysis frame interval is determined so that the displacement of the moving object is projected in pixel units or more,
The step of calculating the optical flow analysis frame interval is,
Calculate the moving distance of the moving object per unit frame interval from the time interval between frames obtained from the video standard and the speed information of the moving object, and calculate the ground distance of the center point pixel of the divided area and the moving distance of the moving object per unit frame interval. Calculate the frame interval at which the moving distance of the moving object is greater than or equal to the ground distance of the pixel.
A computer program including instructions for causing the processor to perform an oblique shooting video moving object detection method.

Images