KR102233494B1 - Multi-object tracking device and method in 360 degree video space - Google Patents

Abstract

It relates to a multi-object tracking apparatus and method in a 360-degree image space, and the method of tracking a multi-object in a 360-degree image space by a multi-object tracking device is (a) within a 360-degree video frame included in the 360-degree image space. Detecting an object presence candidate region, (b) converting and extracting a viewport frame having a predetermined angle of view including the object presence candidate region from the 360-degree video frame into a flat viewport frame form, (c) the extracted Detecting an object region including an object in the plan viewport frame, (d) when the object region is detected, extracts a corresponding image signal corresponding to the detected object region, and determines the position of the detected object region in the 360 Generating a transformed position by transforming to correspond to the image space of the diagram, (e) tracking the position of an object in the 360-degree video frame based on the corresponding image signal and the transformed position, wherein (a) When there are a plurality of the object presence candidate regions detected in step ), steps (b) to (e) may be individually performed for each of the plurality of object presence candidate regions.

Description

Multi-object tracking device and method in 360-degree image space {MULTI-OBJECT TRACKING DEVICE AND METHOD IN 360 DEGREE VIDEO SPACE}

The present invention relates to a multi-object tracking apparatus and method in a 360-degree image space for detecting multiple objects in a 360-degree video and tracking a movement trajectory of each object for a video security system, entertainment application, and the like.

Recently, as crimes frequently occur in public places or restricted areas, 360-degree images including CCTV (Closed Circuit Television) are installed to monitor a designated area. However, in the case of real-time monitoring by an operator using CCTV, it often occurs that a criminal activity is not detected during monitoring. In this case, a method of detecting an event after the death by analyzing the video recorded on a video cassette recorder (VCR) or a digital video recorder (DVR) is used. To compensate for this, an intelligent image detection system that detects an event in real time using an image analysis method using an IP (Internet Protocoal) camera is drawing attention. The detection of an object in the image and tracking technology of the detected object is used to determine the abnormal signs of an event. In general, a method of detecting a moving object by modeling a background image such as GMM (Gaussian Mixture Model) is mainly used as an object detection method. do. In addition, after detecting the object, the detected object is tracked using a Kalman filter, mean shift, or bounding box overlap method. In this case, since a 360-degree image has structural distortion different from that of a planar image, a tracking method different from the conventional tracking method is required.

The technology behind the present application is disclosed in Korean Patent Publication No. 10-1735365.

The present application is to solve the above-described problems of the prior art, ERP (Equi-rectangular Projection) or CMP (Cube) composed by stitching 360-degree videos acquired from one 360-degree omnidirectional camera or videos acquired from several general cameras. -Map Projection) The purpose of a 360-degree video is to provide a screen that detects and displays the positions of a plurality of objects of a type set by a user in a video frame.

The present application is to solve the above-described problems of the prior art, and an object of the present invention is to provide a screen displaying movement trajectories of a plurality of objects of a type set by a user over time in a 360-degree video of an ERP or CMP format.

The present application is to solve the above-described problems of the prior art, and an object of the present invention is to provide viewport screens having a narrow angle of view including one or a plurality of objects designated by a user in a 360-degree video of an ERP or CMP format.

However, the technical problem to be achieved by the embodiments of the present application is not limited to the technical problems as described above, and other technical problems may exist.

As a technical means for achieving the above technical problem, a method for tracking multiple objects in a 360-degree image space by a multiple-object tracking device according to the first aspect of the present application includes: (a) 360 degrees included in the 360-degree image space. Detecting an object presence candidate region within a video frame, (b) converting and extracting a viewport frame having a predetermined angle of view including the object presence candidate region from the 360-degree video frame into a flat viewport frame form, (c ) Detecting an object region including an object with respect to the extracted plan viewport frame, (d) when the object region is detected, extracting a corresponding image signal corresponding to the detected object region, and Converting a position to correspond to the 360-degree image space to generate a transformed position, (e) tracking the position of an object in the 360-degree video frame based on the corresponding image signal and the transformed position, and If there are a plurality of the object presence candidate regions detected in step (a), steps (b) to (e) may be individually performed for each of the plurality of candidate object presence regions.

In addition, the 360-degree image space may be in ERP (Equirectangular Projection) or CMP (Cube Map Projection) format.

In addition, steps (a) to (e) are performed for each of a plurality of 360-degree video frames included in the 360-degree image space, and step (e) includes: A corresponding image signal and a transform position corresponding to the object region detected in the first frame, and a corresponding image signal corresponding to the object region detected in a second frame after the first frame in time series among the plurality of 360-degree video frames, and In consideration of the transformation position, it is possible to track the movement of the object according to the passage of time in the 360-degree image space.

In addition, the predetermined angle of view is a horizontal ratio and a preset width ratio and a vertical ratio occupied by the object presence candidate region within a limit such that some of the object presence candidate regions are not excluded within the plan viewport frame. It can be set to be more than the vertical ratio.

In addition, the corresponding image signal is a signal that separates the detected object region from other regions to enable tracking of the detected object region, and may be in the form of an image feature vector for the detected object region.

In addition, in step (a), in consideration of a possibility that an object exists in the motion region, the motion region may be detected as the object presence candidate region.

In addition, in the step (a), a background is generated and a foreground is extracted for the 360-degree video frame, and the motion region is extracted through a background difference, but only when the motion region is larger than a predetermined region size, the object It can be detected as an existing candidate region.

In addition, in step (a), instead of directly detecting an object area, an object presence candidate area may be detected using an object detector applied to an existing rectangular video frame.

In addition, in step (b), the coordinates of the viewport frame of the predetermined angle of view are converted into u and v coordinates, the yaw and pitch coordinates of the midpoint coordinates are obtained, and then converted into a sphere coordinate system, using a projection equation. After obtaining the coordinates in the old coordinate system, it can be extracted in the form of the plan viewport frame using a transformation including a rotation transformation.

In addition, when the object presence candidate regions detected in step (a) are plural, and two or more of the plurality of object presence candidate regions overlap at least partially, the step (b) includes the at least partially overlapping two or more objects. Transformation and extraction of a flat viewport frame shape may be performed for only one of the existence candidate regions.

In addition, in step (c), the object area may be detected using an object detector for the extracted plan viewport frame, and the type of the object may be identified.

In addition, in the step (e), the first frame corresponds to a frame in which an object to be tracked is located in a first boundary portion of the 360-degree image space, and the second frame is a second frame opposite to the first boundary portion. 2 In the case of object movement tracking on a discontinuous boundary corresponding to a frame in which an object to be tracked in the boundary portion is located, yaw and pitch values corresponding to the first boundary portion are determined so that yaw and pitch values at the discontinuous boundary have continuity. As a reference, yaw and pitch values corresponding to the second boundary portion may be compensated.

In addition, (f) displaying and displaying a display corresponding to the location of the tracked object on the 360-degree video frame, and (g) when the display is selected by the user, a selection object area corresponding to the selected display is included. The method may further include converting and displaying a viewport frame having a predetermined angle of view into a flat viewport frame format.

In addition, when there are a plurality of object regions detected in step (c), the display in step (f) is a plurality corresponding to a plurality of detected object regions, and in step (g), the display is displayed by the user. When two or more are selected, viewport frames having a predetermined angle of view corresponding to each of the selected two or more displays may be converted into flat viewport frames and displayed.

Meanwhile, the apparatus for tracking multiple objects in a 360-degree image space according to the second aspect of the present application includes an object presence candidate region extractor that detects an object presence candidate region within a 360-degree video frame included in the 360-degree image space, and the 360 A viewport image extractor that converts and extracts a viewport frame of a predetermined angle of view including the object presence candidate region from a video frame into a plan viewport frame form, and an object that detects an object region including an object for the extracted plan viewport frame When a detector and an object region are detected, a 360-degree coordinate converter that extracts a corresponding image signal corresponding to the detected object region and converts the position of the detected object region to correspond to the 360-degree image space to generate a transformed position And an object tracker for tracking a position of an object in the 360-degree video frame based on the corresponding video signal and the transformed position, and when there are a plurality of object presence candidate regions detected by the object presence candidate region extractor, The viewport image extractor and the object detector may individually perform each of the plurality of object presence candidate regions.

The above-described problem solving means are merely exemplary and should not be construed as limiting the present application. In addition to the above-described exemplary embodiments, additional embodiments may exist in the drawings and detailed description of the invention.

According to the above-described problem solving means of the present application, when an object of interest specified by users appears and moves in a 360-degree image having a wide viewing angle in all directions, it is possible to automatically detect the object and improve the user's 360-degree image space search efficiency. In addition, it is possible to configure a video security system or a video content generator with a standard angle of view through a separate viewport including objects.

However, the effect obtainable in the present application is not limited to the above-described effects, and other effects may exist.

1 is an ERP-format coordinate system (FIG. 1A) in which a 360-degree image is projected by ERP (Equirectangular Projection) and CMP (Cube Map Projection) according to an embodiment of the present application, and a CMP-format It is a figure showing (b)).
2 is a diagram illustrating an operation flow of a multi-object tracking device in a 360-degree image space according to an embodiment of the present application.
3 is a diagram illustrating an operation flow of an object presence candidate region extractor according to an embodiment of the present application.
4 is a diagram illustrating an operation flow of a viewport image extractor according to an embodiment of the present application.
5 is a diagram illustrating an operation flow of an object detector, a 360 degree coordinate converter, and an object tracker according to an embodiment of the present application.
6 is a diagram illustrating an operation flow for converting a position of an object in a viewport frame into an ERP or CMP space according to an embodiment of the present application.
7 is a diagram illustrating a screen displayed by expressing an object in a 360-degree video frame according to an embodiment of the present application.
8 is a diagram illustrating a 360-degree video frame and a plan viewport frame including a selection object area according to an exemplary embodiment of the present disclosure.
9 is a flowchart illustrating a method of controlling a multi-object tracking apparatus in a 360-degree image space according to an exemplary embodiment of the present disclosure.

Hereinafter, embodiments of the present application will be described in detail with reference to the accompanying drawings so that those of ordinary skill in the art may easily implement the present application. However, the present application may be implemented in various different forms and is not limited to the embodiments described herein. In addition, in the drawings, parts irrelevant to the description are omitted in order to clearly describe the present application, and similar reference numerals are attached to similar parts throughout the specification.

Throughout the present specification, when a part is said to be "connected" with another part, it is not only the case that it is "directly connected", but also "electrically connected" or "indirectly connected" with another element interposed therebetween. "Including the case.

Throughout the present specification, when a member is positioned "on", "upper", "upper", "under", "lower", and "lower" of another member, this means that a member is located on another member. This includes not only the case where they are in contact but also the case where another member exists between the two members.

In the entire specification of the present application, when a certain part "includes" a certain component, it means that other components may be further included rather than excluding other components unless specifically stated to the contrary.

The present application relates to a multi-object tracking apparatus and method in a 360-degree image space, and extracts a viewport for an object presence candidate area detected in a 360-degree video, detects an object in the viewport, and displays it in a 360-degree video. Alternatively, it is possible to provide a screen that detects and displays the position of a plurality of objects of a type set by a user in a video frame in a 360-degree video in CMP format, and one or more user-designated 360-degree videos in ERP or CMP format. It is a technology capable of providing a viewport screen with a predetermined angle of view including objects. Hereinafter, a multi-object tracking device (hereinafter referred to as “the device”) in a 360-degree image space according to an exemplary embodiment of the present disclosure will be described.

1 is an ERP-format coordinate system (FIG. 1A) in which a 360-degree image is projected by ERP (Equirectangular Projection) and CMP (Cube Map Projection) according to an embodiment of the present application, and a CMP-format (b)), and FIG. 2 is a diagram illustrating an operation flow of a multi-object tracking device in a 360-degree image space according to an embodiment of the present application.

Since 360-degree video uses a method of projecting an image from all directions to one frame, it has a characteristic that there is structural distortion in the image. The present apparatus 1 is for resolving an object error detection due to structural distortion of an image and a problem in that a small object cannot be tracked, and may extract an object presence candidate region by detecting a motion region.

1 and 2, the apparatus 1 includes an object presence candidate region extractor 100, a viewport image extractor 200, an object detector 300, a 360 degree coordinate converter 400, and an object tracker 500. It may include.

The object presence candidate region extractor 100 may detect the object presence candidate region within a 360 degree video frame included in the 360 degree image space.

A 360-degree video (image) that simultaneously records an omnidirectional image can be reproduced using multiple projection methods, and the object presence candidate region extractor 100 projects a 360-degree image into an ERP (Equirectangular Projection) coordinate system or A coordinate system in CMP format projected by CMP (Cube Map Projection) can be acquired from a 360-degree image. That is, the 360-degree video space in the device 1 may be in ERP or CMP format.

The object presence candidate region extractor 100 may detect an object presence candidate position within a 360-degree video frame.

3 is a diagram illustrating an operation flow of an object presence candidate region extractor according to an embodiment of the present application.

Referring to FIG. 3A, the object presence candidate region extractor 100 may detect the object presence candidate region from the motion region in consideration of the possibility that the object exists in the motion region. The object presence candidate region extractor 100 performs background generation and foreground extraction for a 360-degree video frame, and extracts a motion region through a background difference, but only when the motion region is larger than a predetermined region size, the object presence candidate region It can be detected as. The object presence candidate region extractor 100 generates a background for a 360-degree video image that is input as an input, extracts the foreground, and uses the background difference to find a region where motion exists between frames, and a viewport image extractor described later. (200) may convert the upper left coordinate of the corresponding area into u, v, yaw, and pitch formats.

First, the object presence candidate region extractor 100 performs background generation and foreground extraction based on 360-degree video in ERP or CMP format ('background creation/foreground extraction' in FIG. 3), and background difference (' Background difference') can be used to detect the location of the candidate existence of the object. In detail, the object presence candidate region extractor 100 generates a background and extracts a foreground in a 360 video frame space in ERP or CMP format ('background creation/foreground extraction' in FIG. 3), and a background difference ('background difference in FIG. 3). Branching') and morphology operation/labeling ('morphology operation/labeling' in FIG. 3) extracted ERP or CMP coordinate system format object region representative coordinates may be used to generate an object presence candidate region.

Here, the morphology operation is one of the techniques for analyzing and processing the shape of an image.It is an operation method that extracts only the desired part in the image by reflecting the information of the target object.In order to extract the object structure in the image, the boundary and skeleton of the image It is a technique used to extract the elements necessary to express the shape of the block, etc. The most basic operation constituting the morphology technique may include an erosion operation and a dilation operation, and the morphology operation may be used in a binary image or a contrast image. In addition, labeling refers to grouping and numbering adjacent pixel values to identify objects in the binarized image, and may refer to a method of grouping adjacent ring pixels to determine an object. Labeling can be divided into 4-way labeling and 8-way labeling. Since morphology calculation and labeling are already known techniques, detailed descriptions will be omitted.

The object presence candidate region extractor 100 generates a background for a 360-degree video frame in ERP or CMP format, extracts a foreground, and performs a background difference, and performs morphology calculation or labeling to determine the object region representative coordinates, that is, the object presence candidate region. Can be generated.

The object presence candidate region extractor 100 performs background generation and foreground extraction for a 360-degree video frame, and extracts a motion region through a background difference. Can be detected. That is, the object presence candidate region extractor 100 generates a background in a 360 video frame space in ERP or CMP format and extracts the foreground, and the region size of the upper left coordinate of the motion region extracted through background difference and morphology calculation/labeling is a certain level. Only in the case of abnormality, it can be detected as an object presence candidate region.

Referring to FIG. 3(b), the object presence candidate region extractor 100 may extract the object presence candidate region through background generation/foreground extraction, background difference branching, and morphology calculation/labeling, but is not limited thereto. The object detector 300, which will be described later, may be used to detect the representative coordinates of an object area in the form of an ERP or CMP coordinate system, that is, an object presence candidate area. The object presence candidate region extractor 100 may detect the candidate existence position of an object using the object detector 300 applied to the existing rectangular video frame, and the object detector ( 300) can extract the upper left coordinate of the object area detected. That is, the object presence candidate region extractor 100 may detect the object presence candidate region instead of directly detecting the object by using the object detector 300 applied to the existing rectangular video frame. The object detector 300 will be described in more detail below.

The present apparatus 1 can eliminate false detection due to structural distortion of the image by detecting an object for the viewport image by creating a viewport with a narrow angle of view for an object presence candidate region.

4 is a diagram illustrating an operation flow of a viewport image extractor according to an embodiment of the present application.

The viewport image extractor 200 may check overlap to prevent multiple viewports having a predetermined angle of view from being generated at one point, and generate a flat viewport image having a predetermined size without distortion at a corresponding position.

Referring to FIG. 4, the viewport image extractor 200 may convert and extract a viewport frame having a predetermined angle of view including an object presence candidate region from a 360-degree video frame into a flat viewport frame form.

The viewport image extractor 200 may convert a viewport frame having a predetermined angle of view centered on an object presence candidate region from a 360-degree video frame. The viewport image extractor 200 may generate a viewport having a predetermined angle of view in a frame unit based on an object presence candidate region.

The viewport image extractor 200 converts the coordinates of the viewport frame at a predetermined angle of view into u and v coordinates, obtains the yaw pitch coordinates of the midpoint coordinates, and converts them into a sphere coordinate system. After obtaining the coordinates, it can be extracted in the form of a flat viewport frame using a transformation including a rotation transformation. Here, the predetermined angle of view is such that the horizontal ratio and the vertical ratio occupied by the object presence candidate region are equal to or greater than the preset horizontal ratio and the preset vertical ratio, within the limit of not excluding some of the object presence candidate regions within the plan viewport frame. Can be set. For example, the viewport image extractor 200 may form a viewport frame with an angle of view in which a large margin is not formed around the object presence candidate region. The viewport frame is preferably formed in a direction that does not include an area other than the object presence candidate area if possible in order to improve the computational speed, so the predetermined angle of view is narrow (for example, a view angle of 54° horizontally and 90° vertically) accordingly. Can be set. That is, the narrow angle of view may be such that the entire object presence candidate region is included without excluding some of the object presence candidate regions.

Referring to FIG. 4, the viewport image extractor 200 is implemented as a single device to generate a viewport of a predetermined angle of view in a frame unit around an object presence candidate region, but is not limited thereto. Using the u- and v-coordinate converter 210, the sphere coordinate system converter 220, and the viewport extractor 230 included in 200), a viewport having a predetermined angle of view may be generated in units of frames. For example, the viewport image extractor 200 may include a u- and v-coordinate converter 210, a sphere coordinate system converter 220, and a viewport extractor 230. The u, v coordinate converter 210 receives a 360-degree video frame ('ERP/CMP coordinates' in FIG. 4) in ERP or CMP format from the object presence candidate region extractor 100, and converts the coordinates of the 360-degree image to u, You can convert it to v coordinates and convert the midpoint coordinates to yaw and pitch coordinates. The sphere coordinate system converter 220 may receive the u, v coordinate system from the u, v coordinate converter 210 and convert it into a sphere coordinate system, and obtain the coordinates in the sphere coordinate system using a projection equation. have. The viewport extractor 230 may generate a plan viewport image using rotation transformation or the like based on the old coordinate system.

In this case, when there are a plurality of object presence candidate regions detected by the object presence candidate region extractor 100, and two or more of the plurality of object presence candidate regions overlap at least partially, the viewport image extractor 200 may at least partially overlap two or more object presence candidate regions. Transformation and extraction of a flat viewport frame shape can be performed for only one of the object presence candidate regions. For example, when there are two object presence candidate regions detected by the object presence candidate region extractor 100 and the two object presence candidate regions overlap each other, the viewport image extractor 200 may determine the existence of the object detected first among the two. The candidate region may be left and the object presence candidate region detected later may be excluded. As another example, when there are three object presence candidate regions detected by the object presence candidate region extractor 100 and the three object presence candidate regions overlap each other, an object presence candidate region located in the middle of the three is left, and relatively both sides It is possible to exclude an object presence candidate area located at. This is to prevent overlapping display of one object in the overlapping area when extracting a viewport frame of a predetermined angle of view for the entire frame, ERP or CMP format, and a viewport of a predetermined angle of view within a certain distance. By extracting the image, it is possible to prevent the occurrence of duplication. When two or more object presence candidate regions are overlapped by the viewport image extractor 200, a method of selecting one is not limited thereto.

5 is a diagram illustrating an operation flow of an object detector, a 360 degree coordinate converter, and an object tracker according to an embodiment of the present application.

Referring to FIG. 5, the object detector 300 may detect an object region including an object with respect to the extracted plan viewport frame.

When the device 1 extracts a viewport frame with a narrow angle of view for an object presence candidate area from one frame, it detects the object coordinates in the viewport by performing object detection on the viewport frame. You can use a method that converts to the coordinates of the image and passes it to the tracker. That is, the present apparatus 1 may detect the position of an object and an image signal by using the viewport frame of each narrow angle of view, and convert the detected position of each object into a 360-degree video frame space.

Specifically, the object detector 300 of the apparatus 1 may detect a rectangle including an object for the converted viewport frame having a predetermined angle of view. The object detector 300 detects a rectangle including an object with respect to a viewport frame having a predetermined angle of view converted from a 360-degree image in the conventional EPR or CMP format, and identifies the type of the object. That is, the apparatus 1 may detect an object area with respect to the extracted plan viewport frame using the object detector 300 and classify the type of the object. The types of objects may be classified into, for example, people and non-human objects. As another example, the type of object may be classified into a person, a means of transportation used by a person, and other objects excluding a person and a means of transportation. In the case of the above-described example, the object detector 300 may detect an object area in the shape of a rectangle including a person or a moving means used by the object to be tracked.

When the object area is detected by the object detector 300, the 360-degree coordinate converter 400 extracts a corresponding image signal corresponding to the detected object area, and converts the position of the detected object area correspondingly in the 360-degree image space. You can create a transform location. Here, the corresponding image signal is a signal that separates the detected object region from other regions to enable tracking of the detected object region, and may be in the form of an image feature vector for the object region detected by the object detector 300.

Referring to FIG. 5, the object detector 300 may detect an object area including an object with respect to the extracted flat viewport frame ('viewport image' in FIG. 5), and the 360 degree coordinate converter 400 is detected. The corresponding image signal in the form of a CNN-based feature vector corresponding to the object region can be extracted, and the position of the detected object region is converted to correspond to a 360-degree image space, and the transformed position ('360 degree image object coordinates' in Fig. 5) Can be created. In summary, when the object detector 300 extracts the location of the tracked object and the image signal in the viewport frame having a narrow angle of view in the candidate area for the existence of the 360 video frame object, the 360-degree coordinate converter 400 The position can be retransformed into a 360-degree video frame space in ERP or CMP format.

The object tracker 500 may track the position of the object in the 360-degree video frame based on the corresponding image signal and the transformed position. The object tracker 500 may track the location of each object using the converted location of each object (ERP or CMP) and an image signal (viewport or ERP/CMP) in a rectangle including the object.

If there are a plurality of object presence candidate regions detected by the object presence candidate region extractor 100, the viewport image extractor 200, the object detector 300, the 360 degree coordinate converter 400, and the object tracker 500 are plural. It can be performed individually for each of the two object presence candidate regions.

The object presence candidate region extractor 100, the viewport image extractor 200, the object detector 300, the 360-degree coordinate converter 400, and the object tracker 500 are each of a plurality of 360-degree video frames included in the 360-degree image space. The object tracker 500 is performed on the corresponding image signal and the transform position corresponding to the object region detected in the first frame among the plurality of 360-degree video frames, and the first plane among the plurality of 360-degree video frames in time series. The movement of the object over time in the 360-degree image space may be tracked in consideration of the corresponding image signal and the transformed position corresponding to the object region detected in the second frame after the time. For example, the second frame may be a frame immediately after the first frame in time series, or may be a frame following at least one frame after the first frame.

The object tracker 500 can track by compensating so that yaw and pitch values are continuous at the discontinuous boundary of ERP or CMP during tracking. The object tracker 500 corresponds to a frame in which an object to be tracked is located at a first boundary portion of a 360-degree image space, and the first frame is an object to be tracked at a second boundary portion opposite to the first boundary portion. In the case of object movement tracking for a discontinuous boundary corresponding to the frame in which is located, the yaw and pitch values at the discontinuous boundary correspond to the second boundary based on the yaw and pitch values corresponding to the first boundary so that the values have continuity. The yaw and pitch values can be compensated. For example, when an object existing at the left border border of the first frame appears at the right border border of the first frame connected to the left border border of the first frame, yaw corresponding to the left border border, You can adjust the yaw and pitch values corresponding to the right border boundary based on the pitch value.

6 is a diagram illustrating an operation flow for converting a position of an object in a viewport frame into an ERP or CMP space according to an embodiment of the present application.

Referring to FIG. 6, the device 1 uses a u, v coordinate converter 210 and a sphere coordinate system converter 220 based on the position coordinates of each object of the viewport obtained from the object tracker 500. The position of the object can be converted to ERP or CMP coordinates. The u, v coordinate converter 210 receives the position coordinates of each object of the viewport, converts them to u, v coordinates, obtains the yaw and pitch coordinates of the central coordinates, and then the old coordinate system converter 220 converts the u, v coordinates to the old coordinates. After converting to, it can be converted to an ERP or CMP coordinate system using a 2D projection equation.

7 is a diagram illustrating a screen displayed by expressing an object in a 360-degree video frame according to an embodiment of the present application, and FIG. 8 is a 360-degree video frame including a selected object area and a plan viewport according to an embodiment of the present application. It is a diagram showing a frame.

The device 1 can display and display a display corresponding to the position of an object converted to an ERP or CMP coordinate system in a 360-degree video frame. When a display is selected by the user, the selected object area corresponding to the selected display is displayed. The included viewport frame having a predetermined angle of view may be converted into a flat viewport frame and displayed. If there are a plurality of object regions detected by the object detector 300, the device 1 may have a plurality of displays corresponding to the plurality of detected object regions, and if two or more displays are selected by the user, A predetermined angle of view viewport frame corresponding to each of the two or more selected displays may be converted into a flat viewport viewport frame and displayed.

The user can use the viewport image for the object being tracked, and if he wants to track, he can create a new viewport image through clicking (selection) to track the object. If, as described above, when there are a plurality of detected object areas, if the user selects several objects, the device 1 can create an arbitrary viewport according to the user's selection, and the general angle of view is 360 degrees. You can enable tracking of objects of interest in the video.

In summary, the device 1 can simultaneously display the positions of objects converted into a 360-degree video frame space on the screen, and a user can select some of the objects displayed in the 360-degree video frame space, and 360-degree video In the frame space, a viewport frame having a predetermined angle of view including an object selected by a user may be converted and reproduced. The device (1) can simultaneously display the positions of objects converted into a 360-degree video frame space on the screen, simultaneously displaying the ERP or CMP format of the object observed in a narrow viewport frame on a 360-degree video frame screen. have. When extracting a viewport frame with a narrow angle of view for an entire frame of ERP or CMP format 360-degree video frame, the device 1 prevents duplicate display of one object in the overlapping area by the above-described object tracker 500. In addition, it is possible to prevent overlapping by extracting a viewport image having a predetermined angle of view again within a certain distance. In addition, the device 1 can process the user's selection for the tracking object displayed in the ERP or CMP format video frame space. When the user selects the object to be tracked using a means such as a mouse click, Multi-object selection may be possible by obtaining the object coordinates of the corresponding selection occurrence location, and tracking and reproduction of the corresponding object may be performed by performing a viewport frame conversion of a predetermined angle of view including the selected object. In this case, there may be a plurality of selected objects, and if the object is being tracked, a viewport video sequence having a predetermined angle of view may be generated accordingly to facilitate observation according to the movement of the object.

According to the multi-object tracking apparatus 1 in a 360-degree image space according to an embodiment of the present application, it is possible to automatically detect the appearance and movement of an object of interest designated by users in a 360-degree video having a wide viewing angle in all directions, It is possible to increase the user's search efficiency in a 360-degree video space, and to configure a video security system or a video content generator with a standard angle of view through a separate viewport including objects.

Hereinafter, based on the details described above, the operation flow of the present application will be briefly described.

9 is a flowchart illustrating a method for tracking multiple objects in a 360-degree image space according to an embodiment of the present application.

The multi-object tracking method in the 360-degree image space shown in FIG. 9 may be performed by the multi-object tracking apparatus 1 in the 360-degree image space described above. Accordingly, even if omitted below, the description of the multi-object tracking apparatus 1 in a 360-degree image space may be equally applied to a description of a method for tracking multiple objects in a 360-degree image space.

Referring to FIG. 9, the multi-object tracking method in a 360-degree image space according to an embodiment of the present application may detect an object presence candidate region within a 360-degree video frame included in the 360-degree image space (S901).

Next, a viewport frame having a predetermined angle of view including an object presence candidate region may be converted into a flat viewport frame form and extracted from the 360-degree video frame (S902).

Next, an object area including an object may be detected with respect to the extracted plan viewport frame (S903).

Next, when the object region is detected, a corresponding image signal corresponding to the detected object region is extracted, and the position of the detected object region is converted to correspond to a 360-degree image space to generate a transformed position (S904). .

Next, the position of the object in the 360-degree video frame may be tracked based on the corresponding video signal and the transformed position (S905).

In this case, when there are a plurality of object presence candidate regions detected in step S901, steps S902 to S905 may be individually performed for each of the plurality of object presence candidate regions.

In the above description, steps 901 to 905 may be further divided into additional steps or may be combined into fewer steps, according to an embodiment of the present disclosure. In addition, some steps may be omitted as necessary, or the order between steps may be changed.

The method for tracking multiple objects in a 360-degree image space according to an exemplary embodiment of the present disclosure may be implemented in the form of a program command that can be executed through various computer means and recorded in a computer-readable medium. The computer-readable medium may include program instructions, data files, data structures, etc. alone or in combination. The program instructions recorded in the medium may be specially designed and configured for the present invention, or may be known and usable to those skilled in computer software. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tapes, optical media such as CD-ROMs and DVDs, and magnetic media such as floptical disks. -A hardware device specially configured to store and execute program instructions such as magneto-optical media, and ROM, RAM, flash memory, and the like. Examples of program instructions include not only machine language codes such as those produced by a compiler, but also high-level language codes that can be executed by a computer using an interpreter or the like. The hardware device described above may be configured to operate as one or more software modules to perform the operation of the present invention, and vice versa.

In addition, the above-described method for tracking multiple objects in a 360-degree image space may be implemented in the form of a computer program or application executed by a computer stored in a recording medium.

The foregoing description of the present application is for illustrative purposes only, and those of ordinary skill in the art to which the present application pertains will be able to understand that it can be easily modified into other specific forms without changing the technical spirit or essential features of the present application. Therefore, it should be understood that the embodiments described above are illustrative and non-limiting in all respects. For example, each component described as a single type may be implemented in a distributed manner, and similarly, components described as being distributed may also be implemented in a combined form.

The scope of the present application is indicated by the claims to be described later rather than the detailed description, and all changes or modified forms derived from the meaning and scope of the claims and their equivalent concepts should be construed as being included in the scope of the present application.

1: Multi-object tracking device in 360-degree image space
100: object existence candidate region extractor
200: viewport image extractor
300: object detector
400: 360 degree coordinate converter
500: object tracker

Claims (15)

A multi-object tracking method in a 360-degree image space by a multi-object tracking device,
(a) in the 360-degree video frame included in the 360-degree video space, detecting a motion region as an object presence candidate region in consideration of a possibility that an object exists in the motion region;
(b) Considering that a 360-degree image has structural distortion different from that of a planar image, converting and extracting a viewport frame having a predetermined angle of view including the object presence candidate region from the 360-degree video frame into a planar viewport frame format. step;
(c) detecting an object region including an object by performing object detection based on the plane viewport frame extracted corresponding to the object presence candidate region;
(d) when the object region is detected, extracting a corresponding image signal corresponding to the detected object region, and converting the position of the detected object region to correspond to the 360-degree image space to generate a transformed position;
(e) tracking the position of the object in the 360-degree video frame based on the corresponding video signal and the transformed position,
When the object presence candidate regions detected in step (a) are plural, steps (b) to (e) are individually performed for each of the plurality of candidate object presence regions,
The predetermined angle of view is set within a limit such that some of the object presence candidate regions are not excluded from the plan viewport frame. The method of claim 1,
The 360-degree image space is ERP (Equirectangular Projection) or CMP (Cube Map Projection) format. The method of claim 1,
Steps (a) to (e) are performed for each of a plurality of 360-degree video frames included in the 360-degree image space,
The step (e) includes: a corresponding image signal and a transform position corresponding to an object region detected in a first frame among the plurality of 360-degree video frames, and a time-sequentially after the first frame among the plurality of 360-degree video frames. Multi-object in 360-degree image space, which tracks the movement of an object according to the passage of time in the 360-degree image space in consideration of a corresponding image signal and a transform position corresponding to the object region detected in the second frame Tracking method. The method of claim 1,
The predetermined angle of view is set such that a horizontal ratio and a vertical ratio occupied by the object presence candidate region are equal to or greater than a preset horizontal ratio and a preset vertical ratio. The method of claim 1,
The corresponding image signal is a signal that separates the detected object region from other regions to enable tracking of the detected object region, and is in the form of an image feature vector for the detected object region. Multi-object tracking method. delete The method of claim 1,
The step (a),
Perform background generation and foreground extraction for the 360-degree video frame, and extract the motion region through background difference
The multi-object tracking method in a 360 degree image space, wherein the motion region is detected as the object presence candidate region only when the motion region is larger than a predetermined region size. The method of claim 1,
The step (a),
Using an object detector applied to a conventional rectangular video frame, instead of directly detecting an object region, an object presence candidate region is detected. The method of claim 1,
In the step (b), the coordinates of the viewport frame of the predetermined angle of view are converted into u and v coordinates, yaw and pitch coordinates of the midpoint coordinates are obtained, and then converted into a sphere coordinate system. A method of tracking multiple objects in a 360-degree image space, which is to extract the coordinates in the plane viewport frame form using a transformation including a rotation transformation. The method of claim 1,
When the object presence candidate regions detected in step (a) are plural, and at least two or more of the plurality of object presence candidate regions overlap at least partially,
In the step (b), transforming and extracting a plan viewport frame type is performed only on one of the at least partially overlapping two or more object presence candidate regions. The method of claim 1,
In the step (c), the object region is detected using an object detector with respect to the extracted plan viewport frame, and the type of the object is classified. The method of claim 3,
In the step (e), the first frame corresponds to a frame in which an object to be tracked is located in a first boundary portion of the 360-degree image space, and the second frame is a second boundary opposite to the first boundary portion. In the case of object movement tracking on a discontinuous boundary corresponding to a frame in which an object to be tracked in the part is located, the yaw and pitch values at the discontinuous boundary are based on the yaw and pitch values corresponding to the first boundary so as to have continuity. Compensating the yaw and pitch values corresponding to the second boundary portion, a multi-object tracking method in a 360-degree image space. The method of claim 1,
(f) displaying and displaying a display corresponding to the location of the tracked object on the 360-degree video frame; And
(g) When the display is selected by the user, in a 360-degree image space, further comprising converting and displaying a viewport frame having a predetermined angle of view including a selected object area corresponding to the selected display into a flat viewport frame format. Multi-object tracking method. The method of claim 13,
When there are a plurality of object areas detected in step (c),
In the step (f), the display is a plurality corresponding to a plurality of detected object areas,
In the step (g), when two or more displays are selected by the user, viewport frames of a predetermined angle of view corresponding to each of the selected two or more displays are converted into flat viewport frames and displayed. Multi-object tracking method in. A multi-object tracking device in a 360-degree image space,
An object presence candidate region extractor for detecting a motion region as an object presence candidate region in consideration of a possibility that an object exists in the motion region within the 360 degree video frame included in the 360 degree image space;
A viewport image extractor that converts and extracts a viewport frame of a predetermined angle of view including the object presence candidate region from the 360-degree video frame into a flat viewport frame form, taking into account that a 360-degree image has structural distortion different from that of a planar image. ;
An object detector configured to detect an object region including an object by performing object detection based on the plane viewport frame extracted corresponding to the object presence candidate region;
A 360-degree coordinate converter for generating a transformed position by extracting a corresponding image signal corresponding to the detected object area, and converting the position of the detected object area to correspond to the 360-degree image space when the object area is detected;
An object tracker for tracking the position of the object in the 360-degree video frame based on the corresponding video signal and the transformed position,
When there are a plurality of object presence candidate regions detected by the object presence candidate region extractor, the viewport image extractor and the object detector are individually performed for each of the plurality of object presence candidate regions,
The predetermined angle of view is set within a limit such that some of the object presence candidate regions are not excluded in the plan viewport frame.

Images