KR20190043452A - Method and apparatus for monitoring moving picture using omni-directional multi-camera - Google Patents

Abstract

The present invention provides a method for monitoring a video in real time by using an omnidirectional multi-camera, which comprises the following steps: receiving an input image from a multi-camera; capturing and reproducing the received input image of the multi-camera in real time based on synchronization; performing real-time stitching image processing based on real-time camera posture and color correction processing for the reproduced input image of the multi-camera; and outputting the input image of the multi-camera in which the image processing is performed.

Description

Technical Field [0001] The present invention relates to a method and apparatus for real-

The present invention relates to a method for performing real-time video monitoring. More particularly, the present invention relates to a method for performing real time video monitoring using a omnidirectional multi-camera.

In the case of acquiring a high-quality omnidirectional camera image, a single camera shooting method is not able to acquire high-quality images due to lens distortion, and thus, a shooting method utilizing a league equipped with two or more cameras is mainstream.

However, in the above-described special shooting method, a viewfinder mounted on a single camera has to be utilized. At this time, it is impossible to match the posture, color, etc. of the mounted multi-camera according to the conventional image monitoring method, so post-correction work may be required. In this case, since the post-correction operation must be performed, it is difficult to relay and capture an omnidirectional video due to problems such as failure in shooting and deterioration in image quality.

It is an object of the present invention to provide a method and apparatus for performing real time video monitoring.

An object of the present invention is to provide a method for solving the inconvenience of shooting when acquiring a high-quality omnidirectional camera image and improving omnidirectional moving image quality.

It is an object of the present invention to provide a real-time monitoring technology suitable for taking a panoramic multi-camera.

It is an object of the present invention to provide a real-time monitoring method and apparatus suitable for omnidirectional multi-camera shooting for improving omnidirectional moving image quality.

It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory and are not restrictive of the invention, unless further departing from the spirit and scope of the invention as defined by the appended claims. It will be possible.

According to an embodiment of the present invention, a video real-time monitoring method using a multi-camera can be provided. In this case, the video real-time monitoring method using a multi-camera includes receiving an input image from a multi-camera, capturing and reproducing the received multi-camera input image in real time based on synchronization, Performing a real-time stitching image processing based on a real-time camera posture and a color correction process for a multi-camera input image in which a mask for a specified ROI area is specified and captured and reproduced, And outputting the multi-camera input image subjected to the image processing.

According to the present invention, it is possible to provide a method and apparatus for performing real-time video monitoring.

According to the present invention, it is possible to provide a method for solving the inconvenience of shooting when acquiring a high-quality omnidirectional camera image and improving omnidirectional moving image quality.

According to the present invention, it is possible to provide a real-time monitoring technique suitable for shooting in omnidirectional multi-cameras.

According to the present invention, it is possible to provide a real-time monitoring method and apparatus suitable for omnidirectional multi-camera shooting for improving omnidirectional moving image quality.

The effects obtained by the present invention are not limited to the above-mentioned effects, and other effects not mentioned can be clearly understood by those skilled in the art from the following description will be.

1 is a diagram showing a method of displaying a time code format on a screen.
2 is a view showing a screen that is cropped based on FOV (Field Of View) with respect to an input image according to a photographer's intention.
3 is a diagram illustrating a screen in which a user manually designates an ROI area.
4 is a diagram illustrating a mask that is created by a user manually designating an ROI area.
5 is a flowchart illustrating a method for real-time monitoring of moving images using a multi-camera.

Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings so that those skilled in the art can easily carry out the present invention. The present invention may, however, be embodied in many different forms and should not be construed as limited to the embodiments set forth herein.

In the following description, well-known functions or constructions are not described in detail since they may obscure the subject matter of the present invention. In the drawings, parts not related to the description of the present invention are omitted, and like parts are denoted by similar reference numerals.

In the present invention, when an element is referred to as being " connected ", " coupled ", or " connected " to another element, this means not only a direct connection relationship but also an indirect connection relationship May also be included. Also, when an element is referred to as " comprising " or " having " another element, it is meant to include not only excluding another element but also another element .

In the present invention, the terms first, second, etc. are used only for the purpose of distinguishing one element from another, and the order or importance of the elements is not limited unless otherwise stated. Thus, within the scope of the present invention, the first component in one embodiment may be referred to as a second component in another embodiment, and similarly the second component in one embodiment may be referred to as a first component .

In the present invention, the components that are distinguished from each other are intended to clearly describe the respective features, and do not necessarily mean that the components are separated. That is, a plurality of components may be integrated into one hardware or software unit, or a single component may be distributed into a plurality of hardware or software units. Accordingly, such integrated or distributed embodiments are included within the scope of the present invention, unless otherwise noted.

In the present invention, the components described in the various embodiments are not necessarily essential components, and some may be optional components. Accordingly, embodiments consisting of a subset of the components described in one embodiment are also included in the scope of the present invention. Also, embodiments that include other components in addition to the components described in the various embodiments are also included in the scope of the present invention.

Hereinafter, real-time monitoring methods and apparatuses suitable for the omni-directional multi-camera shooting according to embodiments of the present invention will be described.

Real-time capture / playback based on multi-camera input video synchronization

For example, images input from a multi-camera can be captured and reproduced in real time based on synchronization. More particularly, the present invention can synchronize input images received from a plurality of cameras to capture and play back combined images in real time. At this time, there is a need to simultaneously capture and reproduce a plurality of image sources having the image format quality supported by the image capture card in real time. For example, the plurality of images may be 10 to 12 or more image sources.

The default video source capture format may be " 1920x1080@59.94fps, yuv422 @ 8bits ". At this time, the image capture format other than the above-described capture format must support all the capture formats provided by the capture card, and the user can select the image format or automatically recognize the input image format to capture, , Should be distinguishable. However, as an example, the capture format of the basic video source may be set differently, and is not limited to the above-described embodiment.

In addition, for example, the time code in the captured image source may be displayed for each image to confirm whether or not the reproduced images are synchronized with each other. More specifically, it is possible to confirm whether or not the images are synchronized by displaying each time code for each of a plurality of images obtained from the multi-camera. At this time, for example, the "RT188 VITC" time code embedded in the captured image source may be displayed for each image to confirm whether or not the reproduced images are synchronized with each other. At this time, when a difference occurs in the time code, the difference can be recognized and confirmed. Thus, synchronization of a plurality of images can be performed in the multi-camera.

In addition, for example, video files and images stored in addition to a camera image monitoring method using a capture board can be loaded and monitored. At this time, various formats such as a moving picture playback format, a compression format, and an image format can be supported. For example, the playback format may support AVI (YUV format such as UYUV), but is not limited to the above-described embodiment. In addition, mp4, HEVC, H264, and the like may be supported as the compression format, but the present invention is not limited to the above-described embodiments. The image format may be BMP, PNG, JPG, TIFF, or the like, but is not limited to the above embodiment. In this case, for example, in the case of a moving image, it is necessary that the individual or all timelines be marked so that the user can control the file playback, such as random access.

In addition, the input image can be converted / controlled according to user's selection such as left / right flip, left / right rotation, enlargement / reduction in individual / In this case, for example, the conversion control described above needs to be implemented on a GPU basis so that there is no speed reduction problem. That is, conversion control on the input image may be possible and is not limited to the above-described embodiment.

In addition, when reproducing the captured input image, various playback views must be supported. At this time, the reproduction can be reproduced based on the projection view. Also, as an example, the modeling scheme may support at least one of a 2D view, Equirectangular view, Sphere view, Cylinderical view, and Cubic view. Further, for the video to be reproduced, the conversion operations such as enlargement or reduction based on the input information can be implemented without degrading the processing speed. In this case, for example, a basic digital zoom reproduction trick mode such as zooming in upon reception of an event such as a mouse can be implemented without degrading the overall processing speed.

In addition, alpha blending can be supported and its density can be controlled in overlapping portions of images according to their positions during reproduction. At this time, the image redundancy automatic calculation mark according to the algorithm according to the embodiments of the present invention can be implemented without degrading the entire processing speed. That is, it is possible to perform processing for the overlapping portion between a plurality of images of the multi-camera so as to be captured and reproduced as described above.

In addition, calculation indications may be required by implementing the color comparison method according to the algorithm and design according to embodiments of the present invention. At this time, there is a need to realize a function of photographing convenience tool such as a grid and a horizontal / vertical system picking method without deteriorating the entire processing speed.

1 is a diagram showing a method of displaying a time code format on a screen.

Referring to FIG. 1, video files and images stored in addition to the camera image monitoring method using the capture board can be loaded and monitored. At this time, a multi-format such as a moving picture playback format, a compression format, and an image format can be supported, as described above. In addition, for example, in the case of a moving picture, an individual or an entire timeline 110 can be displayed. This enables the user to control the file playback such as random access. More specifically, the timeline 110 may be included in a moving picture as shown in FIG. 1, and information on the timeline 110 may be utilized to control file reproduction. That is, the timeline 110 is checked among a plurality of images, and synchronization can be performed based on the timeline. In addition, using the information on the timeline included in the moving picture may help to control the moving picture, and is not limited to the above-described embodiment. In addition, when the project is created through the timeline 110, camera input and video input image input can be selected.

In addition, for example, conversion control may be possible for each input image. In addition, for example, conversion control can be collectively performed on the input image. That is, the conversion control in the multi-camera image can be performed individually or collectively. In this case, the conversion control may be a control performed by a user input such as left / right flip, left / right rotation, and enlargement / reduction, and may be as shown in Table 1 as an example.

[Table 1]

Also, as described above, when reproducing the captured input image, various playback views can be supported. At this time, the reproduction can be reproduced based on the projection view. Also, as an example, the modeling scheme may support at least one of a 2D view, Equirectangular view, Sphere view, Cylinderical view, and Cubic view. Further, for the video to be reproduced, the conversion operations such as enlargement or reduction based on the input information can be implemented without degrading the processing speed. In this case, for example, a basic digital zoom reproduction trick mode such as zooming in upon reception of an event such as a mouse can be implemented without degrading the overall processing speed. At this time, for example, a Media Framework may be implemented based on a GPU.

In addition, for example, when an image is reproduced, alpha blending can be supported in an overlapping portion between images according to the reproduction position of eternal life. For example, alpha blending may be a function of giving a pixel value, and an image redundancy automatic calculation mark according to an algorithm based on alpha blending may be implemented without degrading the overall processing speed. At this time, it is possible to perform blending on overlapping area by activating the check box as a function of performing alpha blending on the screen and inserting an alpha value. For example, the blending value may be a value between 0 and 1.0, but is not limited thereto.

In addition, for example, a color comparison method is required to be calculated and marked according to a predetermined algorithm and design, and a photographing convenience tool function such as a grid horizontal / vertical system pick method can be implemented without degrading the entire processing speed. In this case, for example, a user viewing an image may have difficulty visually confirming whether the object in the image is on a horizontal / vertical line. Therefore, there is a need to provide horizontal lines perpendicular to the image so that the user can easily distinguish them.

Hereinafter, a method for realizing the multi-camera input image real-time lens correction image processing will be described based on the above description.

Implementation of multi-camera input image real-time lens correction image processing

Based on the above description, it is possible to capture and reproduce an input image in real time in a multi-camera. In this case, for example, a method of reflecting a photographic lens type (Rectlinear, Circular Fish eye, Full frame Fisheye) and a lens angle of view in real time according to a predetermined operation procedure may be required. In this case, for example, there is a need to implement a tool capable of manually adjusting the correction value of the lens distortion (Barrel, Pincusion). For example, the input image may be cropped based on a field of view (FOV) Center) can be manually adjusted to perform image conversion playback. That is, the input image can be converted according to the intention of the photographer to perform the reproduction.

For example, FIG. 2 is a view showing a screen in which an input image is cropped based on FOV according to the intention of a photographer. Referring to FIG. 2, a high-speed image data processing implementation method based on memory management is supposed to be supported so that the captured image is processed in the GPU for real-time stitching reproduction. So that the real-time moving image is generated in real time. More specifically, through this, each of the multi-cameras can acquire an input image as shown in FIG. 2 (a). At this time, the stitching reproduction can be performed in real time using a plurality of images obtained from the multi-camera. In this case, for example, a high-speed image data processing implementation scheme based on memory management may be supported so that a plurality of images are processed in the GPU as described above. In addition, for example, a zone of a corresponding image is set up, and up-sampling is performed by a required resolution to realize real-time moving image generation in real time, and output as shown in FIG. 2 (b).

FIG. 3 is a diagram illustrating a screen in which a user manually designates a region of interest (ROI) region. 4 is a screen showing a mask generated by a user manually designating an ROI area. 3 and 4, the meaning of the overlap region of two images may mean that the two images include the same space or the same object. Basically, we can derive a SEAM that can join two images in this overlapping region. In this case, the SEAM that connects the two images by connecting the pixels located at the center of the overlapped region of the two images is derived as SEAM based on the angle of view distance center. However, when only the camera posture coordinate value is used, only the SEMA of the angle of view is insufficient depending on the shooting environment such as the distance of the subject, so that the user can manually create the mask by designating the ROI area. That is, it is possible to designate a specific region in the image as a region of interest by the user, and to assign a weight of the angle of view center distance. At this time, the meaning of the mask may be such that an ROI mask is generated to nonlinearly define the overlapping area so as to add or subtract a weight. At this time, the ROI area can be specified based on the user input. That is, the specific area is set to ROI, and a mask for this area can be set. That is, the user can perform image processing on an area of interest to provide output information on a region of interest, and the present invention is not limited to the above-described embodiments.

In the following, a real-time camera posture and a real-time stitching image processing method based on color correction processing will be described on the basis of the above description.

Real-time stitching image processing based on real-time camera posture and color correction processing

For example, in the above-described real-time monitoring reproduction (projection) system environment, it is possible to manually adjust the 3- to 6-axis position for each individual image input for each model. In addition, GPU image processing can be performed based on the adjusted geometry information and real time reproduction can be realized. At this time, for example, in the above-described real-time monitoring (projection) type environment posture adjustment, between the related projection methods (between 2D and 3D), the sharing and matching function of the attitude information can be supported through the geometric transformation.

At this time, in the alpha blending state of the image overlapping section processed in the real-time monitoring reproduction (projection method), the high-speed GPU image processing is implemented so that the Seam processing and the blending processing of the image overlapping section are controlled according to the blending processing algorithm .

In addition, for real-time stitching playback, a high-speed image data processing method based on memory management should be supported so that the captured image is processed by the GPU. In accordance with the user's request, the corresponding image area is set up, Real-time video creation can be realized. At this time, it can be implemented so that the input image source can be converted in real time for the R / G / B color adjustment and the white balance adjustment individually / collectively and reproduced. In addition, the adjusted geometry, color information, and the like must be implemented in a template-based storage function that stores the state information including the reproduction method, and can be implemented so as to be reproduced as desired when the stored template is loaded. That is, a desired image can be output through the above-described correction operation for a plurality of images, and the present invention is not limited to the above-described embodiments. In addition, a GUI can be implemented based on any workflow.

FIG. 5 is a diagram illustrating a method for real-time monitoring of a moving image using a multi-camera using a panorama based on the present invention.

5, an input image can be received from a multi-camera (S510). At this time, the multi-camera can capture each image with the multi-directional camera and acquire an image based on the image. Next, the multi-camera input image can be captured and reproduced in real time on the basis of synchronization (S520). As described above with reference to FIGS. 1 to 4, in order to perform synchronization on the input image, May be included in each image, and the above-described format may be used. In addition, for example, when performing real-time capturing and playback, a plurality of image sources having image format quality supported by the capture card can be simultaneously captured and reproduced in real time. That is, capture and playback can be performed in real time based on the same image format quality for a plurality of images.

In addition, for example, when performing real-time capturing and playback based on input image synchronization of a multi-camera, alpha blending processing can be performed in overlapping portions between images according to the playback position, and concentration can be controlled based thereon. In this case, for example, overlapping portions between images according to the playback position may be performed through synchronization based on the time code described above, as described above.

Next, the real-time stitching image processing based on the real-time camera position and color correction processing can be performed (S530). Thereafter, the image can be output (S540) As described above, the real-time stitching image processing based on the real-time camera posture and the color correction processing is performed by sharing and matching attitude information through the geometric transformation between the related projection methods (2D-3D) . Also, real-time stitching image processing based on the real-time camera posture and color correction processing is performed by adjusting the seam processing and blending processing of the image overlapping section according to a predetermined blending processing algorithm in the alpha blending state of the image overlapping section processed in real- Gt; GPU < / RTI > In addition, the real-time stitching image processing based on the real-time camera posture and the color correction processing can be performed by creating a real-time moving image in real time by setting up a zone of the image according to a request of the user and up- It is not limited to an embodiment.

That is, in order to simultaneously capture a wide angle of view on the basis of the above-described multi-camera, a plurality of cameras are structured to photograph the input image in real time in a manner of taking an image by dividing the angle of view, thereby outputting an image. But is not limited to an example.

Although the exemplary methods of the present invention are represented by a series of acts for clarity of explanation, they are not intended to limit the order in which the steps are performed, and if necessary, each step may be performed simultaneously or in a different order. In order to implement the method according to the present invention, the illustrative steps may additionally include other steps, include the remaining steps except for some steps, or may include additional steps other than some steps.

The various embodiments of the present invention are not intended to be all-inclusive and are intended to illustrate representative aspects of the present invention, and the elements described in the various embodiments may be applied independently or in a combination of two or more.

Furthermore, various embodiments of the invention may be implemented by means of hardware, firmware, software, or a combination thereof. In the case of hardware implementation, one or more application specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays A general processor, a controller, a microcontroller, a microprocessor, and the like.

The scope of the present invention includes software or machine-executable instructions (e.g., operating system, applications, firmware, programs, etc.) that cause an operation in accordance with the methods of the various embodiments to be performed on a device or computer, Instructions, and the like are stored and are non-transitory computer-readable medium executable on the device or computer.

-

Claims (1)

A video real-time monitoring method using a multi-camera,
Receiving an input image from the multi-camera;
Capturing and reproducing the received multi-camera input image in real time based on synchronization, the ROI (Region Of Interest) region is designated in the received multi-camera input image, a mask for the designated ROI region is generated, Capture and playback are performed;
Performing real-time stitching image processing based on a real-time camera posture and a color correction process on the multi-camera input image to be reproduced; And
And outputting the multi-camera input image on which the image processing has been performed.

Images