KR102222290B1 - Method for gaining 3D model video sequence - Google Patents

Abstract

An embodiment of the present invention provides a multi-view image acquisition process of obtaining a multi-view image toward the center of a target object; A calibration process of acquiring camera parameters and adjusting the acquired camera parameters; A pre-processing process in which image correction and illumination compensation are performed on the corrected multi-view image; It may include a 3D mesh data acquisition process of obtaining rendering of 3D mesh data according to a viewer's viewing position by using the preprocessed multi-view image.

Description

Real-life-based omnidirectional 3D model video sequence acquisition method for dynamic 3D reality data driving in mixed reality environment {Method for gaining 3D model video sequence}

The present invention relates to a method for obtaining a 3D model video sequence, and to a method for obtaining a 3D model video sequence based on a real-world image for driving dynamic 3D reality data in a mixed reality environment.

As shown in FIG. 1, the conventional real-life-based holographic video content acquisition method is a method of extracting a 2D frame after removing a background by photographing a green screen image of a two-dimensional single view (one direction).

Therefore, the conventional 3D model acquisition method based on real-life photographs or scans a stationary object from various angles, so that it is difficult to acquire video content, and therefore, virtual and mixed reality that requires a 360° multi-view experience. It's an inefficient way in the environment.

In general, a 3D scanning service or technology for a stationary object accounts for the majority, and there is no technology (product) that has developed all-round 3D model data for a moving object.

Therefore, as a mixed reality-related technology that will be used in various industries for the forthcoming 4th industrial service (security, energy, display, medical, education, etc.), the development of a sequence acquisition technology that essentially generates real data as 360° 3D data. The need is desperate.

Korean Patent Registration 10-0717065

An object of the present invention is to provide a real-life based omnidirectional 3D model video sequence acquisition means for driving dynamic 3D reality data in a mixed reality environment.

An embodiment of the present invention provides a multi-view image acquisition process of obtaining a multi-view image toward the center of a target object; A calibration process of acquiring camera parameters and adjusting the acquired camera parameters; A pre-processing process in which image correction and illumination compensation are performed on the corrected multi-view image; It may include a 3D mesh data acquisition process of obtaining rendering of 3D mesh data according to a viewer's viewing position by using the preprocessed multi-view image.

The multi-view image acquisition process may be characterized in that a multi-view image is obtained by extracting a feature point from a single-view image captured by each camera and matching the distortion of the feature point of each single-view image.

In the calibration process, parameters of a camera in which each single-view image is captured, and the acquired camera parameters are adjusted to correct the single-view image.

In the pre-processing process, the image correction is performed by calculating a homography by applying an epipolar geometry in which two vectors between the camera position and the image point viewing it are located on a common plane, thereby calculating the viewpoint image of each single viewpoint. It can be calibrated to match.

In the pre-processing process, the illumination compensation detects the depth of the target object using a depth sensor provided in each camera, and uses a depth map for a single view image captured by each camera to determine the brightness of each single view. Can be corrected to a constant value.

The process of obtaining 3D mesh data may include obtaining a 3D mesh model sequence using a corrected multi-view image; It may include a process of obtaining 3D mesh data using a 3D mesh model sequence.

According to an embodiment of the present invention, it is possible to obtain a 360-degree live sequence for a moving object using a multi-view camera system, and to obtain data in the form of a 3D model using an omni-directional video sequence.

1 is an exemplary diagram of a holographic video content acquisition method using a conventional two-dimensional single view green screen image.
2 is a block diagram of a system for acquiring a real-life 3D model video sequence based on a real-world image for driving dynamic 3D reality data in a mixed reality environment according to an embodiment of the present invention.
3 is a flowchart showing a method of obtaining a real-life 3D model video sequence based on a real-world image for driving dynamic 3D reality data in a mixed reality environment according to an embodiment of the present invention.
4 is an example of a result of matching feature points between viewpoints according to an embodiment of the present invention.
5 is an exemplary illustration of a correction plate for camera correction according to an embodiment of the present invention.
6 is an exemplary diagram of the calculation of epipolar geometry at two different points in time according to an embodiment of the present invention.
7 is a diagram showing before and after illumination compensation according to an embodiment of the present invention.
8 is an example of a process of modifying 3D mesh data according to an embodiment of the present invention.
9 is an example of rendering a 3D mesh according to a viewer's position according to an embodiment of the present invention.
10 is an example of conversion of a camera parameter and a coordinate system according to an embodiment of the present invention.
11 is an example of extracting and matching feature points of a multi-view image according to an embodiment of the present invention.
12 is an example of obtaining a 3D Point Cloud using matching points and camera parameter information according to an embodiment of the present invention.
13 is an example of obtaining precise 3D point cloud data using a rough 3D point cloud and a multi-view image according to an embodiment of the present invention.

Hereinafter, advantages and features of the present invention, and a method of achieving them will become apparent with reference to the embodiments described below in detail together with the accompanying drawings. However, the present invention is not limited to the embodiments disclosed below, but will be implemented in a variety of different forms, and is provided to completely inform the scope of the invention to those of ordinary skill in the art to which the present invention pertains. As such, the invention is only defined by the scope of the claims. In addition, in describing the present invention, when it is determined that related known technologies or the like may obscure the subject matter of the present invention, a detailed description thereof will be omitted.

2 is a block diagram of a system for acquiring a real-life 3D model video sequence based on a real-world image for driving dynamic 3D reality data in a mixed reality environment according to an embodiment of the present invention.

The present invention enables acquisition of a 360° live sequence for a moving object using a multi-view camera system, and data acquisition in the form of a 3D model using an omnidirectional video sequence. Therefore, assuming that the existing 3D scan system creates a mannequin in a stationary state, the 3D live sequential system of the present invention can capture the change in motion over time as it is.

To this end, the present invention includes a plurality of cameras for acquiring a multi-view image toward the center of a target object, as shown in FIG. 1. Using multi-view images acquired through a plurality of cameras, the image processing device generates a real-life 3D model video sequence based on real-life for dynamic 3D reality data driving in a mixed reality environment.

The image processing device is a device capable of arithmetic processing by having a CPU and a memory such as a computer or a server, and the image processing device of the present invention acquires a plurality of camera parameters, adjusts and corrects the acquired camera parameters, and then calibrates them. Pre-processing in which image correction and lighting compensation are performed on the multi-view image is performed, and rendering of 3D mesh data according to the viewing position of the viewer is obtained by using the pre-processed multi-view image. It will be described in detail with reference to FIGS. 3 to 13 below.

FIG. 3 is a flowchart showing a method of obtaining a real-world-based omnidirectional 3D model video sequence for driving dynamic 3D reality data in a mixed reality environment according to an embodiment of the present invention, and FIG. 4 is a view point according to an embodiment of the present invention. It is an example of the result of matching feature points between, and FIG. 5 is an example of a correction plate for camera calibration according to an embodiment of the present invention, and FIG. 6 is an example of epipolar geometry calculation of two different viewpoints according to an embodiment of the present invention, 7 is a diagram showing before and after illumination compensation according to an embodiment of the present invention, FIG. 8 is an example of a correction process of 3D mesh data according to an embodiment of the present invention, and FIG. 9 is a viewer according to an embodiment of the present invention. It is an example of rendering a 3D mesh according to the position of, FIG. 10 is an example of conversion of a camera parameter and a coordinate system according to an embodiment of the present invention, and FIG. 11 is an example of extracting and matching feature points of a multi-view image according to an embodiment of the present invention. 12 is an example of obtaining a 3D point cloud using matching point and camera parameter information according to an embodiment of the present invention, and FIG. 13 is a precise 3D point cloud using a rough 3D point cloud and a multi-view image according to an embodiment of the present invention. This is an example of acquiring Point Cloud data.

The method of obtaining a 3D model video sequence based on realism of the present invention includes a multi-view image acquisition process of acquiring a multi-view image toward the center of a target object as shown in FIG. 3, and acquisition of a multi-view image parameter. A calibration process for adjusting one parameter, a pre-processing process in which image correction and lighting compensation are performed for the corrected multi-view image, and 3D mesh data rendering according to the viewing position of the viewer are obtained using the pre-processed multi-view image. It may include a process of obtaining 3D mesh data.

The multi-view image acquisition process is a process in which a plurality of cameras acquire a multi-view image toward the center of a target object.

In consideration of obtaining a 3D mesh, the target object photographed by the camera has the following characteristics.

-Characteristics of target objects to be avoided

ㆍObjects with a transparent surface (soap drops, glass, etc.)

ㆍObjects that emit/reflect strong light (mirror, light bulb, etc.)

ㆍA monotonous object with almost no surface change

-Degree of distortion (overlap) between cameras

ㆍIncrease of matching feature points between time points

ㆍHigh-speed parallelism and maximized feature point through GPGPU programming of feature point extraction technique

Accordingly, in the process of obtaining a multi-view image, a multi-view image can be obtained by extracting a feature point from a single-view image captured by each camera and matching the distortion of the feature point of each single-view image.

For example, when matching the misalignment of the feature points as shown in FIG. 4(a), two images are matched, but if most of the feature points are not matched as shown in FIG. 4(b), a multi-view image is obtained. It becomes impossible to do it.

The calibration process is a process of acquiring camera parameters and adjusting the acquired camera parameters. Since the parameters of each camera may be different, the purpose is to match the shooting properties of the single-view image by matching them.

Preferably, in the calibration process, parameters of a camera that has captured each single-view image are acquired, and the acquired camera parameters are adjusted to correct the single-view image. For example, in-camera parameters including focal length, scale factor, image center, and lens distortion parameter of each camera, and rotation of each camera ( By acquiring the external parameters of the camera including rotation) and translation, the internal parameters and external parameters of each camera are corrected so that they have the same value.

For reference, camera calibration may be performed through a method using a dot/line attribute or a method using a specific pattern (correction plate) as shown in FIG. 5.

The pre-processing is a process in which image correction and illumination compensation are performed on the corrected multi-view image.

The pre-processing process may consist of image correction and illumination compensation.

First, in the image correction in the preprocessing process, the image correction is performed to improve the error due to the slight distortion of the camera, the camera rectification and calibration are performed for each camera pair, and the three-dimensional correction using the three-dimensional correction object can be performed. In addition, an integrated correction of the depth and RGB images may be performed using a depth sensor.

Furthermore, for image correction, homography is calculated by applying an epipolar geometry in which two vectors between the position of the camera and the image point viewing it are located on a common plane to match the viewpoint image of each single viewpoint. Correction to be made can be made.

Epipolar Geometry is the theoretical background of stereo vision, and the image information obtained from two cameras at a point in three dimensions is that the two vectors of the position of the camera and the image point viewing it exist on a common plane. As a theory, this isopolar geometry is applied to calculate the epipolar geometry of two different viewpoints as shown in FIG. 6 to perform correction to match the viewpoint images of each single viewpoint.

Second, the illumination compensation in the preprocessing process detects the depth of the target object using a depth sensor provided in each camera, and uses a depth map for a single view image captured by each camera. It is to correct the brightness of the viewpoint to a certain value. That is, the number of cameras is reduced and improved by using a depth map based on a depth camera so as to maintain a constant brightness due to lighting in a multi-view image. For reference, FIG. 7 is a diagram illustrating before and after illumination compensation.

The process of obtaining 3D mesh data is a process of obtaining rendering of 3D mesh data according to a viewer's viewing position using a preprocessed multi-view image. That is, as shown in FIG. 8, correction and frame connection are performed for high-quality 3D mesh data, and rendering of 3D mesh data according to a viewer's position is obtained as shown in FIG. 9. For reference, 3D mesh data is 3D shape data required to acquire a 3D model, and 3D shape data may be stored in a standard triangulated language (STL) data format.

The process of obtaining 3D mesh data may include a process of obtaining a 3D mesh model sequence using a corrected multi-view image and a process of obtaining 3D mesh data using a 3D mesh model sequence. Various known methods may be applied to the process of acquiring the sequence and the process of acquiring 3D mesh data in the above.

For example, in the process of obtaining a 3D mesh model sequence, a 3D mesh model sequence is obtained by generating a mid-view video sequence between cameras for obtaining high quality 3D mesh data.

-Acquisition of depth information to create an intermediate point of view

ㆍAcquisition of depth information by applying stereo matching to the video sequence of two viewpoints

ㆍAcquisition of depth information using ToF (Time-of-Flight) type depth camera

ㆍAcquisition of depth information using depth information of Microsoft's Kinect2

-Acquisition of mid-view video sequence using pinhole camera model and 3D warping

ㆍCoordinate system transformation using camera parameters (K, R|t) acquired through calibration (Fig. 10)

ㆍGenerating a sequence image at the mid-point through coordinate system conversion

In addition, an example of acquiring 3D mesh data using a 3D mesh model sequence will be described in detail below.

-Acquisition and matching of feature points of multi-view images (Fig. 11)

ㆍImplementation of SIFT algorithm for feature extraction

ㆍImplementation of matching algorithm (RANSAC) using extracted feature points of multi-view image

-Acquisition of 3D Mesh data using matching points (Fig. 12)

ㆍCalculate SfM (Structure from Motion) using matching points and obtain approximate point cloud data Calculate SfM (Structure from Motion) using matching points and acquire approximate point cloud data

-Acquisition of precise point cloud data using approximate point cloud data and multi-view images (Fig. 13)

ㆍPMVS(Patch-based Multi-View Stereoscopic)

The embodiments in the description of the present invention described above are presented by selecting the most preferable examples to aid the understanding of those skilled in the art from among various possible examples, and the technical idea of the present invention is not necessarily limited or limited only by this embodiment. , Various changes and modifications and other equivalent embodiments are possible within the scope of the technical spirit of the present invention.

Claims (6)

A multi-view image acquisition process of obtaining a multi-view image toward the center of the target object;
A calibration process of acquiring camera parameters and adjusting the acquired camera parameters;
A pre-processing process in which image correction and illumination compensation are performed on the corrected multi-view image;
Including; 3D mesh data acquisition process of obtaining rendering of 3D mesh data according to the viewing position of the viewer by using the preprocessed multi-view image,
In the multi-view image acquisition process, feature points are extracted from single-view images captured by each camera, and a multi-view image is obtained by matching distortions of the feature points of each single-view image,
In the calibration process, a focal length, a scale factor of each camera, and a scale factor of each camera are obtained so as to obtain the parameters of a camera that has captured each single-view image and adjust the acquired camera parameters to correct the single-view image. By acquiring camera internal parameters including image center and lens distortion parameter, and camera external parameters including rotation and translation of each camera, Calibrate so that the parameter and the external parameter have the same value,
In the pre-processing process, the illumination compensation detects the depth of the target object using a depth sensor provided in each camera, and uses a depth map for a single view image captured by each camera to determine the brightness of each single view Is corrected to a constant value,
The process of obtaining 3D mesh data may include obtaining a 3D mesh model sequence using a corrected multi-view image; A process of acquiring 3D mesh data using a 3D mesh model sequence; a photorealistic-based omnidirectional 3D model video sequence acquisition method for driving dynamic 3D reality data in a mixed reality environment including.
delete delete The method according to claim 1, wherein the image correction in the pre-processing process,
The two vectors between the position of the camera and the image point viewing it are computed by applying an epipolar geometry to be located on a common plane, and corrected to match the viewpoint image of each single viewpoint. Real-life-based omnidirectional 3D model video sequence acquisition method for driving dynamic 3D reality data in a mixed reality environment.
delete delete

Images