KR102146839B1 - System and method for building real-time virtual reality - Google Patents

Abstract

According to an exemplary embodiment of the present invention, a method of modeling a three-dimensional space includes the steps of: acquiring a plurality of fisheye images photographed through a plurality of fisheye lenses; extracting a plurality of feature points from the plurality of fisheye images; estimating a plurality of three-dimensional coordinates of a plurality of feature points based on a curvature function of the plurality of fisheye lenses and a plurality of fisheye images; and generating output data defining a three-dimensional space based on the plurality of three-dimensional coordinates, thereby enabling an accurate modeling of the three-dimensional space with a reduced amount of computation.

Description

System and method for building real-time virtual reality {SYSTEM AND METHOD FOR BUILDING REAL-TIME VIRTUAL REALITY}

The technical idea of the present invention relates to modeling of a three-dimensional space, and in detail, to an apparatus and method for constructing a real-time virtual reality through modeling of a three-dimensional space.

Virtual reality (VR) may refer to an environment or situation similar to the real world created by a computer or the like. Virtual reality can be used in various fields, and in particular, virtual reality that simulates a real environment can be used in various applications that provide an experience of a real environment to remote users. Virtual reality that simulates such a real environment may be constructed through modeling of a 3D space that converts the real environment into data, and may depend on the performance and efficiency of modeling of the 3D space.

The technical idea of the present invention provides a system and method capable of accurately modeling a three-dimensional space with a reduced amount of computation in order to construct a real-time virtual reality.

In order to achieve the above object, a method of modeling a three-dimensional space according to one aspect of the technical idea of the present invention includes obtaining a plurality of fisheye images photographed through a plurality of fisheye lenses, a plurality of fisheye images Extracting a plurality of feature points from, estimating a plurality of 3D coordinates of a plurality of feature points based on a curvature function of a plurality of fisheye lenses and a plurality of fisheye images, and 3 based on the plurality of 3D coordinates It may include generating output data that defines the dimensional space.

According to an exemplary embodiment of the present invention, the extracting of a plurality of feature points includes converting a plurality of fisheye images into a plurality of panoramic images corresponding to an image photographed with a standard lens, and a plurality of panoramic images. It may include the step of extracting a plurality of feature points from.

According to an exemplary embodiment of the present invention, the converting into a plurality of panoramic images includes a zenith angle (θ) and an azimuth angle (φ) for a feature point included in a fisheye image in a three-dimensional polar coordinate system using a center point of the fisheye lens as an origin. ), and estimating coordinates of the feature points in the panoramic image based on the zenith angle (θ) and the azimuth angle (φ).

According to an exemplary embodiment of the present invention, in the step of extracting a plurality of feature points from a plurality of panoramic images, a plurality of feature points may be extracted based on an AKAZE algorithm.

According to an exemplary embodiment of the present invention, the step of extracting a plurality of feature points,

Extracting first feature points having a feature value exceeding a predefined threshold value from a plurality of fisheye images, a mesh including first feature points and edges connecting the first feature points in each of the plurality of fisheye images Generating, dividing each of the plurality of fisheye images into at least one region based on the plurality of meshes, and extracting at least one second feature point within the regions corresponding to each other of the plurality of fisheye images It may include steps.

According to an exemplary embodiment of the present invention, the step of generating the mesh includes aligning the first feature points based on the feature value, and representing the aligned first feature points as a k-dimensional tree (kd tree). It may include, and dividing each of the plurality of fisheye images into at least one region may include comparing a plurality of k-dimensional trees corresponding to the plurality of fisheye images.

According to an exemplary embodiment of the present invention, the step of estimating a plurality of 3D coordinates includes a zenith angle (θ) and an azimuth angle (φ) for a feature point included in a fisheye image in a 3D polar coordinate system using a center point of the fisheye lens as an origin. ) Calculating, based on the zenith angle (θ) and azimuth angle (φ), defining a vector passing through the center point and the feature point of the fisheye lens, a plurality of vectors corresponding to the plurality of fisheye lenses It may include converting into a 3D polar coordinate system using a center point of as an origin, and estimating 3D coordinates of the feature point based on the converted vectors.

According to an exemplary embodiment of the present invention, the step of estimating the three-dimensional coordinates of a feature point includes calculating a point separated by the shortest distance among points separated by the same distance from a plurality of vectors as the three-dimensional coordinates of the feature point. It may include.

According to an exemplary embodiment of the present invention, the generating of the output data includes applying texture mapping to a plane defined by a plurality of 3D coordinates based on at least one panoramic image generated from a plurality of fisheye images. It may include the step of.

In order to achieve the same object, a system for modeling a three-dimensional space according to one aspect of the technical idea of the present invention includes a first fisheye lens and a second fisheye lens, and includes a first fisheye lens and a second fisheye lens. Generates output data defining a three-dimensional space based on a first camera module and a second camera module for generating a first fisheye image and a second fisheye image captured through, and the first fisheye image and the second fisheye image It may include an image processing module, wherein the image processing module includes a feature point extracting unit for extracting a plurality of feature points from the first fisheye image and the second fisheye image, a curvature function of the first fisheye lens and the second fisheye lens, and the first fisheye A coordinate estimating unit that estimates a plurality of 3D coordinates of a plurality of feature points based on the image and the second fisheye image, and an output data generation unit that generates output data based on the plurality of 3D coordinates.

According to an exemplary embodiment of the present invention, the image processing module may further include a curvature providing unit for estimating and storing a curvature function based on the first fisheye image and the second fisheye image in the correction mode.

The system and method according to an exemplary embodiment of the present invention can enable real-time virtual reality construction by providing 3D spatial modeling with a reduced amount of computation.

In addition, the system and method according to an exemplary embodiment of the present invention enable real-time virtual reality so that a remote user can experience a real environment in real time, and solve spatial constraints on multi-party meetings or visiting tourist attractions. I can.

In addition, the system and method according to an exemplary embodiment of the present invention can significantly alleviate cost and space requirements for modeling a 3D space by reducing the number of camera modules required for modeling a 3D space.

The effects obtained in the exemplary embodiments of the present invention are not limited to the above-mentioned effects, and other effects that are not mentioned are common knowledge in the technical field to which the exemplary embodiments of the present invention pertain from the following description. It can be clearly derived and understood by those who have. That is, unintended effects of implementing the exemplary embodiments of the present invention may also be derived from the exemplary embodiments of the present invention by a person having ordinary skill in the art.

1 is a block diagram schematically showing a system according to an exemplary embodiment of the present invention.
Fig. 2 is a flow chart showing a method of modeling a 3D space according to an exemplary embodiment of the present invention.
3 is a flow chart showing an example of step S40 of FIG. 2 according to an exemplary embodiment of the present invention.
4A and 4B are diagrams illustrating examples of a fisheye image and a panoramic image according to exemplary embodiments of the present invention.
5A to 5C are diagrams for explaining an operation of capturing a feature point using a plurality of fisheye lenses according to an exemplary embodiment of the present invention.
6 is a diagram illustrating an operation of generating a panorama image from a fisheye image according to an exemplary embodiment of the present invention.
7 is a flowchart illustrating an example of step S40 of FIG. 2 according to an exemplary embodiment of the present invention.
8 is a diagram illustrating an example of an operation of generating a mesh from a plurality of fisheye images according to an exemplary embodiment of the present invention.
9 is a diagram illustrating an operation of estimating 3D coordinates of a feature point according to an exemplary embodiment of the present invention.

Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the accompanying drawings. The embodiments of the present invention are provided to more completely describe the present invention to those with average knowledge in the art. In the present invention, various modifications may be made and various forms may be applied, and specific embodiments will be illustrated in the drawings and described in detail. However, this is not intended to limit the present invention to a specific form disclosed, it should be understood to include all changes, equivalents, and substitutes included in the spirit and scope of the present invention. In describing each drawing, similar reference numerals are used for similar elements. In the accompanying drawings, the dimensions of the structures are shown to be enlarged or reduced than in actuality for clarity of the present invention.

The terms used in the present application are only used to describe specific embodiments, and are not intended to limit the present invention. Singular expressions include plural expressions unless the context clearly indicates otherwise. In the present application, terms such as "comprise" or "have" are intended to designate the presence of features, numbers, steps, actions, components, parts, or combinations thereof described in the specification, but one or more other features. It is to be understood that the possibility of addition or presence of elements or numbers, steps, actions, components, parts, or combinations thereof is not preliminarily excluded.

Unless otherwise defined, all terms, including technical or scientific terms, used herein have the same meaning as commonly understood by one of ordinary skill in the art to which the present invention belongs. Terms as defined in a commonly used dictionary should be interpreted as having a meaning consistent with the meaning in the context of the related technology, and should not be interpreted as an ideal or excessively formal meaning unless explicitly defined in this application. .

In the following drawings and description, a component indicated or described as a single block may be a hardware block or a software block. For example, each of the components may be an independent hardware block that transmits and receives signals from each other, or may be a software block executed in one processor. In addition, a component represented or described as one block may be a computing system including a processor and software.

1 is a block diagram schematically illustrating a system 100 according to an exemplary embodiment of the present invention. In this specification, the system 100 may be referred to as a system for modeling a three-dimensional space, a system for real-time virtual reality construction, and the like. As shown in FIG. 1, the system 100 may include a plurality of camera modules CM1, CM2,..., CMn and an image processing module 10, and output data defining a three-dimensional space. (D_OUT) can be generated (n is an integer greater than 1). In some embodiments, different from that shown in FIG. 1, the system 100 may include only the image processing module 10 and communicates with a plurality of external camera modules via wired or wireless communication channels. You may.

The plurality of camera modules CM1, CM2,..., CMn may provide a plurality of fisheye images F_IMG1, F_IMG2,..., F_IMGn to the image processing module 10. For example, the first camera module CM1 may include a first fisheye lens F1, and may include an image sensor that detects light passing through the fisheye lens. A fisheye lens is a lens manufactured to maintain uniform brightness and sharpness over the entire field of view of 180 degrees or more by generating barrel distortion. As illustrated in FIG. 6A, a fisheye lens A subject corresponding to the center of the lens may have a relatively large image, while a subject corresponding to the peripheral part of the fisheye lens may have a relatively small image. In other words, the fisheye lens can provide a wide angle of view while providing a distorted image. The image sensor may convert the detected light into an electrical signal, and the first camera module CM1 may generate a first fisheye image (or first fisheye image data) F_IMG1 based on the output signal of the image sensor. have.

The image processing module 10 may receive a plurality of fisheye images F_IMG1, F_IMG2,..., F_IMGn from a plurality of camera modules CM1, CM2,..., CMn, and a plurality of fisheye images Output data D_OUT defining a 3D space may be generated based on the fields F_IMG1, F_IMG2,..., F_IMGn. Compared with a camera module that includes a standard lens, that is, a lens that captures an image similar to a person's vision, a three-dimensional space can be photographed using two or more fisheye lenses, and due to the wide angle of view of the fisheye lens The number of camera modules required for modeling may be reduced. Also, the amount of image data to be processed by the image processing module 10 may be reduced. As a result, the system 100 can significantly reduce the space and time required for modeling a 3D space. Hereinafter, exemplary embodiments of the present invention will be described with reference mainly to a system including two camera modules, that is, a first camera module CM1 and a second camera module CM2, but the technical idea of the present invention It will be appreciated that a system including three or more camera modules is also possible, depending on. The image processing module 10 may include a feature point extraction unit 12, a coordinate estimation unit 14, an output data generation unit 16, and a curvature providing unit 18, as illustrated in FIG. 1.

The feature point extracting unit 12 may extract a plurality of feature points from a plurality of fisheye images F_IMG1, F_IMG2,..., F_IMGn. A feature point may refer to a point used to identify an object in an image, and may be extracted from the image in various ways. In some embodiments, the feature point may have a feature value (eg, brightness, color, etc.) that is distinguished from the surroundings. The feature point extracting unit 12 may extract feature points common to a plurality of fisheye images F_IMG1, F_IMG2,..., F_IMGn, and provide the extracted feature points to the coordinate estimation unit 14. Examples of the operation of the feature point extraction unit 12 will be described later with reference to FIGS. 3 to 8.

The coordinate estimating unit 14 may estimate a plurality of 3D coordinates of a plurality of feature points provided from the feature point extracting unit 12. For example, the coordinate estimation unit 14 may receive a curvature function of a plurality of fisheye lenses F1, F2,..., Fn from the curvature providing unit 18, and a curvature function and a plurality of fisheye images A plurality of 3D coordinates of a plurality of feature points may be estimated based on the F_IMG1, F_IMG2,..., F_IMGn. An example of the operation of the coordinate estimation unit 14 will be described later with reference to FIG. 9.

The output data generator 16 may generate output data D_OUT defining a 3D space based on a plurality of 3D coordinates provided from the coordinate estimating unit 14. In some embodiments, the output data generation unit 16 may define a fragment based on a plurality of 3D coordinates, and at least one of a plurality of fisheye images F_IMG1, F_IMG2,..., F_IMGn Texture mapping may be performed on a fragment defined based on it. Accordingly, the output data D_OUT may include not only geometric information of the 3D space but also information about the surface of the 3D space (eg, color, texture, etc.).

The curvature providing unit 18 is a curvature function corresponding to the curvature of the plurality of fisheye lenses F1, F2,..., Fn included in the plurality of camera modules CM1, CM2,..., CMn, respectively. Can be provided to the coordinate estimation unit 14. The curvature of the fisheye lens can define the degree of distortion of the fisheye image generated by being photographed through the fisheye lens. Accordingly, as will be described later with reference to FIG. 6, the fisheye lens in a three-dimensional polar coordinate system with the center of the fisheye lens as the origin. It is possible to determine the zenith angle (θ) for a point in the image. In the present specification, the curvature function of the fisheye lens may define a zenith angle θ corresponding to a distance separated from the center of the fisheye image. In some embodiments, a plurality of fisheye lenses F1, F2,..., Fn included in each of the plurality of camera modules CM1, CM2,..., CMn may be the same, and accordingly The curvature providing unit 18 may provide a single curvature function. Further, as will be described later with reference to FIG. 3 and the like, in some embodiments, the curvature providing unit 18 may provide a curvature function to the feature point extracting unit 12.

In some embodiments, the curvature providing unit 18 may pre-store the curvature function of the fisheye lens. For example, a curvature function may be provided in advance from a manufacturer of the first fisheye lens F1, or a curvature function may be obtained in advance by testing the first fisheye lens F1. The curvature function obtained as described above may be injected into the image processing module 10, the curvature providing unit 18 may store the injected curvature function, and the stored curvature function may be provided to the coordinate estimation unit 14, etc. .

In some embodiments, the image processing module 10 may support a correction mode, and the curvature providing unit 18 may generate a curvature function in the correction mode. For example, in the calibration mode, the first camera module CM1 displays a bowl drawn on the inner surface of concentric circles representing a constant zenith angle θ in a three-dimensional polar coordinate system with the center of the first fisheye lens F1 as the origin. You can shoot. The curvature providing unit 18 may estimate the zenith angle θ based on the first fisheye image F_IMG1 generated by photographing the inner surface of the bowl, and the first fisheye lens F1 based on the estimated zenith angle θ. ) Can be created. When the plurality of fisheye lenses F1, F2,..., Fn have different structures, the curvature providing unit 18 is a plurality of fisheye lenses F1, F2,..., Fn. While it is possible to generate curvature functions of, when a plurality of fisheye lenses (F1, F2,..., Fn) have the same structure, a fisheye image (e.g., a fisheye image) captured using one camera module (e.g., CM1) , F_IMG1) can also be used to create a single curvature function.

Fig. 2 is a flow chart showing a method of modeling a 3D space according to an exemplary embodiment of the present invention. In the present specification, the method of FIG. 2 may be referred to as a method for building a real-time virtual reality. As shown in FIG. 2, the method of FIG. 2 may include a plurality of steps S20, S40, S60, and S80. In some embodiments, the method of FIG. 2 may be performed by the system 100 (or image processing module 10) of FIG. 1, and FIG. 2 will be described below with reference to FIG. 1.

Referring to FIG. 2, in step S20, an operation of acquiring a plurality of fisheye images may be performed. For example, the image processing module 10 includes a plurality of camera modules CM1, CM2,..., CMn each including a plurality of fisheye lenses F1, F2,..., Fn. Fisheye images F_IMG1, F_IMG2,..., F_IMGn may be received.

In step S40, an operation of extracting a plurality of feature points may be performed. For example, the feature point extracting unit 12 may extract a plurality of common feature points from the plurality of fisheye images F_IMG1, F_IMG2,..., F_IMGn. Examples of step S40 will be described later with reference to FIGS. 3 and 7.

In step S60, an operation of estimating a plurality of 3D coordinates of a plurality of feature points may be performed. For example, the coordinate estimating unit 14 may calculate a plurality of 3D coordinates of a plurality of feature points extracted in step S40. An example of step S60 will be described later with reference to FIG. 9.

In step S80, an operation of generating output data may be performed. For example, the output data generation unit 16 may generate output data D_OUT defining a 3D space based on a plurality of 3D coordinates and a plurality of fisheye images F_IMG1, F_IMG2,..., F_IMGn. Can be generated.

3 is a flow chart showing an example of step S40 of FIG. 2 according to an exemplary embodiment of the present invention, and FIGS. 4A and 4B are diagrams illustrating examples of a fisheye image and a panoramic image according to exemplary embodiments of the present invention. admit. As described above with reference to FIG. 2, in step S40' of FIG. 3, an operation of extracting a plurality of feature points may be performed. As shown in FIG. 3, step S40' may include steps S42 and S44.

In some embodiments, an operation of converting a plurality of fisheye images obtained from a plurality of camera modules each including a fisheye lens into a plurality of panoramic images, and extracting a common feature point from the plurality of panoramic images is performed. I can. For example, as shown in FIG. 4A, common feature points may be extracted from two fisheye images acquired from two camera modules in an arbitrary manner (eg, AKAZE algorithm), but in FIG. 4A As indicated, feature points that do not actually correspond may be extracted as common feature points. This is because the positions of the 3D space corresponding to the center point of the fisheye lens are different according to the position of the fisheye lens, so that the 3D space is distorted in each of the fisheye images, so the general feature point extraction using the distance between the feature points, etc. This is because there is a limit to applying the technique. On the other hand, as shown in FIG. 4B, two fisheye images are converted into two panoramic images, that is, images each photographed with a standard lens, and common feature points are extracted from the panoramic images in an arbitrary manner. I can.

Referring to FIG. 3, in step S42, an operation of generating a plurality of panoramic images may be performed. The panoramic image may refer to an image photographed with a standard lens, different from a fisheye image generated by photographing with a fisheye lens. For example, the image processing module 10 (or feature point extraction unit 12) of FIG. 1 may generate a plurality of panoramic images corresponding to a plurality of fisheye images F_IMG1, F_IMG2,..., F_IMGn. I can. An example of an operation of generating a panoramic image from a fisheye image will be described later with reference to FIGS. 5A to 5C and FIG. 6.

In step S44, an operation of extracting a plurality of feature points may be performed. For example, the feature point extracting unit 12 of FIG. 1 may extract common feature points from a plurality of panoramic images in an arbitrary manner. For example, the feature point extracting unit 12 may detect edges in each of a plurality of panoramic images, and may extract common feature points based on the detected edges. As described above with reference to FIG. 1, the extracted feature points may be a target for calculating 3D coordinates, and a 3D space may be modeled by 3D coordinates of the feature points.

5A to 5C are diagrams for explaining an operation of capturing a feature point using a plurality of fisheye lenses according to an exemplary embodiment of the present invention. Specifically, FIGS. 5A to 5C illustrate examples of photographing a feature point P through the first fisheye lens F1 and the second fisheye lens F2. Although, examples of photographing feature points through two fisheye lenses in FIGS. 5A to 5C are illustrated, it will be understood that an example using three or more fisheye lenses is possible.

5A and 5B, the first fisheye lens F1 and the second fisheye lens F2 may be spaced apart from each other in the X-axis direction, and feature points P separated in the Z-axis direction may be photographed. Accordingly, as shown in FIG. 5C, the feature point P in the first fisheye image F_IMG1 photographed through the first fisheye lens F1 is r from the center O ₁ of the first fisheye image F_IMG1. _{1 While} displayed at a spaced apart position, the feature point (P) in the second fisheye image (F_IMG2) taken through the second fisheye lens (F2) is r ₂ apart from the center (O ₂ ) of the second fisheye image (F_IMG2) Can be displayed in the designated location.

As described above with reference to FIG. 1, when the curvature functions of the first fisheye lens F1 and the second fisheye lens F2 are given, a feature point in a polar coordinate system with the center of the first fisheye lens F1 as an origin ( The second zenith angle θ ₂ of the feature point P may be calculated in a polar coordinate system using the first zenith angle θ ₁ of P) and the center of the second fisheye lens F2 as an origin. For example, if r ₁ is substituted into the curvature function of the first fisheye lens F1, the first zenith angle θ ₁ can be derived, while the curvature function of the second fisheye lens F2 is substituted with r ₂ . If so, the second ceiling angle θ ₂ may be derived. In addition, the angle formed by the X axis and the line connecting the center O ₁ of the first fisheye lens F1 and the feature point P, that is, the first azimuth angle φ ₁ and the center O2 of the second fisheye lens F2 An angle formed by the X-axis and the line connecting the and feature points P, that is, the second azimuth angle φ ₂ , may be derived from the first fisheye image F_IMG1 and the second fisheye image F_IMG2, respectively. Hereinafter, as will be described later, the zenith angle and the azimuth angle derived from the fisheye image may be used to generate a panoramic image or to estimate the three-dimensional coordinates of the feature point P.

6 is a diagram illustrating an operation of generating a panorama image from a fisheye image according to an exemplary embodiment of the present invention. Specifically, FIG. 6 schematically shows a polar coordinate system with the center of the fisheye lens as the origin. The polar coordinate system of FIG. 6 may be used to generate a panoramic image from a fisheye image in order to extract a common feature point, as described above with reference to FIG. 3, and as will be described later with reference to FIG. 9, the three-dimensional feature point It can also be used to estimate coordinates.

The feature point P may be in a space spaced apart from the origin O corresponding to the center of the fisheye lens in the +Z direction, as described above with reference to FIGS. 5A to 5C. In addition, as described above with reference to FIG. 5C, the zenith angle θ and the azimuth angle φ may be derived from the fisheye image.

In order to generate a panoramic image from a fisheye image, a panoramic image may be generated by mapping at least one pixel of the fisheye image to a virtual hemisphere having a constant radius and spreading the mapped image on the hemisphere. For example, a fisheye image is formed on the XY plane, and a zenith angle (θ) and an azimuth angle (φ) may be derived from one point P _f included in the fisheye image. When there is an imaginary hemisphere convex in the +Z direction with a radius R centered on the origin (O), the point P can be determined by the zenith angle (θ) and the azimuth angle (φ), and the point P is a point on the XY plane It can have a value of P _f (eg, color). In this way, points included in the fisheye image can be mapped onto the virtual hemisphere, and the virtual hemisphere can be filled with the image through a technique such as texture mapping. Then, an operation of unfolding the image mapped on the hemisphere into a flat image, that is, a panoramic image may be performed. For example, a plurality of graphics APIs (eg, OpenGL) may provide a function of unfolding an image mapped to a hemisphere, and a panoramic image may be generated using this.

7 is a flowchart illustrating an example of step S40 of FIG. 2 according to an exemplary embodiment of the present invention. As described above with reference to FIG. 2, an operation of extracting a plurality of feature points may be performed in step S40" of FIG. 7. Compared with step S40" of FIG. 3, step S40" of FIG. 7 Common feature points may be extracted from fisheye images (eg, F_IMG1, F_IMG2), and thus generation of a panoramic image for extracting the feature points may be omitted. As shown in FIG. 7, step S40" may include a plurality of steps S41, S43, S45, and S47, and in some embodiments, step S40" of FIG. 7 is a feature point extraction unit ( 12).

In step S41, an operation of extracting first feature points may be performed. The first feature points may refer to feature points having feature values exceeding a predefined threshold in a plurality of fisheye images. That is, the first feature points may be strong feature points having a relatively high feature value, and accordingly, feature points that are common to each other although extracted from fisheye images may be extracted.

In step S43, an operation of generating a plurality of meshes may be performed. For example, as shown in FIG. 7, in each of the first fisheye image F_IMG1 and the second fisheye image F_IMG2, a mesh including first feature points and edges connecting the first feature points may be generated. have. An example of step S43 will be described later with reference to FIG. 8.

In step S45, an operation of dividing each of the plurality of fisheye images into at least one region may be performed. For example, as illustrated in FIG. 7, an operation of correlating a mesh generated from a first fisheye image F_IMG1 and a mesh generated from a second fisheye image F_IMG2 may be performed. Then, a plurality of regions may be divided into the first fisheye image F_IMG1 and the second fisheye image F_IMG2 in the same manner at the corresponding first feature points. 7 shows an example in which a first fisheye image F_IMG1 and a second fisheye image F_IMG2 are divided up, down, left and right at corresponding first feature points.

In step S47, an operation of extracting the second feature point may be performed. That is, the second feature points that are common in the regions corresponding to each other among the regions divided in step S45 may be extracted. Since common feature points are extracted from the divided regions corresponding to each other, the problem described above with reference to FIG. 4A may not occur. The extraction of the second feature point from the divided regions may be repeatedly performed, and accordingly, common feature points including the first feature point and the second feature point in the first fisheye image (F_IMG1) and the second fisheye image (F_IMG2) are Can be extracted.

8 is a diagram illustrating an example of an operation of generating a mesh from a plurality of fisheye images according to an exemplary embodiment of the present invention. Specifically, FIG. 8 shows an example of step S43 of FIG. 7. As described above with reference to FIG. 7, a mesh including first feature points extracted from a fisheye image and edges connecting the first feature points may be generated.

Referring to the left side of FIG. 8, five first feature points A, B, C, D, and E may be extracted from one fisheye image. The first feature point may be extracted in any manner, for example, based on the AKAZE method. Referring to the middle of FIG. 8, five first feature points A, B, C, D, and E may be aligned according to feature values. For example, as shown in FIG. 8, the five first feature points A, B, C, D, and E are the first feature points D having the lowest feature value from the first feature point D having the highest feature value. It can be sorted in the order of the feature point C. Referring to the right side of FIG. 8, five feature points A, B, C, D, and E may be expressed as a k-d tree based on feature values and coordinates. The k-dimensional tree is a space-division data structure for organizing points in a k-dimensional space, and since it is a technique widely used in the relevant technical field, details of the k-dimensional tree are omitted herein. As a result, a tree structure as shown on the right side of FIG. 8 may be generated, and an operation of matching meshes may be performed by determining whether the trees generated from a plurality of fisheye images have the same shape. When the tree shape is the same, it may be determined that corresponding nodes in the tree, that is, first feature points, correspond to each other.

9 is a diagram illustrating an operation of estimating 3D coordinates of a feature point according to an exemplary embodiment of the present invention. For example, FIG. 9 shows an example of step S60 of FIG. 2. Specifically, FIG. 9 illustrates an example in which the first fisheye lens F1 and the second fisheye lens F2 have a first center O ₁ and a second center O ₂ , respectively, and are spaced l _x apart on the X axis. Show. As shown in FIG. 9, it is assumed that the first center O ₁ is at the center of the X/Y/Z axis, that is, at coordinates (0,0), and accordingly, the second center O ₂ is at the coordinates (l _x , 0).

Referring to FIG. 6, when a point P in FIG. 6 is a feature point in a three-dimensional space and θ'is (90°-θ), the three-dimensional coordinates of the point P are expressed as [Equation 1] below. I can.

In [Equation 1], i, j, and k represent the X-axis, Y-axis, and Z-axis.

Again, referring to FIG. 9, the first vector V ₁ from the first center O ₁ to the feature point P and the second vector V ₁ from the second center O ₂ to the feature point P ₂ ) can be derived as shown in [Equation 2] below based on [Equation 1].

In the formula 2], θ _'1 denotes a value obtained by subtracting a first zenith angle (θ ₁₎ of the characteristic point (P) for a first center (O ₁₎ at 90 °, θ' ₂ is a second center (O _It represents a value obtained by subtracting the second zenith angle θ ₂ of the feature point P for ₂ ) from 90 degrees. Ideally, the first vector (V ₁ ) and the second vector (V ₂ ) should intersect at the feature point (P), but various causes, for example, of the first fisheye lens (F1) and the second fisheye lens (F2) Due to non-ideal characteristics such as an error in the curvature function and misalignment of the image sensor and fisheye lens, the first vector V ₁ and the second vector V ₂ may not intersect in a 3D space. That is, the first vector V ₁ and the second vector V ₂ may be in a twisted position.

In some embodiments, the feature point P may be calculated as a coordinate of a point separated by the shortest distance among points separated by the same distance from the first vector V ₁ and the second vector V ₂ . In other words, the weighted first vector (V ₁₎ and second vector (V ₂₎ the line between that is the distance, that is between a first vector (V ₁₎ and second vector (V ₂₎ of the line perpendicular to the amount of vector It can be calculated as the coordinates of the feature point P. According to [Equation 2], the point P ₁ on the first vector V ₁ and the point P ₂ on the second vector V ₂ may be expressed as [Equation 3] below.

및

를 아래 [수학식 4]와 같이 정의할 수 있다.Based on [Equation 3], the direction vector

And

Can be defined as shown in [Equation 4] below.

는 아래 [수학식 5]과 같이 표현될 수 있다.Accordingly, the vector defined by the point P ₁ and the point P ₂ of [Equation 3]

Can be expressed as [Equation 5] below.

및 방향 벡터

가 수직이고, 벡터

및 방향 벡터

가 수직이므로, 아래 [수학식 6]이 만족할 수 있다.When point P ₁ and point P ₂ correspond to both ends of the line representing the distance of the first vector (V ₁ ) and the second vector (V ₂ ), the vector

And direction vector

Is vertical, vector

And direction vector

Since is vertical, [Equation 6] below may be satisfied.

Accordingly, m and n satisfying [Equation 6] can be calculated, and the coordinates of the points P ₁ and P ₂ can be derived accordingly, and as a result, the midpoints of the points P ₁ and P ₂ are the feature points. It can be calculated as the coordinates of (P).

As described above, exemplary embodiments have been disclosed in the drawings and specifications. In the present specification, the embodiments have been described using specific terms, but these are only used for the purpose of describing the technical idea of the present invention, and are not used to limit the meaning or the scope of the present invention described in the claims. . Therefore, those of ordinary skill in the art will understand that various modifications and equivalent other embodiments are possible therefrom. Accordingly, the true technical scope of the present invention should be determined by the technical spirit of the appended claims.

Claims (12)

As a method of modeling a three-dimensional space,
Acquiring a plurality of fisheye images photographed through the plurality of fisheye lenses;
Extracting a plurality of feature points from the plurality of fisheye images;
Estimating a plurality of 3D coordinates of the plurality of feature points based on a curvature function of the plurality of fisheye lenses and the plurality of fisheye images; And
Generating output data defining the three-dimensional space based on the plurality of three-dimensional coordinates,
The step of extracting the plurality of feature points,
Converting the plurality of fisheye images into a plurality of panoramic images corresponding to an image photographed with a standard lens; And
Including the step of extracting the plurality of feature points from the plurality of panoramic images,
The step of extracting the plurality of feature points,
Extracting first feature points having feature values exceeding a predefined threshold value from the plurality of fisheye images;
Generating a mesh including the first feature points and edges connecting the first feature points in each of the plurality of fisheye images;
Dividing each of the plurality of fisheye images into at least one region based on a plurality of meshes; And
And extracting at least one second feature point in regions corresponding to each other of the plurality of fisheye images. delete The method according to claim 1,
Converting the plurality of panoramic images,
Calculating a zenith angle (θ) and an azimuth angle (φ) for a feature point included in the fisheye image in a three-dimensional polar coordinate system using the center point of the fisheye lens as an origin; And
And estimating coordinates of the feature points in the panoramic image based on the zenith angle (θ) and the azimuth angle (φ). The method according to claim 1,
The extracting the plurality of feature points from the plurality of panoramic images comprises extracting the plurality of feature points based on an AKAZE algorithm. delete The method according to claim 1,
The step of generating the mesh,
Aligning the first feature points based on feature values; And
Representing the aligned first feature points as a k-dimensional tree (kd tree),
The step of dividing each of the plurality of fisheye images into at least one region comprises comparing a plurality of k-dimensional trees corresponding to the plurality of fisheye images. The method according to claim 1,
Estimating the plurality of 3D coordinates,
Calculating a zenith angle (θ) and an azimuth angle (φ) for a feature point included in the fisheye image in a three-dimensional polar coordinate system using the center point of the fisheye lens as an origin;
Defining a vector passing through the center point and the feature point of the fisheye lens based on the zenith angle (θ) and the azimuth angle (φ);
Converting a plurality of vectors corresponding to the plurality of fisheye lenses into a three-dimensional polar coordinate system having a center point of the first fisheye lens as an origin; And
And estimating the three-dimensional coordinates of the feature point based on the transformed plurality of vectors. The method of claim 7,
The step of estimating the three-dimensional coordinates of the feature point comprises calculating a point spaced apart by the shortest distance among points spaced apart from the plurality of vectors by the same distance as the three-dimensional coordinates of the feature point. Way. The method according to claim 1,
The generating of the output data includes applying texture mapping to a plane defined by the plurality of 3D coordinates based on at least one panoramic image generated from the plurality of fisheye images. How to do it. As a system for modeling a three-dimensional space,
A first camera module and a second camera module including a first fisheye lens and a second fisheye lens, and generating a first fisheye image and a second fisheye image captured through the first fisheye lens and the second fisheye lens; And
An image processing module configured to generate output data defining the three-dimensional space, based on the first fisheye image and the second fisheye image,
The image processing module,
A feature point extraction unit for extracting a plurality of feature points from the first fisheye image and the second fisheye image;
A coordinate estimation unit for estimating a plurality of 3D coordinates of the plurality of feature points based on a curvature function of the first fisheye lens and the second fisheye lens, the first fisheye image and the second fisheye image; And
And an output data generator that generates the output data based on the plurality of 3D coordinates,
The coordinate estimation unit,
Calculate the zenith angle (θ) and azimuth angle (φ) for the feature points included in the fisheye image in a three-dimensional polar coordinate system with the center point of the fisheye lens as the origin,
Based on the zenith angle θ and the azimuth angle φ, a vector passing through the center point and the feature point of the fisheye lens is defined,
Converting a plurality of vectors corresponding to the first fisheye lens and the second fisheye lens into a three-dimensional polar coordinate system having a center point of the first fisheye lens as an origin,
Estimating the three-dimensional coordinates of the feature point based on the transformed plurality of vectors, and calculating a point spaced apart by the shortest distance among points spaced at the same distance from the plurality of vectors as the three-dimensional coordinates of the feature point A system, characterized in that. The method of claim 10,
The image processing module,
In the correction mode, the system further comprising a curvature providing unit for estimating and storing the curvature function based on the first fisheye image and the second fisheye image. The method of claim 10,
Including a third fisheye lens, further comprising a third camera module for generating a third fisheye image photographed through the third fisheye lens,
The image processing module generates the output data further based on the third fisheye image,
The feature point extracting unit extracts the plurality of feature points from the first fisheye image, the second fisheye image, and the third fisheye image,
And the coordinate estimating unit estimates the plurality of 3D coordinates further based on a curvature function of the third fisheye lens and the third fisheye image.

Images