KR102249381B1 - System for generating spatial information of mobile device using 3D image information and method therefor - Google Patents

Abstract

A system and a method for generating spatial information of a mobile device by using three-dimensional image information are provided. The system comprises: a reference information database unit which stores reference image data in link with reference geometry information including three dimensional position data and first external geometry data with first posture data; a search and selection unit which obtains device geometry information including second external geometry data with at least two-dimensional position data in relation with image data obtained from the mobile device, which searches for a candidate group of the image data that can be matched with the reference image data by referring to the two-dimensional position data and the three-dimensional position data, and which selects the image data with the highest matching degree with the reference image data among the candidate group; a matching unit which matches the selected image data with the reference image data based on feature geometries of a predetermined form common between the reference image data and the selected image data; a geometry transformation unit which geometrically transforms the device geometry information of the selected image data with respect to the reference image data such that the three-dimensional position data is included to correct the device geometry information to three-dimensional geometry information in the common feature geometries; and a three image information generating unit which generates three-dimensional image information for the image data of the mobile device based on the three-dimensional device geometry information and the matching information with respect to the feature geometry. The present invention can transform the image data, obtained by a low-priced mobile device, into three-dimensional spatial information with convenience and high precision.

Description

System for generating spatial information of mobile device using 3D image information and method therefor}

The present invention relates to a system and method for generating spatial information of a mobile device using pre-stored 3D image information as reference data.

Image camera sensors such as smartphones, HMD (Head Mount Display), and smart glasses, positioning sensors, geomagnetic sensors, gyro sensors, geomagnetic sensors, and communication modules, as well as smart mobile devices equipped with computational computing systems are used for various purposes.

For example, in the case of navigation, vehicle location information is extracted by fusion of existing road network information and GPS location data, and driving information is provided at the starting point and destination based on the vehicle's movement route using road network information and GPS location data. It is delivered to the user.

In this way, in constructing spatial information, the positional accuracy acquired from the observation sensor acts as a very important factor. However, smart devices have limitations in that they can only use low-performance sensors due to problems such as size, power, and price, causing many errors.

And recently, as smart cities, autonomous driving, and digital twins have begun to be emphasized as important technologies in the industry, there are limitations in the form of services based on existing two-dimensional positioning information.

Therefore, although the field of application of 3D image information is increasing, there is a limitation in obtaining accurate dimensional positioning information with a low-cost positioning sensor.

Data such as conventional road networks, CAD drawings, and maps can be used in conventional navigation, but in the application of machine learning, accurate surrounding spatial information has not been recognized and sufficient information has not been provided in response thereto.

In addition, since the accuracy and precision of conventional road networks and CAD drawings were generated at the level of meters, there were limitations in providing services such as autonomous driving and AR/VR that require higher performance. In order to overcome this, it is necessary to expand it to 3D spatial information instead of 2D reference spatial information.

However, it is difficult to connect information acquired from image data of a general single camera-based smart device to a service based on 3D spatial information. This is due to the fact that there are problems in the data capacity and the amount of calculation in the use of 3D spatial information, and a plan to overcome this has been continuously proposed.

The technical problem to be achieved by the present invention is to match the image data of 2D location information acquired by a low-performance image sensor, a positioning sensor, etc. of a mobile device with 3D image information generated from a platform composed of high-performance sensors to transform geometrical and the like. To provide a system and method for generating spatial information of a mobile device using 3D image information, which generates 3D spatial information of convenience and high precision from image data acquired by a low-cost mobile device by performing the method.

The object of the present invention is not limited to the objects mentioned above, and other objects not mentioned will be clearly understood by those skilled in the art from the following description.

According to an aspect of the present invention for achieving the above technical problem, a system for generating spatial information of a mobile device using 3D image information includes 3D position data and first posture data defining a viewing direction set to generate a reference image. A reference image database unit for storing reference image data in association with reference geometric information including first external geometric data. Obtain device geometric information including second external geometric data having at least two-dimensional position data related to the image data acquired from the mobile device, and refer to the two-dimensional position data and the three-dimensional position data to obtain the reference image data and A search selector for searching for a candidate group of matchable image data and selecting the image data having the highest degree of matching with the reference image data among the candidate groups, and a predetermined type common between the reference image data and the selected image data A matching unit that matches the reference image data and the selected image data based on the feature geometries of, and in common feature geometries, geometric information of the device of the selected image data with respect to the reference image data is obtained by geometric transformation modeling. A geometric conversion unit that geometrically transforms to include the 3D location data and corrects it with 3D device geometry information, and based on the 3D device geometry information and matching information for the feature geometry, with respect to the image data of the mobile device And a 3D image information generator that generates 3D image information.

In another embodiment, the second external geometric data further includes second posture data defining a viewing direction set to generate the image, and the search selection unit is based on the 3D position data and the 2D position data. A candidate group of the image data that can be contrasted with the reference image data may be searched, and the image data having the highest degree of matching with the reference image data may be selected from among the candidate groups by referring to the first and second posture data.

In addition, the reference geometric information further includes first internal geometric data based on a geometric parameter defined in a sensor set to generate the reference image, and the device geometric information is a geometric parameter defined in an image sensor set to generate the image The image data having the highest degree of matching by the search selection unit and the selection of the image data having the highest degree of matching by the search selection unit and correction of the 3D device geometry information by the geometry conversion unit are performed by the first and The second internal geometric data may be further included for reference.

In addition, when at least one of the reference image and the image data originates from a wide area image sensor that generates a distortion image, the reference image and the image data are a plane generated by converting the distortion image into flat image data. Image data is used, and the 3D position data of the reference image utilizes the 3D position data of the planar image data, and the 2D position data of the image data is defined as the 2D position data of the planar image data, The first and second posture data may be defined as a viewing direction of the planar image data.

In another embodiment, when the reference image originates from a laser scanning sensor, the reference image uses virtual image data obtained by converting point cloud data obtained from the laser scanning sensor, and the 3D position data Is defined as 3D position data of the virtual image data, and the first posture data may be defined as a viewing direction of the virtual image data.

In another embodiment, a sampling image in which a mutually spaced amount of the sampled feature geometries and corresponding feature geometries of the reference image data is minimized by randomly sampling feature geometries of image data to which the 3D device geometric information is linked A matched optimal value estimating unit that estimates the feature geometry of the data as a matched optimal value, and a device geometry that estimates geometric-transformed three-dimensional device geometry information as final three-dimensional device geometry information in relation to the feature geometry estimated as the matched optimal value. An information estimation unit may be further included, and the final 3D device geometry information may be used as the 3D device geometry information among information based upon generation of the 3D image information.

In another embodiment, based on an object information extraction unit for extracting an object of a specific type from the 3D image information, and an object of the same reference image data as the extracted object, the 3D device geometric information and the reference image A spatial information generator for generating new spatial information including the object information related to the object stored in data in the 3D image information may be further included.

Here, the new spatial information is stored in the spatial information database unit, and when existing spatial information already stored in the spatial information database unit exists, in the object information at the same location, the new spatial information is It may further include a spatial information update unit that determines whether or not they are different, and updates the existing spatial information with the new spatial information for different object information as a result of the determination.

According to an aspect of the present invention for achieving the above technical problem, a method for generating spatial information of a mobile device using 3D image information includes second external geometric data having at least 2D position data related to image data acquired from the mobile device. The 3D position data and the 3D position data of the reference geometric information including first external geometric data defining a viewing direction set to obtain the included device geometry information and to obtain the included device geometry data and to generate the 3D position data associated with the reference image data and the reference image. A search selection step of searching for a candidate group of the image data that can be matched with the reference image data by referring to 2D position data, and selecting the image data having the highest degree of matching with the reference image data from among the candidate groups; and the reference image A matching step of matching the reference image data and the selected image data based on feature geometries of a predetermined shape common between the data and the selected image data, and the reference image data by geometric transformation modeling for the common feature geometries. A geometric conversion step of geometrically transforming the device geometry information of the selected image data to include the 3D position data and correcting the 3D device geometry information, and the 3D device geometry information and matching information on the feature geometry And generating 3D image information for the image data of the mobile device based on the 3D image information.

Details of other embodiments are included in the detailed description and drawings.

According to the present invention, geometric transformation is performed by matching image data of 2D location information acquired by a low-performance image sensor or a positioning sensor of a mobile device with 3D image information generated from a platform composed of high-performance sensors. , It is possible to generate 3D spatial information with convenience and high precision from image data acquired with a low-cost mobile device.

In addition, 3D spatial information may be updated in association with existing spatial information from information extracted from image data of a smart device.

In addition, effects derived through configurations that can be grasped by those skilled in the art through the present specification are not excluded.

1 is a configuration diagram of a system for generating spatial information of a mobile device using 3D image information and a mobile device communicating therewith according to an exemplary embodiment of the present invention.
2 is a flowchart illustrating a method of generating spatial information of a mobile device using 3D image information according to another embodiment of the present invention.
3 is a diagram illustrating an example of 3D image information stored in a reference database unit.
4 is a flowchart of a first process of removing distortion of image data acquired from a wide area image sensor and converting it into flat image data.
5 and 6 are diagrams schematically illustrating a first process of image distortion removal and planar image data conversion.
7 is a flowchart illustrating a second process of removing distortion of image data acquired from a wide area image sensor and converting it into flat image data.
8 is a diagram schematically illustrating a second process of image distortion removal and planar image data conversion.
9 is a form in which the reference image database unit stores virtual image data converted from point cloud data obtained from a laser scanning sensor.
10 is a diagram illustrating a process of searching for a candidate group of image data that can be matched with reference image data.
11 is a diagram illustrating a process of selecting image data having the highest degree of matching with reference image data.
12 is a diagram illustrating a process of calculating 3D position data of reference image data corresponding to 2D position data of image data.
13 is a diagram illustrating an example in which the location and viewing direction of image data of a mobile device are corrected with final 3D device geometric information.
14 is a flowchart illustrating a method of generating spatial information of a mobile device using 3D image information according to another embodiment of the present invention.
15 is a diagram illustrating an example of generating new spatial information including final 3D device geometric information and object information in 3D image information.
16 is a diagram for describing a process of updating spatial information.

Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the accompanying drawings and the following description. However, the present invention is not limited to the embodiments described herein and may be embodied in other forms. Rather, the embodiments introduced herein are provided so that the disclosed contents may be thorough and complete, and the spirit of the present invention may be sufficiently conveyed to those skilled in the art. The same reference numbers throughout the specification indicate the same elements. Meanwhile, terms used in the present specification are for explaining embodiments and are not intended to limit the present invention. In this specification, the singular form also includes the plural form unless specifically stated in the phrase. As used herein, "comprises" and/or "comprising" refers to the presence of one or more other components, steps, actions and/or elements in which the recited component, step, operation and/or element is Or does not preclude additions.

In addition, "sub" to "module" generally refer to components such as software (computer, program) and hardware that can be logically separated. Accordingly, the module in the present embodiment refers not only to a module in a computer or a program, but also a module in a hardware configuration. Therefore, the present embodiment is a computer program for making them function as modules (a program for executing each step in a computer, a program for making a computer function as each means, a program for realizing each function in a computer) ), also describes the system and method. However, for convenience of explanation, phrases equivalent to "save" and "save" are used, but these words are stored in a memory device or stored in a memory device when the embodiment is a computer program. It means to control like that. In addition, the "sub" to the module may correspond to the function one-to-one, but in the implementation, one module may be composed of one program, a plurality of modules may be composed of one program, and conversely, one module may be composed of a plurality of programs. do. Further, a plurality of modules may be executed by one computer, or one module may be executed by a plurality of computers by a computer in a distributed or parallel environment. In addition, another module may be included in one module. In addition, hereinafter, "connection" is adopted in the case of logical connection (data exchange, instruction, reference relationship between data, etc.) in addition to physical connection. The term "predetermined" means that it is determined before the target processing, as well as before the processing according to the present embodiment is started, even after the processing according to the present embodiment is started, and before the target processing, the It includes the meaning of what is determined according to the situation and state of the time, or the situation and state up to that time.

In addition, a system or device is configured by connecting a plurality of computers, hardware, devices, etc. by means of communication such as a network (including a one-to-one communication connection), and a single computer, hardware, device, etc. This includes cases where it is realized. The terms "device" and "system" are used as terms of mutual agreement. Of course, the "system" does not include anything but an artificial decision, a social "mechanism" (social system).

In addition, when processing by each unit or module or a plurality of processing within each unit or module, the target information is read and input from the storage device for each processing, and after the processing is performed, the processing result is displayed. It is to write to the storage device. Therefore, the description of the read input from the storage device before processing and the writing to the storage device after processing may be omitted in some cases. Further, the storage device herein may include a hard disk, a random access memory (RAM), an external storage medium, a storage device through a communication line, a register in a CPU (Central Processing Unit), and the like.

Hereinafter, referring to FIG. 1, by matching the image data of the 2D location information acquired from the mobile device according to the embodiment of the present invention with the 3D image data of the reference image database to perform predetermined processing, A system for generating spatial information of a mobile device for generating image data will be described.

1 is a block diagram of a system for generating spatial information of a mobile device using 3D image information and a mobile device communicating therewith according to an exemplary embodiment of the present invention.

The system 100 collects information in which image data of spatial objects such as terrain and features captured by the mobile device 102 are combined with 2D location data and the estimated viewing direction of the image sensor 104 at the time of shooting. Receive.

First, the mobile device 102 will be described. The mobile device 102 includes an image sensor 104, a positioning sensor 106, an IMU sensor (Inertial Measurement Unit) 106, a central processing unit 110, and a communication unit 112. can do.

The mobile device 102 is a device that photographs a surrounding environment while moving through a person or a moving object, and is a device that generates image data associated with 2D location data and transmits/receives to the system 100 through a wired or wireless network. For example, the mobile device 102 may be a smartphone, a head mount display (HMD), smart glasses, a vehicle black box that recognizes vehicle collisions and accidents, etc., but is not limited thereto. .

The image sensor 104 is equipped with a plurality of sensors mounted on the mobile device 102 to acquire image data and time information for an image at the time the image data was acquired by photographing a surrounding object, such as a topography or a feature, as an image. can do. The image sensor 104 may be a flat image sensor, a wide area image sensor having a wider field of view than the planar image sensor, and a surveying or non-surveying camera, but is not limited thereto. The wide-area image sensor generates a stereo image in which images having different viewing directions are combined, or a panoramic image in which images from a camera disposed within a predetermined viewing angle are combined, and for example, a camera that has a fisheye lens to generate a fisheye lens image Alternatively, it may be a multi-camera system that outputs omnidirectional images by multiple cameras using a plurality of cameras.

In the case where the image data originates from a wide area image sensor that generates a distorted image, the image data is processed by the search selection unit 116 to be described later. Planar image data generated by converting a distorted image to planar image data can be used, and a detailed conversion process will be described later.

The positioning sensor 106 detects navigation information such as the positioning coordinates of the object and the location of the mobile device 102, and includes two-dimensional positioning data related to the object coordinates and positioning time data at the time the position data was acquired. It is a sensor that acquires position data. The positioning sensor 106 may be constituted by, for example, a position acquisition device that acquires a moving position of a platform through a satellite navigation device (GPS) in which an international satellite navigation system (GNSS) is used.

When the positioning sensor 106 uses GNSS, the location information basically includes positioning data and positioning time data, as well as the state and number of navigation satellite information, altitude information of the receiver, speed, positioning precision, and true north information. Can have as ancillary data.

The IMU sensor 108 detects the posture and speed of the mobile device 102, from which the central processing unit 110 transmits posture data such as a photographing direction of the image sensor 104, a viewing direction such as an angle, and a viewing angle. Can be estimated. The viewing direction of the image sensor 104 may be the direction information shown in FIG. 10 as an azimuth angle measured by the IMU sensor 108, and may be an azimuth expressed in the east, west, north and south directions of a predetermined coordinate system, or an azimuth angle determined relative to the movement path. I can. The viewing angle of the image sensor 104 may be generated by measuring an inclination of an accelerometer, an angular accelerometer, and a gravimeter constituting the IMU sensor 108 with respect to the ground of the image sensor 104. The IMU sensor may be configured with a posture acquisition device that acquires or calculates posture data of the mobile device 102 and the image sensor through an inertial navigation system (INS).

The central processing unit 110 includes two-dimensional position data and posture data of the image sensor 104 obtained from the positioning sensor 106 and the IMU sensor 108 on the image data acquired from the image sensor 104 (hereinafter referred to as'second Based on the geometric parameters defined in the image sensor 104, the positioning sensor 106, and the IMU sensor 108 in addition to the external geometric data (hereinafter referred to as'second external geometric data') having'attitude data'). Device geometric information including internal geometric data (hereinafter referred to as “second internal geometric data”) may be generated. The central processing unit 110 may control to transmit the above-described data through the communication unit 112 or may receive specific data from the system 100 and perform necessary processing.

The second external geometric data is based on a parameter value that changes each time image information of the image sensor 104 is acquired due to the position and posture of the mobile device 102 in motion, that is, the position and posture of the image sensor 104 It is an error of the observed data calculated by doing so, and in this case, it can be calculated through a predetermined equation or modeling. When the image data originates from a wide-area image sensor, the 2D position data utilizes 2D position data of the planar image data, and the second posture data may be defined as a viewing direction and a viewing angle of the planar image data.

The second internal geometric data is an intrinsic value of the sensors 104 to 108 itself, and is an error in observed data for each sensor 104 to 108 due to a parameter maintained regardless of whether or not the mobile device 102 is moved. The geometric parameter of the image sensor 104 may include any one or more of a focal length, a main point position, a lens distortion parameter, and a sensor format size. The geometric parameters of the positioning sensor 106 or the IMU sensor 108 may include any one or more of an axial scale and an axial offset.

The system 100 includes a reference image database unit 114, a search selection unit 116, a matching unit 118, a geometric conversion unit 120, an optimal matching value estimating unit 122, and a device geometric information estimating unit 124. ), a 3D image information generation unit 126, an object information extraction unit 128, a spatial information generation unit 130, a spatial information update unit 132, and a spatial information database unit 134.

The reference image database unit 114 is connected with reference geometric information including three-dimensional position data of the reference image and first external geometric data having first posture data defining a viewing direction and a viewing angle set to generate the reference image. Save the reference image data. The 3D location data may be 2D location data, that is, XYZ coordinates obtained by adding an altitude value Z from the XY coordinates. The viewing direction and viewing angle are substantially the same as those described for the image sensor 104. In addition, the reference geometric information may further include first internal geometric data based on a geometric parameter defined in a sensor set to generate a reference image.

The reference image is obtained from the outside, is an image including at least 3D position data, and is processed into reference image data that can be contrasted with the image data obtained from the mobile device 102. The first sensor used to generate the reference image may be at least one of an image acquisition device having higher positioning accuracy and image realization performance than the mobile device 102, and a laser scanning sensor (eg, Lidar). The reference geometric information linked to the reference image includes, for each pixel coordinate, 3D position information, color information such as RGB data, first external geometric data, and first internal geometric data, and the reference image combined with the reference geometric information is It may be stored as reference image data.

When the first sensor used to generate the reference image is an image sensor, the first internal geometric data is substantially the same as the image sensor 104 described in the second internal geometric data. When the first sensor is a laser scanning sensor, the first internal geometric data may be, for example, at least one of an incident angle of each laser, a distance scale, a distance direction offset, and an axial offset.

When the reference image originates from a wide area image sensor that generates a distorted image, the reference image uses planar image data generated by converting the distorted image to planar image data, and the three-dimensional position data is the three-dimensional position of the planar image data. Data is utilized, and the first posture data may be defined as a viewing direction and viewing angle of the planar image data. The detailed conversion process will be described later.

When the reference image originates from the laser scanning sensor, the reference image may use virtual image data obtained by converting point cloud data obtained from the laser scanning sensor, and the 3D position data is a 3D position of the virtual image data. It is defined as data, and the first posture data may be defined as a viewing direction of the virtual image data. The conversion of the point cloud data may be performed by virtual camera modeling in consideration of the reflection intensity and altitude value of the point cloud data. When an image sensor is additionally involved in generating the reference image, color data obtained by the image sensor when the point cloud data is converted may be included in the virtual image data.

In addition, the reference image database unit 114 connects and stores information related to a specific type of object, that is, object information in the reference image data. For example, the object may correspond to a building that needs to be displayed on a map, a traffic sign that needs to be recognized for autonomous driving, a traffic light, a lane, and the like. In this case, the object information may include the type of the object, the overall shape, a specific geometric shape such as a line, surface, circle, or sphere constituting at least a part of the object, a color, and texture-related properties.

Meanwhile, the search selection unit 116 acquires device geometric information in relation to the image data acquired from the mobile device 102, and includes first and second external geometric data including 3D and 2D position data and posture data, A candidate group of image data that can be matched with the reference image data is searched by referring to the first and second internal geometric data, and image data having the highest degree of matching with the reference image data is selected from among the candidate groups.

Here, the image data may be used as a descriptor having a characteristic capable of representing a specific object as well as data in a matrix form having color information. In the case of a descriptor having a characteristic representing a specific object, SIFT (Scale Invariant Feature Transform) can be used as an image comparison algorithm.

In order to calculate a descriptor according to SIFT, scale-space extrema, key-point localization & filtering, orientation assignment, and descriptor generation may be performed in the order of scale-space extrema.

Specifically, if a simple boundary is detected in the image data, a lot of meaningless edges including noise are also detected. Therefore, first, a Gaussian filter is applied at various levels (σ value is adjusted) to blur the image data. When the difference between the image data and the reference image data is calculated, a characteristic robust to noise is obtained (Difference of Gauassian: DoG). When the work on the current scale is finished, the same work is performed while changing the size of the image so that it can respond to the scale change, and extrema, which is a strong feature at different sizes, is obtained.

Even though a relatively strong extrema is extracted through DoG, it still contains a lot of unnecessary information, so only meaningful information is extracted through keypoint localization and filtering. Next, a histogram according to the direction of the gradient is calculated.

Descriptors of dimensions NxNm are generated in m directions in the NxN gradient window, and the image selection and consistency are determined by comparing these descriptors.

Specifically, the search selector 116 searches for a candidate group of image data that can be contrasted with the reference image data based on the 3D position data and the 2D position data, and refers to the first and second position data to determine the reference image among the candidate groups. You can select the image data that has the highest degree of matching with the data.

When the image data acquired from the mobile device 102 originates from a wide area image sensor, the search selector 116 removes the distortion of the wide area image and converts it into flat image data. This can be done before the data selection process.

In the above embodiment, the search selector 116 selects a candidate group of image data by analyzing the first and second external geometric data including the posture data, and the first and second internal geometric data, and selects the image data having the highest degree of matching. Although it is described as selecting, the process of selecting a candidate group and selecting image data may be performed based on values other than the altitude value of the 3D position data together with the 2D position data.

The matching unit 118 matches the reference image data and the selected image data based on feature geometries of a predetermined type common between the reference image data and the selected image data, and uses a geometric model such as a collinear conditional equation with reference to the feature geometry. A plurality of selected image data may be combined.

The feature geometry of a predetermined shape may be extracted based on the selected image data estimated to be the same location and the specific geometric shape of the object expressed in the reference image data. The object may be randomly selected as an object from which feature geometry such as points, lines, surfaces, pillars, etc. can be easily extracted from the image data, such as road signs, unusually shaped objects, and building boundaries. Such a feature geometry adopts a predetermined shape with a high probability of being estimated to be the same object even though the attitude data between the selected image data and the reference image data are slightly different, and may be, for example, a lane displayed on a road surface, a road sign, a building outline, or the like.

The geometric conversion unit 120 geometrically converts the device geometric information of the selected image data for the reference image data to include 3D position data by geometric transformation modeling for common feature geometries and corrects it with 3D device geometry information. do.

For geometric transformation modeling, a collinear conditional expression such as Equation 1 may be used, or alternative modeling may be used for one of perspective transformation, affine transformation, and homography transformation that may reflect the geometry of image data in a matrix form.

[Equation 1]

Image coordinates are two-dimensional position data of image data related to the same feature geometry, and object coordinates are three-dimensional position data of reference image data with respect to the same feature geometry. The camera position and posture are first external geometric data of the reference image data, and the main point position, focal length, and lens distortion are first internal geometric data.

The matching optimal value estimating unit 122 randomly samples the feature geometries of the image data to which 3D device geometric information is linked, and the sampled image data in which the mutually spaced amount of the sampled feature geometries and the feature geometries of the corresponding reference data is minimized. The feature geometry of can be estimated as the matching optimal value.

Specifically, image data sharing the feature geometry for each of the same estimated feature geometry is sampled in plurality, and reference image data having the feature geometry is interlocked to be selected. Subsequently, when the image data sharing the same estimated feature geometry and the reference image data are arranged according to the 3D device geometry information, the amount of separation between the image data and the reference image data of the same estimated feature geometry is determined to be the minimum. The feature geometry of the image data to be obtained is estimated as a matching optimal value.

The device geometry information estimating unit 124 may estimate geometrically transformed 3D device geometry information as final 3D device geometry information in relation to the feature geometry estimated as the matching optimal value.

The 3D image information generator 126 generates 3D image information for the image data of the mobile device 102 based on the final 3D device geometry information and matching information for the feature geometry.

The 3D image information, like the reference image data of the reference image database unit 114, includes image data corrected by geometric transformation, 3D position data in connection therewith, and corrected second posture data. It may include geometric data, final 3D device geometric information including corrected second internal geometric data, color data, time data, and the like.

The object information extraction unit 128 may extract an object of a specific type from 3D image information resulting from the image data.

Specifically, the object information extraction unit 128 may extract object information related to an attribute of the extracted object based on data included in the 3D image information.

The object information extracting unit 128 detects candidate group data related to the object identified from image data included in the 3D image information, final 3D device geometric information, color data, time data, etc., using a machine learning technique, and detects the detected candidate group. Information on the object can be recognized by sequentially applying additional machine learning models to the data.

Machine learning is performed in two stages: object object detection and object object recognition, and in the object object detection stage, candidate groups are detected in the data based on properties such as the shape, color, and texture of the object object through machine learning. And, by sequentially applying an additional machine learning model to the detected candidate group, the information represented by the target object is recognized and classified.

The spatial information generation unit 130 includes, in the 3D image information, the final 3D device geometric information and object information related to the object stored in the reference image data, based on the object of the same reference image data as the extracted object, For example, new spatial information according to an absolute coordinate system may be generated and transmitted to the spatial information database unit 134 for storage. The spatial information is spatial data to be described later, and detailed information will be described later.

The spatial information database unit 134 structured and stores data previously generated for drawing as spatial data based on observation data and observation data previously acquired from heterogeneous sensors according to a predetermined coordinate system, for example, an absolute coordinate system. Spatial data is data generated in the corresponding coordinates of a predetermined coordinate system, such as observation data, internal geometry, external geometry, intrinsic data, time information, acquisition environment information, and the amount of separation, accuracy, and precision due to the positional error of sensor data between heterogeneous sensors. It may include related probability information, point cloud data related information, image data related information, object information, and map related information. In the present embodiment, the spatial data is described as including all of the above-described data or information, but is not limited thereto, and may be set to include only some of the above-described data/information according to design specifications.

When the existing spatial information already stored in the spatial information database unit 134 exists, the spatial information update unit 132 determines whether the new spatial information is different from the existing spatial information in the object information at the same location, As a result of the determination, for different object information, existing spatial information may be updated with new spatial information.

According to the present embodiment, a two-dimensional image matching method may be applied to match the image data having the two-dimensional position data of the mobile device 102 and the reference image data having the three-dimensional position data. So that the matching process can be performed quickly with low complexity.

In addition, 3D image information can be generated by geometrically transforming and correcting the position data of the image data with 3D position data with high precision and accuracy included in the reference image data. It can be usefully used in subsequent data processing.

In addition, in the present embodiment, the final 3D device geometry information, etc., is generated with feature geometries with the minimum mutual separation between the feature geometries estimated to be the same through the matched optimal value estimator 122, but the second external geometric data Is data capable of satisfactorily realizing the matching between the image data, and if the condition that the mutually spaced amount between the feature geometries is less than or equal to the threshold value is satisfied, the geometric conversion unit 120 may be omitted.

Hereinafter, a method of generating spatial information of a mobile device using 3D image information according to another embodiment of the present invention will be described with reference to FIGS. 2 to 13. In the present embodiment, the first and second posture data are used in the process of selecting the image data having the highest degree of matching with the reference image data, and the function of the geometric conversion unit 120 is performed. However, as explained above, these can be omitted.

2 is a flowchart illustrating a method of generating spatial information of a mobile device using 3D image information according to another embodiment of the present invention. 3 is a diagram illustrating an example of 3D image information stored in a reference database unit.

First, the search selection unit 116 retrieves device geometric information including second external geometric data and second internal geometric data having two-dimensional position data and second posture data related to the image data acquired from the mobile device 102. Acquire (S205).

The second external geometric data is based on a parameter value that changes each time image information of the image sensor 104 is acquired due to the position and posture of the mobile device 102 in motion, that is, the position and posture of the image sensor 104 It is an error of the observed data calculated by doing so, and in this case, it can be calculated through a predetermined equation or modeling. The second internal geometric data is an intrinsic value of the sensors 104 to 108 itself, and is an error in observed data for each sensor 104 to 108 due to a parameter maintained regardless of whether or not the mobile device 102 is moved. The geometric parameters for each of the sensors 104 to 108 are omitted in detail.

Next, the search selector 116 includes at least three-dimensional position data and an image from among the reference geometric information of the reference image data including the first external geometric data and the first internal geometric data having the 3D position data and the first posture data. A candidate group of image data is searched based on the 2D position data of the data (S210).

Here, the image data may be used as a descriptor having a characteristic capable of representing a specific object as well as data in a matrix form having color information. In the case of a descriptor having a characteristic representing a specific object, SIFT (Scale Invariant Feature Transform) can be used as an image comparison algorithm. This will be described in detail in the description of the system 100 and omitted.

The reference image data is stored in the reference image database unit 114, and, as shown in FIG. 3, the reference image data stores the above-described data including 3D position data in a pixel displayed in dark color. The 3D location data may be 2D location data, that is, XYZ coordinates obtained by adding an altitude value Z from the XY coordinates. The first external geometric data includes the above-described first posture data together with 3D position data, and the first internal geometric data is based on a geometric parameter defined in a sensor set to generate a reference image.

The reference image is obtained from the outside, is an image including at least 3D position data, and is processed into reference image data that can be contrasted with the image data obtained from the mobile device 102. The first sensor used to generate the reference image may be at least one of an image acquisition device having higher positioning accuracy and image realization performance than the mobile device 102, and a laser scanning sensor (eg, Lidar). The reference geometric information linked to the reference image includes, for each pixel coordinate, 3D position information, color information such as RGB data, first external geometric data, and first internal geometric data, and the reference image combined with the reference geometric information is It may be stored as reference image data.

When the first sensor used to generate the reference image is an image sensor, the first internal geometric data is substantially the same as the image sensor 104 described in the second internal geometric data. When the first sensor used is a laser scanning sensor, the first internal geometric data may include, for example, one or more of an incident angle of each laser, a distance scale, a distance direction offset, and an axial offset.

When the reference image originates from a wide area image sensor that generates a distorted image, the reference image uses planar image data generated by converting the distorted image to planar image data, and the three-dimensional position data is the three-dimensional position of the planar image data. Data is utilized, and the first posture data may be defined as a viewing direction and viewing angle of the planar image data.

In this regard, a process of removing distortion and generating planar image data performed by the reference image database unit 114 will be described with reference to FIGS. 4 to 8.

4 is a flowchart illustrating a first process of removing distortion of image data acquired from a wide area image sensor and converting it into flat image data, and FIGS. 5 and 6 schematically illustrate a first process of removing image distortion and converting planar image data. It is a drawing shown. 7 is a flowchart illustrating a second process of removing distortion of image data acquired from a wide area image sensor and converting it into flat image data, and FIG. 8 is a diagram schematically showing a second process of removing image distortion and converting planar image data to be.

Referring to FIG. 4 related to the first process, when image data for a fisheye lens distorted by a wide-area image sensor having a fisheye lens is input (S405), the reference image database unit 114 is as shown in FIG. , Each pixel is given an index to define a correspondence relationship between the distorted image data from the image data and the reference plane image data (S410).

Next, the reference image database unit 114 decomposes the distorted image data to which the index is assigned into preset sections (S415).

Subsequently, the reference image database unit 114 generates flat image data by removing distortion of the distorted image data based on the distortion correction model of the decomposed section by referring to the reference plane image data (S420).

The distortion correction model may be modeling based on an equation or index of the decomposed section.

If the omnidirectional image data is input, the reference image database unit 114 decomposes the omnidirectional image data into preset sections as in the second step of FIG. 6.

Next, the reference image database unit 114 re-decomposes the decomposed omnidirectional image into preset sections, as in the third step of FIG. 6. Subsequently, the reference image database unit 114 removes the distortion based on a distortion correction model of the re-decomposed section, for example, an equation or an index, and generates flat image data as shown in the right diagram of FIG. 6.

This is substantially the same process even when the image data received from the mobile device 102 includes a distorted image due to the wide area image sensor, and this process can be performed in the search selection unit 116 for inputting the image data. have.

On the other hand, referring to FIG. 7 related to the second process, when omnidirectional image data distorted by a wide area image sensor generating an omnidirectional image is input to the reference image database unit 114 (S705), the reference image database unit ( 114) restores a virtual 3D shape from the distortion image data based on the camera geometry based on the distortion image data from the image data (S710).

Next, the reference image database unit 114 creates planar image data by defining a virtual camera geometry and projecting it onto a virtual plane camera through the geometry (S715).

In contrast, when the reference image originates from a laser scanning sensor, the reference image may use virtual image data obtained by converting point cloud data obtained from the laser scanning sensor. It is defined as 3D position data, and the first posture data may be defined as a viewing direction of the virtual image data.

Conversion of the point cloud data may be performed by modeling a virtual camera in consideration of the reflection intensity and altitude values of the point cloud data located under the drawing as shown in FIG. 9. 9 is a form in which the reference image database unit stores virtual image data converted from point cloud data obtained from a laser scanning sensor. As a result, virtual image data may be generated as in the example above of FIG. 9. When an image sensor is additionally involved in generating the reference image, color data obtained by the image sensor when the point cloud data is converted may be included in the virtual image data.

As described above, the search for a candidate group of image data based on the three-dimensional position data of the reference image data and the two-dimensional position data of the image data performed by the search selector 116 may be performed by a search process as shown in the steps below in FIG. 10. have. 10 is a diagram illustrating a process of searching for a candidate group of image data that can be matched with reference image data.

Next, the search selector 116 selects the image data having the highest degree of matching with the reference image data from among the candidate groups by referring to the 2D position data, the 3D position data, and the first and second posture data.

Specifically, if the first and second posture data of the first and second external geometric data are different according to the viewing direction and viewing angle of the image data having the same or very similar two-dimensional position data as the reference image data, due to the image geometry The same feature geometry is recognized as a different geometry, making it difficult to match the reference image data and the image data. Accordingly, the accuracy and processing speed of the subsequent matching step may be improved by selecting only image data that is similar to the first and second posture data within a predetermined range from the same position from the candidate group. In this example, a search based on direction information is performed as shown in FIG. 10, and as shown in FIG. 11, image data adjacent to the position and posture of the reference image data in a predetermined range are finally selected. FIG. 10 is a diagram illustrating a process of searching for a candidate group of image data that can be matched with reference image data, and FIG. 11 is a diagram illustrating a process of selecting image data having the highest degree of matching with the reference image data.

Subsequently, the matching unit 118 matches the reference image data and the selected image data based on feature geometries of a predetermined type that are common between the selected image data and the reference image data (S220).

The matching unit 118 may combine a plurality of selected image data through a geometric model such as a collinear conditional equation with reference to the feature geometry.

The feature geometry of a predetermined shape may be extracted based on the selected image data estimated to be the same location and the specific geometric shape of the object expressed in the reference image data. The object may be randomly selected as an object from which feature geometry such as points, lines, surfaces, pillars, etc. can be easily extracted from the image data, such as road signs, unusually shaped objects, and building boundaries. Such a feature geometry adopts a predetermined shape with a high probability of being estimated to be the same object even though the attitude data between the selected image data and the reference image data are slightly different, and may be, for example, a lane displayed on a road surface, a road sign, a building outline, or the like.

The geometric conversion unit 120 geometrically converts the device geometric information of the selected image data for the reference image data to include 3D position data by geometric transformation modeling for common feature geometries and corrects it with 3D device geometry information. do.

For geometric transformation modeling, a collinear conditional expression such as Equation 1 above may be used, or alternative modeling may be used for one of perspective transformation, affine transformation, and homography transformation capable of reflecting the geometry of image data in a matrix form.

12 is a diagram illustrating a process of calculating 3D position data of reference image data corresponding to 2D position data of image data. In the case of using the collinear conditional equation, in the same feature geometry, the image data is geometrically transformed into a three-dimensional positional coordinate of the reference image data and corrected. 13 is a diagram illustrating an example in which the location and viewing direction of image data of a mobile device are corrected with final 3D device geometric information. The dotted line is the 2D position data of the image data acquired from the mobile device 102, and the solid line is corrected to 3D position data by geometric transformation correction of the image data, and the 2D coordinate is moved to a more accurate position, and the mobile device ( 102) indicates that the conversion is performed with a higher precision than that of the positioning sensor 106.

Next, the matching optimal value estimating unit 122 randomly samples the feature geometries of the image data to which the 3D device geometric information is linked, so that the mutually spaced amount of the sampled feature geometries and the feature geometries of the corresponding reference data is minimized. The feature geometry of the sampled image data is estimated as a matching optimal value (S230). The detailed process of the matching optimal value will be omitted in detail.

Subsequently, the device geometry information estimating unit 124 estimates the geometrically transformed 3D device geometry information as the final 3D device geometry information in relation to the feature geometry estimated as the matching optimal value (S235).

Next, the 3D image information generation unit 126 generates 3D image information for the image data of the mobile device 102 based on the final 3D device geometry information and matching information for the feature geometry (S240). .

The 3D image information, like the reference image data of the reference image database unit 114, includes image data corrected by geometric transformation, 3D position data in connection therewith, and corrected second posture data. It may include geometric data, final 3D device geometric information including corrected second internal geometric data, color data, time data, and the like.

According to the present embodiment, image data of two-dimensional position information acquired by the low-performance image sensor 104, the positioning sensor 106, etc. of the mobile device 102 is a three-dimensional image generated from a platform composed of high-performance sensors. By matching with information and performing geometric transformation, it is possible to generate 3D image information with convenience and high precision from image data acquired by a low-cost mobile device.

Hereinafter, a method of generating spatial information of a mobile device using 3D image information according to another embodiment of the present invention will be described with reference to FIGS. 14 to 16.

14 is a flowchart illustrating a method of generating spatial information of a mobile device using 3D image information according to another embodiment of the present invention. 15 is a diagram illustrating an example of generating new spatial information including final 3D device geometric information and object information in 3D image information, and FIG. 16 is a diagram for explaining a process of updating spatial information.

First, the object information extracting unit 128 extracts an object of a specific shape from 3D image information originating from the image data (S1405).

Specifically, the object information extraction unit 128 may extract object information related to an attribute of the extracted object based on data included in the 3D image information.

The object information extracting unit 128 detects candidate group data related to the object identified from image data included in the 3D image information, final 3D device geometric information, color data, time data, etc., using a machine learning technique, and detects the detected candidate group. Information on the object can be recognized by sequentially applying additional machine learning models to the data.

Next, the spatial information generation unit 130 includes, in the 3D image information, the final 3D device geometric information and object information related to the object stored in the reference image data, based on the object of the same reference image data as the extracted object. New spatial information is generated according to a predetermined coordinate system, for example, an absolute coordinate system (S1410).

The spatial information generation unit 130 may transmit and store new spatial information to the spatial information database unit 134. Spatial information is data generated in the corresponding coordinates of a predetermined coordinate system, such as observation data, internal geometry, external geometry, unique data, time information, acquisition environment information, and the amount of separation, accuracy, and precision due to the positional error of sensor data between heterogeneous sensors. It may include related probability information, point cloud data related information, image data related information, object information, and map related information. In the present embodiment, the spatial information is described as including all of the above-described data or information, but is not limited thereto, and may be set to include only some of the above-described data/information according to design specifications.

As shown in FIG. 15, new spatial information may be generated by including final 3D device geometric information and object information in 3D image information, and based on this, through a drawing process, a map on the right side may be produced.

Next, when the existing spatial information already stored in the spatial information database unit 134 exists, the spatial information update unit 132 determines whether the new spatial information is different from the existing spatial information in the object information at the same location. As a result of the determination (S1415), with respect to the different object information as a result of the determination, the existing spatial information is updated with new spatial information as shown in FIG. As a result of the determination, when the object information is the same, the existing spatial information may be maintained (S1420).

The components constituting the device shown in Fig. 1 or the steps according to the embodiments shown in Figs. 2 and 14 are computer-readable recording media in the form of a program for realizing the function.

Can be recorded on the sieve. Here, the computer-readable recording medium refers to a recording medium that can be read by a computer by storing information such as data or programs by electrical, magnetic, optical, mechanical, or chemical action. Examples of such recording media that can be separated from a computer include portable storage, flexible disks, magneto-optical disks, CD-ROMs, CD-R/Ws, DVDs, DATs, and memory cards. Further, as a recording medium fixed to mobile devices and computers, there are solid state disks (SSDs), hard disks, and ROMs.

Further, in the above, even if all the constituent elements constituting the embodiments of the present invention have been described as being combined into one operation, the present invention is not necessarily limited to these embodiments. That is, within the scope of the object of the present invention, all the constituent elements may be selectively combined and operated in one or more. In addition, all of the components may be implemented as one independent hardware, but some or all of the components are selectively combined to provide a program module that performs some or all of the combined functions in one or a plurality of hardware. It may be implemented as a computer program having.

Although the present invention has been described in detail through exemplary embodiments above, a person of ordinary skill in the art to which the present invention pertains will realize that various modifications can be made to the above-described embodiments without departing from the scope of the present invention. I will understand. Therefore, the scope of the present invention is limited to the described embodiments and should not be determined, and should be determined by all changes or modifications derived from the claims and the concept of equality as well as the claims to be described later.

Claims (10)

A reference image database unit for storing reference image data in association with reference geometric information including 3D position data and first external geometric data having first posture data defining a viewing direction set to generate the reference image;
Obtain device geometric information including second external geometric data having at least two-dimensional position data related to the image data acquired from the mobile device, and refer to the two-dimensional position data and the three-dimensional position data to obtain the reference image data and A search selection unit that searches for a candidate group of matchable image data and selects the image data having the highest degree of matching with the reference image data from among the candidate groups;
A matching unit for matching the reference image data and the selected image data based on feature geometries of a predetermined shape common between the reference image data and the selected image data;
In common feature geometries, a geometric conversion unit that geometrically transforms the device geometry information of the selected image data with respect to the reference image data to include the 3D position data and corrects it into 3D device geometry information by geometric transformation modeling. ;
The feature geometry of the image data to which the 3D device geometry information is linked is randomly sampled to match the feature geometry of the sampled image data in which the mutually spaced amount of the sampled feature geometry and the corresponding feature geometry of the reference image data is minimum A matched optimum value estimating unit that estimates an optimum value;
A device geometry information estimating unit for estimating geometrically transformed 3D device geometry information as final 3D device geometry information in relation to the feature geometry estimated as the matching optimal value; And
A mobile using 3D image information, comprising a 3D image information generator for generating 3D image information for image data of the mobile device based on the final 3D device geometry information and matching information for the feature geometry Device spatial information generation system. The method of claim 1,
The second external geometric data further includes second posture data defining a viewing direction set to generate the image,
The search selection unit searches for a candidate group of the image data that can be contrasted with the reference image data based on the 3D position data and the 2D position data, and refers to the first and second posture data to determine the reference group among the candidate groups. A system for generating spatial information of a mobile device using 3D image information for selecting the image data having the highest degree of matching with the image data. The method of claim 2,
The reference geometric information further includes first internal geometric data based on a geometric parameter defined in a sensor set to generate the reference image, and the device geometric information is based on a geometric parameter defined in an image sensor set to generate the image. The selection of the image data having the highest degree of matching by the search selection unit and correction of the 3D device geometry information by the geometric conversion unit further includes second internal geometric data, and the first and second A system for generating spatial information of a mobile device using 3D image information to further include and refer to internal geometric data. The method of claim 2,
When at least one of the reference image and the image data originates from a wide-area image sensor that generates a distortion image, the reference image and the image data are flat image data generated by converting the distortion image into planar image data. The three-dimensional position data of the reference image uses three-dimensional position data of the planar image data, and the two-dimensional position data of the image data is defined as two-dimensional position data of the planar image data, and the first And the second posture data is defined as a viewing direction of the planar image data. A system for generating spatial information of a mobile device using 3D image information. The method of claim 1,
When the reference image originates from a laser scanning sensor, the reference image uses virtual image data converted from point cloud data obtained from the laser scanning sensor, and the 3D position data is the virtual image data. The system for generating spatial information of a mobile device using 3D image information, which is defined as 3D position data of, and the first posture data is defined as a viewing direction of the virtual image data. delete The method of claim 1,
An object information extracting unit for extracting an object of a specific type from the 3D image information; And
Generates new spatial information including the final 3D device geometric information and the object information related to the object stored in the reference image data in the 3D image information based on the object of the same reference image data as the extracted object A system for generating spatial information of a mobile device using 3D image information, further comprising: a spatial information generating unit to perform. The method of claim 7,
If the new spatial information is stored in the spatial information database unit, and there is existing spatial information already stored in the spatial information database unit, whether the new spatial information is different from the existing spatial information in the object information at the same location. A spatial information generation system of a mobile device using 3D image information, further comprising a spatial information update unit that determines whether or not, and updates the existing spatial information with the new spatial information with respect to different object information as a result of the determination. A viewing direction set to acquire device geometry information including second external geometry data having at least 2D position data related to image data acquired from a mobile device, and to generate 3D position data and a reference image linked to the reference image data Searching for a candidate group of the image data that can be matched with the reference image data by referring to the 3D location data and the 2D location data among reference geometric information including first external geometric data defining A search selection step of selecting the image data having the highest degree of matching with the image data;
A matching step of matching the reference image data and the selected image data based on feature geometries of a predetermined shape common between the reference image data and the selected image data;
In common feature geometries, a geometric transformation step of geometrically transforming the device geometry information of the selected image data with respect to the reference image data to include the 3D position data by geometric transformation modeling, and correcting it into 3D device geometry information. ;
The feature geometry of the image data to which the 3D device geometry information is linked is randomly sampled to match the feature geometry of the sampled image data in which the mutually spaced amount of the sampled feature geometry and the corresponding feature geometry of the reference image data is minimum A matched optimal value estimating step of estimating an optimal value;
A device geometry information estimating step of estimating geometry-transformed 3D device geometry information as final 3D device geometry information in relation to the feature geometry estimated as the matching optimal value; And
Using 3D image information comprising the step of generating 3D image information for the image data of the mobile device based on the final 3D device geometry information and matching information for the feature geometry. How to generate spatial information on mobile devices. The method of claim 9,
The second external geometric data further includes second posture data defining a viewing direction set to generate the image,
In the search and selection step, a candidate group of the image data that can be contrasted with the reference image data is searched based on the 3D position data and the 2D position data, and the candidate group among the candidate groups is A method of generating spatial information of a mobile device using 3D image information for selecting the image data having the highest degree of matching with the reference image data.

Images