KR101853652B1

KR101853652B1 - Around view genegation method and apparatus performing the same

Info

Publication number: KR101853652B1
Application number: KR1020150188078A
Authority: KR
Inventors: 황정훈; 김승훈
Original assignee: 전자부품연구원
Priority date: 2015-12-29
Filing date: 2015-12-29
Publication date: 2018-05-03
Also published as: KR20170078005A

Abstract

The apparatus includes an environment view processor for generating an environment view through a plurality of camera sensors, a distance information processor for receiving distance information about at least one main subject in the environment view through the distance measurement sensor, And an ambient view processor for generating an ambient view by fusing the ambient environment view and the distance information to a predefined ambient view projection model. Accordingly, the surrounding view generating apparatus can provide more accurate information about the surround view through the plurality of camera sensors and the distance measuring sensor, thereby allowing the user to accurately recognize the surrounding environment.

Description

Field of the Invention [0001] The present invention relates to a method,

Field of the Invention [0002] The present invention relates to a technique for creating an overview view, and more particularly, to a method for generating an overview view by merging information obtained using a heterogeneous sensor, which can be used in, for example, .

The smart car or the robot can finally go to the destination without being boarded by the corresponding device of the person, or can be controlled by the user at the remote place. To this end, the smart car or robot can be equipped with an image sensor, such as a camera, to perceive surrounding obstacles or information about the target.

Conventionally, in the process of synthesizing 2D information, which is a peripheral image obtained from the four wide-angle cameras, with a three-dimensional image view around the vehicle, it properly reflects information about the surroundings of the vehicle, for example, surrounding pillars and walls The user can not pass the visual sense of space properly.

Korean Patent No. 10-1579100 relates to a vehicle surround view providing apparatus and a vehicle having the same, wherein the vehicle surround view providing apparatus includes first to fourth cameras mounted on a vehicle, first to fourth cameras mounted on the first to fourth cameras, Based on the difference between the reference image corresponding to each of the first to fourth cameras from the memory and each shot image from the first to fourth cameras, And a processor for calculating the offset information and synthesizing the respective images from the first to fourth cameras using the offset information to generate the surround view image. This makes it possible to provide an accurate arousal view image based on the calibrated image.

Korean Patent No. 10-1504335 discloses an apparatus, method, and vehicle for providing an around view. The camera section includes a plurality of cameras installed in a vehicle and capturing an image. The control unit performs calibration for generating the surround view using the images photographed by the plurality of camera units. According to the present invention, there is an effect that a straight line on an image is detected using Hough transform, and an image is matched through a homography matrix.

Korean Patent No. 10-1566964 discloses an approach view monitoring method capable of tracking a moving object, an apparatus for monitoring an ambient view for performing the method, and a recording medium storing the apparatus. The surround view monitoring apparatus includes a plurality of cameras for imaging a periphery of a working vehicle to generate a plurality of images, an at least one of the plurality of images or an ambient view generated based on the plurality of images, And an object recognizing unit for recognizing a moving object due to at least the working environment of the working vehicle and determining the necessity of tracking the recognized object in the working environment view. Therefore, the surround view monitoring apparatus can prevent a safety accident by increasing the recognition speed or recognition accuracy of the object according to the working environment of the working vehicle.

Korean Registered Patent No. 10-1579100 (registered on December 15, 2015) Korean Registered Patent No. 10-1249274 (Registered on Mar. 13, 2015) Korean Registered Patent No. 10-1566964 (registered on Nov. 13, 2015)

One embodiment of the present invention seeks to create an ambient view by fusing information obtained using heterogeneous sensors, which may be used, for example, in an automotive or robot device. For example, the heterogeneous sensor may include a plurality of camera sensors and a distance measurement sensor such as a radar, a lidar, or an ultrasonic sensor.

An embodiment of the present invention provides more precise information about the surround view through a plurality of camera sensors and a distance measuring sensor, so that the user intends to accurately recognize the surrounding environment.

In embodiments, the method for generating an overview view includes generating an ambient view through a plurality of camera sensors, receiving distance information about at least one primary subject in the ambient view through the distance measurement sensor, And creating an ambient view by fusing the ambient environment view and the distance information to a predefined environment view projection model.

The step (c) includes the step of measuring the direction of the main subject positioned from the moving object equipped with the surround view generating apparatus by analyzing the surrounding view, and measuring the absolute distance of the measured main subject from the distance information can do.

The step (c) may include the step of measuring the distance of the object having the longest absolute distance and determining the projection range of the environmental view projection model based on the distance of the object.

The step (c) may include overlaying each subject on the environmental view projection model according to a relative distance reflecting the position and direction of the at least one main subject based on the determined projection range.

The step (c) may include distorting the surrounding view projection model based on the surrounding shape of the surrounding view, and reflecting the distance information to the distorted surrounding view projection model.

In an embodiment of the present invention, the surround view generating apparatus includes a surrounding view processor for generating a surrounding view through the plurality of camera sensors, distance information about at least one main subject in the surrounding view through the distance measuring sensor, And a surrounding view processor for generating a surrounding view by fusing the surrounding environment view and the distance information to a predefined surrounding view projection model.

The disclosed technique may have the following effects. It is to be understood, however, that the scope of the disclosed technology is not to be construed as limited thereby, as it is not meant to imply that a particular embodiment should include all of the following effects or only the following effects.

The method of generating an approach view according to an exemplary embodiment of the present invention can generate an overview view by fusing information obtained using a heterogeneous sensor, which can be used in, for example, an automobile or a robot. For example, the heterogeneous sensor may include a plurality of camera sensors and a distance measurement sensor such as a radar, a lidar, or an ultrasonic sensor.

The method of generating an overview view according to an exemplary embodiment of the present invention provides more accurate information about the surround view through a plurality of camera sensors and a distance measurement sensor so that the user can accurately recognize the surrounding environment.

1 is a view for explaining a moving object having an apparatus for generating an overview view according to an embodiment of the present invention.
2 is a block diagram illustrating an apparatus for generating an overview view in FIG.
FIG. 3 is a flowchart of an overview view generation process performed by the surround view generation apparatus of FIG. 1;
Fig. 4 is a view for explaining a surrounding view projection model in the projection model database shown in Fig. 1. Fig.

The description of the present invention is merely an example for structural or functional explanation, and the scope of the present invention should not be construed as being limited by the embodiments described in the text. That is, the embodiments are to be construed as being variously embodied and having various forms, so that the scope of the present invention should be understood to include equivalents capable of realizing technical ideas. Also, the purpose or effect of the present invention should not be construed as limiting the scope of the present invention, since it does not mean that a specific embodiment should include all or only such effect.

Meanwhile, the meaning of the terms described in the present application should be understood as follows.

The terms "first "," second ", and the like are intended to distinguish one element from another, and the scope of the right should not be limited by these terms. For example, the first component may be referred to as a second component, and similarly, the second component may also be referred to as a first component.

It is to be understood that when an element is referred to as being "connected" to another element, it may be directly connected to the other element, but there may be other elements in between. On the other hand, when an element is referred to as being "directly connected" to another element, it should be understood that there are no other elements in between. On the other hand, other expressions that describe the relationship between components, such as "between" and "between" or "neighboring to" and "directly adjacent to" should be interpreted as well.

It is to be understood that the singular " include " or "have" are to be construed as including the stated feature, number, step, operation, It is to be understood that the combination is intended to specify that it does not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, or combinations thereof.

In each step, the identification code (e.g., a, b, c, etc.) is used for convenience of explanation, the identification code does not describe the order of each step, Unless otherwise stated, it may occur differently from the stated order. That is, each step may occur in the same order as described, may be performed substantially concurrently, or may be performed in reverse order.

The present invention can be embodied as computer-readable code on a computer-readable recording medium, and the computer-readable recording medium includes all kinds of recording devices for storing data that can be read by a computer system . Examples of the computer-readable recording medium include a ROM, a RAM, a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, and the like, and also implemented in the form of a carrier wave (for example, transmission over the Internet) . In addition, the computer-readable recording medium may be distributed over network-connected computer systems so that computer readable codes can be stored and executed in a distributed manner.

All terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs, unless otherwise defined. Commonly used predefined terms should be interpreted to be consistent with the meanings in the context of the related art and can not be interpreted as having ideal or overly formal meaning unless explicitly defined in the present application.

1 is a view for explaining a moving object having an apparatus for generating an overview view according to an embodiment of the present invention.

Referring to FIG. 1, a moving object 1 includes a plurality of camera sensors 110 and an overview view generating apparatus 100 connected to a distance measuring sensor 120. Here, the moving object 1 may be implemented as an automobile or a robot, and the surrounding view generating apparatus 100 may be installed in the moving object 1. [

Fig. 1A assumes a situation in which the moving object 1 moves on an ordinary road, and Fig. 1B assumes a situation in which the moving object 1 moves on an alleyway with walls on both sides. The related description will be described later with reference to Fig.

A plurality of camera sensors 110 may be disposed on the front, back, right, and left sides of the mobile object 1, respectively, and each image sensor picks up the corresponding area to generate the corresponding image. Here, the number and position of the plurality of camera sensors 110 are not necessarily limited to these, and may be changed according to the embodiment. In addition, the plurality of camera sensors 110 may be mounted with a forward camera or a rear camera of the moving object 1 to perform a function of preventing lane departure or recognizing a traffic sign.

The distance measuring sensor 120 may be disposed at front, rear, left, and right sides of the moving object 1, and may sense distance information regarding at least one main subject in the surrounding environment. Here, the distance measuring sensor 120 may be implemented as an ultrasonic sensor, a lidar sensor, or a radar sensor.

The surround view generating apparatus 100 may be connected to the plurality of camera sensors 110 and the distance measuring sensor 120 and generates an overview view by fusing subject image information and corresponding distance information received from the sensors. Here, the surround view may include a depth image that visually reflects the distance of the object away from the moving object 1. [

2 is a block diagram illustrating an apparatus for generating an overview view in FIG.

2, the surround view generating apparatus 100 is connected to the plurality of camera sensors 110 and the distance measuring sensor 120, and includes an environment view processor 210, a distance information processor 220, A view processing unit 230, a projection model database 240, and a control unit 250.

The surround environment view processing unit 210 may synthesize the subject image information received through the plurality of camera sensors 110 to generate a peripheral environment view. More specifically, the surrounding environment view processing unit 210 can receive an image (included in the subject image information) of the corresponding area from each of the plurality of camera sensors 110, and can superimpose the subject or background in each image May be considered. Here, the main subject may correspond to a target or an obstacle preset by the user, and may include, for example, a pedestrian, a vehicle, and a cycle list around the mobile 1. [

The distance information processing unit 220 may receive the distance information through the distance measuring sensor 120. Here, the distance information may correspond to an absolute distance to at least one main subject with respect to the moving object 1. For example, the distance measurement sensor 120 may be implemented as a Lidar sensor, and the Lidar sensor can measure the absolute distance by measuring the time reflected from the object within the measurement range by emitting a laser pulse signal . Here, the Lidar sensor can be replaced by a radar sensor or an ultrasonic sensor.

In one embodiment, the distance information processing unit 220 may be operated independently of the surrounding view processing unit 210, and the surrounding view processing unit 230 may fuse the subject image information and the distance information. In another embodiment, the distance information processing unit 220 may receive the direction of the main object from the surrounding environment view processing unit 210 in advance, measure the absolute distance of the main object in the direction, and transmit it to the surround view processing unit 230 ).

The surround view processing unit 230 may generate the surround view by fusing the view of the surrounding environment received from the surrounding environment view processing unit 210 and the distance information received from the distance information processing unit 210. [

In one embodiment, the surround view processing unit 230 may measure a direction of a main subject positioned from the moving object 1 by analyzing the surrounding environment view generated by the surrounding view processing unit 210. For example, the direction may correspond to the front surface of the moving body 1. [ Also, in one embodiment, the surround view processing unit 230 can measure the absolute distance of the main subject from the distance information received by the distance information processing unit 220. [ For example, the absolute distance may correspond to 5 meters from the moving object 1. Finally, the surround view processing unit 230 can generate the surround view by fusing the direction and absolute distance information of each object in the surrounding environment.

The projection model database 240 may correspond to a memory device that stores the ambient view projection model 20 defined by the user. The environment view projection model 20 defined herein may be selected by the surround view processing unit 230, may be determined according to the shape of the surrounding environment view, and may be distorted or deformed by the surround view processing unit 230 .

Fig. 4 is a view for explaining a surrounding view projection model in the projection model database shown in Fig. 1. Fig.

In Fig. 4, the ambient view projection model 20 may include a parabolic projection model, a dish projection model, or a cylindrical projection model.

The surround view processing unit 230 can select the surrounding view projection model 20 from the projection model database 240 according to the surrounding environment.

For example, as shown in FIG. 1A, when the moving object 1 is traveling in an open space such as an ordinary road or a highway, the surrounding view processing unit 230 may convert the surrounding environment view projection model 20 into a dish projection model You can choose.

As another example, as shown in FIG. 1B, when the moving object 1 moves on an alleyway including walls on both sides, the surrounding view processing unit 230 recognizes walls on both sides through object image information and distance information And the surrounding view projection model 20 selected based on the distance to the wall according to the recognition can be modified.

Hereinafter, an overview view generation process by the surround view processing unit 230 will be described in detail with reference to FIGS. 2 and 4. FIG.

Assuming that a wall is detected in the periphery of the moving object 1, the surround view processing unit 230 recognizes an obstacle (for example, a wall) having a vertical wall or a specific inclination while the moving object 1 moves The surrounding environment view projection model 20 can be partially modified. The surround view processing unit 230 may reflect the main subject to the modified surrounding view projection model 20 to generate the surrounding view.

Assuming that a plurality of main subjects are detected in the vicinity of the moving object 1, the surround view processing unit 230 determines the distance from the first main subject 10 to 10 and the distance from the second main subject 11 to 8, the first and second main subjects 10 and 11 are projected onto the surrounding view projection model 20 by reflecting the directions and the absolute distances of the first and second main subjects 10 and 11, respectively, .

In one embodiment, the first main subject 10 may correspond to the object farthest from the moving object 1, and the surround view processing unit 230 may determine the relative distance required in the projection process based on the farthest object . That is, the surround view processing unit 230 can overlay each main subject in the surrounding environment on the surrounding view projection model 20 by reflecting the respective directions and absolute distances.

The control unit 250 controls the overall operations of the plurality of camera sensors 110 and the distance measurement sensor 120 connected to the surround view generating apparatus 100. The surround monitoring unit 210 includes a surrounding view processor 210, ), The surround-view processing unit 230, and the projection model database 240, as shown in FIG.

FIG. 3 is a flowchart of an overview view generation process performed by the surround view generation apparatus of FIG. 1;

The surround view generating apparatus 100 may generate a surrounding view through the plurality of camera sensors 110 (step S310). The plurality of camera sensors 110 capture an image of each corresponding area to generate an image, and the surrounding environment view processing unit 210 synthesizes the object image information received through the plurality of camera sensors 110, Lt; / RTI >

The surround view generating apparatus 100 may receive the distance information for at least one main subject in the environment view through the distance measuring sensor 120 (step S320). The distance measuring sensor 120 may measure an absolute distance to at least one main subject in the surrounding environment. Here, the distance measurement sensor 120 may be implemented as an ultrasonic sensor, a lidar sensor, a radar sensor, and the distance information processing unit 220 may receive distance information through the distance measurement sensor 120.

The surrounding view generating apparatus 100 may generate an ambient view by fusing the surrounding environment view and the distance information to the predefined environment view projection model 20 (step S330). The surround view processing unit 230 may generate the surround view by fusing the view of the surrounding environment received from the surrounding view processor 210 and the distance information received from the distance information processing unit 210, The projection range of the surrounding environment view projection model 20 can be determined. Based on the projection range of the determined surrounding view projection model 20, the surround view processing unit 230 reflects the directions and absolute distances of at least one main subject and displays each subject in the surrounding environment in a surrounding view Can be overlaid on the projection model (20).

Finally, the surround view generating apparatus 100 can generate the surround view by fusing the information obtained using the heterogeneous sensor, which can be used, for example, in an apparatus such as an automobile or a robot. For example, the heterogeneous sensor may include a plurality of camera sensors 110 and a distance measurement sensor 120 such as a radar, a lidar, or an ultrasonic sensor. The surrounding view generating apparatus 100 can provide more accurate information on the surrounding environment view through the plurality of camera sensors 110 and the distance measuring sensor 120 so that the user can accurately recognize the surrounding environment.

It will be apparent to those skilled in the art that various modifications and variations can be made in the present invention without departing from the spirit and scope of the present invention as defined by the following claims It can be understood that

100: Around view generation device
110: a plurality of camera sensors
120: Distance measuring sensor
210: Ambient environment view processor
220: distance information processing unit 230:
240: Projection model database 250:
1: mobile body
10: first main subject 11: second main subject
20: Surrounding environment view projection model

Claims

A method of generating an ambient view, which is performed in an ambient view generating apparatus connected to a plurality of camera sensors and a distance measuring sensor,
(a) synthesizing object image information received through the plurality of camera sensors to generate a peripheral environment view;
(b) measuring an absolute distance to at least one main subject through the distance measurement sensor, and receiving distance information regarding at least one main subject in the perimeter view; And
(c) determining a projection range of the surrounding view projection model by measuring a distance of the object having the longest absolute distance, and based on the determined projection range, reflecting the position and direction of the at least one main subject, Generating a surrounding view by fusing the peripheral view and the distance information to a peripheral view projection model defined as one of the parabolic, dish or cylindrical projection models of each of the at least one main subject Way.

2. The method of claim 1, wherein step (c)
And measuring the direction of the main subject positioned on the moving object equipped with the surround view generating apparatus by analyzing the surrounding view and measuring the absolute distance of the measured main subject from the distance information. How to create a view.

delete

2. The method of claim 1, wherein step (c)
Distorting the surrounding environment view projection model based on the peripheral shape of the surrounding environment view, and reflecting the distance information to the distorted surrounding view projection model.

An environment view processor for synthesizing object image information received through a plurality of camera sensors to generate a peripheral environment view;
A distance information processing unit for receiving distance information about at least one main subject in the environment view through a distance measuring sensor; And
Determining a projection range of the surrounding environment view projection model by measuring a distance of the object having the longest absolute distance and determining a projection range of the at least one main object based on the determined projection range, An ambient view processing unit that includes an ambient view processing unit that generates an ambient view by fusing the surrounding view and the distance information to a surrounding view view projection model defined as one of a main subject, a parabolic type, a dish type, or a cylindrical projection model Device.