KR101549165B1 - Apparatus and method for estimating pose of vehicle - Google Patents

Abstract

The present invention relates to a vehicle pose estimation device and a vehicle pose estimation method. The vehicle pose estimation device comprises: an image capturing device installed in a vehicle driving on a road and capturing lanes parallel to each other on the road to generate an image; an image analysis unit extracting the lanes from the image to calculate each gradient of the lane and a coordinate of a vanishing point where extension lines of each lane are intersect with each other; and a vehicle pose estimation unit calculating a direction vector of the lanes on the road based on the coordinate of the vanishing point and calculating the normal vector of the road based on an interval of the coordinate of the vanishing point and the direction vector and the lanes on the road to estimate a pose of the vehicle.

Description

[0001] APPARATUS AND METHOD FOR ESTIMATING POSE OF VEHICLE [0002]

The present invention relates to an apparatus and a method for estimating a posture of a vehicle on a road.

The present invention is a research conducted as part of the industrial core technology development project (IT convergence) of the Korea Industrial Technology Evaluation and Management Agency of the Ministry of Commerce, Industry and Energy (Project No. 201403250000, development of augmented reality information providing system for driver's vision centered vehicle for driving safety and convenience) ) And Future Creation Science Department of Information Technology Research Promotion Agency (ITU Project 201403020000, development of AUTOSAR-based internal and external communication platform and application technology for smart cars).

When driving a vehicle on the road, the driver must operate the vehicle while visually confirming whether the vehicle is correctly traveling between the lanes in the driver's seat. This is because, if the posture of the vehicle is inclined with respect to the lane or the position of the lane is close to a certain lane side, the possibility of collision with an adjacent vehicle may increase. By using GPS (Global Positioning System), the position of the vehicle can be measured on the road, and the posture of the vehicle can be estimated from the change of the vehicle position. However, since the GPS data has an error of about several meters, it is difficult to accurately estimate the posture and the position of the vehicle with respect to the lane depending on the GPS data.

An object of the present invention is to provide a vehicle posture estimation apparatus and method for estimating a posture of a vehicle using image information of lanes of a road.

Another object of the present invention is to provide a vehicle posture estimating apparatus and method which can efficiently estimate a posture of a vehicle based on image analysis without using special equipment.

The problems to be solved by the present invention are not limited to the above-mentioned problems. Other technical subjects not mentioned will be apparent to those skilled in the art from the description below.

A vehicle posture estimating apparatus according to an aspect of the present invention includes a video photographing apparatus installed in a vehicle running on a road and photographing parallel lanes on the road to generate an image; An image analyzer for extracting the lanes from the image and calculating coordinates of a slope of each of the lanes and a vanishing point where an extension line of each of the lanes crosses; And calculating a normal vector of the road based on the slope of the lanes in the image, the direction vector, and the interval of the lanes on the road based on the coordinates of the vanishing point, And a vehicle posture estimating section for estimating a posture of the vehicle.

Wherein the vehicle posture estimating unit comprises: a lane direction vector calculating unit for calculating a direction vector of the lanes on the road based on the coordinates of the vanishing point; And a road normal vector calculating unit for calculating a normal vector of the road based on the slope of the lanes in the image, the direction vector, and the interval between the lanes on the road.

The vehicle posture estimating unit may further include a coordinate system moving conversion relation calculating unit for calculating a moving conversion relationship between the coordinate system of the image photographing apparatus and the coordinate system of the lanes on the road.

The lane direction vector calculating unit may calculate the direction vector using the focal length of the image photographing apparatus, the center coordinates of the image, and the coordinates of the vanishing point.

The road normal vector calculating unit may calculate the normal vector according to the following equations (1) to (3)

[Equation 1]

[Equation 2]

[Equation 3]

M ₁ , m ₂ , and m ₃ represent slopes of three lanes in the image, a _x , a _y , and a _z represent a three-dimensional component of the direction vector, and w ₁ , w ₂ denotes an interval of three lanes on the _{road, d x, d y, d} z denotes a three-dimensional components of a unit vector perpendicular to the lane in a plan view on the road, b _x, b _y is Wherein the normal vector is a vector orthogonal to the direction vector and the unit vector, wherein the normal vector is a motion vector between the coordinate system of the image photographing device and the coordinate system of the lanes on the road.

According to another aspect of the present invention, there is provided an image analysis method for extracting the lanes from an image of parallel lanes on a road and calculating coordinates of a vanishing point at which the lanes of each of the lanes cross each other, ; A lane-direction vector calculating unit for calculating a direction vector of the lane on the road based on the coordinates of the vanishing point, and a normal vector calculating unit calculating a normal vector of the road based on the slope of the lane in the image, And a vehicle posture estimating unit that estimates a posture of the vehicle that captured the image according to the direction vector and the normal vector.

According to still another aspect of the present invention, there is provided a method of driving a vehicle, comprising the steps of: extracting the lanes from an image of parallel lanes on a road and calculating a slope of each of the lanes and a coordinate of a vanishing point at which an extension of each of the lanes intersects; Calculating a direction vector of the lanes on the road based on the coordinates of the vanishing point; Calculating a normal vector of the road based on an inclination of the lanes in the image, an interval between the direction vector and lanes on the road; And estimating a posture of the vehicle that has taken the image according to the direction vector and the normal vector.

The vehicle posture estimation method may further include a step of calculating a conversion relationship between a coordinate system of the image photographing apparatus that has captured the image and a coordinate system of the lanes on the road.

The calculating of the direction vector may calculate the direction vector using the focal length of the image photographing apparatus that has captured the image, the center coordinates of the image, and the coordinates of the vanishing point.

The step of calculating the normal vector may calculate the normal vector according to Equations (1) to (3).

According to another aspect of the present invention, there is provided a computer-readable recording medium having recorded thereon a program for executing the above vehicle orientation estimation method.

According to the embodiment of the present invention, the posture of the vehicle can be estimated using the image information of the lanes of the road.

In addition, according to the embodiment of the present invention, it is possible to efficiently estimate the posture of a vehicle based on image analysis without using special equipment.

The effects of the present invention are not limited to the effects described above. Unless stated, the effects will be apparent to those skilled in the art from the description and the accompanying drawings.

1 is a plan view showing a vehicle having a vehicle orientation estimating apparatus according to an embodiment of the present invention running on a road.
2 is a configuration diagram of a vehicle orientation estimation apparatus according to an embodiment of the present invention.
3 is a view for explaining a vehicle posture estimation method according to an embodiment of the present invention.
4 is a diagram for explaining a method of estimating a posture of a vehicle by analyzing a photographed image according to an embodiment of the present invention.
5 is a configuration diagram of a vehicle posture estimating unit that constitutes a vehicle posture estimating apparatus according to an embodiment of the present invention.

Other advantages and features of the present invention and methods of achieving them will be apparent by referring to the embodiments described hereinafter in detail with reference to the accompanying drawings. However, the present invention is not limited to the embodiments disclosed below, and the present invention is only defined by the scope of the claims. Although not defined, all terms (including technical or scientific terms) used herein have the same meaning as commonly accepted by the generic art in the prior art to which this invention belongs. A general description of known configurations may be omitted so as not to obscure the gist of the present invention. In the drawings of the present invention, the same reference numerals are used as many as possible for the same or corresponding configurations.

As used throughout this specification, the term " to " may refer to a hardware component, such as a software, FPGA or ASIC, unit that processes at least one function or operation. However, the term '~' is not meant to be limited to software or hardware. &Quot; to " may be configured to reside on an addressable storage medium and may be configured to play one or more processors. Thus, by way of example, 'parts' may refer to components such as software components, object-oriented software components, class components and task components, and processes, functions, , Subroutines, segments of program code, drivers, firmware, microcode, circuitry, data, databases, data structures, tables, arrays, and variables. The functions provided in the " part " may be provided separately from the plurality of components, or may be integrated with the functions of the other components.

1 is a plan view showing a vehicle having a vehicle orientation estimating apparatus according to an embodiment of the present invention running on a road. Referring to FIG. 1, a vehicle orientation estimation apparatus 100 according to an embodiment of the present invention is installed in a vehicle 10 running on a road 20. The vehicle attitude estimation apparatus 100 extracts lanes from the image of the lanes 21, 22, and 23 on the road 20 and calculates a slope of each of the lanes and a vanishing point ), Calculates a direction vector of the lane on the road based on the coordinates of the vanishing point, calculates a normal vector of the road based on the slope of the lanes in the image, the direction vector and the interval of the lanes on the road, Estimate the posture. According to the embodiment of the present invention, the posture of the vehicle can be estimated efficiently by analyzing the images of the lanes on the road without using special equipment.

2 is a configuration diagram of a vehicle orientation estimation apparatus according to an embodiment of the present invention. 1 and 2, a vehicle orientation estimation apparatus 100 according to an embodiment of the present invention includes a video photographing apparatus 120, an image analysis unit 140, a vehicle position estimation unit 160, and a memory 180, .

The image photographing apparatus 120 is installed in the vehicle 10 and photographs the parallel lanes 21, 22, and 23 on the road 20 to generate an image. The image capturing apparatus 120 is illustratively provided on the front side of the vehicle 10 and can photograph the lanes 21, 22, and 23 located in front of the vehicle 10. [ Unlike the image capturing apparatus 120, the image capturing apparatus 120 may photograph an image of the rear side or the side of the vehicle 10 to generate an image.

FIG. 3 is a view for explaining a vehicle posture estimation method according to an embodiment of the present invention. FIG. 4 is a flowchart illustrating a method of estimating a posture of a vehicle by analyzing a captured image according to an embodiment of the present invention. FIG. Referring to Figure 1 to Figure 4, the image analysis unit 140 to extract the in-lane in the captured image (IMG) (ℓ _1, ℓ _2, ℓ ₃₎ Lane (ℓ _1, ℓ _2, ℓ ₃₎ The coordinates of the vanishing point (v) at which the extension of each of the slopes and each of the lanes (l ₁ , l ₂ , l ₃ ) intersect is calculated.

The vehicle attitude estimating unit 160 calculates the direction vectors of the lanes 21, 22 and 23 on the road 20 based on the coordinates of the vanishing point v and calculates the direction vectors of the lanes l ₁ and l _2, ℓ ₃₎ of the slope and the direction vector and distance of the road 20. the lane (21,22,23) on the (w _1, w ₂₎ normal to the plane (Π) on the road based on the 20 vector And estimates the attitude of the vehicle 10.

The memory 180 may store a program for executing the vehicle posture estimation method according to an embodiment of the present invention and information necessary for vehicle posture estimation. The information necessary for vehicle posture estimation includes information such as the interval (w ₁ , w ₂ ) of the lanes 21, 22, 23, internal parameters such as the focal length of the image pickup apparatus (camera) .

5 is a configuration diagram of a vehicle posture estimating unit that constitutes a vehicle posture estimating apparatus according to an embodiment of the present invention. 5, the vehicle attitude estimating unit 160 includes a lane direction vector calculating unit 1622, a road normal vector calculating unit 1624, and a coordinate system moving conversion relation calculating unit 164. The lane direction vector calculating section 1622 and the road normal vector calculating section 1624 constitute the coordinate system rotational conversion relation calculating section 162. [

1 to 5, the image IMG obtained by photographing three parallel lanes 21, 22 and 23 existing on the plane? On the road 20 is analyzed to determine the posture of the vehicle 10 Will be described. The interval of the first lane 21 and the second lane (22) distance (w _1), the second lane (22) and the third lane (23) (w ₂₎ is defined as a value that is known in advance.

The lane (21,22,23) on the route (20) are each linear equation _{L 1: t a + b -w} 1 d, L 2: t a + b, L 3: t a + b + w 2 d ( t is a real number). _{a = (a x, a y} , a z) is a direction vector of the lane (21,22,23), the unit vector _{d = (d x, d y} , d z) is a plane (Π on the road 20 ), And is a vector perpendicular to the direction vector a . In the illustrated example, b is defined as the intersection of the XY plane of the coordinate system of the image photographing apparatus 120 and the second lane 22 on the road 20. That is, the linear equation L ₂ of the second lane 22 passes the point b = (b _x , b _y , 0). In the linear equations L ₁ , L ₂ , L ₃ , the relationship between the components is expressed by eliminating t, which can be expressed by the following equations (1) to (3).

[Formula 1]

[Formula 2]

[Formula 3]

Formula 1 in to Equation _{3, a x, a y,} a z is the lane on the plane (Π) on a three-dimensional components of the direction vector a, the road _{(20) d x, d y} , d z (21, B _x and b _y are the three-dimensional components of the unit vector d perpendicular to the coordinate system of the image capturing apparatus 120 and the coordinate systems of the lanes 21, 22 and 23 on the road 20 X, y coordinate transformation value indicating the relationship.

The vector n = (n _x , n _y , n _z ) is the normal vector of the plane Π containing the lanes L ₁ , L ₂ , L ₃ on the road 20. The normal vector is a vector perpendicular to both the direction vector a and the unit vector d . That is, the normal vector d = (d _x , d _y , d _z ) satisfies the condition of n × a = d . The vectors a , n , and d satisfy the condition | a | = | n | = | d | = 1.

The image analyzing unit 140 extracts straight lines l ₁ , l ₂ and l ₃ that are images of lanes in the image IMG in which the lanes 21, 22 and 23 on the road 20 are projected, (M ₁ , m ₂ , m ₃ ) of the straight lines (l ₁ , l ₂ , l ₃ ) representing the lanes in the image IMG can be calculated.

When the angles between the lanes (ℓ ₁ , ℓ ₂ , ℓ ₃ ) and the X axis in the image IMG are θ ₁ , θ ₂ and θ ₃ , the lane (ℓ ₁ , ℓ ₂ , ℓ ₃₎ gradient _{(m 1, m 2, m} 3) is the _{tanθ 1, tanθ 2, tanθ 3} . (x, y) on the lanes l ₁ , l ₂ , l _{3 in the} image IMG is defined as m _i = tan _i (i = ), Each straight line (ℓ ₁ , ℓ ₂ , ℓ ₃ ) corresponding to the lane can be expressed by the following Equation (4).

[Formula 4]

At this time, the straight lines representing all the lanes (l ₁ , l ₂ , l ₃ ) in the image IMG have different slopes and the extension lines of the lanes l ₁ , l ₂ and l ₃ meet at the vanishing point v . That is, the coordinates (v _x , v _y ) of the vanishing point v can be calculated by finding the intersection of the extension lines of the straight lines (ℓ ₁ , ℓ ₂ , ℓ ₃ ) in the image IMG.

Assuming that skew does not occur in the image capturing apparatus 120, the focal distance of the image capturing apparatus 120 may be represented by f _x = f _y = f, and the inner matrix of the image capturing apparatus 120 may be represented by (5) View the MathML source

[Formula 5]

Here, f is the focal length, and c _x and c _y are the coordinates of the center of the image. If the point (X, Y, Z) projected on the three-dimensional straight line using the projection matrix K is (x, y), the following equation (6) holds.

[Formula 6]

[Equation 7]

Taking the third lane (L ₃ ) as an example, the equation ( ₃ ) can be summarized with respect to X and Y as shown in the following equations (8) to (9).

[Equation 8]

[Equation 9]

Substituting Equations 8 to 9 into equations 6 to 7 described above, the equations 10 to 11 can be summarized as follows.

[Equation 10]

[Equation 11]

When the equation 10 is divided by the equation 11 and then the condition of b _z = 0 is applied, the following equation 12 can be derived.

[Equation 12]

Here, the relational expression of Expression 12 represents a straight line ₃ obtained by projecting the third lane L ₃ on the image (IMG) plane. A comparison of Equation 4 with Equation 13 to Equation 14 can be made.

[Formula 13]

[Equation 14]

Lane direction vector calculating unit 1622 to the image taking device a focal length (f), and the coordinates of the center of the image (c _x, c _y) and the coordinates of the vanishing point (v _x, v _y) according to Equation 13 to Equation 14 can be calculated (= 1 | | a) utilized, the direction vector a = the lane of (21,22,23) on a road (20) (a _x, a _y, a _z).

If the first lane L ₁ and the second lane L ₂ are summarized in the same manner as described above, the following relations (15) to (17) can be obtained.

[Formula 15]

[Formula 16]

[Formula 17]

The road normal vector calculating unit 1624 calculates the slope m ₁ , m ₂ and m ₃ of the three lanes l ₁ , l ₂ and l ₃ in the image IMG according to equations 15 to 17, vector _{a = (a x, a y} , a z) and the road (20) units is based on the distance (w _1, w ₂₎ of the lane (21,22,23) on the vector d = (d _x, d _y , _dz ) (| d | = 1, a · d = 0), and the normal vector n of the road perpendicular to both the direction vector and the unit vector can be calculated.

If the variables A, B, and C are defined as shown in the following Equation 18, Equations 15 to 17 can be summarized as Equations 19 to 21 below.

[Formula 18]

[Formula 19]

[Formula 20]

[Formula 21]

A is divided into B and summarized as shown in Equation 22 below.

[Formula 22]

Since the condition of nxa = d and a · d = 0 is established, the following equation (23) can be obtained.

[Equation 23]

을 계산할 수 있으므로, 도로 법선벡터 산출부(1624)는 계산된

를 식 22, 식 23에 적용하여 정리함으로써,

를 계산할 수 있다. ｜a｜=｜d｜= 1 의 조건으로부터 방향 벡터 a = (a_x,a_y,a_z), 단위 벡터 d = (d_x,d_y,d_z) 를 결정할 수 있으며, a × d = n 의 조건으로부터 법선 벡터 n 을 산출할 수 있다.Using the vanishing point (v) and the inner matrix found in the image (IMG)

The road normal vector calculation unit 1624 calculates the road normal vector

Is applied to Equation (22) and Equation (23)

Can be calculated. | A | = | d | = direction vector a = from condition _{1 (a x, a y,} a z), the unit vector d = can determine the _{(d x, d y, d} z), a × d = the normal vector n can be calculated from the condition of n .

After the a, d, n vector is determined, the calculation unit 164 coordinate movement conversion relationship is to calculate the b _x are summarized A in Equation 18, by calculating the b _y by substituting it to C, the image taking device (120 ) it is possible to determine the movement conversion relationship between the coordinate system for the b plane (Π) on the coordinate system and a road 20 in the.

The transformation relation [R | t] can be represented in the coordinate system of the image capturing apparatus 120 in the coordinate system having the vector d , n , and a as three axes, with the coordinate b as a center, The rotational transformation relation R can be estimated as ( d n a ) and the motion transformation relation t as b . Finally, the intersection between the extension line of any one of the lanes and the XY plane of the image capturing apparatus coordinate system is used as the center of the lane direction (direction vector), the road surface direction (normal vector of the road surface plane) And the posture of the vehicle 10 can be estimated based on the conversion relationship between the coordinate system and the image pickup device.

On the other hand, if the attitude of the vehicle 10 is estimated, the attitude of the vehicle 10 can be notified to the driver at regular intervals through a speaker (not shown) or a display (not shown) The warning signal such as a warning sound or a warning light may be generated through an alarm generating unit (not shown).

The vehicle attitude estimation method according to an embodiment of the present invention can be realized in a general-purpose digital computer which can be formed into a program that can be executed by a computer, for example, and which operates the program using a computer-readable recording medium. The computer readable recording medium may be a volatile memory such as SRAM (Static RAM), DRAM (Dynamic RAM), SDRAM (Synchronous DRAM), ROM (Read Only Memory), PROM (Programmable ROM), EPROM A non-volatile memory such as an electrically erasable programmable read-only memory (EEPROM), a flash memory device, a phase-change RAM (PRAM), a magnetic RAM (MRAM), a resistive RAM (RRAM) Or optical storage media, such as, but not limited to, CD-ROMs, DVDs, and the like.

It is to be understood that the above-described embodiments are provided to facilitate understanding of the present invention, and do not limit the scope of the present invention, and it is to be understood that various modifications are possible within the scope of the present invention. It is to be understood that the technical scope of the present invention should be determined by the technical idea of the claims and the technical scope of protection of the present invention is not limited to the literary description of the claims, The invention of the present invention.

10: Vehicle
20: Road
21, 22, 23:
100: vehicle posture estimating device
120:
140: Image analysis section
160:
162: coordinate system rotation conversion relation calculating unit
1622: lane direction vector calculating section
1624: Road normal vector calculation unit
164: coordinate system moving conversion relation calculating unit
180: Memory

Claims (13)

A video photographing device installed in a vehicle traveling on a road and photographing parallel lanes on the road to generate an image;
An image analyzer for extracting the lanes from the image and calculating coordinates of a slope of each of the lanes and a vanishing point where an extension line of each of the lanes crosses; And
Calculating a normal vector of the road based on the coordinates of the vanishing point, the direction vector, and the interval between the lanes on the road based on the coordinates of the vanishing point, And estimates a vehicle posture of the vehicle. The method according to claim 1,
Wherein the vehicle-
A lane direction vector calculating unit for calculating a direction vector of the lanes on the road based on the coordinates of the vanishing point; And
And a road normal vector calculating unit for calculating a normal vector of the road based on the slope of the lanes in the image, the direction vector, and the interval between the lanes on the road. 3. The method of claim 2,
Wherein the vehicle posture estimating section further includes a coordinate system movement conversion relation calculating section for calculating a movement conversion relationship between a coordinate system of the image photographing device and a coordinate system of lanes on the road. 3. The method of claim 2,
Wherein the lane direction vector calculating unit calculates the direction vector using the focal length of the image photographing apparatus, the center coordinates of the image, and the coordinates of the vanishing point. 5. The method according to any one of claims 2 to 4,
The road normal vector calculating unit calculates the normal vector according to the following equations (1) to (3)
[Equation 1]

[Equation 2]

[Equation 3]

M ₁ , m ₂ , and m ₃ represent slopes of three lanes in the image, a _x , a _y , and a _z represent a three-dimensional component of the direction vector, and w ₁ , w ₂ denotes an interval of three lanes on the _{road, d x, d y, d} z denotes a three-dimensional components of a unit vector perpendicular to the lane in a plan view on the road, b _x, b _y is Wherein the normal vector represents a conversion relation between a coordinate system of the image photographing apparatus and a coordinate system of lanes on the road, and the normal vector is a vector perpendicular to the direction vector and the unit vector. An image analyzer for extracting the lanes from an image of parallel lanes on the road and calculating coordinates of a slope of each of the lanes and an intersection point of extension lines of the lanes; And
A lane direction vector calculating unit for calculating a direction vector of the lane on the road based on the coordinates of the vanishing point and a normal vector of the road based on the slope of the lane in the image and the interval between the direction vector and the lane on the road And a vehicle posture estimating unit that estimates a posture of the vehicle that captured the image in accordance with the direction vector and the normal vector. The method according to claim 6,
Wherein the vehicle posture estimating unit further includes a coordinate system movement conversion relation calculating unit for calculating a movement conversion relation between a coordinate system of the image photographing apparatus that photographs the image and a coordinate system of the lanes on the road. The method according to claim 6,
Wherein the lane direction vector calculating unit calculates the direction vector using the focal length of the image photographing apparatus, the center coordinates of the image, and the coordinates of the vanishing point. Extracting the lanes from the image of the parallel lanes on the road and calculating the slope of each of the lanes and the coordinates of the vanishing point at which the extension of each of the lanes intersects; And
Calculating a direction vector of the lanes on the road based on the coordinates of the vanishing point;
Calculating a normal vector of the road based on an inclination of the lanes in the image, an interval between the direction vector and lanes on the road; And
And estimating a posture of the vehicle that photographed the image according to the direction vector and the normal vector. 10. The method of claim 9,
Wherein the vehicle posture estimating method further includes calculating a conversion relationship between a coordinate system of the image photographing apparatus that photographs the image and a coordinate system of the lanes on the road. 10. The method of claim 9,
Wherein the calculating of the direction vector calculates the direction vector using the focal length of the image photographing apparatus that has captured the image, the center coordinates of the image, and the coordinates of the vanishing point. 10. The method of claim 9,
The step of calculating the normal vector may include calculating the normal vector according to the following expressions (1) to (3)
[Equation 1]

[Equation 2]

[Equation 3]

M ₁ , m ₂ , and m ₃ represent slopes of three lanes in the image, a _x , a _y , and a _z represent a three-dimensional component of the direction vector, and w ₁ , w ₂ denotes an interval of three lanes on the _{road, d x, d y, d} z denotes a three-dimensional components of a unit vector perpendicular to the lane in a plan view on the road, b _x, b _y is Wherein the normal vector represents a conversion relationship between a coordinate system of the image photographing apparatus and a coordinate system of lanes on the road, and the normal vector is a vector perpendicular to the direction vector and the unit vector. A computer-readable recording medium having recorded thereon a program for executing the vehicle position estimation method according to any one of claims 9 to 12.

Images