JP6787758B2 - Light beam recognition device - Google Patents

Description

The present disclosure relates to a technique for detecting a beam of light, which is a streak of light reflected in an image acquired from an optical camera.

There is known a technique related to an object detection device that captures an image of the front of a vehicle with an optical camera or the like and detects an object existing in front of the vehicle.
However, when there is a light source in front of the vehicle, there is a possibility that the object detection device erroneously detects the light beam generated by the light source as an object. In particular, the light beam is generated when the unwiped water droplets adhering to the windshield of the vehicle scatter the light from the light source as a cylindrical lens.

Patent Document 1 discloses a technique for suppressing false detection of a light beam as an object. That is, the light source in the image captured by an optical camera or the like is detected, and the detected light source is approximated to an ellipse. The light beam is detected by recognizing a point with high brightness existing in the direction linearly extended in the long side direction of the approximated ellipse as a light beam.

U.S. Patent Application Publication No. 2015/0161469

However, the light beam recognition device of Patent Document 1 has a problem that the light beam cannot be detected correctly in the following situations, for example.
・ Situation where the direction in which the light beam appears and the long side direction when the light source is approximated to an ellipse do not match ・ Situation where the light beam is curved instead of straight line ・ There is a low brightness part between the light source and the light source, and the light source Situations where the light beams are separated The purpose of the present disclosure is to provide a technique for improving the detection accuracy of the light rays even in the situations illustrated above.

The light beam recognition device (12), which is one aspect of the present disclosure, includes an image acquisition unit, a light source detection unit (S100), a maximum detection unit (S330), a point detection unit, and a generation unit (S370). The image acquisition unit acquires an image from the optical camera (11) and calculates the pixel brightness, which is the brightness of each pixel, with respect to the acquired image. The light source detection unit detects light source pixels whose pixel brightness calculated by the image acquisition unit is equal to or higher than the light source threshold value which is a preset brightness threshold value. The maximum detection unit detects pixels whose pixel brightness calculated by the image acquisition unit satisfies the maximum condition, which is a preset brightness condition, as the maximum pixel. The point detection unit detects one or more light beam pixels having different distances from the light source pixels and indicating light rays from the light source pixels from among the maximum pixels detected by the maximum detection unit. The generation unit generates a polygonal line in which the light source pixels detected by the light source detection unit and the light beam pixels detected by the point detection unit are connected in a straight line in order from the one closest to the light source pixel as a light beam straight line. Further, the point detection unit includes a range setting unit (S320), an extraction unit (S350), and an update unit (S360), and operates the point detection unit using the prediction straight line updated by the update unit. Is repeated. Range setting unit, from the latest shaft of light pixels detected by the detected light source pixels or points detecting section at the light source detector, apart preset unit distance above, and, located a preset predicted straight line The detection range centered on the predicted pixel is set with the pixel to be predicted as the predicted pixel. The extraction unit extracts the maximum pixels included in the detection range set by the range setting unit, and uses the extracted maximum pixels as the detection result of the light beam pixels. The update unit updates the prediction straight line based on the amount of deviation between the light beam pixels extracted by the extraction unit and the prediction pixel.

According to such a configuration, it is possible to improve the detection accuracy of the light beam.

It is a block diagram which shows the structure of a vehicle control system. It is a flowchart of a light beam recognition process. It is a flowchart of a light source detection process. It is a figure which shows the circumference of a selected pixel. It is a flowchart of a light beam detection process. It is a figure which showed the state which the light beam detection processing is performed.

Hereinafter, embodiments of the present disclosure will be described with reference to the drawings.
[1. Embodiment]
[1-1. Constitution]
The vehicle control system 1 shown in FIG. 1 is provided in a vehicle and includes an optical camera 11, a light beam recognition device 12, an object recognition device 13, and a vehicle control device 14. Hereinafter, the vehicle provided with the vehicle control system 1 will be referred to as the own vehicle.

The optical camera 11 takes an image of the front of the own vehicle.
The light beam recognition device 12 recognizes the light beam existing in front of the own vehicle based on the image captured by the optical camera 11.

The optical beam recognition device 12 is mainly composed of a well-known microcomputer having a CPU 121 and semiconductors such as RAM, ROM, and flash memory. Various functions of the optical beam recognition device 12 are realized by the CPU 121 executing a program stored in a non-transitional substantive recording medium. In this example, the memory 122 corresponds to a non-transitional substantive recording medium in which a program is stored. Moreover, when this program is executed, the method corresponding to the program is executed. The number of microcomputers constituting the optical beam recognition device 12 may be one or a plurality.

The light beam recognition device 12 realizes the light beam recognition process described later by executing a program by the CPU 121. The method for realizing various functions of the light beam recognition device 12 is not limited to software, and a part or all of the elements may be realized by using one or more hardware. For example, when the above function is realized by an electronic circuit which is hardware, the electronic circuit may be realized by a digital circuit including a large number of logic circuits, an analog circuit, or a combination thereof.

The object recognition device 13 recognizes an object existing in the traveling direction of the own vehicle. That is, the object included in the image is recognized based on the image acquired from the optical camera 11 via the light beam recognition device 12. Specifically, the object recognition device 13 detects the difference in brightness and chromaticity from the acquired image, recognizes the object existing in front of the own vehicle, and predicts whether or not the recognized object collides with the own vehicle. To do. The operation of the object recognition device 13 is known, and detailed description thereof will be omitted.

The vehicle control device 14 instructs each ECU, which is an electronic control device that controls the operation of the own vehicle, to control the own vehicle based on the prediction by the object recognition device 13 whether or not the vehicle collides with an object in front of the vehicle. To send. Here, each ECU performs steering control and braking control of the own vehicle according to an instruction from the vehicle control device 14. The operation of the vehicle control device 14 is known and detailed description thereof will be omitted.

[1-2. processing]
Next, the light beam recognition process executed by the light beam recognition device 12 will be described with reference to the flowchart of FIG. The light beam recognition process is executed when an image is captured by the optical camera 11 and the light beam recognition device 12 acquires the image. Further, at the stage of acquiring the image, the pixel brightness, which is the brightness of each pixel in the image, is acquired. The arrangement of pixels in the left-right direction of the image acquired here is a row, and the arrangement of pixels in the up-down direction is a column.

In S100, the light beam recognition device 12 detects a light source region, which is a region in which a light source is reflected, and executes a light source detection process for storing the position of the light source region.
The light beam recognition device 12 repeats the processes from S200 to S400, which will be described later. Here, the process repeated from S200 to S400 is referred to as a light beam recognition cycle.

The light beam recognition device 12 in S200 selects a light source region to be processed in the current light beam recognition cycle from the light source regions detected in S100. As the light source region selected at this time, a light source region not selected in the light beam recognition cycle performed before the current light beam recognition cycle is selected.

In S300, the light beam recognition device 12 executes a light beam detection process for detecting the light beam extending from the light source region selected in S200. The light beam detection process is executed using a Kalman filter.
In S400, the light beam recognition device 12 determines whether or not there is a light source region that has not been selected before the current light beam recognition cycle.

When the light source region not selected in S400 exists, the light beam recognition device 12 returns to S200 and performs the subsequent processing.
When the light beam recognition device 12 does not have a light source region not selected in S400, the light beam recognition device 12 in S500 captures the light beam straight line detected by the processes from S100 to S400 in the image acquired from the optical camera. Output is performed to the object recognition device 13 so as to suppress recognition of the existing pixel and its surroundings as an object. The term “periphery” as used herein refers to a range in which the expected light beam spreads with respect to the pixels in which the light beam straight line is projected.

The process of acquiring an image corresponds to the image acquisition unit, the process of S100 corresponds to the light source detection unit, and the process of S500 corresponds to the recognition suppression unit.
<Light source detection process>
Here, the details of the light source detection process executed by the light beam recognition device 12 in S100 will be described with reference to the flowchart shown in FIG.

The light beam recognition device 12 repeats the processes from S110 to S160 described later. Here, the process repeated from S110 to S160 is referred to as a light source detection cycle.
In S110, the light beam recognition device 12 selects pixels in the acquired image. In this pixel selection, the pixel in the upper left corner of the acquired image is selected for the first time, and in the second and subsequent light source detection cycles, the pixel on the right side of the same row as the pixel selected in the previous light source detection cycle is selected. If the pixel to the right does not exist, the leftmost pixel in the row one row below the pixel selected in the light source detection cycle before the light source detection cycle is selected. Hereinafter, the selected pixel will be referred to as a selected pixel.

In S120, the light beam recognition device 12 determines whether or not the selected pixel selected in S110 is a light source pixel. The light source pixel referred to here is a pixel in which the pixel brightness of the selected pixel is equal to or higher than the light source threshold value which is the threshold value of the brightness preset by the light beam recognition device 12.

When the light beam recognition device 12 determines in S120 that the selected pixel is a light source pixel, the light beam recognition device 12 in S130 determines whether or not the detection light source pixel exists around the selected pixel. As shown by the solid line in FIG. 4, the periphery of the selected pixel here means a range within 1 pixel in each of the row direction and the column direction centering on the selected pixel, and the detection light source pixel is the current light source detection. A light source pixel detected in a light source detection cycle prior to the cycle.

When the light beam recognition device 12 determines in S130 that the detection light source pixel does not exist around the selected pixel, the process shifts to S140.
In S140, the light beam recognition device 12 stores the position of the selected pixel as the position of a new light source region in the memory 122, and shifts the processing to S160.

When the light beam recognition device 12 determines in S130 that the detection light source pixel exists around the selected pixel, the process shifts to S150.
In S150, the light beam recognition device 12 determines the position of the light source region to which the detection light source pixel belongs, assuming that the selected pixel is a light source pixel belonging to the same light source region as the detection light source pixel. Here, the position of the light source region obtained is the center position of the light source belonging to the same light source region. The center position referred to here may be the position of the center of gravity of a light source belonging to the same light source region. Then, the position of the obtained light source area is updated with the position of the detection light source stored in the memory 122, and the process shifts to S160.

When the light beam recognition device 12 determines in S120 that the selected pixel is not a light source pixel, whether or not there is a pixel in the image acquired in S160 that has not been selected in the previous light source detection cycle. It is judged.

If it is determined in S160 that a pixel not selected exists, the process returns to S110 and the subsequent processing is repeated.
By performing the light source detection process shown above, the light beam recognition device 12 detects the light source region, which is a region having pixels having pixel brightness equal to or higher than the luminance threshold reflected in the image acquired from the optical camera 11, and the light source. The position of the area can be memorized.

<Kalman filter>
Next, a general Kalman filter used in the light beam detection process will be described.
Here, the pixel existing at the upper left corner of the image is defined as the origin of coordinates (x, y) = (0,0), the position in the row direction on the image is x, and the position in the column direction is y. The straight line on the image of this time shall be expressed in x = ay + b, the vector gradient a of the straight line, the x-intercept b is the intersection of the x-axis and the element and the state vector s _t.

Hereinafter, t is the time, P _t is the covariance vector, K _t is the Kalman filter gain, H _t is the observation matrix (y, 1), q is the prediction error, and r is the observation error. The state vector s _t and the system model is expressed by the following equation (1) (3).

The system noise vs. the observed noise vz follow the normal distributions N (0, q) and N (0, r), respectively.
In the following, in state estimation, estimating the future state is referred to as prediction, and estimating the current state is referred to as filtering.

Then, the prediction vector, which is a state vector representing the predicted state at time t, is s _{t + 1} , the covariance matrix is P _{t | t-1,} and the estimation vector is a state vector representing the filtered state at time t. Is s _t , the covariance matrix is P _{t | t} , and the Kalman gain at time t is K _{t. As} is well known, it is expressed by equations (4) to (7).

<Light beam detection process>
Here, the details of the light beam detection process executed by the light beam recognition device 12 in S300 will be described with reference to the flowchart shown in FIG. 5, and the state of the process in each step is shown in FIG. Hereinafter, the line defined by parameters representing the state vector s _t of the Kalman filter that the expected line S _t.

Shaft of light recognition apparatus 12 performs initial setting of the expected line _{S 0} in S310. The specific prediction straight line S ₀ is shown below. That is, the coordinates of the position z ₀ of the light source region A selected by the light beam recognition device 12 in S200 are set to (x, y) = (x ₀ , y ₀ ). The predicted straight line S ₀ is a straight line extending in the column direction of the image from the position of the light source region A selected by the light beam recognition device 12 in S200. That is, the predicted straight line S ₀ is a straight line defined by the state vector s ₀ = (0, x ₀ ) ^T.

The light beam recognition device 12 repeats the processes from S320 to S360 described later. Here, the process repeated from S320 to S360 is referred to as a light beam detection cycle. It is assumed that the first light beam detection cycle is t = 1, and t is incremented by 1 each time the light beam detection cycle is repeated.

Shaft of light recognition device 12 in S320 performs the setting of the detection range _{C t.} Detection range C _t here is, in the row that belongs predicted pixel B _t, is set a predetermined range around the predicted pixel B _t, is defined by the expression (2) v _s. The term prediction pixel B _t is that the pixels that intersect the unit distance Δy between rows apart in the column direction and the expected line S _t is the distance light streak pixel z _t-1 is predetermined from belongs row. The position of the predicted pixel B _t referred to here is obtained by H _t × _st-1 which is a part of the equation (6). Further, the unit distance Δy is set to be larger than the distance assumed when the light source and the light beam are separated from each other, and is, for example, a distance of 5 pixels. Further, as the y coordinate of the predicted pixel B ₁ , y = y ₀ + Δy holds using the y coordinate y ₀ of the light source pixel. Further, also in the subsequent cycles, y = y ₀ + Δy × t holds as the y coordinate of the predicted pixel B _t using the y coordinate y ₀ of the light source pixel.

Shaft of light recognition device 12 in S330, in the detection range _{C t} set in S320, detects a maximum pixel. The maximum pixel referred to here is a pixel that satisfies the maximum condition, which is a preset brightness condition. Specifically, the maximum condition is to take the maximum brightness in the preset range of each pixel and to take the brightness equal to or higher than the light beam threshold which is the preset brightness threshold. Here, the predetermined range of each pixel means a range of 3 pixels on the left and right of each pixel. Further, the light beam threshold value may be set smaller than the light source threshold value.

S340 in shaft of light recognition device 12, the detection range _{C t} set in S320, determines whether or not the maximum pixel exists. If it is determined that the maximum pixel exists at S340, shaft of light recognition device 12 in S350 extracts the maximum pixel present in the detection range, to set a light streak pixel z _t based on the extracted maximum pixel. When there is one maximum pixel extracted here, the maximum pixel is set as a light beam pixel. On the other hand, the extracted maximum pixel when there are a plurality of pixels present at a position obtained by averaging the respective maxima between pixels is set as the shaft of light pixels z _t.

Next, in S360, shaft of light recognition device 12 updates the expected line _{S t.} The updating of the expected line S _t here is calculated based on the distance between the prediction pixel B _t and light streak pixel z _t shift amount D _t. Here, the deviation amount D _t corresponds to (z _t −H _t × s _t-1 ), which is the second term on the right side of the equation (6). The updating of the expected line _{S t} is (6) correspond. That is, by adding the multiplied by the Kalman gain _{K t} sought (5) to the displacement amount _{D t} predicted pixel _{B t} and shaft of light pixels _{z t} to the prediction vector _{s t-1,} the prediction vector _{s t} calculate. Thus, by updating the prediction vector s _t, it becomes to be updated expected line S _t specified by the updated predicted vector s _t.

When it is determined in S340 that the maximum pixel does not exist, the light beam recognition device 12 generates a light beam straight line in S370 and ends the process. The light beam straight line E ₁ referred to here starts from the position z ₀ of the light source region A selected by the light beam recognition device 12 in S200, performs a light beam detection cycle, and connects the light beam points set in S350 with a straight line in the set order. Is generated by. That is, the position z ₀ of the light source region A and the light beam point z ₁ are connected by a straight line, the light beam point z ₁ and the light beam point z ₂ are connected, and so on, the light source and the light beam point are connected by a straight line in a set order. This sets the light beam straight line.

By performing the light source detection process shown above, the light beam extending from the light source selected in S200 is detected.
The processing in S320 corresponds to the range setting unit, the processing in S330 corresponds to the maximum detection unit, the processing in S350 corresponds to the extraction unit, the processing in S360 corresponds to the update unit, and the processing in S370 corresponds to the update unit. Process corresponds to the generation unit.

[1-3. effect]
According to the embodiment described in detail above, the following effects are obtained.
(1a) According to the light beam recognition device of the present embodiment, the direction in which the light beam appears and the light source region are approximated to an ellipse because it is determined whether or not there is a light beam in the pixels around the light source region regardless of the shape of the light source region. It is possible to detect the light beam even in a situation where the long side direction does not match, and it is possible to provide a technique for improving the detection accuracy of the light beam.

(1b) According to the light beam recognition device 12 of the present embodiment, a polygonal line formed by connecting the maximum pixel and the center of the surrounding light source or the light beam pixel with a straight line is recognized as the light beam. Therefore, the light beam can be recognized even in a situation where the entire light beam is curved instead of a straight line. This makes it possible to provide a technique for improving the detection accuracy of the light beam.

(1c) Further, according to the light beam recognition device 12 of the present embodiment, the unit distance Δy is set to be larger than the distance between the assumed light source and the light beam. As a result, the light beam can be recognized even in a situation where the light source region and the light beam are separated from each other, where there is an assumed low-luminance portion between the light source region and the light beam. This makes it possible to provide a technique for improving the detection accuracy of the light beam.

[2. Other embodiments]
Although the embodiments of the present disclosure have been described above, the present disclosure is not limited to the above-described embodiments, and can be implemented in various modifications.

(2a) In the above embodiment, when a plurality of maximum pixels are present, the average position of each maximum pixel is set as the light beam position, but the setting of the light beam position is not limited to this. For example, it may be the pixel having the highest brightness among the plurality of maximum pixels, or the pixel closest to the predicted pixel. Further, the Kalman filter may be used to set the light beam position. That is, the position of the light beam may be set by updating the positions of the plurality of maximum pixels extracted with respect to the predicted pixel B _t using a Kalman filter. In other words, the straight line represented by the state vector _st-1 may be set as the predicted straight line _St as the straight line updated by using the Kalman filter from the positional relationship with the detected light beam pixels. Specifically, the predicted straight line S ₁ may be a straight line obtained by updating the initially set state vector s ₀ with the position z _{1 of the} light beam pixel by using a Kalman filter.

(2b) In the above embodiment, the maximum condition is to take the maximum brightness in the preset pixel range and to take the brightness equal to or higher than the light beam threshold which is the preset brightness threshold. did. However, the maximum condition is not limited to this. For example, the maximum condition may be a pixel that becomes the maximum within a preset pixel range. Further, the preset pixel range is not limited to the left and right 3 pixel range, but may be 1 pixel range or may be set to the upper and lower range.

(2c) In the above embodiment, the maximum pixel is detected in S330, but the timing at which the processing in S330 is performed is not limited to this. For example, it may be performed before the light source detection cycle after the processing of S100.

(2d) A plurality of functions possessed by one component in the above embodiment may be realized by a plurality of components, or one function possessed by one component may be realized by a plurality of components. .. Further, a plurality of functions possessed by the plurality of components may be realized by one component, or one function realized by the plurality of components may be realized by one component. Further, a part of the configuration of the above embodiment may be omitted. In addition, at least a part of the configuration of the above embodiment may be added or replaced with the configuration of the other above embodiment. It should be noted that all aspects included in the technical idea specified from the wording described in the claims are embodiments of the present disclosure.

(2e) In addition to the above-mentioned light beam recognition device, a system having the light beam recognition device as a component, a program for operating a computer as the light beam recognition device, a non-transitional substantive record of a semiconductor memory or the like in which this program is recorded The present disclosure can also be realized in various forms such as a medium and a light beam recognition method.

1 ... Vehicle control system, 11 ... Optical camera, 12 ... Light beam recognition device, 13 ... Object recognition device, 14 ... Vehicle control device, 121 ... CPU, 122 ... Memory.

Claims (3)

An image acquisition unit configured to acquire an image from the optical camera (11) and calculate the pixel brightness, which is the brightness of each pixel, with respect to the acquired image.
A light source detection unit (S100) configured to detect a light source pixel whose pixel brightness calculated by the image acquisition unit is equal to or higher than a light source threshold value which is a preset brightness threshold.
A maximum detection unit (S330) configured so that the pixel brightness calculated by the image acquisition unit detects a pixel satisfying a maximum condition, which is a preset brightness condition, as a maximum pixel.
From the maximum pixels detected by the maximum detection unit, point detection configured to detect one or more light beam pixels having different distances from the light source pixels and indicating light rays from the light source pixels. Department and
It is configured to generate a polygonal line in which the light source pixel detected by the light source detection unit and the light beam pixel detected by the point detection unit are connected in a straight line in order from the one closest to the light source pixel as a light beam straight line. Generation unit (S370) and
With
The point detection unit
Pixels located on a preset prediction straight line at a distance of a preset unit distance or more from the light source pixel detected by the light source detection unit or the latest light beam pixel detected by the point detection unit. A range setting unit (S320) configured to set a detection range centered on the predicted pixel, and
An extraction unit (S350) configured to extract the maximum pixel included in the detection range set by the range setting unit and use the extracted maximum pixel as the detection result of the light beam pixel.
An update unit (S360) configured to update the predicted straight line based on the amount of deviation between the light beam pixels extracted by the extraction unit and the predicted pixel.
The light beam recognition device (12), which is configured to repeatedly execute the operation of the point detection unit by using the prediction straight line updated by the update unit. The light beam recognition device according to claim 1.
A light beam recognition device that updates the predicted straight line by using the Kalman filter by the updating unit. The optical beam recognition device according to claim 1 or 2.
The light beam recognition device is further connected to an object recognition device (13) that recognizes an object existing in the traveling direction of the vehicle.
A light beam recognition device further comprising a recognition suppression unit (S500) configured to suppress recognition of a pixel of the light beam straight line as an object in the image acquired from the optical camera.

Images