KR20240105212A - Surround view monitoring system and method for reduce color differences of multi-camera - Google Patents

Abstract

A surround view monitoring system according to an embodiment of the present invention includes a plurality of cameras mounted on a vehicle to photograph the surroundings of the vehicle and generate a color temperature estimate of the captured image; a color temperature setting unit that receives the color temperature estimates from the plurality of cameras and sets a reference color temperature estimate among the received color temperature estimates; and an image correction unit that receives the images from the plurality of cameras and corrects the color temperature of the images to correspond to the reference color temperature estimate based on the reference color temperature estimate set in the color temperature setting unit.

Description

Surround view monitoring system and method for reducing color differences in multi-channel cameras {SURROUND VIEW MONITORING SYSTEM AND METHOD FOR REDUCE COLOR DIFFERENCES OF MULTI-CAMERA}

The present invention relates to a surround view monitoring system and method for reducing color differences in multi-channel cameras. More specifically, the present invention relates to a surround view monitoring system and method for photographing the surroundings of a vehicle using multi-channel cameras mounted on the vehicle, and capturing the captured image. The purpose of the present invention is to provide a surround view monitoring system and method for reducing the color difference between each image synthesized in the process of combining images.

Recently, due to the increase in automobiles and traffic volume, the number of traffic accidents has rapidly increased and has become a social problem, and research on driver assistance systems that can prevent traffic accidents in advance is being actively conducted.

Driver assistance systems include devices that provide image information so that the driver can understand the situation outside the vehicle while sitting in the driver's seat.

Additionally, the driver assistance system can photograph the surroundings of the vehicle using a plurality of cameras installed in the vehicle, then synthesize and convert the images captured by each camera and provide them to the driver.

However, in the case of synthesized images provided by the conventional alpha blending method, there is a problem in that color differences occur due to color differences between the synthesized images.

The present invention seeks to provide a system and method that can reduce color differences in surround view images by improving a method of processing images captured by a plurality of cameras.

A surround view monitoring system according to the present invention for achieving the above object includes a plurality of cameras mounted on a vehicle to photograph the surroundings of the vehicle and generate a color temperature estimate of the captured image; a color temperature setting unit that receives the color temperature estimates from the plurality of cameras and sets a reference color temperature estimate among the received color temperature estimates; and an image correction unit that receives the images from the plurality of cameras and corrects the color temperature of the images to correspond to the reference color temperature estimate based on the reference color temperature estimate set in the color temperature setting unit.

A surround view monitoring method according to the present invention for achieving the above object includes the steps of estimating color temperatures of images captured by a plurality of cameras; setting a reference color temperature estimate among the estimated color temperature estimates; correcting the color temperature of the images to correspond to the reference color temperature estimate; and synthesizing the corrected images to generate a surround view monitoring image.

The surround view monitoring system and method of the present invention sets a reference color temperature estimate and corrects the images of all channels to correspond to the reference color temperature, even if the color temperature estimates of each image captured from a plurality of cameras mounted on a vehicle do not match. It has the effect of reducing color differences when combining images.

1 is a block diagram of a surround view monitoring system according to an embodiment of the present invention.
Figure 2 is a flowchart of a surround view monitoring method according to an embodiment of the present invention.
FIG. 3 shows images actually captured by a plurality of cameras using the surround view monitoring method of FIG. 2.

Since the present invention can be modified in various ways and can have various embodiments, specific embodiments will be illustrated in the drawings and detailed descriptions will be given for carrying out the invention. However, this is not intended to limit the present invention to specific embodiments, and should be understood to include all changes, equivalents, and substitutes included in the spirit and technical scope of the present invention. The terms used in this specification are merely used to describe specific embodiments and are not intended to limit the invention. Singular expressions include plural expressions unless the context clearly dictates otherwise. In this application, terms such as “comprise” or “have” are intended to designate the presence of features, numbers, steps, operations, components, parts, or combinations thereof described in the specification, but are not intended to indicate the presence of one or more other features. It should be understood that this does not exclude in advance the possibility of the existence or addition of elements, numbers, steps, operations, components, parts, or combinations thereof. Unless otherwise defined, all terms (including technical and scientific terms) used in this specification may be used with meanings that can be commonly understood by those skilled in the art to which the present invention pertains.

Additionally, terms defined in commonly used dictionaries are not interpreted ideally or excessively unless they are defined.

Hereinafter, preferred embodiments of the present invention will be described with reference to the attached drawings.

1 is a block diagram of a surround view monitoring system according to an embodiment of the present invention.

Referring to FIG. 1, the surround view monitoring system according to an embodiment of the present invention may include a plurality of cameras 10, a color temperature setting unit 20, an image correction unit 30, and an image synthesis unit 40. You can.

The color temperature setting unit 20, the image correction unit 30, and the image synthesis unit 40 according to an exemplary embodiment of the present invention are an algorithm configured to control the operation of various components of a vehicle or software that reproduces the above algorithm. It may be implemented through a non-volatile memory (not shown) configured to store data regarding instructions and a processor (not shown) configured to perform operations described below using data stored in the memory. Here, the memory and processor may be implemented as individual chips. Alternatively, the memory and processor may be implemented as a single chip integrated with each other. A processor may take the form of one or more processors.

The plurality of cameras 10 include a first camera 11 installed in the front of the vehicle to capture images around the front of the vehicle, and a second camera 12 installed at the rear of the vehicle to capture images around the rear of the vehicle. It may include a third camera 13 installed on the left side of the vehicle to capture images around the left side of the vehicle, and a fourth camera 14 installed on the right side of the vehicle to capture images around the right side of the vehicle.

When capturing images around a vehicle, the plurality of cameras 10 may perform independent auto white balance (AWB) operations. Here, when receiving white color, AWB uses red, green, and blue colors so that the color temperature is constant in bright or dark areas, that is, electrically achromatic. This may mean automatically adjusting the gain.

Additionally, images surrounding the vehicle captured by each of the plurality of cameras 10 may partially overlap each other.

The plurality of cameras 10 may provide camera exposure information, brightness information of captured images, color temperature estimates of captured images, accumulated pixel data, etc. to the color temperature setting unit 20 .

The plurality of cameras 10 may provide captured surrounding images of the vehicle (front surrounding image, rear surrounding image, left surrounding image, right surrounding image) to the image correction unit 30.

The plurality of cameras 10 select only pixels in the captured images having the same RGB ratio as the RGB (Red, Green, Blue) ratio of the light source color, and select the selected pixels. After accumulating data (pixel data), the color temperature of the light source in the captured image can be estimated by collecting the accumulated pixel data. Here, light source color may refer to the color of light emitted from the light source.

For example, the plurality of cameras 10 may continuously estimate the color temperature of a captured image at every frame at 1/30 second intervals and provide the estimated color temperature to the color temperature setting unit 20 .

The color temperature setting unit 20 determines the information provided from the plurality of cameras 10 (camera exposure information, image brightness information, color temperature estimates of captured images, accumulated pixel data), reliability of the color temperature estimate, frequency, A reference color temperature estimate can be set according to distribution or preset priorities.

The color temperature setting unit 20 may set the color temperature estimate with the highest reliability among the color temperature estimates of images captured by the plurality of cameras 10 as the reference color temperature estimate.

For example, the color temperature setting unit 20 may determine the reliability of the color temperature estimate in proportion to the amount of pixel data selected and accumulated from the index portion of the light source of the images.

More specifically, as the amount of accumulated pixel data increases, the reliability of the color temperature estimate may increase, and as the amount of accumulated pixel data decreases, the reliability of the color temperature estimate may decrease.

Also, when the brightness of the images captured by the plurality of cameras 10 is dark, the reliability of the color temperature estimate may be low due to the influence of noise.

Additionally, the color temperature setting unit 20 may set the color temperature estimate that appears most frequently among the color temperature estimates provided from the plurality of cameras 10 as the reference color temperature estimate.

For example, when the plurality of cameras 10 capture images around a vehicle, the color temperature estimate is updated for each frame, typically at 1/30 second intervals, so that various types of light sources are used over time (light sources when the vehicle is driving on the road). When the light source is sunlight or fluorescent light when driving inside a tunnel is captured, the color temperature estimate of the image captured by the plurality of cameras 10 may continue to change.

The color temperature setting unit 20 may calculate the frequency of light sources for the color temperature estimates of images provided from the plurality of cameras 10 and set the color temperature estimate of the most measured light source as the reference color temperature estimate.

Additionally, the color temperature setting unit 20 may set the color temperature estimate with the highest distribution among the color temperature estimates provided from the plurality of cameras 10 as the reference color temperature estimate.

For example, assuming that the color temperature estimates estimated from the images captured by each of the cameras 11, 12, 13, and 14 are 'A, B, C', the cameras 11, 12, 13, and 14 If the color temperature estimate of the images captured by three or more cameras 11, 12, and 13 is 'A', and the color temperature estimate of the images and videos captured by one camera is 'B', the color temperature setting unit 20 ) can set the 'A' color temperature estimate as the reference color temperature estimate.

Among the cameras 11, 12, 13, and 14, the color temperature estimate of the image captured by two or more cameras 11 and 12 is 'A', and the color temperature estimate of the image captured by one camera 13 is 'A'. B', and if the color temperature estimate of the image captured by one camera 14 is 'C', the color temperature setting unit 20 may set the 'A' color temperature estimate as the reference color temperature estimate.

Additionally, the color temperature setting unit 20 may set a reference color temperature estimate according to a preset priority.

For example, the color temperature setting unit 20 assigns priorities to the color temperature estimates provided by the user of the surround view monitoring system 100 from the plurality of cameras 10 and then uses the color temperature estimate with the highest priority as the basis. It can be set as an estimate of the color temperature.

If the color temperature estimates provided from the cameras 11, 12, 13, and 14 are the same, the color temperature setting unit 20 may set the same color temperature estimate as the reference color temperature estimate.

The color temperature setting unit 20 may provide the set reference color temperature estimate to the image correction unit 30.

The image correction unit 30 may correct the image surrounding the vehicle based on the reference color temperature estimate provided from the color temperature setting unit 20.

For example, the image correction unit 30 may correct the images provided from the plurality of cameras 10 so that they converge to a white point corresponding to the reference color temperature estimate set by the color temperature setting unit 20.

More specifically, since objects can be photographed in different colors from different light sources (sunlight, tungsten lamp, fluorescent lamp), the image correction unit 30 captures the images provided from the cameras 11, 12, 13, and 14. Images can be corrected so that their color temperature converges to the white point corresponding to the reference color temperature estimate. Here, the white point is uniquely defined for a given light source and may mean a correction value that allows an object to display its original color without being distorted by the light source.

The image synthesis unit 40 synthesizes the images corrected in the image correction unit 30 (front peripheral image, rear peripheral image, left peripheral image, right peripheral image) to create a surround view monitoring (Surround) with reduced color temperature difference. View Monitoring) video can be created.

Figure 2 is a flowchart of a surround view monitoring method according to an embodiment of the present invention.

Referring to FIG. 2, the surround view monitoring method according to an embodiment of the present invention monitors surrounding areas of the vehicle (e.g., front area, rear area, left area, right area) using a plurality of cameras 10, respectively. You can take pictures (S201).

When step S201 begins, the cameras 11, 12, 13, and 14 may independently perform an Auto White Balance (AWB) operation on the captured images.

Some of the surrounding images of the vehicle captured by the plurality of cameras 10 may overlap with each other.

The plurality of cameras 10 can estimate the color temperature of captured images (S202).

For example, the plurality of cameras 10 select only pixels in the captured images having the same RGB ratio as the RGB ratio of the index portion of the light source, accumulate data for the selected pixels, and then accumulate By collecting pixel data (accumulated pixel data), the color temperature of the light source in the captured image can be estimated.

When the color temperature setting unit 20 receives color temperature estimates from the plurality of cameras 10, it can set a reference color temperature estimate among the received color temperature estimates (S203).

For example, the color temperature setting unit 20 uses information provided from the plurality of cameras 10 (camera exposure information, image brightness information, color temperature estimates of captured images, accumulated pixel data) and color temperature estimates. Reference color temperature estimates can be set according to reliability, frequency, distribution, or preset priorities.

The image correction unit 30 may correct the images provided from the plurality of cameras 10 to correspond to the reference color temperature estimate provided from the color temperature setting unit 20 (S204).

For example, the image correction unit 30 may correct the images provided from the plurality of cameras 10 so that they converge to a white point corresponding to the reference color temperature estimate set by the color temperature setting unit 20.

When images taken from a plurality of cameras 10 are corrected to correspond to the reference color temperature estimate set in the color temperature setting unit 20, the image synthesis unit 40 synthesizes the corrected images to generate one surround view monitoring image. You can do it (S205).

FIG. 3 exemplarily shows video images before and after the images captured by the cameras 11, 12, 13, and 14 using the surround view monitoring method of FIG. 2 are corrected to correspond to the reference color temperature estimate.

Figure 3 (a) shows images taken by the cameras 11, 12, 13, and 14 before using the surround view monitoring method, and (b) shows images taken by the cameras 11, 14 after using the surround view monitoring method. This shows images taken in steps 12, 13, and 14) corrected to correspond to the standard color temperature estimate.

By correcting all images captured by the cameras 11, 12, 13, and 14 to correspond to the same color temperature estimate (reference color temperature estimate), the color difference in images captured as shown in (b) of FIG. 3 can be reduced.

Claims (10)

A plurality of cameras mounted on the vehicle to photograph the surroundings of the vehicle and generate color temperature estimates of the captured images;
a color temperature setting unit that receives the color temperature estimates from the plurality of cameras and sets a reference color temperature estimate among the received color temperature estimates; and
A surround view monitoring system comprising an image correction unit that receives the images from the plurality of cameras and corrects the color temperature of the images to correspond to the reference color temperature estimate based on the reference color temperature estimate set in the color temperature setting unit. The method of claim 1, wherein the color temperature setting unit,
A surround view monitoring system that sets the color temperature estimate with the highest distribution among the color temperature estimates as the reference color temperature estimate. The method of claim 1, wherein the color temperature setting unit,
A surround view monitoring system that calculates the frequency of light sources for the color temperature estimates and sets the color temperature estimate of the most measured light source as the reference color temperature estimate. The method of claim 1, wherein the color temperature setting unit,
A surround view monitoring system that sets the color temperature estimate with the highest reliability among the color temperature estimates as the reference color temperature estimate. The method of claim 4, wherein the color temperature setting unit,
A surround view monitoring system that determines the reliability of the color temperature estimates in proportion to the amount of pixel data selected and accumulated from the index portion of the light source of the images. The method of claim 1, wherein the color temperature setting unit,
A surround view monitoring system that sets a color temperature estimate with the highest priority among the color temperature estimates as the reference color temperature estimate. The method of claim 1, wherein the image correction unit,
A surround view monitoring system that corrects the images to converge to a white point corresponding to the reference color temperature estimate. According to paragraph 1,
A surround view monitoring system further comprising an image synthesis unit that synthesizes the images corrected by the image correction unit to generate a surround view monitoring image. estimating color temperatures of images captured by a plurality of cameras;
setting a reference color temperature estimate among the estimated color temperature estimates;
correcting the color temperature of the images to correspond to the reference color temperature estimate; and
A surround view monitoring method including the step of synthesizing corrected images to create a surround view monitoring image. The method of claim 9, wherein setting the reference color temperature estimate comprises:
Surround view monitoring that sets the reference color temperature estimate according to camera exposure information provided from the plurality of cameras, image brightness information, the color temperature estimate, and reliability, frequency, distribution, and preset priority for the color temperature estimate. method.

Images