JP6991664B2 - Photographed image display system and electronic mirror system - Google Patents

Description

The present invention relates to a captured image display system that displays an image captured by a camera.

As a shooting image display system that displays images taken by the camera, an electronic mirror system that displays images taken by the camera that shoots the rear of the car on a display placed above the front windshield instead of the rearview mirror. It is known (for example, Patent Documents 1 and 2).

Further, as a technique in a captured image display system, in a system for capturing an image including a headlight of an automobile, when a region of pixels having a saturated brightness value exists in the image captured by the camera, the gain of the camera is reduced. There is known a technique of increasing the shutter speed of a camera to prevent the image of a headlight from blurring due to saturation and causing two headlights to appear as one light source (for example, Patent Document 3). ).

In addition, as a technique in a captured image display system, a camera whose gain can be adjusted for each pixel is used as a camera, and when a pixel region having a saturated brightness value occurs in an image captured with a standard gain, There is also known a technique for suppressing the appearance of two headlights as one light source by taking a picture with a small gain only for the pixels in the pixel region where the brightness value is saturated (for example). The same patent document 3).

Japanese Unexamined Patent Publication No. 2016-60411 Japanese Unexamined Patent Publication No. 2007-214749 Japanese Unexamined Patent Publication No. 2006-339994

When there is a pixel area where the brightness value is saturated in the image taken by the above-mentioned camera, the two headlights can be used as one light source by reducing the gain of the camera or increasing the shutter speed of the camera. According to the technique for suppressing the appearance, the entire image is darkened, so that the dark portion of the imaged area is crushed to black and may not be visible.

Further, according to the above-mentioned technique of reducing the gain only for the pixels in the area of the pixel where the luminance value is saturated, it is necessary to use a special camera whose gain can be adjusted for each pixel.

Further, in any of the techniques, the gain may be lowered even when the area of the pixel whose luminance value is saturated is not due to a plurality of light sources but due to a single light source. In such a case, the brightness of the displayed image is lowered without any merit, and the visibility of the displayed image is deteriorated. Therefore, the present invention provides a captured image display system for displaying an image captured by a camera. It is an object of the present invention to prevent two light sources from appearing as one light source without reducing the visibility of the displayed image as much as possible.

In order to achieve the above-mentioned problems, the present invention presents a plurality of different shooting settings in a shot image display system for displaying an image shot by a camera, with the gain of the camera × the exposure time of the camera as the shooting setting values. There is a high-brightness region, which is a region in which a high-brightness image appears, in the shooting control means for causing the camera to shoot an image with a value and the first image, which is an image shot with the first shooting setting value. The first shooting setting value, which is different from the first shooting setting value, is also a plurality of corresponding areas in the second image, which is an image shot with a small shooting setting value, which is a region corresponding to the high brightness region. When the separated high-brightness image appears, the image portion of the high-brightness region of the first image is used as the image portion of the corresponding region of the second image, and the image of the high-brightness region is used. The portion is modified to represent a plurality of separated high-intensity images, and is provided with a modified image display means for displaying the modified first image on the display.

Here, in such a shooting image display system, the shooting control means is assumed to cause the camera to shoot an image with three or more different shooting setting values, and the first shooting setting value is set to the maximum shooting setting value. In addition, in the modified image display means, there is a high-brightness region, which is a region in which a high-brightness image appears, in the first image, which is an image taken with the first shooting set value. For any of the shooting setting values different from the first shooting setting value, a plurality of separated high-brightness images are formed in the corresponding area which is the area corresponding to the high-brightness area in the image shot with the shooting setting value. When is appearing, the image taken with the largest shooting setting value among the images in which a plurality of separated high-intensity images appear in the corresponding area is referred to as the second image, and the first image is described as the second image. The image portion of the high-brightness region is modified by using the image portion of the corresponding region of the second image so that the image portion of the high-brightness region represents a plurality of separated high-brightness images. The later large image may be displayed on the display.

Here, the above-mentioned captured image display system replaces the correction with the image portion of the high-luminance region of the first image with the image portion of the corresponding region of the second image in the modified image display means. It may be configured to do so.

Alternatively, in the above-mentioned captured image display system, in the modified image display means, the modification is combined with the image portion of the high-luminance region of the first image and the image portion of the corresponding region of the second image. It may be configured to do so.

Alternatively, in the modified image display means, the above-mentioned captured image display system performs the modification, the image portion of the high-brightness region of the first image, and the image portion of the corresponding region of the second image. The higher-brightness pixel portion of the portion may be configured to be masked so that the pixel color of the first image appears.

In addition, the present invention also provides an electronic mirror system provided with the above-mentioned captured image display system. Here, in the electronic mirror system, the camera is a camera for photographing the rear, left rear, or right rear of the vehicle mounted on the vehicle, and the display is arranged at a position visible to the driver of the vehicle. ing.

According to the shooting image display system and the electronic mirror system as described above, an image is shot by the camera with a plurality of shooting setting values different from each other, with the gain of the camera × the exposure time of the camera as the shooting setting value. In the first image, which is the image taken with the shooting setting value of, the high-brightness area, which is the area where the high-brightness image appears, is due to a single light source, or multiple light sources are blurred by saturation. Whether it appears as one light source is determined by the corresponding area, which is the area corresponding to the high-brightness area in the second image, which is an image taken with a shooting setting value smaller than the first shooting setting value. Identify by whether multiple separated high-intensity images appear.

Then, when it is identified that the high-luminance region in the first image is one in which a plurality of light sources are blurred by saturation and appear as one light source, the high-luminance region in the first image is separated into a plurality of parts. Using the second image in which the high-brightness image appears, the image portion of the high-brightness region of the first image is modified and displayed so that the image portion represents a plurality of separated high-brightness images. do.

Therefore, the image portion of the high-luminance region in which the plurality of light sources of the first image are blurred by saturation and appear as one light source can be modified so as to represent the images of the plurality of separated light sources. It is also possible to suppress the deterioration of the visibility of the displayed image by reducing the brightness of the image portion in the high-luminance region by a plurality of single light sources of one image without any merit.

As described above, according to the present invention, in a captured image display system that displays an image captured by a camera, two light sources appear as one light source without reducing the visibility of the displayed image as much as possible. Can be suppressed.

It is a block diagram which shows the structure of the electronic mirror system which concerns on embodiment of this invention. It is a flowchart which shows the night display processing which concerns on this invention. It is a figure which shows the processing example of the night display processing which concerns on this invention. It is a figure which shows the processing example of the night display processing which concerns on this invention.

Hereinafter, embodiments of the present invention will be described.
FIG. 1a shows the configuration of the electronic mirror system according to the present embodiment.
The electronic mirror system is a system mounted on an automobile, and includes a camera 1, a display control device 2, and a display 3 as shown in the figure.
Here, the camera 1 is arranged at the rear of the automobile and photographs the rear view of the automobile, for example, as shown in FIG. 1b. Further, as shown in FIG. 1c, the display 3 is arranged so that the display surface faces the rear side at a position above the front windshield, that is, at a position where the rearview mirror is traditionally arranged.

Next, in such an electronic mirror system, a nighttime display process performed by the display control device 2 at nighttime will be described. Whether or not it is currently at night is detected according to the overall brightness of the image taken by the camera 1, the illuminance detected by the illuminance sensor provided separately, the lighting of the automobile, and the like.

FIG. 2 shows the procedure of night display processing.
First, the constants used in this night display processing are shown.
In this night display processing, MinG, MidG, and MaxG are used as constants representing the gain of the camera 1. However, MinG <MidG <MaxG.
As shown in the figure, in the nighttime display process, the display control device 2 first sets the gains of MaxG, MidG, and MinG to the camera 1 in sequence, and shoots an image with each gain (step 202).

Then, whether or not there is a high-luminance region in the image taken with the gain MaxG in step 202, which is a region having a predetermined size or more in which high-luminance pixels having a luminance value of a predetermined threshold value ThL or more are gathered. Examine (step 204). Here, the threshold value ThL is set to a value close to the saturation value of the luminance or the saturation value less than the saturation value of the luminance. Further, the predetermined size is, for example, a size that includes two points in the captured image in which two points 1.3 m apart from the camera 1 at a position 250 m away are captured.

Then, if there is no high-luminance region in the image captured by the gain MaxG (step 204), the image captured by the gain MaxG is displayed on the display 3 (step 206), and the process returns from step 202.

On the other hand, if there is a high-luminance region in the image captured with the gain MaxG (step 204), the following processing is performed for each of the existing high-luminance regions (steps 208, 218, 220).

That is, in the image taken with the gain MidG in step 202, the region having the same position and size as the high-luminance region is divided into the regions of the pixels having relatively low brightness, and the regions of a plurality of relatively high-luminance pixels. It is examined whether or not it is separated into images of a plurality of light sources, that is, whether or not it is separated into images of a plurality of light sources (step 210).

Then, if the images are separated into images of a plurality of light sources, the image captured by the gain MaxG is corrected by using the image portion of the region in the image captured by the gain MidG (step 212), and the high-luminance region is described. End the process.

Here, this modification replaces the image portion of the high-luminance region in the image captured with the gain MaxG with the image portion of the region having the same position and size as the high-luminance region in the image captured with the gain midG. Do it by. However, this modification is to combine the image part of the high-brightness region in the image taken with the gain MaxG with the image part of the region having the same position and size as the high-brightness area in the image taken with the gain midG. Or, the image part of the high-brightness region in the image taken with the gain MaxG is corrected by using the image part of the region having the same position and the same size as the high-brightness area in the image taken with the gain midG as a mask. It may be done by such as.

On the other hand, if the region having the same position and size as the high-luminance region in the image captured by the gain MidG is not separated into images of a plurality of light sources (step 210), the image captured by the gain MinG in step 202. It is examined whether or not the region having the same position and the same size as the high-luminance region is separated into images of a plurality of light sources (step 214).

Then, if the images are separated into images of a plurality of light sources, the image captured by the gain MaxG is corrected by using the image portion of the region in the image captured by the gain MinG (step 216), and the high-luminance region is described. End the process.

Here, the correction of the image taken with the gain MaxG using the image portion of the region in the image taken with the gain MinG is performed in the same manner as in step 212.
On the other hand, if the region having the same position and size as the high-luminance region in the image taken with the gain MinG is not separated into images of a plurality of light sources (step 214), the processing for this high-luminance region is performed as it is. finish.

Then, if the above processing of steps 210-216 is performed for each high-luminance region detected in step 204, the image taken with the gain MaxG is displayed on the display 3 (step 222), and the processing from step 202 is performed. return. However, the image taken with the gain MaxG displayed in step 222 is an image taken with the gain MaxG after the correction, if the correction is made in step 212 or step 216.

The nighttime display process performed by the display control device 2 at night has been described above.
FIG. 3 shows an example of such night display processing.
Here, FIGS. 3a, 3b, and c show images taken when a following vehicle is present at night, FIG. 3a is an image taken with a gain maxG, and FIG. 3b is an image taken with a gain midG. 3c is an image taken with a gain of minG.

As shown in the figure, in this example, in the image taken with the gain maxG of FIG. 3a, there is a high-luminance region 301 in which the two headlights of the following vehicle appear as an image of one continuous light source due to saturation.

On the other hand, as shown in FIG. 3b, in the image taken with the gain midG, in the region 302 having the same position and the same size as the high-luminance region 301 in FIG. 3a, the two headlights of the following vehicle are images of two light sources. Appears separately as. Further, as shown in FIG. 3c, in the image taken with the gain minG, the two headlights of the following vehicle are images of two light sources in the region 303 at the same position and the same size as the high-luminance region 301 in FIG. 3a. Appears separately as.

Then, in this night display process, in such a case, the image portion of the high-intensity region 301 of the image taken with the gain maxG is placed in the regions 302 and 303 at the same position and the same size as the high-intensity region 301 with two light sources. Of the images taken with the gain midG and the images taken with the gain minG, the images appear separated, and the image of the area 302 at the same position and size as the high-brightness area 301 in the image taken with the larger gain midG. The image shown in FIG. 3d modified by replacing with a portion is displayed.

As described above, in the night display processing, the correction of the high-luminance region 301 of the image taken with the gain maxG was taken with the gain midG / minG in the image portion of the high-luminance region 301 of the image taken with the gain maxG. The image portion of the region 302/303 having the same position and the same size as the high-luminance region 301 in the image may be combined at a predetermined ratio. That is, for example, in the high-luminance region 301 of the image taken with the gain maxG shown in FIG. 4a1, the image portion of the region 302 in the image taken with the gain midG shown in FIG. 4a2 is shown with a predetermined transparency as shown in FIG. 4a3. The correction may be made by superimposing the display.

Further, as described above, in the night display processing, the correction of the high-luminance region 301 of the image taken with the gain maxG is performed at the same position and the same size as the high-luminance region 301 in the image taken with the gain midG / minG. The image portion of 302/303 may be used as a mask, and the image portion of the high-luminance region 301 of the image taken with the gain maxG may be masked.
That is, in this case, for example, as a mask of the image portion of the high-luminance region 301 of the image taken with the gain maxG of FIG. 4b1, the image portion of the region 302 in the image taken with the gain midG shown in FIG. 4b2 is used. The image portion of the high-luminance region 301 of the image taken with the mask at a gain of max G is masked. Further, in this case, in the image portion of the region 302 in the image taken with the gain midG, the larger the brightness value of the pixel in the image portion, the higher the brightness of the image taken with the gain maxG of FIG. 4b1 overlapping the pixel. It is used as a mask that displays the color of the pixel in the area 301 in a large proportion, and displays black in a large proportion as the brightness value of the pixel in the image portion is smaller.

The embodiment of the present invention has been described above.
As described above, in the present embodiment, the high-luminance region in the image taken with the gain MaxG is due to a single light source, or a plurality of light sources are blurred by saturation and appear as one light source. This is identified by whether or not a plurality of separated high-luminance images appear in the region corresponding to the high-luminance region in the image taken with the gain MidG or the gain MinG.

Then, of the high-luminance regions 301 in the image taken with the gain Max G, only the high-luminance regions that can be identified as having a plurality of light sources blurred by saturation and appearing as one light source have a plurality of separated heights. Using the image portion of the region corresponding to the high-brightness region in the image taken with the gain MidG or the gain MinG in which the image of the brightness appears, the image portion of the high-brightness region is divided into a plurality of high-brightness areas. Modify and display to represent the image.

Therefore, it is possible to modify the image part of the high-brightness region where multiple light sources in the image taken with the gain MaxG are blurred by saturation and appear as one light source so as to represent images of multiple separated light sources. At the same time, the brightness of the image portion in the high-luminance region due to a single light source in the image taken with the gain Max G is not reduced without any merit, and the visibility of the displayed image is not deteriorated.

Further, since the correction is performed by preferentially using the image taken with the gain MidG over the image taken with the gain MinG, it is possible to suppress the decrease in brightness in the high-luminance region due to the correction as small as possible.

By the way, in the above embodiment, in the night display processing, a plurality of images are taken with different gains of the camera 1, but this is the exposure time (shutter speed) of the camera 1 or the exposure time of the camera 1. A plurality of images may be taken with different gains.

That is, for example, in the night display processing, an image is taken with the exposure time of the camera 1 different in three stages, and instead of the image taken with the gain MaxG in the above embodiment, the maximum exposure time used for the shooting is the maximum. Using an image, replace the image taken with Gain MidG with the image with the second longest exposure time used for shooting, and replace the image taken with Gain MinG with the image with the shortest exposure time used at the time of shooting. Try to use it.

Alternatively, for example, in the night display processing, the exposure time of the camera 1 × the exposure time used for shooting instead of the image taken with the gain MaxG in the above embodiment in which the images were taken with the gains different in three stages × Use the image with the maximum gain, replace the image taken with the gain MidG, use the image with the second longest exposure time x gain used for shooting, and replace the image taken with the gain MinG, the exposure used for shooting. Try to use the image with the smallest time x gain.

Further, in the above embodiment, the electronic mirror system for displaying the image taken by the camera 1 on the display 3 arranged at a position above the front windshield has been described. However, the above nighttime display process is taken by the camera 1. An image including a light source is taken by the camera 1, such as an electronic mirror system that displays the left rear / right rear image on the display 3 arranged on the left / right, that is, an electronic mirror system that replaces a traditional side mirror. It can be similarly applied to any other captured image display system.

1 ... camera, 2 ... display control device, 3 ... display.

Claims (6)

It is a shot image display system that displays the image taken by the camera on the display.
A shooting control means for causing the camera to shoot an image with a plurality of shooting setting values different from each other, with the gain of the camera × the exposure time of the camera as the shooting setting value.
The first image, which is an image taken with the first shooting setting value, has a high-brightness region, which is a region where a high-brightness image appears, and is different from the first shooting setting value. When a plurality of separated high-brightness images appear in the corresponding area, which is the area corresponding to the high-brightness area in the second image, which is an image taken with a small shooting setting value. Using the image portion of the high-brightness region of the first image and the image portion of the corresponding region of the second image, the image portion of the high-brightness region represents a plurality of separated high-brightness images. A photographed image display system comprising a modified image display means for displaying the corrected first image on the display. The photographed image display system according to claim 1.
The shooting control means causes the camera to shoot an image with three or more different shooting setting values.
The first shooting setting value is the maximum shooting setting value.
The modified image display means has a high-brightness region, which is a region in which a high-brightness image appears, in the first image, which is an image taken with the first shooting set value, and the first shooting. For any of the shooting setting values different from the set value, a plurality of separated high-brightness images appear in the corresponding area, which is the area corresponding to the high-brightness area in the image shot with the shooting setting value. In this case, the image taken with the largest shooting setting value among the images in which a plurality of separated high-brightness images appear in the corresponding area is set as the second image, and the high-brightness area of the first image is used. The image portion of the second image is modified by using the image portion of the corresponding region of the second image so that the image portion of the high-intensity region represents a plurality of separated high-intensity images . 1 A photographed image display system characterized by displaying an image on the display. The captured image display system according to claim 1 or 2.
The corrected image display means is characterized in that the correction is performed by replacing the image portion of the high-luminance region of the first image with an image portion of the corresponding region of the second image. system. The captured image display system according to claim 1 or 2.
The corrected image display means is characterized in that the correction is performed by synthesizing the image portion of the corresponding region of the second image with the image portion of the high-luminance region of the first image. system. The captured image display system according to claim 1 or 2.
The modified image display means performs the modification by displaying the image portion of the high-brightness region of the first image, the image portion of the corresponding region of the second image, and the high-brightness pixel portion of the image portion. A captured image display system characterized in that it is performed by masking so that the pixel color of the first image appears. An electronic mirror system including the captured image display system according to claim 1, 2, 3, 4 or 5.
The camera is a camera that captures the rear, left rear, or right rear of the vehicle mounted on the vehicle.
The display is an electronic mirror system characterized in that the display is arranged at a position visible to the driver of the automobile.

Images