EP2556676A1

EP2556676A1 - Color mask for an image sensor of a vehicle camera

Info

Publication number: EP2556676A1
Application number: EP11705836A
Authority: EP
Inventors: Ulrich Seger; Alexander Wuerz-Wessel; Paul Ruhnau
Original assignee: Robert Bosch GmbH
Current assignee: Robert Bosch GmbH
Priority date: 2010-04-07
Filing date: 2011-02-09
Publication date: 2013-02-13
Also published as: JP5591395B2; DE102010003668B4; WO2011124406A1; JP2013524664A; US9883148B2; CN102835115B; CN102835115A; US20130027560A1; DE102010003668A1

Abstract

The invention relates to a color mask (8) for an image sensor (7) of a vehicle camera (2), wherein the color mask (8) has a matrix arrangement consisting of filter pixels (Pij) having different spectral transmission behaviors, wherein several filter pixels are transparent filter pixels which are at least substantially uniformly transparent for optical light, and additional filter pixels are color pixels having wavelength selective transmission behavior, wherein the color mask (8) has a color pattern as a self-repeating arrangement made of filter pixels (Pij) having specific transmission behavior. According to the invention, the color mask (8) comprises a middle area having a first color pattern and a lower area (8b) and/or upper area (8c) having a second color pattern, wherein the first color pattern and the second color pattern are different, and the first color pattern contains transparent filter pixels.

Description

description

title

Color mask for an image sensor of a vehicle camera prior art

Vehicle cameras are used in particular for detecting a vehicle environment in front of the vehicle and for displaying an image on a display device, as well as for evaluating the captured images and for use in driver assistance systems, wherein objects in the captured image are identified. Thus, z. B. other road users, lane markers, traffic signs and traffic lights are determined. For the assessment of the relevance of objects, their color is of particular importance. So have the simultaneous occurrence of yellow and white lane markings, z. For example, in a construction site area, generally the yellow marks take precedence. Furthermore, light signals can be assigned according to their color value, wherein z. B. Taillights with red color value of headlamps with white or yellow color value and taillights with yellow color value can be distinguished; Also traffic signs and traffic lights such as traffic lights and flashing lights are relevant according to their respective color value.

For this purpose, it is known to perform a color classification of the detected objects. However, the color in the captured image may differ due to the color temperature of the environment, which depends on the color of the lighting or the average color value. Different color temperatures of the captured image may be due to different lighting, z. As a function of the time of day, different street lighting and also by tints of the vehicle windows, behind which the vehicle cameras are generally mounted, the result. The tints of the

Vehicle windows are initially unknown, as they vary greatly depending on the type are different and vary in part depending on the position on the vehicle window. In general, therefore, the assumption of a "gray world" in which the histograms (statistical assignments) of the color pixels and intensity pixels are uniformly occupied is not correct over the entire image. In order to differentiate the colors, color masks are generally applied in front of the sensitive sensor surface whose filter pixels have a specific color value in front of each sensor pixel, in order to differentiate the colors. In this case, the color masks generally have specific color patterns or color patterns, which are formed by periodic repetition of specific basic patterns, for example as blocks of four pixels with a specific color value en, eg R (red), G (green), B (blue). JP 2004304706 A and WO 2009/027134 A1 show such color masks with specific color patterns. Through the use of such Farbpattern, however, the local resolution drops because z. For example, a block of four pixels with different color values is used for one pixel. Furthermore, the filtering reduces the intensity of the incident light and thus the sensitivity. For night applications, such. As a light control function for selbstätigen switching between high beam and low beam sometimes far away light sources must be de- tektiert, which a high resolution and high sensitivity are required. Especially with full-color pattern such. B. RGGB, however, the spatial resolution is reduced and the intensity of the incoming light attenuated. Partial driving patterns as described in WO 2009/027134 A1 form a compromise of color classification by the respective color filter pixels and resolution or high sensitivity through the transparent filter pixels.

Thus, in addition to the accuracy of the color determination, the spatial resolution and photosensitivity, furthermore the spatial resolution of the color values or chrominance and the orst resolution of the brightness are relevant features for vehicle cameras and driver assistance systems.

Disclosure of the invention According to the invention, the color mask is formed in different areas with different color patterns. In this case, a subdivision is carried out essentially in the vertical direction, with a central region and a - preferably smaller - upper region and / or lower region.

In this case, the invention is based on the finding that, in a typical image, which differs from a z. B. behind the windshield arranged vehicle camera is detected, a lower image area detects the road or lane in front of the vehicle, which is generally formed of asphalt or possibly concrete and thus represents a gray value without major coloration, depending on the road composition and the depending on the lighting situation, from anthracite black to light gray. In contrast, an upper image area generally reflects the sky, which is also the main source of illumination, especially in daytime situations. In this case, these areas can generally be rated as not so relevant for the evaluation for driver assistance systems, the lower image area often reproducing the roadway section directly in front of the vehicle, which is also used when initiating immediate driving dynamics interventions.

From this upper and / or lower image area, in particular also a combination of these image areas, an ambient temperature or color temperature of the environment can thus be determined. Thus, according to the invention, color patterns with a lower relative number of transparent filter pixels are used for these image areas, in particular full-color patterns without transparent ones

Filter pixels, z. B. RGGB. It is recognized according to the invention that the spatial resolution in these areas is generally not so relevant, since more relevant objects are not to be expected as frequently here or can not be reproduced with a very high local resolution.

The color determination in the upper region and / or lower region can thus be used for the white balance or the color correction of the pixel signals of the middle image region. For this middle image area, a color pattern with a higher proportion of transparent filter pixels, ie in particular a partial color pattern, is provided, which offers a compromise between higher resolution and color determination, preferably even at high spatial resolution. Solution of the chrominance and spatial resolution of the luminance, ie color-differentiated spatial resolution and brightness-differentiated spatial resolution.

Thus, according to the invention, there are some advantages. It is determined by the sub-color pattern with transparent filter pixels detection of objects in the relevant central image area with a high spatial resolution and yet good color determination or high spatially resolved chrominance, and a correction or white balance of the color values of the central image area by the upper and / or lower screen area allows. This adjustment can be carried out dynamically or in the respective driving situations and lighting situations, without additional sensors and without relevant impairment of the image processing. This white balance or the correction can basically be made purely software in the control and evaluation and thus requires no additional hardware. Furthermore, the control and evaluation used also permanently occurring tints and colorations can be assessed as permanent, z. B. as a tint of the vehicle window or a protracted pollution.

It thus preferably also eliminates a more complex consideration of different Farbpattern or color pattern in the middle image area, since there may be provided in particular a single Teilfarbpattern so that the histogram creation for the camera control or edge detection for lane detection and recognition of objects on this Teilfarbpattern can be adjusted. Thus, according to the invention a fast processing of the data with relatively little hardware effort is possible.

In principle, the second color pattern of the upper region may also be different from a second color pattern of the lower region. Furthermore, the upper and lower portions of the color mask may be different in size. They may each extend over entire rows of the pixel array, or even in the horizontal direction each comprise only central pixels, so that the central area laterally to the upper area and to the upper edge of the color mask, and / or laterally to the next lower area may extend to the lower edge of the color mask. That is the

Understanding that in the upper and lower image area in lateral If necessary, objects may occur that may be relevant to the detection and classification and may also affect the determination of the color temperature of the environment. Furthermore, objects that have been detected and classified in the middle image area may be extrapolated into the upper and lower image areas, e.g. As lane markers or larger objects such as buildings and trees, so that these parts of the upper and / or lower area are not used for the determination of the color temperature, where appropriate, for subsequent algorithms can be taken into account.

From the second pixel signals of the lower and / or upper image area, a light sensor function of the camera can additionally be displayed and used for further functions in the vehicle.

Brief description of the drawings

The invention will be explained in more detail below with reference to the accompanying drawings to ai gene embodiments. Show it:

Fig. 1 is a street scene of a vehicle with a camera in

Top view;

2 shows the camera arrangement of the vehicle;

Fig. 3 different Farbpattern; an image captured by the camera and the pixel array of the color mask with subdivision according to the invention; a flowchart of a method according to the invention.

A vehicle 1 with a camera 2 travels on a roadway 3. In a detection area 4 of the camera 2, various objects 10, 11, 12, 13, 14, 15, 16 on or outside the roadway 3 are detected. According to Figure 2, the camera

2 behind a pane 5 of the vehicle 1, in particular the windshield 5, and in addition to an imaging optics 6, which are not shown here in more detail, has an image sensor 7 with a sensitive sensor surface, on which a matrix arrangement of sensor pixels (imager pixels) 7a, not shown here in detail, is formed. On the sensitive sensor surface with the sensor pixels 7a, a color mask (filter mask) 8 is placed, which has a matrix arrangement of filter pixels Pij according to FIG. Each filter pixel Pij sits in a conventional manner directly in front of a sensor pixel 7a and filters according to its color value, ie its spectral transmission behavior, incident light. The image sensor 7 picks up the optical light incident through the color mask 8 and outputs image signals S1 to a control and evaluation device 9, which can thus carry out an evaluation in a corresponding local and color resolution, whereupon the display of the captured image takes place on an in-vehicle display device can and / or in particular by image processing a qualitative evaluation can be made to z. B.

To issue warning messages to the driver and, in particular, to intervene in vehicle management. 4 shows on the left the detected driving scene or the image 18 created from the image signals S1. B. distance control systems for setting a constant distance to a preceding vehicle 10, as well as a lane detection by identifying lane markings 1 1, 12, continue to detect traffic signs 13 and other road users 15 done. In particular, color values of the detected objects 10, 11, 12, 13, 15 can be determined and used to identify these objects or partial areas of these objects.

The color mask 8 according to the invention is divided according to FIG. 4 into regions 8 a, 8 b, 8 c, which thus correspond to image regions 18 a, 18 b and 18 c of the image 18. In this case, a central region 8a comprises a major part of the color mask 8, so that in the corresponding central image area 18a in particular relevant objects 10, 1 1, 12, 13, 15 fall, in particular other road users 10, 15, traffic signs 13 and lane markings 1 1 12. A lower area 8b of the color mask 8 corresponds to a lower image area 18b in which a part of the roadway 3 or road lies in front of the vehicle 1. In this lower area 8b, therefore, no relevant objects are to be expected initially; if necessary, parts of the traffic lane markings 1 1, 12 can extend into the lower area 8 b. An upper area 8c of the color mask 8 corresponds in the image 18 to an upper image area 18c, in which the sky is substantially imaged. The representation 4 allows the direct assignment of the image 18 to the color mask 8, wherein actually in an inverting imaging optics 6, the image 18 can also be inverted or mirror-symmetrically imaged on the image sensor 7 and thus z. B. the lower portion 8b - physically - is arranged on top of the color mask 8. In the following, however, the lower region 8b is understood to mean the region corresponding to the lower image region 18b, correspondingly also to the upper region 8c.

The regions 8b and 8c may extend over a plurality of complete lines of the color mask 8, as shown, or may comprise middle parts of the upper and lower lines, respectively, since in the lateral regions of a captured image 18, more objects, e.g. As building 14 or trees 16 occur on the edge of a road that can extend to the top of the image 18.

According to the invention, the lower area 8b and / or the upper area 8c are used to determine a color temperature or an average color value Fa of the image 18. According to the invention, the upper image area 18c detects the sky and thus a uniformly bright illumination, without significant impairment of other objects and its brightness value and spectral composition affect the brightness of the entire image 18. Here, the detected in the upper image area 18c sky z. B. white, or be reddish in the morning or in the evening, at night possibly black. The roadway 3 detected in the lower image area 18b generally represents a gray value of the asphalt or of a road surface, which may vary between light gray and black depending on the composition and overall brightness. In particular, asphalt here represents a suitable object for determining a color temperature or average color value Fa, which accordingly also contains data on the total brightness. In this case also tints of the windshield 5 and by dirt on the windshield 5 and optionally in the imaging optics 6 are detected.

According to the invention, the central area 8a of the color mask 8 contains a different color pattern than the areas 8b and 8c. FIG. 3 shows, by way of example, color patterns 20a to 20f, the respective filter pixels Pij being characterized according to the color value. Here, a transparent pixel is a value I, a red pixel is R, a cyan pixel shown as C, a blue pixel as B and a green pixel as G. The pattern 20a represents a 41 pattern, which thus serves as a gray imager and only allows the determination of gray values. The color patterns 20b, 20c, 20d, and 20f represent partial color patterns having both color values, G, B, C, and intensity pixels I, where 20b represents R3I, 20c represents RC2I, 20d represents RGBl patterns. In 20e is a

Full color pattern shown here as RGGB or Bayer pattern. 20f again represents a partial color pattern known as the sparse Bayer pattern.

The central area 8a contains more transparency values I than the areas 8b and 8c and may in particular be one of the partial color patterns 20b to 20d or 20f.

The areas 8b and / or 8c are z. B. full color pattern 20f. In this case, different full-color patterns may also be provided in the regions 8b and 8c, or the pattern 20d which contains a small proportion of transparency values I may be provided in one of the regions 8b or 8c.

By means of the partial color patterns 20b, c, d or f used, objects can thus be detected in the area 18a and their color values can be determined. So z. B. taillights 10a, 15a as red values, headlights 15b as white or yellow values, according to flashing lights for turning operations are determined as temporally pulsating yellow values, the colors of traffic signs 13 and possibly also the color representation of a Traffic lights are detected. Furthermore, color values of the lane markings 1 1 and 12 can be detected to z. For example, to distinguish a white tone from a yellow tone. In the case of temporary lane changes, yellow lane markings 1 1 are additionally attached to lane 3 without removing the white lane markings, so that yellow lane markings 1 1 should then take precedence. By according to the invention in the central region 8a of the Teilfarbpattern 20b to 20e is attached, the colors can be white or substantially colorless and yellow differentiated. This can be z. B. be achieved by the R3I Farbpattern 20b.

The imager pixels 7a of the image sensor 7 behind the middle region 8a output first pixel signals S1a, the imager pixels 7a behind the regions 8b and 8c corresponding to second pixel signals S1b and Sic, so that the image signals S1 receive these pixel signals S1 a, S1 b and Si c, as indicated in Figure 2 at the signal arrow of S1. The control and evaluation device 9 uses the Pixelsig- nals S1 b and Si c for determining an average color value Fa and / or an average brightness value Ha for the white balance, ie for the correction of all the image signals S1 and thus of the pixel signals S1 a. On the basis of these corrected image signals S1, detection and recognition of the objects 10, 11, 12, 13, 14, 15 takes place. The control and evaluation device 9 accordingly outputs output signals S2 which are used for driving management interventions and also as image signals for graphical representation of the image 18 on a display device.

In this case, the output signals S2 or the detected objects in the middle image area 18a can additionally be used to evaluate the image areas 18b and 18c. In particular, z. Example, when detecting a lane marker 1 1 or 12 in the middle image area 18a they are extrapolated down into the image area 18b, so that possibly detected in the image area 18b contours these lane markings 1 1 and 12 can be assigned. Furthermore, it can be seen in FIG. 4 that the right tree 16 extends upward from the central image area 18a and thus a corresponding right subregion 16-1 of the upper image area 18c is to be assigned to this tree or object 16. Thus, these respective sections 16-1 are not used to determine the average color value Fa and / or the average brightness value Ha. Furthermore, the objects detected in these subregions may possibly also be used for the representation of an image and vehicle management interventions by means of the output signals S2.

The method according to the invention therefore starts according to FIG. 5 in step StO, in particular when the camera 2 is switched on, whereupon in step St1 image signals S1 are output with the pixel signals S1a, S1b, S1c, in step St2 a determination of a color temperature, i. an average color value Fa and / or an average brightness value Ha for the white balance, by which then corrected pixel values or image signals can be formed in step St3 and objects 10, 1 1, 12, 13, 14, 15, 16 in the image 18 can be identified, so that in step St4 below the output signals S2 are output

According to a further embodiment, in the imaging optics 6 z. For example, a diffuser may be introduced which specifically images the upper image area 18c and / or the lower image area 18b more blurred, so that structures in this image are not enough color measurement and determination of the average color value Fa disturb.

From the second pixel signals S1 b and Si c, a light sensor function of the camera can additionally be represented, i. H. These second pixel signals can be used as light sensors for further functions in the vehicle.

The control and evaluation device 9 can be designed so that it recognizes permanent color base values or colorations from longer measurement periods, for. B. by a tinted windshield 5, and thus without current pixel signals S1 a and S1 b, z. B. at night driving with dark image areas 18b and 18c, can make a corresponding white balance.

Claims

claims

1 . Color mask (8) for an image sensor (7) of a vehicle camera (2),

wherein the color mask (8) has a matrix arrangement of filter pixels (Pij) with different spectral transmission behavior,

wherein some filter pixels are transparent filter pixels (I) that are at least substantially uniformly transparent to optical light, and further filter pixels (R, G, B, C) are color pixels (R, G, B, C) with wavelength-selective transmission behavior,

the color mask (8) having a color pattern (20a, 20b, 20c, 20d, 20e, 20f) as a repeating arrangement of filter pixels (Pij) with specific transmission behavior,

characterized in that

the color mask (8) has a central area (8a) with a first color pattern (20b, 20c, 20d, 20f) and a lower area (8b) and / or upper area (8c) with a second color pattern (20e),

wherein the first color pattern (20b, 20c, 20d, 20f) and the second color pattern (20e) are different and the first color pattern contains transparent filter pixels (I).

2. A color mask according to claim 1, characterized in that the first color pattern (20b, 20c, 20d, 20e) has a larger relative proportion of transparent filter pixels (I) than the second color pattern (20f).

3. color mask according to claim 1 or 2, characterized in that the second color pattern (20 f) as a full-color pattern (20 f) without transparent filter pixels, z. B. as RGGB pattern (20f) is formed.

4. A color mask according to one of the preceding claims, characterized in that the lower region (8b) and / or the upper region (8c) comprises a plurality of complete rows of the matrix arrangement of the filter pixels (Pij), wherein the middle region (8a) contains more lines than the upper region (8c) and / or the lower region (8b).

A color mask according to one of the preceding claims, characterized in that the first color pattern is a partial color pattern (20b, 20c, 20d, 20f) which has both transparent and colored filters.

Vehicle camera (2), comprising:

an image sensor (7) with sensor pixels (7a),

a color mask (8) arranged in front of the sensor pixels (7a) according to one of the preceding claims, and

a control and evaluation device (9),

wherein the image sensor (7) outputs image signals (S1) to the control and evaluation device (9), the first pixel signals (S1 a) of the sensor pixels (7a) covered with the filter pixels (Pij) of the middle region (8a) and second pixel signals (S 1b, Si c) of the filter pixels (7a) of the image sensor (7) covered with the filter pixels (Pij) of the lower region (8b) and / or upper region (8c),

wherein the control and evaluation device (9) derives from the second pixel signals (S1 b, Si c) an average color value (Fa) and / or a color temperature and / or an average gray value, for a white balance or a color correction of the first pixel value. Signals (S1 a) determined.

Vehicle camera (2) according to claim 6, characterized in that the control and evaluation device (9) in a central image area (18a) of the image signals (S1) transmitted image (18) from the corrected first pixel signals (S1 a) Objects (10, 1 1, 12, 13, 14, 15, 16) and determined color values of the objects (10, 1 1, 12, 13, 14, 15, 16) and subregions (10 a, 15 a, 15 b) of the objects (10, 1 1, 12, 13, 14, 15, 16) determined.

Vehicle camera (2) according to claim 6 or 7, characterized in that the control and evaluation device (9) checks whether an object (16) extends into the lower image area (18b) or upper image area (8c), and Subareas (16-1) of the lower and / or upper image area (18b, 18c), in which a detected object lies, are not used to determine the average color value (Fa) and / or the color temperature and / or the average gray value.

Vehicle camera (2) according to one of claims 6 to 8, characterized in that in front of the image sensor (7) an imaging optics (6) a blur generating means (17), for. A diffuser, wherein the blurring means (17) in the lower image area (18b) and / or upper image area (18c) produces a diffuse or blurred image.

0. A method of producing an image (18) from image signals (S1) of a vehicle camera (2) and detecting objects (10, 11, 12, 13, 14, 15, 16) in the image (18) at least the following steps:

Detecting a vehicle environment (4) by means of a camera (2) comprising an image sensor (7) and a color mask (8) arranged in front of the image sensor (7), the color mask comprising a central area (8a) with a first color pattern (20b, 20c, 20d, 20f) and a lower region (8b) and / or upper region (8c) with a second color pattern (20e),

Picking up image signals (S1), the first pixel signals (S1 a) from behind the first region (8a) arranged sensor pixels (7a) and second pixel signals (S 1 b, Sic) from behind the second color pattern (S 1 b, Si c) arranged sensor pixels (7a) included, and

Determining an average color value (Fa) and / or a color temperature and / or an average gray value and performing a white balance or a color correction of at least the first pixel signals (S1 a).