JP5967165B2 - Image processing system, wiper control system equipped with image processing system, vehicle, and program - Google Patents

Description

The present invention relates to an image processing system for optically detecting foreign matter adhering to the surface of a windshield of a vehicle , a wiper control system including the image processing system, a vehicle , and a program.

  In recent years, an image processing system has been installed in a vehicle such as an automobile that combines a sensing camera for acquiring vehicle periphery information and a raindrop detection camera for detecting raindrops attached to the windshield of the vehicle. (See, for example, Patent Document 1). Such an image processing system is used, for example, in a system that automatically removes raindrops attached to a windshield of a vehicle with a wiper.

  Patent Document 1 includes an image processing device, a light source, and an optical filter for the purpose of using both a sensing camera and a raindrop detection camera, and shoots raindrops outside the windshield by irradiating light from the light source. An image processing system is disclosed. In this image processing system, a band pass filter for detecting raindrops is provided in a part of the optical filter. Normally, it is difficult to change the exposure amount for only a part of the photographed image, so the exposure is adjusted to the part that is not affected by the outside light without the bandpass filter, and the part with the bandpass filter is also adjusted to the exposure value. Further, the light emission intensity of the light source is changed.

  However, the amount of light incident on the camera from the outside of the vehicle varies greatly depending on, for example, day and night, weather, and the like. The amount of change is as great as 0.1 lux to 100,000 lux. In the above image processing system, it is necessary to increase the light emission intensity of the light source in proportion to the outside light, so there is a problem of power consumption, and there is a problem in safety because strong light is projected outside the vehicle. there were.

  Therefore, for the purpose of realizing low power consumption and improving safety, there has been proposed an image processing apparatus and method for inserting a photographing frame with different exposure for raindrop detection between sensing photographing frames (for example, Patent Document 2).

  Since responsiveness is important for detection of other vehicles and white lines, it is important to keep the frame rate of the sensing frame for sensing high. However, the image processing apparatus disclosed in Patent Document 2 has a problem that the frame rate for sensing (the number of frames per unit time) decreases due to the insertion of a raindrop detection frame.

The present invention has been made in order to solve such a conventional problem, and in a system in which foreign matter detection and vehicle peripheral information are detected by a single imaging device, low power consumption and improvement in safety are achieved. And an image processing system capable of suppressing a decrease in sensing frame rate due to the insertion of a foreign object detection frame, a wiper control system including the image processing system, a vehicle , and a program And

In order to solve the above-described problems, an image processing system according to the present invention is mounted on a vehicle and has an imaging unit that images light from an imaging range including an adhering substance attached to a surface of a transparent member of the vehicle, and the imaging The readout for detecting the surrounding information of the vehicle and the detection processing of the surrounding information of the vehicle, and the reading for detecting the attached matter and the detection processing of the attached matter are performed for a predetermined period on the captured image data by the means In the image processing system, the image processing unit includes: a resolution of the captured image data for detection of the attached matter in the predetermined period; smaller than the resolution of the captured image data for the detection of information, the readout of the image data for the detection of the deposits near the vehicle And performing in the shorter time than the read captured image data for output.

  With this configuration, the imaging of the first image area frame and the imaging of the second image area frame are switched, and the readout rule of the pixel data of the first image area frame is different from the readout rule of the second image area frame. Therefore, in a system that performs foreign object detection and vehicle peripheral information detection with a single imaging device, low power consumption and improved safety can be realized, and the sensing frame rate can be reduced by inserting a foreign object detection frame. Can be suppressed.

The present invention can realize low power consumption and improved safety in a system that performs foreign object detection and vehicle periphery information detection with a single imaging device, and also has a sensing frame rate by inserting a foreign object detection frame. Image processing system, a wiper control system including the image processing system, a vehicle , and a program are provided.

The schematic diagram which shows schematic structure of an on-vehicle apparatus control system provided with the image processing system which concerns on this invention 1 is a schematic diagram showing a schematic configuration of an imaging unit included in an image processing system according to the present invention. Schematic diagram showing the schematic configuration of the imaging device included in the imaging unit Explanatory drawing which shows the infrared light image data which is the imaged image data for raindrop detection Cross-sectional schematic diagram along the light transmission direction of the optical filter, imaging device, and sensor substrate Front schematic diagram for explaining area division of the effective imaging area of the optical filter Explanatory drawing which illustrates a captured image Graph showing the spectral characteristics of the first and second spectral filter layers Graph showing another example of spectral characteristics of the first and second spectral filter layers The flowchart which shows the procedure of the image processing method which concerns on this invention Explanatory drawing illustrating the correspondence between the region division of the optical filter and the pixel of the photodiode Explanatory drawing showing an example of pixel thinning Enlarged view of the wire grid structure constituting the polarizing filter layer Explanatory drawing which shows the longitudinal direction of the metal wire of the wire grid structure in a polarizing filter layer Explanatory drawing which shows the optical path of the emitted light from the light source which an imaging device has Explanatory drawing which shows the image of the experiment result which the present inventors conducted

Embodiments of an image processing system, a wiper control system including the image processing system, a vehicle , and a program according to the present invention will be described below with reference to the drawings. In addition, the dimensional ratio of each structure on each drawing does not necessarily correspond with the actual dimensional ratio.

FIG. 1 is a schematic diagram showing a schematic configuration of an in-vehicle device control system including an image processing system 110 according to the present invention. The in-vehicle device control system uses the captured image data of the front area in the traveling direction of the vehicle 100 captured by the imaging unit 101 mounted on the vehicle 100 such as an automobile to control the light distribution of the headlamp 104, the windshield ( The transparent member is a drive control of the wiper 107 for removing the foreign matter adhering to the transparent member 105, and other vehicle-mounted devices.

  The in-vehicle device control system shown in FIG. 1 mainly includes an imaging unit 101, an image analysis unit (image analysis means) 102, a headlamp control unit 103, a wiper control unit 106, and a vehicle travel control unit 108. Prepare.

  The image processing system 110 according to the present embodiment includes an imaging unit 101 and an image analysis unit 102. The image analysis unit 102 has a function of controlling the imaging unit 101 and a function of analyzing captured image data transmitted from the imaging unit 101.

  The image analysis unit 102 analyzes the captured image data transmitted from the imaging unit 101 to detect foreign matters such as raindrops adhering to the windshield 105, or to detect other vehicles existing in front of the vehicle 100 in the captured image data. The position, direction, and distance are calculated, and a detection target such as a white line (partition line) on the road surface existing in the imaging range is detected. Hereinafter, information such as the position, direction, distance, and white line (division line) on the road surface of other vehicles in front of the vehicle 100 is also referred to as vehicle peripheral information. In the detection of other vehicles, a preceding vehicle that travels in the same traveling direction as the vehicle 100 is detected by identifying the tail lamp of the other vehicle, and an opposite traveling that travels in the opposite direction to the vehicle 100 by identifying the headlamp of the other vehicle. Detect the vehicle.

  The calculation result of the image analysis unit 102 is sent to the headlamp control unit 103. For example, the headlamp control unit 103 generates a control signal for controlling the headlamp 104 that is an in-vehicle device of the vehicle 100 from the distance data calculated by the image analysis unit 102. Specifically, for example, the driver of the vehicle 100 can avoid the dazzling of the driver of the other vehicle while avoiding the strong light of the headlamp of the vehicle 100 entering the eyes of the driver of the preceding vehicle or the oncoming vehicle. Therefore, the switching of the high beam and the low beam of the headlamp 104 is controlled, or partial light shielding control of the headlamp 104 is performed.

  The calculation result of the image analysis unit 102 is also sent to the wiper control unit 106. The wiper control unit 106 controls the wiper 107 to remove deposits such as raindrops attached to the windshield 105 of the vehicle 100. The wiper control unit 106 receives a foreign object detection result detected by the image analysis unit 102 and generates a control signal for controlling the wiper 107. When the control signal generated by the wiper control unit 106 is sent to the wiper 107, the wiper 107 is operated to ensure the visibility of the driver of the vehicle 100.

  The calculation result of the image analysis unit 102 is also sent to the vehicle travel control unit 108. Based on the white line detection result detected by the image analysis unit 102, the vehicle travel control unit 108 notifies the driver of the vehicle 100 of a warning when the vehicle 100 is out of the lane area defined by the white line. Or driving support control such as controlling the steering wheel and brake of the host vehicle.

FIG. 2 is a schematic diagram illustrating a schematic configuration of the imaging unit 101 included in the image processing system 110 according to the present embodiment. As shown in FIG. 2, the imaging unit 101 includes a cover 210 fixed in the vehicle 100, an imaging device 201 (imaging means), and an inner wall surface (one surface) 105a of the windshield 105 of the vehicle 100 from the front side. And a light source 202 that emits light toward the glass 105. The imaging device 201 and the light source 202 are accommodated inside the cover 210. 2 illustrates an example in which one light source 202 is accommodated in the cover 210, the number of light sources is not limited to one, and a plurality of light sources may be provided.

  FIG. 3 is a schematic diagram illustrating a schematic configuration of the imaging device 201 included in the imaging unit 101. The imaging apparatus 201 mainly includes a light source 202 (reflected by a foreign matter (hereinafter, a case where the foreign matter is raindrops) 203 attached to the outer wall surface (the other surface) 105b of the windshield 105 (referred to as an example). The imaging lens 204 that collects the light from the vehicle 100 (see FIG. 2) and the light that has passed through the windshield 105 from the outside of the vehicle 100 (from the outer wall surface 105b side), and the light that is collected by the imaging lens 204 is imaged. An image sensor 206, an optical filter 205 that is disposed between the image pickup lens 204 and the image sensor 206 and transmits only light in a predetermined wavelength range in a predetermined region of the effective image pickup region, and a sensor substrate 207 on which the image sensor 206 is mounted. And the analog electrical signal output from the sensor substrate 207 (the amount of light received by each light receiving element on the image sensor 206) is converted into a digital electrical signal. A signal processing unit 208 for generating and outputting the captured image data, and a.

  The imaging lens 204, the optical filter 205, the imaging element 206, and the sensor substrate 207 are arranged in this order from the windshield 105 side. The signal processing unit 208 is electrically connected to the image analysis unit 102. 3 shows an example in which the image sensor 206 and the signal processing unit 208 are provided independently, but the configuration of the imaging device 201 is not limited to this. For example, when an image sensor 206 having an A / D converter for each pixel is used, the A / D converter becomes the signal processor 208. That is, in this case, the signal processing unit 208 is built in the image sensor 206.

  Further, the image sensor 206 includes a register (sensor register) (not shown). Control of pixel data readout rules, which will be described later, is realized by the image analysis unit 102 changing the parameter value of the sensor register.

  The light source 202 is arranged such that the field angle range of the imaging lens 204 and the irradiation area of the light source 202 overlap with the inner wall surface 105 a of the windshield 105. As the light source 202, a light source having a wavelength and a light amount in the eye-safe band is used. The imaging lens 204 is composed of, for example, a plurality of lenses, and the focal position is set at infinity or between the infinity and the outer wall surface 105 b of the windshield 105.

  The light source 202 is for detecting raindrops attached to the outer wall surface 105 b of the windshield 105. When the raindrop 203 is attached to the outer wall surface 105 b of the windshield 105, the light emitted from the light source 202 is reflected at the interface between the raindrop 203 and the air, and the reflected light is incident on the imaging device 201. On the other hand, when raindrops 203 are not attached to the outer wall surface 105 b of the windshield 105, a part of the light emitted from the light source 202 passes through the windshield 105 and leaks to the outside, and the remaining light is emitted from the windshield 105. Is reflected at the interface between the inner wall surface 105 a or the outer wall surface 105 b and the outside air, and the reflected light enters the imaging device 201.

  As the light source 202, a light emitting diode (LED), a semiconductor laser (LD), or the like can be used. Further, the emission wavelength of the light source 202 may be, for example, a wavelength in the visible light region or the infrared light region. However, in order to avoid dazzling the driver or pedestrian of the oncoming vehicle with the light of the light source 202, the wavelength is longer than the visible light, and the wavelength within the range where the light receiving sensitivity of the image sensor 206 extends, for example, 750 nm or more It is preferable to select a wavelength in the infrared light region of 1000 nm or less. Hereinafter, a case where the light source 202 emits light having a wavelength in the infrared light region will be described as an example.

The light source 202 may perform continuous light emission (CW light emission), or may emit pulse light at a specific timing. In particular, a configuration that performs pulsed light emission is preferable because the influence of disturbance light can be further reduced by synchronizing the timing of light emission and the timing of image capturing.

  Light from the imaging range including the subject (detection target) passes through the imaging lens 204, passes through the optical filter 205, and is converted into an electrical signal corresponding to the light intensity by the imaging element 206. Here, the subject (detection target) is a foreign object such as a landscape in front of the vehicle 100 or raindrops attached to the outer wall surface 105b of the windshield 105. In the signal processing unit 208, when an electrical signal (analog signal) output from the image sensor 206 is input, the brightness (luminance information) of each pixel on the image sensor 206 is obtained as captured image data from the electrical signal. The included digital signal is output to the subsequent unit along with the horizontal / vertical synchronizing signal of the image.

  As already described, in the present embodiment, the focal position of the imaging lens 204 is set to infinity or between infinity and the outer wall surface 105 b of the windshield 105. Thereby, not only when detecting the raindrop 203 attached on the windshield 105, but also when detecting the preceding vehicle and the oncoming vehicle and detecting the white line, appropriate information is obtained from the captured image data of the imaging device 201. Can be acquired.

  For example, when the raindrop 203 attached on the windshield 105 is detected, the shape of the raindrop image on the captured image data is often circular, so whether the raindrop candidate image on the captured image data is circular. A shape recognition process is performed to determine whether the raindrop candidate image is a raindrop image. When such a shape recognition process is performed, rather than the focus of the imaging lens 204 on the raindrop 203 on the outer wall surface 105b of the windshield 105, as described above, In the case where the focus is in between, a slight blurring occurs, so that the raindrop shape recognition rate (circular shape) increases and the raindrop detection performance increases.

  FIG. 4 is an explanatory diagram showing infrared light image data that is captured image data for raindrop detection. 4A shows infrared image data when the imaging lens 204 is focused on the raindrop 203 on the outer wall surface 105b of the windshield 105, and FIG. 4B shows infrared image data when the imaging lens 204 is focused at infinity. Respectively.

  When the image pickup lens 204 is focused on the raindrop 203 on the outer wall surface 105b of the windshield 105, the background image 203a reflected in the raindrop is picked up as shown in FIG. Such a background image 203 a causes erroneous detection of the raindrop 203. In addition, as shown in FIG. 4A, only a part 203b of raindrops may have a bow-like brightness, and the shape of the large brightness part, that is, the shape of the raindrop image is the direction of sunlight or the position of the streetlight. It changes with things. In order to cope with the shape of such variously changing raindrop images by the shape recognition process, the processing load becomes large and the recognition accuracy is lowered.

  On the other hand, when the image is focused at infinity, some blurring occurs as shown in FIG. For this reason, the reflection of the background image 203a is not reflected in the captured image data, and erroneous detection of the raindrop 203 is reduced. In addition, the occurrence of some defocusing reduces the degree of change in the shape of the raindrop image depending on the direction of sunlight, the position of the streetlight, and the like, and the shape of the raindrop image is always substantially circular. Therefore, the load of the raindrop 203 shape recognition process is small, and the recognition accuracy is high.

However, when focusing at infinity, when identifying the tail lamp of a preceding vehicle traveling far, there may be about one light receiving element that receives the light of the tail lamp on the image sensor 206. In this case, the light from the tail lamp may not be received by the red light receiving element that receives the tail lamp color (red). In this case, the tail lamp cannot be recognized and the preceding vehicle cannot be detected. In order to avoid such a problem, it is preferable that the imaging lens 204 is focused before infinity. As a result, the tail lamp of the preceding vehicle traveling far is out of focus, so that the number of light receiving elements that receive the light of the tail lamp can be increased, the recognition accuracy of the tail lamp is increased, and the detection accuracy of the preceding vehicle is improved.

  Here, when infrared imaging light from the light source 202 reflected by the windshield 105 is imaged by the imaging device 201, the imaging device 206 of the imaging device 201 uses, for example, sunlight as well as infrared wavelength light from the light source 202. A large amount of disturbance light including the infrared wavelength light is also received. Therefore, in order to distinguish the infrared wavelength light from the light source 202 from such a large amount of disturbance light, it is necessary to make the light emission amount of the light source 202 sufficiently larger than the disturbance light. It is often difficult to use an amount of light source 202.

  Therefore, in the present embodiment, for example, a cut filter that cuts light with a wavelength shorter than the light emission wavelength of the light source 202 or a bandpass filter whose transmittance peak substantially matches the light emission wavelength of the light source 202 is used. Thus, the image sensor 206 receives the light from the light source 202. As a result, light other than the light emission wavelength of the light source 202 can be removed and received, so the amount of light from the light source 202 received by the image sensor 206 is relatively large with respect to disturbance light. As a result, it is possible to distinguish the light from the light source from the disturbance without using a light source with a large light emission amount.

  However, in the present embodiment, not only the foreign matter such as the raindrop 203 on the windshield 105 is detected from the captured image data, but also the preceding vehicle and the oncoming vehicle and the white line are detected. Therefore, if the wavelength band other than the infrared wavelength light emitted by the light source 202 is removed from the entire captured image, the imaging element 206 receives light in the wavelength band necessary for detecting the preceding vehicle and the oncoming vehicle and for detecting the white line. This cannot be done and hinders these detections. Therefore, in the present embodiment, the image area of the captured image data includes the raindrop detection image area (first image area) for detecting the raindrop 203 on the windshield 105, the detection of the preceding vehicle and the oncoming vehicle, and the white line. A vehicle peripheral information detection image region (second image region) for detection is divided into two, and a wavelength band other than the infrared wavelength light emitted by the light source 202 is removed only for a portion corresponding to the raindrop detection image region. It is configured.

  FIG. 5 is a schematic cross-sectional view along the light transmission direction of the optical filter 205, the image sensor 206, and the sensor substrate 207.

  As shown in FIG. 5, the optical filter 205 includes a substrate 220 that is transparent with respect to light in a use band (in this embodiment, a visible light region and an infrared light region), and is effective on the imaging lens 204 side on the substrate 220. A spectral filter layer (first spectral filter layer) 221 formed on the entire surface of the imaging region and transmitting only light having a wavelength component in the range of wavelengths λ1 to λ2, λ3 to λ4 (λ1 <λ2 <λ3 <λ4), and a substrate 220, the polarizing filter layer 223 formed on the surface of the imaging element 206 side, the filler 224 filled on the polarizing filter layer 223, and a part of the effective imaging area on the imaging element 206 side of the substrate 220. A spectral filter layer (second spectral filter layer) 222 that is formed through the polarizing filter layer 223 and the filler 224 and transmits only light having a wavelength component in the wavelength range of λ3 to λ4. Surface of the image pickup element 206 side is formed by close contact to the image sensor 206 of.

  That is, the optical filter 205 has a structure in which the spectral filter layer 221 and the spectral filter layer 222 are overlapped in the light transmission direction.

FIG. 6 is a schematic front view for explaining the region division of the effective imaging region of the optical filter 205 having the spectral filter layer 221 and the spectral filter layer 222. As shown in FIG. 6, the effective imaging area is divided into a visible light transmission area 211 corresponding to the vehicle surrounding information detection image area and an infrared light transmission area 212 corresponding to the raindrop detection image area. Has been.

  For example, the visible light transmissive region 211 may be a region in the center half of the effective imaging region, while the infrared light transmissive region 212 may be a region above and below the effective imaging region. Alternatively, the infrared light transmission region 212 may be provided at the upper part, the lower part, or the side part of the effective imaging region, but it is particularly preferable to be provided at the upper part and the lower part of the effective imaging region as shown in FIG. . The reason will be described below. FIG. 7 is a diagram illustrating captured image data.

  The headlamp (not shown) of the oncoming vehicle, the tail lamp of the preceding vehicle, and the white line image are often present mainly in the center of the captured image, and the image of the nearest road surface in front of the host vehicle is present below the captured image. It is normal to do. Therefore, the information necessary for identifying the headlamp (not shown) of the oncoming vehicle, the tail lamp of the preceding vehicle, and the white line is concentrated in the center of the captured image, and the information below the captured image is not so important in the identification. . On the other hand, since it is normal that the sky appears in the upper part of the captured image, information on the upper part of the captured image is not so important.

  Therefore, when both the oncoming vehicle, the preceding vehicle, or the detection of the white line and the detection of the raindrop 203 are performed simultaneously from a single captured image data, as shown in FIG. The detection image region 213 is preferably used, and the upper and lower portions of the captured image are preferably the raindrop detection image region 214, respectively. That is, as shown in FIG. 6, the upper and lower portions of the effective imaging area are the infrared light transmission area 212 for detecting raindrops, and the central part of the remaining effective imaging area is the visible light transmission area 211 for detecting vehicle peripheral information. Accordingly, it is preferable to divide the spectral filter layer 222 into regions correspondingly.

  Note that a filter that cuts infrared wavelength light emitted by the light source 202 may be formed in an area of the effective imaging area where the spectral filter layer 222 on the filler 224 is not formed.

  The image sensor 206 is an image sensor using a CCD (Charge Coupled Device), a CMOS (Complementary Metal Oxide Semiconductor), or the like, and a photodiode 206A is used as the light receiving element. The photodiodes 206A are two-dimensionally arranged for each pixel, and in order to increase the light collection efficiency of the photodiodes 206A, the incident side of each photodiode 206A corresponds to each pixel of the photodiode 206A. A microlens 206B is provided. The image sensor 206 is bonded to a PWB (Printed Wiring Board) by a technique such as wire bonding to form a sensor substrate 207.

  FIG. 8 is a graph showing the spectral characteristics of the spectral filter layer 221 and the spectral filter layer 222. As shown in FIG. 8A, the spectral filter layer 221 includes light in a so-called visible light region having a wavelength range of 400 nm to 670 nm (here, λ1 = 400 nm, λ2 = 670 nm) and a wavelength range of 940 nm to 970 nm (here, λ3 = 940 nm, λ4 = 970 nm) in the infrared region. The light in the visible light region is used for detecting vehicle peripheral information, and the light in the infrared light region is used for detecting raindrops.

However, each of the photodiodes 206A constituting the image sensor 206 of the present embodiment is sensitive to light in the infrared wavelength band, so that when the image sensor 206 receives light including the infrared wavelength band, the photodiode 206A is obtained. The captured image is reddish as a whole. As a result, it may be difficult to identify the red image portion corresponding to the tail lamp. Therefore, in the present embodiment, the spectral filter layer 221 does not transmit infrared light having a wavelength range of 670 nm to 940 nm (a transmittance of 5% or less is desirable). Thereby, since the infrared wavelength band is excluded from the captured image data portion used for identifying the tail lamp, the identification accuracy of the tail lamp is improved.

  Further, the spectral filter layer 222 has a transmission band in an infrared light region having a wavelength range of 940 nm to 970 nm as shown in FIG. Therefore, the optical filter 205 transmits only light in the wavelength range of 940 nm (= λ3) to 970 nm (= λ4) by the combination of the spectral filter layer 222 and the spectral filter layer 221 described above. Note that it is desirable that the transmittance peak in the wavelength range λ3 to λ4 and the emission wavelength of the light source 202 are substantially equal (or the emission wavelength is included in the wavelength range λ3 to λ4).

  Note that the spectral filter layer 221 may have the spectral characteristics shown in FIG. 9A instead of the spectral characteristics shown in FIG. Further, the spectral filter layer 222 is shown in FIG. 9B in place of the spectral characteristics shown in FIG. 8B (characteristics including a bandpass filter whose transmittance peak substantially matches the emission wavelength of the light source 202). It may have spectral characteristics. That is, the spectral characteristic of the spectral filter layer 222 may be a characteristic that cuts light having a shorter wavelength than the emission wavelength of the light source 202.

  By the way, when the imaging direction of the imaging device 201 is tilted downward, the hood of the host vehicle may enter the lower part of the imaging range. In this case, sunlight reflected by the bonnet of the host vehicle or the tail lamp of the preceding vehicle becomes disturbance light, which is included in the captured image data, which may cause misidentification of the head lamp of the oncoming vehicle, the tail lamp of the preceding vehicle, and the white line. Become. Even in such a case, in the present embodiment, the spectral filter layer 221 that blocks the light in the wavelength range of 670 nm to 940 nm is formed over the entire effective imaging region, so that the sunlight reflected by the bonnet, the tail lamp of the preceding vehicle, etc. The disturbance light is removed. Therefore, the head lamp of the oncoming vehicle, the tail lamp of the preceding vehicle, and the white line identification accuracy are improved.

  Note that the warpage of the optical filter 205 can be suppressed by forming the spectral filter layer 221 and the spectral filter layer 222 on both surfaces of the substrate 220 of the optical filter 205. For example, when the spectral filter layer is formed only on one surface of the substrate 220, stress is applied to the substrate 220 and warping occurs. However, as shown in FIG. 5, when the spectral filter layers are formed on both surfaces of the substrate 220, warping can be suppressed because the effect of stress is offset.

  Further, as shown in FIGS. 5 and 6, a pattern in which the spectral filter layer 222 is provided on the upper and lower portions of the effective imaging region is desirable. When the infrared light transmission region 212 for raindrop detection is provided only in either the upper part or the lower part of the effective image pickup region, it is difficult to bond the optical filter 205 and the image sensor 206 in parallel. If the optical filter 205 and the image sensor 206 are tilted and bonded, the optical path length changes between the upper and lower parts of the effective image area, and vehicle periphery information, for example, white line coordinate reading errors when white line detection is performed, is recognized. It causes deterioration of accuracy.

  Note that the optical filter 205 and the image sensor 206 are bonded in parallel even if the pixel for detecting raindrops and the pixel for detecting vehicle peripheral information are formed in a checkered pattern or a stripe pattern over the entire effective image pickup area. Is possible. In this case, since the infrared light transmission region 212 can be made large, it is possible to further improve the detection accuracy of raindrops.

The polarizing filter layer 223 is formed in order to cut out disturbance light reflected from the inner wall surface 105 a of the windshield 105 after being irradiated from the light source 202. In general, it is known that most of the polarization components of such disturbance light are S-polarized components. That is, the polarization axis of the polarizing filter layer 223 is formed so as to shield light having a polarization direction component (S polarization component) orthogonal to the normal line of the inner wall surface 105 a of the windshield 105. In other words, the polarizing filter layer 223 is the optical axis 2 of the light emitted toward the windshield 105 of the light source 202.
2 and the optical axis 23 of the imaging lens 204 are designed to transmit only a polarized component (P-polarized component) parallel to a plane formed by two optical axes. The polarizing filter layer 223 designed in this way can also block reflected light reflected by a dashboard or the like.

  As described above, the optical filter 205 is disposed close to the surface of the image sensor 206 on the microlens 206B side. Although there may be a configuration in which there is a gap between the optical filter 205 and the image sensor 206, the configuration in which the optical filter 205 is in close contact with the image sensor 206 and the visible light transmission region 211 of the optical filter 205 and the infrared light transmission. It becomes easy to make the boundary of the region 212 coincide with the boundary between the photodiodes 206 </ b> A on the image sensor 206. Thereby, the boundary between the infrared light transmission region 212 and the visible light transmission region 211 becomes clear, and the detection accuracy of raindrops can be increased.

  The optical filter 205 and the image sensor 206 may be bonded with, for example, a UV adhesive, or UV bonding or thermocompression bonding is performed on the four side areas outside the effective pixel range while being supported by the spacer outside the effective pixel range used for imaging. May be.

  Further, as in the present embodiment, when there are a plurality of light sources, the plurality of light sources may emit light simultaneously or sequentially. When the light is emitted sequentially, the influence of disturbance light can be reduced by synchronizing the light emission timing and the image capturing timing.

  By the way, although the image which image | photographed vehicle periphery information and raindrop simultaneously was shown in FIG. 7, vehicle periphery information and raindrop may be image | photographed separately. For example, the imaging device 201 captures an image with a first exposure amount (exposure time) for capturing a foreign matter such as raindrops attached to the outer wall surface 105b of the windshield 105 in the infrared light transmission region 212, In the visible light transmission region 211, an image may be taken with a second exposure amount (exposure time) for taking an image far from the position of the outer wall surface 105b of the windshield 105.

  The visible light transmissive region 211 and the infrared light transmissive region 212 differ in the amount of light required for shooting, but with the above-described configuration for shooting two images with different exposure times, the optimal exposure for each image can be achieved. An image can be taken.

  Specifically, when taking a distant image, the image analysis unit 102 automatically detects the amount of light transmitted through the effective imaging region (visible light transmission region 211) where only the spectral filter layer 221 is formed. When exposure adjustment is performed and a raindrop image is taken, automatic exposure is performed while detecting the amount of light transmitted through the effective imaging region (infrared light transmission region 212) in which the spectral filter layer 221 and the spectral filter layer 222 are formed. Adjust it.

  Note that an effective imaging region in which only the spectral filter layer 221 is formed has a large change in light amount. Specifically, since the illuminance around the vehicle changes from tens of thousands of lux in the daytime to 1 lux or less at night, it is necessary to adjust the exposure time according to the shooting scene. For this, known automatic exposure control may be performed. In the imaging apparatus 201 described in the present embodiment, since the subject is in the vicinity of the road surface, it is desirable to perform exposure control based on the image of the road surface area.

  On the other hand, the effective imaging region in which the spectral filter layer 221 and the spectral filter layer 222 are formed is designed to capture only the reflected light from foreign matters such as raindrops, so the change in the amount of light due to the surrounding environment is small and fixed. It is also possible to shoot with exposure time.

As described above, the vehicle peripheral information detection image region is used by the image analysis unit 102 for recognition processing such as white line detection and vehicle detection. Since these recognition processes also use information between input image frames, it is necessary to input frames taken at predetermined time intervals or rules to the image analysis unit 102.

  In addition, the result of raindrop detection does not change in a short time compared to, for example, detection of a lane departure or a front inter-vehicle distance, and the safety has a low priority. Therefore, raindrop detection frames (hereinafter referred to as raindrop detection image region frames) are inserted at regular intervals into vehicle periphery information detection frames (hereinafter referred to as vehicle periphery information detection image region frames). It is desirable.

  FIG. 10 is a flowchart showing the procedure of the image processing method of the present embodiment. First, the imaging device 201 continuously performs imaging with the second exposure amount for imaging the vehicle surrounding information detection image region frame (second image region frame) n times (step S120). Next, the image analysis unit 102 reads out the pixel data of the vehicle peripheral information detection image region frame imaged by the imaging device 201 (step S121).

  Here, the reading of the pixel data is not performed over the entire frame of the vehicle surrounding information detection image area frame (that is, the entire effective imaging area), but only with respect to the vehicle surrounding information detection image area. Is desirable. This eliminates the need to read out the pixel data of the raindrop detection image area that is not necessary for sensing applications, and can shorten the readout time of the vehicle peripheral information detection image area frame (reducing the number of clocks required for readout). it can). Therefore, the frame rate of the vehicle peripheral information detection image region frame can be increased.

  Then, the image analysis unit 102 analyzes the vehicle peripheral information detection image region frame read out in step S121 (step S122), and causes the headlamp control unit 103, the vehicle travel control unit 108, and the like to perform various controls. Is sent (step S123).

  Next, the imaging apparatus 201 continuously performs imaging with the first exposure amount for imaging the raindrop detection image region frame (first image region frame) m times (step S124). Next, the image analysis unit 102 reads out the pixel data of the raindrop detection image area frame imaged by the imaging device 201 (step S125).

  Here, it is desirable that the readout of the pixel data is not performed over the entire raindrop detection image region frame (that is, the entire effective imaging region) but only the raindrop detection image region. As a result, it is not necessary to read out the pixel data of the vehicle periphery information detection image area which is unnecessary for the raindrop detection application, and the time for reading out the raindrop detection image area frame can be shortened. Therefore, the frame rate of the vehicle peripheral information detection image region frame can be increased.

  Then, the image analysis unit 102 analyzes the frame for the raindrop detection image area read in step S125 (step S126), and sends out instruction signals for causing the wiper control unit 106 and the like to perform various controls (step S127). ).

  Then, the processes of steps S120 to S127 are repeatedly executed until there is a predetermined end instruction (such as an end instruction by the driver of the vehicle 100) (step S128).

In the above processing, the number m of continuous imaging at the first exposure amount is set to be smaller than the number n of continuous imaging at the second exposure amount. That is, the number of raindrop detection image area frames is smaller than the number of vehicle peripheral information detection image area frames. For example, when n = 30 and m = 1, the vehicle periphery information detection image region frame × 30 → the raindrop detection image region frame × 1
→ Frame for vehicle surrounding information detection image region × 30 → Frame for raindrop detection image region × 1 → Frame for vehicle surrounding information detection image region × 30 →... .

  Alternatively, the initial values of n and m are set to n = 120 and m = 1, respectively, and when raindrops are detected in step S126, n = 30 and m = 1, and the frequency of imaging the raindrop detection image region frame The structure which raises may be sufficient.

  As described above, in the image processing method shown in the flowchart of FIG. 10, the pixel data readout rule for the raindrop detection image region frame is different from the readout rule for the vehicle surrounding information detection image region frame. Specifically, for the vehicle periphery information detection image region frame, the rule that pixel data is read out only for the vehicle periphery information detection image region is applied, and for the raindrop detection image region frame, raindrop detection is performed. The rule that pixel data is read out only for the image area is applied.

  Next, the reading of the pixel data of the raindrop detection image area frame in step S125 will be described in detail. FIG. 11 is an explanatory view illustrating the pixel arrangement of the photodiode 206A arranged in the infrared light transmission area 212 corresponding to the raindrop detection image area of the raindrop detection image area frame. In FIG. 11, each pixel of the photodiode 206A is illustrated in a simplified manner, but actually, the photodiode 206A is composed of about several hundred thousand pixels arranged two-dimensionally.

  In the configuration of the present embodiment, since the focus of the imaging lens 204 (see FIG. 3) is not at the position of the windshield 105, the raindrop image obtained from the raindrop detection image area is originally out of focus, and the raindrop detection image area There is no need to increase the resolution. For this reason, in step S125, it is desirable that the pixel data corresponding to the raindrop detection image area is read out by thinning out the pixels rather than reading out all the pixel data corresponding to the raindrop detection image area.

  FIG. 12 shows an example of pixel thinning. For example, as shown in FIG. 12A, reading of pixel data of a specific column may be partially or entirely skipped. Note that white squares in the figure indicate skipped pixels. Alternatively, reading of pixel data of a specific row may be partially or entirely skipped (FIG. 12B), or reading of pixel data of a specific row and column may be partially or entirely skipped ( FIG. 12 (c)). Alternatively, as shown in FIG. 12D, reading of pixel data may be skipped in a checkered pattern.

  Thereby, compared with the case where all the pixel data corresponding to a raindrop detection image area are read, the read time of the frame for raindrop detection image areas can be shortened. In particular, it is desirable that pixel data corresponding to only the raindrop detection image region is read from the raindrop detection image region frame, and pixel thinning is performed as described above. In this case, the frame rate of the vehicle peripheral information detection image region frame can be further increased.

  For example, in the case of an image sensor having a resolution of WXGA (having a pixel number of 1280 × 800 dots), the time required for reading can be shortened to ¼ by reading out by thinning out to 640 × 400 dots.

  Furthermore, in order to increase the frame rate of the vehicle peripheral information detection image region frame, it is desirable that the area of the infrared light transmission region 212 (see FIG. 6) corresponding to the raindrop detection image region is small. For this reason, it is desirable that the area of the infrared light transmission region 212 is smaller than the area of the visible light transmission region 211 corresponding to the vehicle periphery information detection image region.

  Hereinafter, each part of the optical filter 205 will be described in detail. The substrate 220 is made of a transparent material, such as glass, sapphire, or quartz, that can transmit light in a use band (visible light region and infrared light region in the present embodiment). In the present embodiment, glass, particularly quartz glass (refractive index 1.46) and Tempax glass (refractive index 1.51) which are inexpensive and durable can be suitably used.

  The polarizing filter layer 223 formed on the substrate 220 is composed of a polarizer formed with a wire grid structure as shown in FIG. The wire grid structure is a structure in which metal wires (conductor lines) made of metal such as aluminum and extending in a specific direction are arranged at a specific pitch. By making the wire pitch of the wire grid structure a sufficiently small pitch (for example, ½ or less) compared to the wavelength band of incident light (for example, 400 nm to 800 nm), the wire grid structure vibrates in parallel to the longitudinal direction of the metal wire. Therefore, it can be used as a polarizer for producing a single polarized light because it reflects most of the light of the electric field vector component and transmits almost the light of the electric field vector component that vibrates in the direction orthogonal to the longitudinal direction of the metal wire.

  In a wire grid polarizer, the extinction ratio generally increases as the cross-sectional area of the metal wire increases, and the transmittance decreases for metal wires having a predetermined width or more with respect to the period width. Further, when the cross-sectional shape orthogonal to the longitudinal direction of the metal wire is a tapered shape, the wavelength dispersion of the transmittance and the polarization degree is small in a wide band, and a high extinction ratio characteristic is exhibited.

  FIG. 14 is an explanatory diagram showing the longitudinal direction of a metal wire having a wire grid structure in the polarizing filter layer 223 of the optical filter 205. Since the windshield 105 is generally curved, the polarization direction of reflected light from the dashboard changes at each location in the effective imaging region, and therefore the polarizing filter layer 223 has a polarization axis corresponding to the change. Is desirable.

  Specifically, as shown in FIG. 14, in order to realize a polarization axis corresponding to the curvature of the windshield 105, the longitudinal direction (groove direction) of the wire grid structure can be changed at each location of the polarization filter layer 223. That's fine.

  In the present embodiment, the polarizing filter layer 223 is formed in a wire grid structure, and thus has the following effects.

  The wire grid structure can be formed using a widely known semiconductor manufacturing process. Specifically, after depositing an aluminum thin film on the substrate 220, patterning is performed, and the sub-wavelength uneven structure of the wire grid may be formed by a technique such as metal etching. By such a manufacturing process, it becomes possible to adjust the longitudinal direction of the metal wire, that is, the polarization direction (polarization axis) corresponding to the imaging pixel size (several μm level) of the imaging element 206. Therefore, as in the present embodiment, it is possible to create the polarizing filter layer 223 in which the longitudinal direction of the metal wire, that is, the polarization direction (polarization axis) is different for each imaging pixel.

  Further, since the wire grid structure is made of a metal material such as aluminum, there is an advantage that it is excellent in heat resistance and can be suitably used even in a high-temperature environment such as a vehicle interior that is likely to become high temperature.

  The filler 224 used for planarizing the upper surface of the polarizing filter layer 223 in the stacking direction is filled in the recesses between the metal wires of the polarizing filter layer 223. As the filler 224, an inorganic material having a refractive index lower than or equal to that of the substrate 220 can be suitably used. In addition, the filler 224 in this embodiment is formed so as to cover the upper surface in the stacking direction of the metal wire portion of the polarizing filter layer 223.

The specific material of the filler 224 is preferably a low refractive index material whose refractive index is as close as possible to the refractive index of air (refractive index = 1) so as not to deteriorate the polarization characteristics of the polarizing filter layer 223. . For example, a porous ceramic material formed by dispersing fine pores in ceramics is preferable. Specifically, porous silica (SiO ₂ ), porous magnesium fluoride (MgF), porous alumina (Al ₂ O) ₃ ). The degree of these low refractive indexes is determined by the number and size of pores in the ceramic (porosity). When the main component of the substrate 220 is made of silica crystal or glass, porous silica (n = 1.2-1.26) can be preferably used.

As a method for forming the filler 224, an inorganic coating film (SOG: Spin On Glass) method can be suitably used. Specifically, a solvent in which silanol (Si (OH) ₄ ) is dissolved in alcohol is spin-coated on the polarizing filter layer 223 formed on the substrate 220, and then the solvent component is volatilized by heat treatment, whereby the silanol itself Is formed in such a manner as to cause a dehydration polymerization reaction.

  The polarizing filter layer 223 has a sub-wavelength sized wire grid structure, has a low mechanical strength, and a metal wire is damaged by a slight external force. Since it is desired that the optical filter 205 of the present embodiment is placed in close contact with the image sensor 206, there is a possibility that the optical filter 205 and the image sensor 206 come into contact with each other in the manufacturing stage. In the present embodiment, since the upper surface of the polarizing filter layer 223 in the stacking direction, that is, the surface on the imaging element 206 side is covered with the filler 224, the situation where the wire grid structure is damaged when contacting the imaging element 206 is suppressed. The

  Further, by filling the filler 224 into the recesses between the metal wires in the wire grid structure of the polarizing filter layer 223 as in the present embodiment, it is possible to prevent foreign matter from entering the recesses.

  In the present embodiment, the spectral filter layer 222 laminated on the filler 224 is not provided with a protective layer like the filler 224. According to the experiments by the present inventors, even if the spectral filter layer 222 is in contact with the image sensor 206, damage that affects the captured image did not occur. The layer is omitted. The height of the metal wire (convex portion) of the polarizing filter layer 223 is generally as low as half or less of the wavelength used, while the height of the spectral filter layer 222 increases the transmittance at the cutoff wavelength as the height (thickness) increases. Since the characteristics can be sharpened, the height is about the same as the used wavelength to several times higher. As the thickness of the filler 224 increases, it becomes more difficult to ensure the flatness of the upper surface, which affects the characteristics of the optical filter 205, so that there is a limit to increasing the thickness of the filler 224. Therefore, in the present embodiment, the spectral filter layer 222 is not covered with a filler. That is, in this embodiment, since the spectral filter layer 222 is formed after the polarizing filter layer 223 is covered with the filler 224, the layer of the filler 224 can be stably formed. In addition, the spectral filter layer 222 formed on the upper surface of the layer of the filler 224 can be optimally formed.

  The spectral filter layer 221 and the spectral filter layer 222 according to the present embodiment are formed in a multilayer film structure in which high refractive index thin films and low refractive index thin films are alternately stacked. If such a multilayer film structure is adopted, the degree of freedom in setting the spectral transmittance is increased by utilizing the interference of light, and the reflection is nearly 100% with respect to a specific wavelength (for example, a wavelength band other than red). It is also possible to realize the rate.

In the present embodiment, since the use wavelength range of the captured image data is a wavelength band from substantially visible light to infrared light, the image sensor 206 having sensitivity in the use wavelength range is employed. Since the spectral filter layer 222 only has to transmit infrared light, the transmission wavelength range of the multilayer film portion is set to 9 for example.
What is necessary is just to form the cut filter (refer FIG.9 (b)) which sets to 40 nm or more and reflects other than that wavelength band.

Such a cut filter can be obtained by fabricating a multilayer film having a configuration of “substrate / (0.125L0.25H0.125L) p / medium A” in order from the lower side of the optical filter 205 in the stacking direction. it can. The “substrate” here means the filler 224 described above. Further, “0.125L” is a film thickness marking method of a low refractive index material (for example, SiO ₂ ) in which nd / λ is 1 L. Therefore, the film of “0.125L” has an optical path of 1/8 wavelength. It means that the film is of a low refractive index material having such a long film thickness. “N” is a refractive index, “d” is a thickness, and “λ” is a cutoff wavelength. Similarly, “0.25H” is obtained by setting nd / λ to 1H in the film thickness marking method of a high refractive index material (for example, TiO ₂ ).
. The “25H” film means a film of a high refractive index material having a film thickness that provides an optical path length of ¼ wavelength. “P” indicates the number of times the film combination shown in parentheses is repeated (laminated), and the more “p”, the more the influence of ripple and the like can be suppressed. The medium A is intended for air or a resin or adhesive for tight bonding with the image sensor 206.

As already shown in FIG. 8B, the spectral filter layer 222 may be a bandpass filter having a transmission wavelength range of 940 nm to 970 nm. With such a bandpass filter, it is possible to distinguish the near-infrared region and red region on the longer wavelength side than red. Such a band pass filter is, for example, “substrate / (0.125L0.5M0.125L) p (0.125L0.5H0.125L) q (0.125L0.5M0.125L) r / medium A”. It can be obtained by producing a multilayer film having the structure. As described above, when using titanium dioxide (TiO ₂ ) as the high refractive index material and silicon dioxide (SiO ₂ ) as the low refractive index material, the spectral filter layer 222 with high weather resistance can be realized.

  An example of a method for producing the spectral filter layer 222 of this embodiment will be described. First, the multilayer film described above is formed on the layer of the filler 224 formed on the substrate 220 and the polarizing filter layer 223. As a method for forming such a multilayer film, a well-known method such as vapor deposition may be used. Subsequently, the multilayer film is removed from the portion corresponding to the non-spectral region. As this removal method, a general lift-off processing method may be used. In the lift-off processing method, a pattern opposite to the target pattern is formed on a layer of the filler 224 in advance with a metal, a photoresist, or the like, a multilayer film is formed thereon, and then non-spectral The multilayer film corresponding to the region is removed together with the metal and photoresist.

  In the present embodiment, since a multilayer film structure is employed as the spectral filter layer 222, there is an advantage that the degree of freedom in setting spectral characteristics is high. In general, a color filter used in a color sensor or the like is formed of a resist agent, but it is difficult to control spectral characteristics with such a resist agent as compared with a multilayer film structure. In the present embodiment, since the multilayer structure is employed as the spectral filter layer 221 and the spectral filter layer 222, the wavelength of the light source 202 and the wavelength band of the infrared light transmission region 212 can be substantially matched.

  FIG. 15 is an explanatory diagram illustrating an optical path of light emitted from the light source 202 included in the imaging apparatus 201. The incident angle of the light emitted from the light source 202 on the windshield 105 is set so that the light emitted from the light source 202 and reflected by the boundary surface between raindrops and air is imaged by the image sensor 206. According to the experiment results of the present inventors, the layout in which the reflected light from the raindrop is the strongest is an example in which the light source 202 is arranged so that the imaging lens 204 and the optical axis of the light source 202 are substantially parallel, and The optical axis 23 of the imaging lens 204 and the light of the light source 202 so as to sandwich the normal 10 of the outer wall surface 105b of the windshield 105 passing through the intersection of the optical axis 23 of the imaging lens 204 and the optical axis 22 of the light source 202. An example in which the shaft 22 is arranged is given.

  FIG. 15 shows the latter of the above two layouts. Hereinafter, the function of the image processing system 110 according to the present embodiment detecting foreign matters such as raindrops attached to the outer wall surface 105b of the windshield 105 will be described with reference to FIG.

・ Ray A
A part of the light of the light ray A directed to the portion where the raindrops on the outer wall surface 105 b of the windshield 105 are not attached leaks to the outside of the vehicle 100 as it is. The remaining part is reflected by the inner wall surface 105a of the windshield 105 (not shown).

・ Ray B
As described above, part of the light emitted from the light source 202 is reflected by the inner wall surface 105 a of the windshield 105. As described above, most of the polarized light components of such reflected light are S-polarized light components. Further, such reflected light is unnecessary light for original raindrop detection, and may cause erroneous detection. In the present invention, since the polarizing filter layer 223 that cuts the S-polarized component is disposed in the optical filter 205, unnecessary light can be removed.

・ Ray C
Of the light emitted from the light source 202, the P-polarized component is larger than the S-polarized component as the component of the light transmitted through the windshield 105 without being reflected by the inner wall surface 105a of the windshield 105. When the raindrops are attached to the outer wall surface 105b of the windshield 105, the light incident on the windshield 105 is reflected multiple times inside the raindrop and again passes through the windshield 105 toward the image pickup apparatus 201. Then, the light reaches the optical filter 205 of the imaging device 201.

  Further, the light that reaches the optical filter 205 passes through the spectral filter layer 221, and its P-polarized component passes through the polarizing filter layer 223 having a wire grid structure. Of the P-polarized component light (infrared light) transmitted through the polarizing filter layer 223, the light that has reached the spectral filter layer 222 in the infrared light transmission region 212 for raindrop detection passes through the spectral filter layer 222 and is imaged. The light is incident on the element 206, and the image analysis unit 102 shown in FIG. 1 and the like recognizes that raindrops are attached to the outer wall surface 105b of the windshield 105. Note that the P-polarized component light that has passed through the polarizing filter layer 223 can also enter the visible light transmitting region 211, but if a filter that cuts infrared wavelength light emitted by the light source 202 is formed in this region. , The incident on the image sensor 206 can be prevented.

・ Ray D
Of the light that reaches the imaging device 201 by entering from the outer wall surface 105b side of the windshield 105 instead of the light from the light source 202, the light that reaches the infrared light transmission region 212 is the spectral filter layer 221 and the spectral filter layer. Most of it is cut by 222. As described above, the infrared light transmission region 212 is configured to be able to cut disturbance light outside the windshield 105.

・ Ray E
Of the components of the light that enters from the outer wall surface 105b side of the windshield 105 and passes through the visible light transmission region 211, not the light from the light source 202, the visible light component and the wavelength band of the infrared wavelength light irradiated by the light source 202 Only the component is transmitted through the spectral filter layer 221 to become only the P-polarized component, reaches the image sensor 206 in a state where unnecessary light is cut, and is detected as a signal for detecting vehicle peripheral information.

FIG. 16 is an explanatory diagram showing an image of a result of an experiment conducted by the present inventors. Raindrop detection image areas are provided at the top and bottom of the image. FIG. 16A is an image when raindrops are attached to the windshield 105, and FIG. 16B is an image when raindrops are not attached to the windshield 105.

  A portion surrounded by a white square in FIG. 16 corresponds to a raindrop detection image region. When raindrops are attached to this region, LED light from the light source 202 enters the image sensor 206 and the image analysis unit 102 causes the raindrops to fall. Is attached to the outer wall surface 105 b of the windshield 105. At this time, it is preferable to display a display such as “Rain detected!” For notifying the driver of the attachment of raindrops in the image.

  On the other hand, when raindrops are not attached to the outer wall surface 105b of the windshield 105, the LED light from the light source 202 does not enter the image sensor 206, so the image analysis unit 102 does not recognize the attachment of raindrops. In this case, it is preferable that a display such as “Rain not detected” for notifying the driver that no raindrops are attached is displayed in the image.

  The above recognition processing by the image analysis unit 102 is based on, for example, a threshold set in advance with respect to the amount of LED light received by the image sensor 206, and recognizes that raindrops have adhered when the amount of received light exceeds the threshold. When the amount of received light is less than or equal to the threshold value, it may be recognized that raindrops are not attached.

  Note that the above threshold value does not have to be a predetermined constant value, and may be sequentially calculated based on exposure adjustment information of the vehicle periphery information detection image. Specifically, the light output power of the light source 202 may be increased and the threshold value may be increased during high illuminance such as daytime when the surroundings of the vehicle are bright. This makes it possible to detect raindrops without the influence of disturbance light.

  As described above, the image processing system, the vehicle including the image processing system, and the image processing method according to the present embodiment detect foreign matter (mainly raindrops) and vehicle peripheral information with a single imaging device. In this case, the image analysis is performed by switching between continuous imaging of the vehicle surrounding information detection frame and continuous imaging of the foreign object detection frame, so that power consumption can be reduced and safety can be further improved. Can be improved.

  In addition, the image processing system, the vehicle including the image processing system, and the image processing method according to the present embodiment make the pixel data readout rule of the foreign object detection frame different from the readout rule of the frame for detecting vehicle surrounding information. Thus, it is possible to suppress a decrease in the vehicle peripheral information detection frame due to the insertion of the foreign object detection frame.

  The image analysis unit 102 (see FIG. 1) includes a CPU (not shown) and a memory (not shown), and is configured by software by installing a program in the memory. In this case, the program itself realizes the functions of the above-described embodiments, and the program itself and means for supplying the program to the computer, for example, a storage medium storing the program, the Internet transmitting the program, etc. This transmission medium is also a constituent of the present invention. As the storage medium, a CD-ROM, a flexible disk (FD), a CD-R, a DVD (Digital Versatile Disk), or the like can be used.

10 Normal 11 CPU
12 Memory 13 Hard Disk (HDD)
14 Input interface (I / F)
15 Display panel 16 Output I / F
17 Bus 22, 23 Optical axis 100 Vehicle 101 Imaging unit 102 Image analysis unit (image analysis means)
103 Headlamp control unit 104 Headlamp 105 Windshield (transparent member)
105a Inner wall surface (one surface)
105b Outer wall surface (the other surface)
DESCRIPTION OF SYMBOLS 106 Wiper control unit 107 Wiper 108 Vehicle travel control unit 110 Image processing system 201 Imaging device (imaging means)
202 Light source 203 Raindrop (foreign matter)
203a Background image 204 Imaging lens 205 Optical filter 206 Imaging element 206A Photodiode (light receiving element)
206B Microlens 207 Sensor board 208 Signal processor 210 Cover 211 Visible light transmission region 212 Infrared light transmission region 213 Vehicle periphery information detection image region (second image region)
214 Raindrop detection image area (first image area)
220 substrate 221 spectral filter layer (first spectral filter layer)
222 Spectral filter layer (second spectral filter layer)
223 Polarizing filter layer 224 Filler

Japanese Patent No. 4326999 Specification Japanese Patent Application No. 2011-241698

Claims (10)

An imaging means mounted on a vehicle for imaging light from an imaging range including deposits attached to the surface of the transparent member of the vehicle;
Read-out for detecting the surrounding information of the vehicle and detection processing of the surrounding information of the vehicle, and readout for detection of the attached matter and detection processing of the attached matter with respect to the captured image data by the imaging means An image processing system comprising: an image processing unit that repeatedly executes a predetermined period as one cycle;
In the predetermined period, the image processing means makes the resolution of the captured image data for detection of the attached matter smaller than the resolution of the captured image data for detection of the surrounding information of the vehicle . An image processing system comprising: reading out the captured image data for detecting a kimono in a shorter time than reading out the captured image data for detecting surrounding information of the vehicle.   In the predetermined period, the image processing unit is configured to read data of the captured image data for detection of the adhering matter, the data being smaller than a read data amount of the captured image data for detection of surrounding information of the vehicle. The readout of the captured image data for detecting the attached matter is performed in a shorter time than the readout of the captured image data for detecting surrounding information of the vehicle by reading out an amount. 2. The image processing system according to 1. A light source that emits light toward the transparent member from a side different from the surface of the transparent member;
An optical filter that is arranged between the imaging element of the imaging means and the transparent member, and transmits at least part of the light of the light source and cuts off other light,
The image processing unit is an area corresponding to a portion where the optical filter of the image sensor is arranged when reading the captured image data for detecting the attached matter, and an image used for detecting the attached matter. By reading out only the captured image data of the corresponding area, the readout of the captured image data for detecting the attached matter is performed in a shorter time than the readout of the captured image data for detecting the surrounding information of the vehicle. The image processing system according to claim 2. The area corresponding to the part where the optical filter of the image sensor is arranged is smaller than the area of the area corresponding to the part where the optical filter is not arranged,
The image processing means, when reading the captured image data for detection of the attached matter, reads only the captured image data of a region corresponding to a portion where the optical filter of the image sensor is disposed, and the periphery of the vehicle When reading the captured image data for information detection, by reading only the captured image data of a region corresponding to a portion where the optical filter of the image sensor is not disposed, the detection for the attached matter is performed. 4. The image processing system according to claim 3, wherein the captured image data is read out in a shorter time than the readout of the captured image data for detecting the surrounding information of the vehicle. 5. The image processing system according to claim 1, wherein the image processing unit reads out the captured image data for detection of the attached matter by thinning out pixels . The light source emits infrared wavelength light,
The image processing system according to claim 3, wherein the optical filter transmits at least part of the infrared wavelength light and cuts other light . 7. The image processing system according to claim 1, wherein the detection of the surrounding information of the vehicle by the image processing unit includes detection of a preceding vehicle of the vehicle and detection of a white line. An image processing system according to claim 1;
A wiper for removing the deposit attached to the surface of the transparent member,
A wiper control system that controls the wiper in response to a result of the deposit detection process by the image processing means. An image processing system according to claim 1;
A wiper for removing the deposit attached to the surface of the transparent member,
The wiper is controlled in response to the result of the detection process of the attached matter by the image processing means, and the vehicle travel control is performed in response to the result of the detection process of the surrounding information of the vehicle by the image processing means. vehicle. A program for causing a computer to function as the image processing system according to claim 1,
In the computer,
An imaging step of imaging light from an imaging range that is mounted on a vehicle and that includes deposits attached to the surface of the transparent member of the vehicle;
For the captured image data obtained by the imaging step, reading for detecting the surrounding information of the vehicle and detection processing for the surrounding information of the vehicle, and reading for detecting the attached matter and detection processing for the attached matter are performed. An image processing step that is repeatedly executed with a predetermined period as one cycle, and
In the image processing step, in the predetermined period, the resolution of the captured image data for detecting the attached matter is made smaller than the resolution of the captured image data for detecting surrounding information of the vehicle, and A program for reading out the captured image data for detecting a kimono in a shorter time than reading out the captured image data for detecting surrounding information of the vehicle.