JP3994954B2 - Object detection apparatus and object detection method - Google Patents

Description

  The present invention relates to an object detection device and an object detection method mounted on a vehicle.

  Conventionally, an object detection device mounted on a vehicle is known (for example, Patent Document 1).

  In this technique, a thermal image (an image having thermal data around the vehicle) is acquired using a camera, and these thermal images are binarized based on the thermal distribution. Specifically, a reference range is set in advance, and the thermal image is distinguished into a first region having a thermal data value within the reference range and a second region having a thermal data value exceeding the reference range.

Then, the first area is extracted, and an object is detected from the extracted first area. Furthermore, the speed of the object is detected, and when the speed is about the walking speed of a person, it is determined that the detected object is a person. Thereby, a person around the vehicle is detected.
JP 2001-6096 A

  However, in this technique, when the temperature is about the temperature of a person, such as a day when the temperature is high, most regions on the thermal image have thermal data values that are within the reference range. Most regions are extracted as the first region. For this reason, when the temperature is about the temperature of a person, there is a problem that it is difficult to detect a person because the first region becomes too wide.

  Also, since thermal data is lost from the thermal image due to binarization, only a simple technique can be applied as a technique for detecting a person. For example, in this technique, as a technique for detecting a person, a technique for detecting a person from the aspect ratio of the first region is applied. For this reason, other objects having an aspect ratio comparable to that of a person, such as a low-powered telephone pole, a sign having a thick pillar, and a vending machine may be detected as a person. In other words, it may be difficult to detect a person.

  The present invention has been made to solve such a conventional problem, and a main object thereof is to provide an object detection device and an object detection method capable of detecting a person more reliably. It is.

In order to achieve the above object, the invention described in the claims of the present application is a first object for acquiring a first image including a plurality of pixels having state data around a vehicle in an object detection device mounted on the vehicle. 1st image acquisition means, 2nd image acquisition means to acquire the 2nd image which consists of a plurality of pixels which have state quantity data around the vehicle, and 1st image acquired by the 1st image acquisition means Calculating a parallax between the upper small area and a small area corresponding to the small area on the first image among the small areas on the second image acquired by the second image acquisition unit; and Based on the calculated parallax, distance calculation means for calculating a distance from the vehicle to an object drawn in a small area on the first image, a distance from the vehicle to a predetermined detection object, a state quantity data of the detected object pixels have to configure the image Based on the corresponding data, the distance calculated by the distance calculating means, and the corresponding data, a small area at a distance having the state quantity data of the detection target is extracted from the small area on the first image. The main feature is that it comprises a small area extracting means and a detecting means for detecting a detection object from the small area extracted by the small area extracting means.

In the invention described in the claims of the present application, the following effects can be mainly obtained. That is, the state quantity data (for example, thermal data) included in the detection target image on the first image is in accordance with the distance from the vehicle P to the detection target (for example, a person) corresponding to the detection target image. Different. The present invention uses such a feature to calculate the distance from the vehicle to the object drawn on the first image, and based on the calculated distance and the corresponding data described above, the first A person candidate area (small area at a distance having state quantity data of the detection target) is extracted from the small area on the image. Therefore, since the person candidate area and the other area can be accurately distinguished from each other in the first image, the person candidate area can be reliably extracted. Further, even if the temperature is close to the temperature of the person, the person candidate area can be reliably extracted.

(First embodiment)
Hereinafter, a description will be given of a first form state of the present invention with reference to the accompanying drawings. First, based on FIGS. 1-21, the main function of the component of the object detection apparatus 1 which concerns on one embodiment of this invention and each component is demonstrated.

  FIG. 1 is a block diagram showing the configuration of the object detection apparatus 1, FIG. 2 is a side view showing the installation position of the camera 100, and FIG. 3 is a plan view showing the installation positions of the cameras 100-101. 4 to 5 are explanatory diagrams illustrating an example of the thermal image A acquired by the camera 100. FIG. In the following description, as shown in FIGS. 2 to 3, the midpoint of the straight line connecting the cameras 100 to 101 is the origin, the direction parallel to the optical axis of the cameras 100 to 101 is the z axis, and the z axis is perpendicular to the z axis. A coordinate system in which the direction parallel to the road surface is the x axis, the z axis, and the direction perpendicular to the road surface is the y axis is a reference coordinate system. Here, the optical axis of the cameras 100 to 101 is an axis that passes through the center of the camera lens of the cameras 100 to 101 and is perpendicular to the camera lens. Details will be described later.

Also, as shown in FIG. 4, a thermal image is a coordinate system in which the upper left end of the thermal image is the origin, the direction from the upper end to the lower end of the thermal image is the y axis, and the direction from the left end to the right end of the thermal image is the x axis. Coordinate system.

  As shown in FIG. 1, the object detection apparatus 1 includes a camera (first image acquisition unit) 100, a camera (second image acquisition unit) 101, image memories 102 to 103, and a small region setting unit 104. A small area parallax or distance calculation unit (distance calculation means) (hereinafter referred to as “distance calculation unit”) 105, and a small area parallax or distance storage unit (hereinafter referred to as “distance storage unit”) 106. , A temperature-distance relationship graph (corresponding data) storage unit (hereinafter referred to as a “relation graph storage unit”) 107, a target small region extraction unit (small region extraction unit) 108, and a person detection unit (detection means) 109. And a result output unit 110.

  The camera 100 is installed in the front part of the vehicle P as shown in FIGS. And the vehicle P front is image | photographed and the thermal image A which has the thermal data ahead of the vehicle P is acquired in the format of thermal image data. An example of the thermal image A is shown in FIG. The thermal image data is composed of a plurality of pixel data, and each pixel data includes coordinate data and thermal data. Then, the camera 100 outputs the thermal image data to the image memory 102.

  As shown in FIGS. 2 to 3, the camera 101 is installed in the front part of the vehicle P, photographs the front of the vehicle P, and generates a thermal image B having thermal data in front of the vehicle P in the form of thermal image data. (An example of the thermal image B will be described later). Then, the thermal image data is output to the image memory 103. Here, in the thermal image B, the same object as the object drawn in the thermal image A is drawn, but the installation position of the camera 100 is different from the installation position of the camera 101, so that a certain object is drawn on the thermal image A. Is different from the position where the object is drawn on the thermal image B.

  The image memory 102 stores thermal image data supplied from the camera 100, and the image memory 103 stores thermal image data supplied from the camera 101.

  The small region setting unit 104 acquires thermal image data from the image memory 102, and, as shown in FIG. 5, based on the acquired thermal image data, the thermal image A is converted into several pixels * several pixels (for example, four pixels). * 4 pixels or 1 pixel * 1 pixel, etc.). In other words, the thermal image data is divided for each calculation area a1. Then, the thermal image data divided for each calculation area a 1, that is, the divided image data is output to the distance calculation unit 105.

  The distance calculation unit 105 calculates the parallax for each pixel (small region) included in the thermal image A based on the divided image data given from the small region setting unit 104 and the thermal image data stored in the image memory 103. To do. Here, the parallax calculation processing performed by the distance calculation unit 105 will be described with reference to FIGS. 6 is a plan view showing the positional relationship and the like between the object Q and the camera lenses 100a to 101a. FIGS. 7 to 12 are explanatory diagrams showing examples of thermal images A to B acquired by the cameras 100 to 101. FIG. is there.

  As shown in FIG. 6, when the object Q exists in front of the vehicle P, the camera 100 acquires the thermal image A shown in FIG. 7, and the camera 101 acquires the thermal image B shown in FIG. Here, the thermal image A includes an object image A1 in which the object Q is drawn, and the thermal image B includes an object image B1 in which the object Q is drawn. The camera lens 100 a is a camera lens of the camera 100, and the camera lens 101 a is a camera lens of the camera 101. In this case, as shown in FIG. 9, the distance calculation unit 105 divides the thermal image A into regions of a predetermined size, and sets each region generated by the division as a template A2. Here, the size of the template A2 is set so that an edge or a texture is included in the template A2. For example, when the calculation area a1 has a certain size (a size to which a normalized correlation method described later can be applied), the calculation area a1 is set as the template A2 as it is.

  Then, as shown in FIG. 10, the following processing is performed for each template A2. That is, an image region having the same size as the template A2 is extracted from the thermal image B, and the similarity to the template A2 is calculated for the image region. This is performed for all portions of the thermal image B.

  Here, the similarity is calculated by a normalized correlation method using thermal data of the template A2. Then, the image area B2 having the highest similarity is extracted, and the parallax is calculated for each pixel included in the template A2 based on the template A2 and the image area B2. Here, the x coordinate of the pixel a included in the template A2 is xa, and the x coordinate of the pixel b corresponding to the pixel a (having the same thermal data as the pixel a) among the pixels included in the image region B2 is xb. Then, the parallax of the pixel a is | xa−xb |.

  Then, based on the calculated parallax, a z-direction distance z1 from the object drawn on each pixel to the vehicle P is calculated. Here, as shown in FIG. 6, the z-direction distance z1 is calculated by the following equation (1). Here, in Expression (1), f is a focal length, and D is a distance from the camera lens 100a to the camera lens 101a.

z1 = f * D / | xa−xb | (1)
Then, similarity data related to the similarity and distance data related to the z-direction distance z1 are added to the divided image data and output to the distance storage unit 106.

  The distance storage unit 106 compares the similarity calculated by the distance calculation unit with a preset reference similarity for each calculation region a1 based on the divided image data given from the distance calculation unit 105, and calculates the heat A calculation area a1 in which the similarity is equal to or higher than the reference similarity is extracted from the image A. Then, for each extracted calculation area a1, the variance of the thermal data included in the calculation area a1 is calculated, and the calculated variance is compared with the reference variance. As a result, a calculation area a1 whose variance is greater than or equal to the reference variance is extracted, and among the divided image data given from the distance calculation unit 105, pixel data and distance data that the extracted calculation area a1 has, that is, a small area image Save the data.

  Here, among the divided image data given from the distance calculation unit 105, the pixel data and the distance data included in the extracted calculation area a1 are stored for the following reason. That is, if the similarity between the template A2 and the image region B2 is low, the parallax cannot be accurately calculated, and therefore the z-direction distance z1 cannot be accurately calculated. In addition, when the template A2 includes the calculation area a1 having low variance, there is a possibility that a plurality of image areas B2 corresponding to the template A2 exist, and therefore the z-direction distance z1 cannot be accurately calculated. . As a case where there are a plurality of image areas B2, for example, a case where the cameras 100 to 101 obtain the thermal images A to B shown in FIGS. Here, FIG. 11 is an explanatory diagram showing an example of a thermal image A acquired by the camera 100, and the thermal image A includes a rectangular image A3. FIG. 12 is an explanatory diagram showing an example of a thermal image B acquired by the camera 101. The thermal image B includes a rectangular image B3 corresponding to the rectangular image A3. In this case, when the distance calculation unit 105 sets an area including a part of the rectangular image A3 as the template A2, there are a plurality of image areas B2 corresponding to the template A2. Accordingly, in this case, the distance calculation unit 105 cannot accurately calculate the parallax, and thus cannot accurately calculate the z-direction distance z1.

  The relationship graph storage unit 107 illustrated in FIG. 1 stores, for example, the relationship graph 170 illustrated in FIG. 13 according to an environment such as temperature. The relation graph 170 shows the relation between the z-direction distance z2 from the vehicle P to the person and the thermal data value of the person detected by the camera 100, that is, the temperature T. The relationship graph 170 is created based on the actual measurement value of the person. Specifically, a person is photographed with the camera 100 in various environments and z-direction distances z2, and thermal data (temperature data) of the person is acquired by the photographing. The relationship graph 170 is created by plotting the thermal data on a plane with the temperature T as the vertical axis and the z-direction distance z2 as the horizontal axis. As this relationship graph 170 shows, the longer the z-direction distance z2, the lower the temperature T or the closer to the air temperature.

  The target small region extraction unit 108 detects the environment around the vehicle P, and acquires the relationship graph 170 corresponding to the detected environment from the relationship graph storage unit 107. Further, small area image data is acquired from the distance storage unit 106.

  Then, based on the acquired relation graph 170 and the small area image data, an image area having human thermal data is extracted as a human candidate area from the calculation area a1. Then, the pixel data and distance data of the extracted person candidate area, that is, the person candidate area data is output to the person detection unit 109.

  An example of the process will be described with reference to FIGS. FIG. 14 is an explanatory diagram illustrating an example of the thermal image A, and FIG. 15 illustrates that each pixel included in the thermal image A has a darker color as the distance data value (that is, the z-direction distance z1) of the pixel increases. It is explanatory drawing drawn. FIG. 16 is an explanatory diagram showing person candidate areas a2 to a9. 14 to 16, the x axis on the thermal image A is the i axis, and the y axis on the thermal image A is the j axis.

  That is, when the camera 100 acquires the thermal image A shown in FIG. 14, the target small region extraction unit 108 uses, for each pixel included in the thermal image A, the thermal data included in the pixel as the y coordinate, and the distance data included in the pixel. It is determined whether or not a coordinate point whose value is the x coordinate is included in the relationship graph 170. For example, for a pixel belonging to the coordinate (i, j) on the thermal image A, the thermal data T (i, j) that the pixel has is the y coordinate, and the distance data value z1 (i, j) that the pixel has is the x coordinate. It is determined whether or not the coordinate point to be included in the relation graph 170. Then, pixels whose coordinate points are included in the relation graph 170 are extracted, and among the extracted pixels, adjacent pixels are combined to extract the image regions a2 to a9 as person candidate regions. Then, the pixel data and distance data of the extracted person candidate areas a <b> 2 to a <b> 9, that is, the person candidate area data is output to the person detection unit 109.

  The person detection unit 109 detects a person from the person candidate area based on the person candidate area data given from the target small area extraction unit 108. Then, the person detection unit 109 outputs the person candidate area data in which the person is detected, that is, the person area data to the result output unit 110.

  Specifically, the person detection unit 109 detects a person by the following method. That is, the person candidate area is labeled, and the aspect ratio of the labeled person candidate area is calculated. If the calculated aspect ratio matches the aspect ratio of the person, the person is detected from the person candidate area. For example, when the person candidate area data corresponds to the person candidate areas a2 to a9 shown in FIG. 16, a person is detected from the person candidate area a5.

  The person detection unit 109 can also detect a person based on distance data included in the person candidate area data. The detection process will be described with reference to FIGS. Here, FIG. 17 is a side view showing the positional relationship between the person Q1 in front of the vehicle P, the camera lens 100a, and the imaging surface 100b, and FIG. 18 is an explanatory view showing an example of the thermal image A. Note that the imaging surface 100b illustrated in FIG. 17 is a surface on which the thermal image A is captured.

  As shown in FIG. 17, assuming that the distance from the person Q1 to the camera lens 100a is zd1, the focal length is f, and the height of the person image in which the person Q1 is drawn in the thermal image A is hcd1, the person Q1 actually The height H1 is expressed by the following formula (2).

H1 = zd1 * hcd1 / f (2)
Therefore, when the camera 100 acquires the thermal image A shown in FIG. 18 and the target small area extraction unit 108 extracts the person candidate areas a10 to a11, the person detection unit 109 sets the person candidate areas a10 to a11, respectively. Estimate that the object is drawn. Then, by substituting the distance data values of the person candidate areas a10 to a11 for the z-direction distance zd1 in the formula (2), and substituting the heights hc0 to hc1 of the person candidate areas a10 to a11 for the height hcd1, respectively. The actual height of the object is calculated. And when the said height turns into the height about a person, a person is detected from person candidate area | region a10-a11. In this case, in any of the person candidate areas a10 to a11, since the actual height of the object is about the height of the person, the person detection unit 109 detects a person from each of the person candidate areas a10 to a11.

  The result output unit 110 calculates the position coordinates (x0, z0) of the person detected by the person detection unit 109 based on the person area data given from the person detection unit 109. Then, the position coordinate data related to the calculated position coordinates (x0, z0) is output to a device that requires the position coordinate data, for example, a driving support device.

  Here, a specific method for calculating the position coordinates (x0, z0) will be described. That is, the result output unit 110 calculates the z coordinate z0 by using the distance data value z1 of the person area data as the z coordinate z0 as it is. Also, the x coordinate x0 is calculated based on the x coordinate data value xc0 of the person area data. A specific method for calculating the x-coordinate will be described with reference to FIGS. FIG. 19 is a plan view showing the positional relationship between the person Q1 and the camera lens 100a and the imaging surface 100b, and FIG. 20 is an explanatory view showing an example of the thermal image A.

  As shown in FIG. 19, when the distance in the z direction from the person Q1 to the camera lens 100a is zd1, the focal length is f, and the x coordinate of the person image in which the person Q1 is drawn in the thermal image A is xcd0, The distance xd1 (corresponding to the x coordinate x0) to the optical axis (the axis passing through the central part of the camera lens 100a and the central part of the imaging surface 100b and perpendicular to the camera lens 100a) is expressed by the following equation (3). .

xd1 = zd1 * xcd0 / f (3)
Therefore, when the camera 100 acquires the thermal image A shown in FIG. 20 and the person detection unit 109 detects a person from the person candidate area a12, the result output unit 110 displays the person area data corresponding to the person candidate area a12. The x-coordinate data value xc0 and the distance data value z1 are acquired from the above, the x-coordinate data value xc0 is substituted into the x-coordinate xcd0 in the equation (3), and the distance data value z1 is substituted into the distance zd1 in the equation (3). Thus, the x coordinate x0 is calculated.

  Next, a processing procedure by the object detection apparatus 1 will be described with reference to a flowchart shown in FIG.

  In step S <b> 1 shown in FIG. 21, the camera 100 captures the front of the vehicle P and acquires the thermal image A in which the heat distribution in front of the vehicle P is drawn in the form of thermal image data. Next, the acquired thermal image data is output to the image memory 102, and the image memory 102 stores the thermal image data given from the camera 100. On the other hand, the camera 101 captures the front of the vehicle P and acquires the thermal image B in which the heat distribution in front of the vehicle P is drawn in the form of thermal image data. Next, the thermal image data is output to the image memory 103, and the image memory 103 stores the thermal image data given from the camera 101.

  Next, in step S2, the small area setting unit 104 acquires thermal image data from the image memory 102, and divides the acquired thermal image data for each calculation area a1, as shown in FIG. Next, the thermal image data divided for each computation area a 1, that is, the divided image data is output to the distance calculation unit 105.

  Next, in step S <b> 3, the distance calculation unit 105 is included in the thermal image A by the above-described processing based on the divided image data given from the small region setting unit 104 and the thermal image data stored in the image memory 103. The parallax and z-direction distance z1 are calculated for each pixel. Further, the similarity is calculated for each calculation area a1. Next, similarity data relating to the similarity and distance data relating to the z-direction distance z <b> 1 are added to the divided image data and output to the distance storage unit 106.

  Next, the distance storage unit 106 stores data that satisfies the above-described requirements, that is, small area image data, among the divided image data provided from the distance calculation unit 105 by the above-described processing.

  Next, in step S <b> 4, the target small region extraction unit 108 detects the environment around the vehicle P and acquires the relationship graph 170 corresponding to the detected environment from the relationship graph storage unit 107. Further, small area image data is acquired from the distance storage unit 106.

  Then, based on the acquired relation graph 170 and the small area image data, a person candidate area having human thermal data is extracted from the calculation area a1 by the above-described processing. Next, the pixel data and distance data of the extracted person candidate area, that is, the person candidate area data is output to the person detection unit 109.

  Next, in step S5, the person detection unit 109 detects a person from the person candidate area by the above-described processing based on the person candidate area data given from the target small area extraction unit. Next, the person candidate area data in which the person is detected, that is, the person area data is output to the result output unit 110.

  Next, in step S6, the result output unit 110, based on the person area data given from the person detection unit 109, the position coordinates (x0, z0) of the person detected by the person detection unit 109 by the above-described processing. Is calculated. Subsequently, the position coordinate data regarding the calculated position coordinates (x0, z0) is output to an apparatus that requires the position coordinate data, for example, a driving support apparatus. Thereafter, the object detection apparatus 1 ends this process.

  As described above, in the first embodiment, the object detection apparatus 1 acquires the thermal images A to B in which the heat distribution in front of the vehicle P is drawn with the cameras 100 to 101, and is shown in FIGS. As described above, the parallax between the acquired pixel on the thermal image A and the pixel corresponding to the pixel on the thermal image A among the pixels on the thermal image B is calculated. Then, based on the calculated parallax, a z-direction distance z1 from the vehicle P to the object drawn on the thermal image A is calculated, and the calculated z-direction distance z1 and the relationship graph shown in FIG. As shown in FIG. 16, a person candidate area is extracted from the calculation area a1 on the thermal image A. Then, a person is detected from the person candidate area.

  Here, as shown in FIG. 13, the thermal (temperature) data of the person image on the thermal image A differs according to the z-direction distance from the vehicle P to the person corresponding to the person image. The object detection device 1 uses such characteristics to calculate the z-direction distance z1 from the vehicle P to the object drawn on the thermal image A. The calculated z-direction distance z1 is shown in FIG. Based on the relationship graph, a person candidate area is extracted from the calculation area a1 on the thermal image A. Accordingly, since the person candidate area and the other area can be accurately distinguished on the thermal image A, the person candidate area can be reliably extracted. Further, even if the temperature is close to the temperature of the person, the person candidate area can be reliably extracted. Even when the distance from the vehicle P to the person is long and the thermal data value of the person image on the thermal image A is lower than normal (for example, normal heat of the person), the calculation area a1 in which the person is drawn is displayed. It can be reliably extracted as a person candidate area.

  Further, conventionally, when detecting a person according to the z-direction distance from the vehicle P to the person, the thermal data of the person image on the thermal image varies depending on the z-direction distance, so the above-described reference range is widened. There is a need to. However, in this case, an area in which an object other than a person is drawn is also set as the first area described above, and thus an object other than a person may be detected as a person. On the other hand, the object detection device 1 calculates the z-direction distance z1 from the vehicle P to the object drawn on the thermal image A, and based on the calculated z-direction distance z1 and the relationship graph shown in FIG. Thus, the person candidate area is extracted from the calculation area a1 on the thermal image A. Accordingly, even when a person is detected according to the z-direction distance from the vehicle P to the person, the size of the area extracted as the person candidate area is smaller than that of the conventional first area. Therefore, a person can be detected easily and reliably.

  Further, since the person detection unit 109 can detect a person from the person candidate area based on the z-direction distance z1 from the vehicle P to the object drawn in the person candidate area, a simple method (for example, an aspect ratio) The person can be detected by a technique with higher accuracy than the conventional technique in which a person can be detected only by the method of detecting a person based on the above. For this reason, it is possible to prevent other objects having an aspect ratio comparable to that of a person from being detected as a person as in the past. In other words, a person can be detected more reliably.

  Further, since the relation graph 170 is created based on the actual measurement values of each part of the person, the person candidate area can be reliably detected. Further, for example, by setting the actual measurement value as the temperature of only the person's head, the object detection apparatus 1 can extract the calculation area a1 in which the person's head is drawn as the person candidate area. Further, by using the actual measurement value as the temperature of the person wearing the hat, the object detection apparatus 1 can extract the calculation area a1 in which the person wearing the hat is drawn as a person candidate area.

(Second Embodiment)
It will be described below based on the shape condition of the second embodiment of the present invention with reference to the accompanying drawings. First, based on FIG.1 and FIG.22, the component of the object detection apparatus 2 which concerns on one embodiment of this invention and the main functions of each component are demonstrated. FIG. 22 is an explanatory diagram showing the relationship graphs 271 to 272.

  As shown in FIG. 1, the object detection device 2 adds to the object detection device 1 a relationship graph storage unit 207, target small region extraction instead of the relationship graph storage unit 107, target small region extraction unit 108, and person detection unit 109. A unit 208 and a person detection unit 209 are provided.

  The relationship graph storage unit 207 illustrated in FIG. 1 stores, for example, the relationship graphs 271 to 272 illustrated in FIG. 22 according to an environment such as temperature. The relationship graph 271 shows the relationship between the z-direction distance z2 from the vehicle P to the person and the heat of the face of the person detected by the camera 100, that is, the temperature T1 of the face. The relationship graph 271 is created based on the actual measurement value of the human face. Similarly, the relationship graph 272 shows the relationship between the z-direction distance z2 from the vehicle P to the person and the heat of the body of the person detected by the camera 100, that is, the temperature T2 of the body. The relationship graph 271 is created based on the actual measurement value of the human torso. Note that, as the relationship graphs 271 to 272 indicate, the region of the relationship graph 271 is different from the region of the relationship graph 272. This is because the temperature of the torso is smaller than the temperature of the face because the torso of the person is usually covered with clothes.

  The target small region extraction unit 208 detects the environment around the vehicle P, and acquires the relationship graphs 271 to 272 corresponding to the detected environment from the relationship graph storage unit 207. Further, small area image data is acquired from the distance storage unit 106.

  Then, it is determined whether it is sufficient to detect only the head of the person. This determination is made based on settings made for the object detection device 2 and the like.

  As a result, when only the head of the person is detected, a person candidate area having the thermal data of the person's head is extracted from the thermal image A based on the acquired relation graph 271 and the small area image data. To do. On the other hand, when a part other than the head of the person is also detected, it is determined whether or not the back of the person is drawn on the thermal image A based on the environment around the vehicle P. As a result, when a person's occipital region is drawn, based on the acquired relationship graph 272 and small area image data, a person candidate area having thermal data of the human torso is extracted from the thermal image A. In this case, since the thermal data value of the occipital region is larger than the thermal data value of the face portion, when an attempt is made to detect a person candidate area based on the relationship graph 271, the calculation area a1 in which the person's head is drawn becomes the person candidate area This is because it may not be extracted as. On the other hand, when the back of the person is not drawn and when it is not possible to determine whether or not the back of the head is drawn, the thermal image A based on the acquired relationship graphs 271 to 272 and the small area image data. Then, a person candidate area having thermal data of the person's head and torso is extracted.

  Then, the pixel data and distance data of the extracted person candidate area, that is, the person candidate area data is output to the person detection unit 209.

  The person detection unit 209 detects a person from the person candidate area based on the person candidate area data given from the target small area extraction unit 208. Then, the person detection unit 209 outputs the person candidate area data in which the person is detected, that is, the person area data to the result output unit 110.

  Specifically, the person detection unit 209 detects a person by the following method. That is, the person candidate area is labeled, and the aspect ratio of the labeled person candidate area is calculated. If the calculated aspect ratio matches the aspect ratio of the person, the person is detected from the person candidate area. Here, when only the head of the person is drawn in the person candidate area, the aspect ratio of the person candidate area is compared with the aspect ratio of the person's head. As a result, when the aspect ratio of the person candidate area matches the aspect ratio of the person's head, the person is detected from the person candidate area and its surrounding area. Further, when only the human torso is drawn in the person candidate area, the aspect ratio of the person candidate area is compared with the aspect ratio of the person torso. As a result, when the aspect ratio of the person candidate area matches the aspect ratio of the torso of the person, a person is detected from the person candidate area and its surrounding area. Further, when the human head and torso are drawn in the human candidate area, the aspect ratio of the human candidate area is compared with the aspect ratio of the human head and torso. As a result, when the aspect ratio of the person candidate area matches the aspect ratio of the head and torso of the person, the person is detected from the person candidate area and the surrounding area.

  Note that the person detection unit 209 can detect a person based on the distance data included in the person candidate area data and the above-described equation (2), similarly to the person detection unit 109. Here, when only the head of the person is drawn in the person candidate area, the height H1 in the expression (2) is the height of the head of the person. Further, when only the human torso is drawn in the person candidate area, the height H1 in the expression (2) is the height of the torso of the person. When a person's head and torso are drawn in the person candidate area, the height H1 in Equation (2) is the height of the person's head and torso.

  Since the procedure of the process by the object detection device 2 is the same as the procedure of the process by the object detection device 1, the description of the procedure is omitted.

  As described above, in the second embodiment, the object detection apparatus 2 stores the relationship graph 271 related to the human face and the relationship graph 272 related to the human torso. Thereby, as described above, it is possible to extract a person candidate area having thermal data of the person's head, or to extract a person candidate area having thermal data of the person's torso. In this case, since the person candidate area to be processed by the person detection unit 109 becomes smaller, the person detection unit 109 can more easily detect a person.

  In addition, when extracting a human candidate region having thermal data of the torso, it is possible to detect a person regardless of the state of the head (for example, regardless of whether the person is wearing a hat). Therefore, a person can be detected more reliably.

(Third embodiment)
It will be described below with reference third form status of implementation of the present invention with reference to the drawings. First, based on FIG. 1, FIG. 23, and FIG. 24, the component of the object detection apparatus 3 which concerns on one embodiment of this invention, and the main functions of each component are demonstrated. 23 to 24 are explanatory diagrams illustrating an example of a person candidate area.

  As illustrated in FIG. 1, the object detection device 3 includes the object detection device 1 including a target small region extraction unit 308 instead of the target small region extraction unit 108.

  The target small area extraction unit 308 extracts an image area having human thermal data as a person candidate area by the same processing as the target small area extraction unit 108. Further, a peripheral area around the extracted image area is extracted as a person candidate area. Then, the pixel data and distance data of the extracted person candidate area, that is, the person candidate area data is output to the person detection unit 109.

  An example of the process will be described with reference to FIGS. That is, as illustrated in FIG. 23, the target small area extraction unit 308 extracts the image areas a <b> 2 to a <b> 9 as person candidate areas by the same processing as the target small area extraction unit 108. Further, as shown in FIG. 24, peripheral areas around the image areas a2 to a9 (for example, peripheral areas a3-1, a5-1, and a8-1 around the human candidate areas a3, a5, and a8) are set as human candidate areas. Extract. Then, the pixel data and distance data of the extracted person candidate area, that is, the person candidate area data is output to the person detection unit 109.

  Since the procedure of the process by the object detection device 3 is the same as the procedure of the process by the object detection device 1, description of the procedure is omitted.

  As described above, in the third embodiment, the target small area extraction unit 308 extracts not only the image area having the thermal data of the person but also the peripheral area around the image area as the person candidate area. Therefore, when the image area in which each part of the person is drawn is not connected (for example, as shown in FIG. 23, the image area in which the body part of the person is drawn and the image area in which the foot part is drawn are not connected). Even so, it is possible to extract a human candidate region including all these image portions. Therefore, a person can be extracted more reliably from the person candidate area.

  Further, when the temperature is close to the temperature of the person, the image portion having the heat data of the person may be smaller than the normal size of the person image. Since it can be larger than the portion, it is possible to reliably detect a person from the person candidate area.

(Fourth embodiment)
It will be described below with reference fourth form status of implementation of the present invention with reference to the drawings. First, based on FIG. 1, FIG. 22, FIG. 25, FIG. 26, and FIG. 27, components of the object detection device 4 according to an embodiment of the present invention and main functions of each component will be described. FIG. 25 to FIG. 27 are explanatory diagrams illustrating an example of a region extracted by the target small region extraction unit 408.

  As shown in FIG. 1, the object detection device 4 adds a relationship graph storage unit 207, target small region extraction to the object detection device 1 instead of the relationship graph storage unit 107, the target small region extraction unit 108, and the person detection unit 109. A unit 408 and a person detection unit 409. Here, the relation graph storage unit 207 has been described in the second embodiment.

  The target small region extraction unit 408 detects the environment around the vehicle P, and acquires the relationship graphs 271 to 272 corresponding to the detected environment from the relationship graph storage unit 207. Further, small area image data is acquired from the distance storage unit 106.

  Then, based on the relationship graph 271 and the small area image data, the head image area having the thermal data of the person's head is extracted from the thermal image A as the person candidate area. Further, based on the relationship graph 272 and the small area image data, the body part image area having the thermal data of the human body part is extracted from the thermal image A as the person candidate area. When the torso image area exists below the head image area, a vertical edge area (an area corresponding to a person's foot) in which a straight line is drawn exists below the torso image area. Determine whether or not. This determination is performed by a general vertical edge detection process such as applying a Sobel filter or performing a differentiation process in the thermal image A in a region below the body image region. If a vertical edge area exists, the vertical edge area is extracted as a person candidate area. Then, the pixel data and distance data included in the extracted person candidate areas, that is, the person candidate area data is output to the person detection unit 409.

  A specific example of this processing will be described with reference to FIGS. That is, as shown in FIG. 25, based on the relationship graph 271 and the small area image data, head image areas a2 to a3, a51 to a52, a54 to a56, and a6 to a7 having thermal data of the human head are represented. Extracted as a person candidate area. Also, as shown in FIG. 26, based on the relationship graph 272 and the small area image data, the body image areas a4, a52 to a54, a7, and a9 having the thermal data of the human torso are extracted.

  Since the torso image area a53 having the heat data of the torso exists below the head image area a51 having the heat data of the head, it is determined whether or not there is a vertical edge area below the image area a53. To do. In this example, as shown in FIG. 25, since the image areas a55 to a56 are vertical edge areas, there are vertical edge areas. Therefore, the target small area extraction unit 408 extracts the image areas a55 to a56 as person candidate areas. Then, the pixel data and distance data included in the extracted person candidate areas, that is, the person candidate area data is output to the person detection unit 409.

  The person detection unit 409 detects a person from the person candidate area based on the person candidate area data given from the target small area extraction unit 408. Then, the person detection unit 409 outputs the person candidate area data in which the person is detected, that is, the person area data to the result output unit 110.

  Specifically, the person detection unit 409 detects a person by the following method. That is, a person candidate area is labeled, and when the labeled person candidate area is composed of a head image area, a torso image area, and a vertical edge area, a person is detected from the person candidate area.

  Since the procedure of the process by the object detection device 4 is the same as the procedure of the process by the object detection device 1, description of the procedure is omitted.

  As described above, in the fourth embodiment, when the person candidate area is composed of the head image area, the torso image area, and the vertical edge area, the object detection apparatus 4 starts from the person candidate area. Is detected. Therefore, since a person can be detected from the person candidate area based on the shape of the person candidate area, the person can be detected more reliably. Accordingly, for example, it is possible to prevent a person from being erroneously detected from a person candidate region having a thermal data equivalent to that of a person but having a shape different from that of the person.

(Fifth embodiment)
It will be described below with reference fifth form status of implementation of the present invention with reference to the drawings. First, based on FIG. 1, FIG. 28 and FIG. 29, components of the object detection device 5 according to one embodiment of the present invention and main functions of each component will be described. 28 to 29 are explanatory diagrams illustrating an example of an image region extracted by the target small region extraction unit 408.

  As illustrated in FIG. 1, the object detection device 5 includes the object detection device 4 including a result output unit 510 instead of the result output unit 110.

  The result output unit 510 calculates the position coordinates (x0, z0) of the person detected by the person detection unit 109 by the same processing as the result output unit 110 based on the person area data given from the person detection unit 109. To do. Further, the inclination of the vertical edge image detected by the person detection unit 109 is calculated by a general vertical edge detection process. Then, it is determined whether or not the calculated inclination of the vertical edge region changes with time. As a result, when it changes with the passage of time, the inclination is compared with a predetermined reference value. As a result, when the inclination becomes larger than a predetermined reference value, it is determined that the person detected by the person detection unit 409 is walking. And the position coordinate data regarding the position coordinate (x0, z0) mentioned above and the determination result data regarding the said determination result are output to the apparatus which requires the said position coordinate data and determination result data, for example, a driving assistance apparatus.

  Here, an example of the determination process will be described with reference to FIGS. That is, as shown in FIG. 28, the inclinations of the vertical edge images a55 to a56 detected by the person detection unit 109 are calculated by a general vertical edge detection process. Then, as shown in FIG. 29, when the calculated inclination of the vertical edge region changes with time, the inclination is compared with a predetermined reference value. As a result, when the inclination becomes larger than a predetermined reference value, it is determined that the person detected by the person detection unit 409 is walking. In this example, it is determined that the person is walking. Then, the position coordinate data related to the position coordinates (x0, z0) and the determination result data related to the determination result are output to a device that requires the position coordinate data and the determination result data, such as a driving support device.

  As described above, in the fifth embodiment, the object detection device 5 can also output determination result data on whether or not the person is walking in addition to the position coordinate data of the person to the driving support device or the like. As a result, the driving support device or the like can easily and quickly determine whether or not the person has jumped forward of the vehicle P (that is, whether or not the person has moved in the lateral direction) and whether or not the person has moved away from the vehicle P. can do. As a result, the driving assistance device or the like can perform various alarms and vehicle control more quickly when, for example, a person has jumped out ahead of the vehicle P. Further, when a person is moving away from the vehicle P, a predetermined alarm or vehicle control (for example, control for reducing the speed of the vehicle P) becomes unnecessary. Since the object detection device 5 can easily and quickly determine whether or not a person is moving away from the vehicle P, it can easily and quickly determine whether or not the predetermined alarm or vehicle control is necessary.

  Needless to say, the techniques described in the first to fifth embodiments can be combined with each other.

  In the first to fifth embodiments, a person is detected as a detection target. However, other objects such as road structures may be detected.

It is a block diagram which shows the structure of the object detection apparatus which concerns on one embodiment of this invention. It is the side view which showed the installation position of the camera. It is the top view which showed the installation position of the camera. It is explanatory drawing which shows an example of the thermal image acquired with a camera. It is explanatory drawing which shows an example of the thermal image acquired with a camera. It is a top view which shows the positional relationship of an object and a camera lens. It is explanatory drawing which shows an example of the thermal image acquired with a camera. It is explanatory drawing which shows an example of the thermal image acquired with a camera. It is explanatory drawing which shows an example of the thermal image acquired with a camera. It is explanatory drawing which shows an example of the thermal image acquired with a camera. It is explanatory drawing which shows an example of the thermal image acquired with a camera. It is explanatory drawing which shows an example of the thermal image acquired with a camera. It is explanatory drawing which shows an example of a relationship graph. It is explanatory drawing which shows an example of the thermal image acquired with a camera. It is explanatory drawing which shows the distance calculated about each pixel on a thermal image. It is explanatory drawing which shows an example of the image area | region extracted by the object small area extraction part. It is a side view which shows the positional relationship of a person and a camera lens. It is explanatory drawing which shows an example of the thermal image acquired with a camera. It is a top view which shows the positional relationship of a person and a camera lens. It is explanatory drawing which shows an example of the thermal image acquired with a camera. It is the flowchart which showed the procedure of the process by an object detection apparatus. It is explanatory drawing which shows an example of a relationship graph. It is explanatory drawing which shows an example of the image area | region extracted by the object small area extraction part. It is explanatory drawing which shows an example of the image area | region extracted by the object small area extraction part. It is explanatory drawing which shows an example of the image area | region extracted by the object small area extraction part. It is explanatory drawing which shows an example of the image area | region extracted by the object small area extraction part. It is explanatory drawing which shows an example of the image area | region extracted by the object small area extraction part. It is explanatory drawing which shows an example of the image area | region extracted by the object small area extraction part. It is explanatory drawing which shows an example of the image area | region extracted by the object small area extraction part.

Explanation of symbols

1 to 5: Object detection device 100: Camera (first image acquisition means)
101 ... Camera (second image acquisition means)
102 to 103 ... image memory 104 ... small area setting section 105 ... distance calculation section (distance calculation means)
106: Distance storage unit 107, 207 ... Relation graph storage unit 108, 208, 308, 408 ... Target small region extraction unit (small region extraction means)
109, 209, 409 ... person detection unit (detection means)
110, 510 ... result output unit (determination means)
170, 271-272 ... relation graph (corresponding data)

Claims (7)

In an object detection device mounted on a vehicle,
First image acquisition means for acquiring a first image comprising a plurality of pixels having state quantity data around the vehicle;
Second image acquisition means for acquiring a second image comprising a plurality of pixels having state quantity data around the vehicle;
Of the small area on the first image acquired by the first image acquisition means and the small area on the second image acquired by the second image acquisition means, Distance for calculating the parallax between the small area corresponding to the small area and calculating the distance from the vehicle to the object drawn in the small area on the first image based on the calculated parallax A calculation means;
Correspondence data between a distance from the vehicle to a predetermined detection object and state quantity data of the detection object included in pixels constituting the first image;
Based on the distance calculated by the distance calculation means and the corresponding data, a small area is extracted from a small area on the first image that is at a distance having the state quantity data of the detection target. Region extraction means;
An object detection apparatus comprising: detection means for detecting the detection target object from a small area extracted by the small area extraction means. The object detection device according to claim 1,
The correspondence data is created based on an actual measurement value of each part of the detection target. The object detection apparatus according to claim 1 or 2,
The small area extracting unit extracts, from the small area on the first image, a small area at a distance having the state quantity data of the detection target object and a small area around the small area. Object detection device. In the object detection device according to any one of claims 1 to 3,
The small area extracting means includes a head image area at a distance having the state quantity data of the person's head and a state quantity of the human torso as a small area at a distance having the state quantity data of the detection target A torso image area at a distance having data, and
In the case where the small region extracting unit extracts the head image region and the body part image region is extracted from below the extracted head image region, the detecting unit extracts the small region extracting unit. An object detection apparatus for detecting a person from a head image area and a torso image area extracted by the above. The object detection apparatus according to claim 4, wherein
When the vertical edge region in which a straight line is drawn is present below the trunk image region extracted by the small region extraction unit, the detection unit and the head image region extracted by the small region extraction unit and An object detection apparatus for detecting a person from a torso image area and the vertical edge area. The object detection apparatus according to claim 5,
And determining means for determining that the person detected by the detecting means is walking when the inclination of the vertical edge region changes with time and becomes larger than a predetermined reference value. An object detection apparatus characterized by that. A first step of obtaining a first image comprising a plurality of pixels having state quantity data around the vehicle;
A second step of obtaining a second image comprising a plurality of pixels having state quantity data around the vehicle;
Corresponds to the small area on the first image among the small area on the first image acquired by the first step and the small area on the second image acquired by the second step. A third step of calculating a parallax between the vehicle and the small area to be calculated, and calculating a distance from the vehicle to an object drawn in the small area on the first image based on the calculated parallax; ,
Corresponding data between the distance from the vehicle to the predetermined detection object and the state quantity data of the detection object included in the pixels constituting the first image, and the distance calculated by the third step A fourth step of extracting, from the small area on the first image, a small area at a distance having the state quantity data of the detection target;
A fifth step of detecting the detection object from the small region extracted by the fourth step;
An object detection method comprising: