JP2023003924A - Arithmetic device and speed calculation method - Google Patents

Abstract

To calculate a speed of a measurement object using a sensing result of a time when the distance information may not be directly calculated and a sensing result of a time when the distance information may be directly calculated.SOLUTION: An arithmetic device is mounted on a moving object and includes: a crossing angle identification unit that identifies a crossing angle, which is an angle at which the traveling direction of the moving object and the traveling direction of a detected object intersect; a sensing information acquisition unit which acquires a sensing result which includes at least a captured image taken by a camera; a ranging unit which calculates a distance from the detected object at a first time on the basis of the sensing result; an angle of view identification unit which specifies an angle of view, which is an angle with respect to the traveling direction of the moving object at a second time on the basis of the captured image; and a speed calculation unit which calculates a speed of the detected object on the basis of the crossing angle, the distance from the moving object at the first time, the angle with the moving object at the second time, the time difference between the first and second times, and the speed of the moving object.SELECTED DRAWING: Figure 3

Description

The present invention relates to an arithmetic device and a speed calculation method.

2. Description of the Related Art Various techniques are known for detecting a moving object in an image, such as an image captured by a camera. Japanese Patent Laid-Open No. 2004-100001 discloses an object region extracting means for extracting an object region from an image captured by an image capturing means, and distance information to an object outside the image capturing range from a reflected wave of an electromagnetic wave irradiated outside the image capturing range of the image capturing means. Using the distance information acquisition means to acquire and the distance information acquired by the distance information acquisition means, when the object outside the imaging range enters the imaging range of the imaging means, the object area extraction means extracts the object. and information creating means for creating prediction area information for extracting an object area.

JP 2015-195018 A

In the invention described in Patent Document 1, it is necessary to acquire distance information at different times in order to calculate the speed of the object to be measured, and the speed cannot be calculated in situations where distance information cannot be obtained.

An arithmetic device according to a first aspect of the present invention is an arithmetic device mounted on a mobile object, and is an intersection angle specifying an intersection angle, which is an angle at which the traveling direction of the mobile object and the traveling direction of a detected object intersect. a specifying unit, a sensing information acquisition unit that acquires a sensing result including at least an image captured by a camera, and a distance measurement unit that calculates a distance to the detected object at a first time based on the sensing result; an angle-of-view specifying unit that specifies an angle of view that is an angle with respect to the moving direction of the moving body at a second time based on the captured image; the intersection angle; the distance from the moving body at the first time; a velocity calculation unit configured to calculate the velocity of the detected object based on the angle with the moving body at the time point, the time difference between the first time and the second time, and the velocity of the moving body.
A speed calculation method according to a second aspect of the present invention is a speed calculation method executed by an arithmetic device mounted on a moving body, and is an angle at which the moving direction of the moving body and the moving direction of the detected object intersect. identifying the crossing angle; obtaining sensing results including at least an image captured by a camera; calculating a distance to the detected object at a first time based on the sensing results; specifying an angle of view that is an angle with respect to a traveling direction of the moving body at a second time based on the captured image, the intersection angle, a distance from the moving body at the first time, and the moving body at the second time and calculating the speed of the detected object based on the angle between and, the time difference between the first time and the second time, and the speed of the moving body.

According to the present invention, the speed of the object to be measured can be calculated using the sensing result at the time when the distance information cannot be directly calculated and the sensing result at the time when the distance information can be directly calculated. Problems, configurations, and effects other than those described above will be clarified by the following description of the mode for carrying out the invention.

Diagram explaining the positional relationship between the field of view and the vehicle Visual field correlation diagram FIG. 2 is a functional block diagram showing functions of the camera system according to the first embodiment; FIG. Conceptual diagram explaining the operation of the monocular detection unit Conceptual diagram for explaining parallax image generation by the stereo matching unit Conceptual diagram explaining how the stereo detector detects three-dimensional objects Flowchart showing speed calculation processing Flowchart showing details of steps S705 and S706 in FIG. Flowchart showing details of speed calculation processing shown in step S708 of FIG. A diagram showing a first specific example The figure which replaced the example of FIG. 10 with the relative position centering on the own vehicle. A diagram explaining the distance in FIG. 11 A diagram showing a second specific example The figure which replaced the example of FIG. 13 with the relative position centering on the own vehicle. A diagram showing a third specific example A diagram showing a fourth specific example Flowchart showing details of template selection processing shown in step S804 of FIG. Diagram explaining the size of the template Functional configuration diagram of a camera system in modification 1 Functional configuration diagram of a camera system in modification 2 FIG. 3 is a functional block diagram showing functions of a camera system according to a second embodiment; FIG. The figure which shows the calculation method of the speed by a speed calculation part.

Embodiments of the present invention will be described below with reference to the drawings. The examples are exemplifications for explaining the present invention, and are appropriately omitted and simplified for clarity of explanation. The present invention can also be implemented in various other forms. Unless otherwise specified, each component may be singular or plural.

When there are a plurality of components having the same or similar functions, they may be described with the same reference numerals and different suffixes. Further, when there is no need to distinguish between these constituent elements, the subscripts may be omitted in the description.

In the embodiments, there are cases where the processing performed by executing the program will be described. Here, the computer executes a program by means of a processor (eg, CPU, GPU) and performs processing determined by the program while using storage resources (eg, memory) and interface devices (eg, communication port). Therefore, the main body of the processing performed by executing the program may be the processor. Similarly, a main body of processing executed by executing a program may be a controller having a processor, a device, a system, a computer, or a node. The subject of the processing performed by executing the program may be an arithmetic unit, and may include a dedicated circuit for performing specific processing. Here, the dedicated circuit is, for example, FPGA (Field Programmable Gate Array), ASIC (Application Specific Integrated Circuit), CPLD (Complex Programmable Logic Device), or the like.

The program may be installed on the computer from a program source. The program source may be, for example, a program distribution server or a computer-readable storage medium. When the program source is a program distribution server, the program distribution server may include a processor and storage resources for storing the distribution target program, and the processor of the program distribution server may distribute the distribution target program to other computers. Also, in the embodiment, two or more programs may be implemented as one program, and one program may be implemented as two or more programs.

-First Embodiment-
A first embodiment of an arithmetic device will be described with reference to FIGS. 1 to 18. FIG. The arithmetic device according to the present embodiment detects a detected object (hereinafter also referred to as a “measurement target”) from the detection and distance measurement results at the time when it exists at an identifiable distance, and from the sensing results other than that time. The speed of the detected object is specified by using the angle of view of the detected object and the speed of the own vehicle. In this embodiment, it is assumed that the course of the own vehicle and the course of the detected object intersect substantially at right angles. The velocity of the detected object is obtained by dividing the time difference from the amount of movement of the detected object between two times.

Of the amount of movement of the detected object, the amount of movement in the depth direction viewed from the computing device is determined to be the amount of movement of the own vehicle. By specifying the depth distance at the other time from the distance and the amount of movement at one time, and specifying the distance in the horizontal direction from the angle of view of the object at the other time, the amount of movement in the horizontal direction can also be determined. Identify. The above-mentioned identification of the near-orthogonal movement direction is performed in object type identification. When discriminating the type of the object with the discriminator, the direction in which the object is facing is discriminated, and whether the direction is a direction of movement that is nearly orthogonal at the angle of view is specified.

The image processing system according to the present embodiment detects a target object from an image (hereinafter referred to as "captured image") captured by a camera. The target object is, for example, a moving object, a three-dimensional object, or a specific object. This "moving object" refers to an object that is moving relative to the background. However, the “background” is not limited to an area composed of pixels that do not change in the captured image. For example, in the case of an image captured by a camera mounted on a running vehicle, the background is the road surface whose imaging position changes as the vehicle runs, and pedestrians and vehicles moving on the road surface are moving objects. . Further, for example, in the case of a fixed surveillance camera, moving objects include people, animals, and objects that move within the camera image.

A "three-dimensional object" is, for example, an object having a height relative to the road surface, such as a vehicle, a pedestrian, a two-wheeled vehicle, and a utility pole. A “specific object” refers to a specific object such as a vehicle, a pedestrian, a motorcycle, etc., which a designer intentionally determines to detect. Templates, patterns, machine learning dictionaries, and the like are prepared in advance for specific objects, and objects that have a high degree of similarity with the templates, patterns, and machine learning dictionaries are detected. The image processing system performs this detection as well as detects an object within a certain range from the camera and calculates the distance and velocity of the object.

FIG. 1 is a diagram for explaining the positional relationship between the viewing angle and the vehicle. Vehicle 100 includes camera system 101 , left imaging section 102 , and right imaging section 103 . The left imaging unit 102 is an imaging unit arranged on the left side of the vehicle 100. The optical axis 102Ax faces the front of the vehicle 100 and has a wide right field of view 1021 on the right side of the central axis of the vehicle 100 in the figure. The right imaging unit 103 is arranged on the right side of the vehicle 100. The optical axis 103Ax faces the front of the vehicle 100 and has a wide left field of view 1031 on the left side of the central axis of the vehicle 100 in the figure. Vehicle 100 includes a vehicle speed sensor and at least one of a steering angle sensor and a yaw rate sensor. The image captured by the right imaging unit 103 is hereinafter referred to as a right image 203 , and the image captured by the left imaging unit 102 is hereinafter referred to as a left image 202 .

As shown in FIG. 1 , right field of view 1021 and left field of view 1031 overlap in front of vehicle 100 . This overlapping visual field is hereinafter referred to as "stereo visual field", the visual field obtained by removing the stereo visual field from the right visual field 1021 is referred to as the "right monocular visual field", and the visual field obtained by removing the stereo visual field from the left visual field 1031 is referred to as the "left monocular visual field". call.

FIG. 2 is a diagram showing the correlation of fields of view. The upper part of the figure shows a left image 202 captured by the left imaging unit 102, the middle part of the figure shows a right image 203 that is an image captured by the right imaging unit 103, and the lower part of the figure shows a composition created using the left image 202 and the right image 203. An image 204 is shown. The composite image 204 is divided into a right monocular region 204R, a stereo region 204S, and a left monocular region 204L.

The right monocular region 204R corresponds to the region of the right monocular visual field described with reference to FIG. The stereo region 204S corresponds to the stereo field of view region described with reference to FIG. The left monocular region 204L corresponds to the left monocular visual field region described with reference to FIG.

FIG. 3 is a functional block diagram showing functions of the camera system 101 according to the first embodiment. The camera system 101 may be realized as a single piece of hardware, an arithmetic device, or may be realized by a plurality of hardware devices. The camera system 101 includes a left imaging unit 102, a right imaging unit 103, a stereo matching unit 301, a monocular detection unit 302, a monocular distance measurement unit 303, a template generation unit 304, an image storage unit 305, a similar part A search unit 306, a view angle identification unit 307, a stereo detection unit 308, a speed calculation unit 309, a vehicle speed input unit 310, a vehicle steering angle input unit 311, a yaw rate input unit 312, and an intersection angle identification unit 1901Z. and Camera system 101 is connected to vehicle speed sensor 313 , vehicle steering angle sensor 314 , and yaw rate sensor 315 mounted on vehicle 100 .

Crossing angle specifying unit 1901Z specifies the crossing angle, which is the angle at which the traveling direction of vehicle 100 intersects the traveling direction of another vehicle. In the present embodiment, the crossing angle is always assumed to be 90 degrees. Therefore, the crossing angle specifying unit 1901Z in the present embodiment does not perform any particular calculation, and may be configured to hold that the value of the crossing angle is "90" degrees. For example, the crossing angle specifying unit 1901Z can be realized by a nonvolatile storage device.

The left imaging unit 102 and the right imaging unit 103 include imaging sensors, and are configured as known cameras, for example. A lens is attached to the imaging sensor and images the outside world of the device. The imaging sensor includes, for example, a CMOS (Complementary Metal Oxide Semiconductor) and converts light into electrical signals.

The information converted into electrical signals by the imaging sensors of the left imaging unit 102 and the right imaging unit 103 is further converted into image data representing captured images in the left imaging unit 102 and the right imaging unit 103 . The image data includes luminance values of pixels. A luminance value can be expressed as a digital value, for example, and is expressed as a luminance value for each color such as RGB (Red Green Blue) or RC (Red Clear), or as a monochrome luminance value. The left imaging unit 102 and the right imaging unit 103 convert image data into a stereo matching unit 301, a monocular detection unit 302, a monocular distance measurement unit 303, a template creation unit 304, an image storage unit 305, a similar part search unit 306, and an angle of view specification. 307. Left imaging unit 102 and right imaging unit 103 transmit image data at predetermined intervals, for example, every 17 milliseconds.

The left imaging unit 102 and the right imaging unit 103 have a function of changing exposure conditions of the imaging sensor. For example, an image sensor has an electronic shutter such as a global shutter or a rolling shutter, and can shoot with an arbitrary exposure time. When a photodiode that accumulates charge when receiving light is used as the light receiving element, the exposure time is the time from resetting the accumulated charge of the photodiode to extracting the charge to read out information related to the luminance value. Point.

If the exposure time is lengthened, a large amount of electric charge is accumulated, so that the read luminance value is increased. On the other hand, if the exposure time is shortened, less charge is stored, so the readout brightness is lower. Therefore, the brightness of the image obtained from the imaging sensor changes in correlation with the exposure time.

The left imaging unit 102 and the right imaging unit 103 convert the amount of charge taken out from the photodiode into a voltage and perform A/D (Analog Digital) conversion to obtain a digital value. The left imaging unit 102 and the right imaging unit 103 have amplifiers used for A/D conversion, and may be configured so that the gain of the amplifiers can be changed as part of the exposure conditions. In that case, the read luminance value changes according to the gain setting of the amplifier. A higher gain results in a higher luminance value, and a lower gain results in a lower luminance value.

Here, in general, when the luminance value decreases due to changes in exposure conditions (exposure time and gain in the above example), the luminance value of a dark object becomes 0, or the luminance value of an object with a low contrast ratio becomes 0. It may become uniform, and it may not be possible to distinguish outlines, shading, etc. This problem is conspicuous when the luminance value is expressed as a digital value, but the same problem can occur when it is expressed as an analog value. Similarly, when the luminance value increases due to a change in exposure conditions, the luminance value of a bright object becomes the maximum value, and the outline of the object and the shading of the object cannot be determined in some cases. Therefore, it is necessary to set the exposure conditions according to the brightness of the object to be photographed.

The left imaging unit 102 and the right imaging unit 103 can shoot moving images by repeatedly releasing the electronic shutters in time series and acquiring image data for each shutter. The number of times the electronic shutter is released and image data is output per unit time is called a frame rate, and the frame rate per second is expressed in units of FPS (Frame Per Second).

The captured images acquired by the left imaging unit 102 and the right imaging unit 103 are the result of sensing by the imaging elements, and therefore the captured images are also referred to as "sensing results" below. Also, since the left imaging unit 102 and the right imaging unit 103 acquire sensing information, they can be called "sensing information acquisition units". Also, hereinafter, in order to distinguish between the left imaging unit 102 and the right imaging unit 103, the former may be referred to as a "first imaging unit" and the latter may be referred to as a "second imaging unit". However, this classification is for convenience, and the names are interchangeable.

The stereo matching unit 301 receives data including image data from the left imaging unit 102 and the right imaging unit 103 and processes the data to calculate parallax. Parallax is a difference in image coordinates in which the same object is captured, which is caused by differences in the positions of a plurality of imaging units. The parallax is large at short distances and small at long distances, and the distance can be calculated from the parallax.

The stereo matching unit 301 corrects distortion of image data of the left image 202 and the right image 203 . For example, distortion of image data is corrected so that objects having the same height and the same depth distance, which are called a central projection model or a perspective projection model, are arranged horizontally in the image coordinates. It should be noted that the reason why the left imaging unit 102 and the right imaging unit 103 are arranged side by side in the left-right direction is corrected so that they are aligned in the horizontal direction. Using the corrected left image 202 and right image 203, one of them is used as reference image data as a reference, and the other is used as comparison image data to be compared, and a parallax is obtained.

In the above description, an example using central projection has been described, but when the angle of view is wide, projection may be performed on a cylindrical or spherical surface in order to reduce the size of the image data. When performing matching at that time, it is necessary to perform matching using a search line along which objects at the same distance on the projection plane line up.

When obtaining the parallax, first, the vertical coordinates of the vanishing points of the reference image data and the comparison image data are matched. For each coordinate of the reference image data, which horizontal coordinate of the same vertical coordinate of the comparison image data shows the same object is determined by, for example, SSD (Sum of Squared Difference) or SAD (Sum of Absolute Difference). method.

However, it is not limited to SSD or SAD, and other methods may be used. Matching of the same object may be performed by combining methods.

A monocular detection unit 302 detects a specific object appearing in the left image 202 or the right image 203 . A specific object is a three-dimensional object or a moving object, specifically a pedestrian, a vehicle, a bicycle, or the like. Detection detects a specific object within a certain range from the camera. The detected result holds image coordinate information of the left image 202 or the right image 203 in which the detected object is captured. For example, they are held as vertical and horizontal image coordinates of the upper left and lower left of a rectangular frame surrounding the object to be detected (hereinafter referred to as "monocular detection frame").

A type identification unit 317 identifies the type of an object detected by the monocular detection unit 302 or the stereo detection unit 308 . For the type of object, which is an object assumed to perform a crossing motion, such as a four-wheeled vehicle, a two-wheeled vehicle, and a pedestrian, the orientation of the front surface of the object, which is the direction of movement, is also specified. The front orientations of these moving bodies are the front orientations of the objects with respect to the camera system 101 . In this embodiment, the operation is limited to the directions of movement that are nearly orthogonal. Whether the direction of movement is nearly orthogonal or not is determined by the angle of view and the orientation of the front surface of the object with respect to the camera system 101 . For example, in the case of an object captured on the right side of the camera system 101, the surface and side of the object in the advancing direction are captured. In the case of an angle of 45°, an object with a direction of motion close to orthogonal can be identified when the front and side faces are viewed at a 45° orientation. The type identification unit 317 transmits the determination result as to whether or not the moving direction of the object is nearly orthogonal to the speed calculation unit 309 .

A monocular ranging unit 303 identifies the position of a specific object detected by the monocular detection unit 302 and obtains the distance and direction from the left imaging unit 102 or the right imaging unit 103 . For example, the specified distance and direction are represented by a coordinate system that can specify the position on the plane of the depth distance in front of the vehicle 100 and the lateral distance in the lateral direction. However, information expressed in a polar coordinate system expressed by a Eugrid distance, which is a distance from the camera, and a direction may be held, and a trigonometric function may be used for interconversion between the depth and two horizontal axes. A monocular distance measurement unit 303 uses an overhead image obtained by projecting the left image 202, the right image 203, and the combined image 204 onto the road surface, and calculates the vertical and horizontal coordinates of the overhead image and the vertical and horizontal coordinates of the overhead image with respect to the actual road surface. Locating a location from a directional scale.

However, it is not essential to use a bird's-eye view image to specify the position. It may be specified by performing geometric calculations using extrinsic parameters of the position and orientation of the camera, and information on the focal length, the pixel pitch of the imaging device, and the distortion of the optical system.

The template creation unit 304 selects one of the captured images, cuts out a specific region from this captured image (hereinafter referred to as a “detected image”), and uses it as a template. Specifically, the template creation unit 304 creates a template for searching for similar portions from pixel information inside and around the monocular detection frame of the object detected by the monocular detection unit 302 . This template is scaled to a predetermined image size. When scaling a template, maintain the aspect ratio or do not change it significantly. After the enlargement/reduction process, either the brightness value of the pixel itself or the reduced image is divided into a plurality of small regions (hereinafter also referred to as "kernels"), and the relationship between the brightness values in the kernel is stored as a feature of the image. . Image features are extracted in various forms, and there are various feature amounts such as left-right and top-bottom average brightness differences in the kernel, average brightness differences between the periphery and the center, and brightness averages and variances. Any form may be used.

The image holding unit 305 holds the left image 202 and the right image 203 for a certain period of time, or until a certain number of left images 202 and right images 203 at different times are accumulated. A storage device such as a DRAM (Dynamic Random Access Memory) is used for this holding. A plurality of addresses and ranges in the storage device to be held may be determined in advance, and the transfer destination address may be changed for each captured image, and after one cycle, the area where the old image is stored may be overwritten. Since the address is determined in advance, there is no need to notify the address when reading out the held image.

A similar part searching unit 306 searches for a part similar to the template created by the template creating unit 304 from the image stored in the image holding unit 305 . Hereinafter, an image in which the similar portion search unit 306 searches for a similar portion to the template will be referred to as a “search target image”. The search target image is an image different from the image at the time detected by the monocular detection unit 302 . The selection of the image at which time is based on the selection of the image captured at the time immediately preceding the image at the detected time. It is highly probable that the similar part exists near the coordinates of the monocular detection frame at the time of detection, and it is expected that there will be little change in brightness due to changes in the orientation of the detection object and changes in exposure with respect to the template. , a highly accurate search is possible.

However, if the imaging interval is short, an image taken two or more times before may be selected as the search target image. Also, the image to be selected may be changed according to the vehicle speed. For example, if the vehicle speed is greater than or equal to the threshold, an older image that is closer to the captured time of the detected image is selected. You can choose an old image. When the vehicle speed is high, if the image is taken at a time close to the detected time, the appearance does not change significantly with respect to the template, and it is easy to ensure search accuracy. In addition, when the vehicle speed is slow, the positional change of the detected object in the image for searching for a similar part increases with respect to the detected time, and the accuracy of the calculated speed can be improved.

The angle-of-view specifying unit 307 specifies the horizontal angle of view of the camera for a specific object appearing in the similar location specified by the similar location searching unit 306 . However, if the height is specified along with the position, then the vertical image is also specified. Also, even if there is a possibility that the camera rolls, the speed can be calculated with high accuracy by specifying both the horizontal and vertical angles of view. The horizontal angle of view can be obtained by a trigonometric function from the ratio of the depth distance and the horizontal distance.

A stereo detection unit 308 detects, from the parallax image created by the stereo matching unit 301, portions within a certain size range at the same distance as a three-dimensional object. Detected objects carry parallax image coordinate information. For example, they are held as vertical and horizontal image coordinates of the upper left and lower left of a rectangular frame (hereinafter referred to as "stereo detection frame") surrounding the object to be detected.

A stereo ranging unit 316 identifies the position of the object detected by the stereo detection unit 308 to identify the distance and direction. For example, the specified distance and direction are represented by a coordinate system that can specify the position on the plane of the depth distance in front of the vehicle 100 and the lateral distance in the lateral direction. However, information expressed in a polar coordinate system expressed by a Eugrid distance, which is a distance from the camera, and a direction may be held, and a trigonometric function may be used for interconversion between the depth and two horizontal axes.

A stereo ranging unit 316 calculates the distance in the depth direction from the parallax of the object. The stereo ranging unit 316 uses an average or mode when there is variation in the calculated parallax of the object. When the disparity of parallax is large, a method of taking a specific outlier may be used. The horizontal distance is obtained from the horizontal angle of view and the depth distance of the detection frame of the stereo detection unit 308 using a trigonometric function.

The speed calculation unit 309 receives the distance measurement result from the monocular distance measurement unit 303 or the stereo distance measurement unit 316, receives the angle of view of the similar part from the angle of view identification unit 307, and inputs the vehicle speed input unit 310 and the vehicle steering angle input unit 311. and the vehicle behavior information from the yaw rate input unit 312, and calculates the speed of the detected specific object. However, instead of receiving the vehicle speed from the vehicle speed input unit 310, the speed calculation unit 309 may use the calculated differential value of the relative depth distance as the vehicle speed. This is useful when the depth distance can be measured with high accuracy and the horizontal distance is measured with low accuracy. Also, if the depth velocity is used instead of the vehicle velocity, it is possible to accurately predict a collision with a crossing vehicle that is not orthogonal.

Vehicle speed input unit 310 receives vehicle speed information from vehicle speed sensor 313 . Vehicle steering angle input unit 311 receives vehicle steering angle information from vehicle steering angle sensor 314 . Yaw rate input unit 312 receives the turning speed of the vehicle from yaw rate sensor 315 .

Vehicle speed sensor 313 is a device mounted on vehicle 100 that measures the speed of the vehicle. Vehicle speed sensor 313 measures the speed of the vehicle from, for example, the rotational speed of the wheels and the outer circumference of the wheels. The vehicle steering angle sensor 314 is a device mounted on the vehicle 100 that measures the steering angle of the steered wheels. Vehicle steering angle sensor 314 is, for example, a steering rotation angle sensor mounted on vehicle 100 . Yaw rate sensor 315 is a device that measures the turning speed of vehicle 100 . Yaw rate sensor 315 is, for example, an inertial sensor mounted near the center of gravity of vehicle 100 or at a plurality of locations.

Stereo matching unit 301, monocular detection unit 302, monocular distance measurement unit 303, template creation unit 304, similar part search unit 306, angle of view identification unit 307, stereo detection unit 308, and speed calculation unit 309 are realized as follows. may be That is, these functional blocks are stored in the ROM by the CPU using a CPU as a central processing unit (not shown), a ROM as a read-only storage device (not shown), and a RAM as a readable/writable storage device (not shown). It may be realized by deploying the program to be executed in RAM and executing it.

However, these functional blocks may be implemented by FPGA, which is a rewritable logic circuit, or ASIC, which is an application-specific integrated circuit, instead of the combination of CPU, ROM, and RAM. Also, these functional blocks may be realized by a combination of different configurations, for example, a combination of CPU, ROM, RAM and FPGA, instead of the combination of CPU, ROM, and RAM.

Vehicle speed input unit 310, vehicle steering angle input unit 311, and yaw rate input unit 312 may be realized by a communication port, for example, a communication module compatible with IEEE 802.3, or by an AD converter capable of reading voltage and current. may be implemented.

4A and 4B are conceptual diagrams for explaining the operation of the monocular detection unit 302. FIG. A first right image 203-1 shown in the upper left of FIG. 4 is the right image 203 captured at time t1. The first projected image 401 shown in the upper right of FIG. 4 is obtained by projectively transforming the first right image 203-1 onto the road surface. In this projective transformation, one pixel is transformed so as to correspond to a constant distance in the depth and width of the road surface. For the conversion, the camera's height from the road surface, which is the camera's posture, and the orientation of the camera's optical axis with respect to the road surface are calculated using tilt rotation, pan rotation, and roll rotation information. corresponds to where in the first right image 203-1, an affine table for conversion is created. Affine transformation is performed using this affine table to obtain a first projected image 401 in which the first right image 203-1 is projected onto the road surface.

A second right image 203-2 shown in the middle left of FIG. 4 is the right image 203 captured at time t2. Time t2 is a time later than time t1 described above. A second projected image 402 shown in the middle right of FIG. 4 is obtained by projectively transforming the second right image 203-2 onto the road surface. A differential image 404 shown in the lower part of FIG. 4 is created using the first projected image 401 and the second projected image 402 as follows.

The difference image 404 is obtained by translating the second projection image 402 in a direction that cancels out the movement of the vehicle 100 between time t1 and time t2, and superimposing it on the first projection image 401 to extract the difference. obtained by This difference can be obtained not only by comparing the brightness of the first projected image 401 and the second projected image 402, but also by comparing the intensity and direction of the edge of each image and extracting the difference. The change in the angle at which an object is reflected can be obtained as a difference. A three-dimensional object can be detected by using the difference image 404 generated in this way.

FIG. 5 is a conceptual diagram for explaining parallax image generation by the stereo matching unit 301. As shown in FIG. A parallax image 500 shown in the lower part of FIG. 5 is an image obtained by using the right image 203 as a reference image and calculating the parallax with the left image 202 . Parallax is a difference in image coordinates in which the same object is captured, which is caused by differences in the positions of a plurality of imaging units. The parallax is large at short distances and small at long distances, and the distance can be calculated from the parallax.

6A and 6B are conceptual diagrams for explaining a three-dimensional object detection method by the stereo detection unit 308. FIG. A strip parallax image 600 is obtained by dividing the parallax image 500 into vertically long strips, in other words, into a plurality of vertically long regions 601 aligned in the horizontal direction of the figure. The stereo detection unit 308 creates a histogram that indicates the frequency of presence of the same distance or parallax for each divided area and evaluates it. In this histogram, the horizontal axis, such as the reciprocal of the distance, is set such that the frequency is constant in the area where only the road surface is shown. Since only the road surface is shown in the strip area indicated by reference numeral 603, the frequency is constant as indicated by the histogram indicated by reference numeral 6031. FIG.

In the histogram 6021 of the area 602 including the three-dimensional object, the parallax of the road surface up to the distance where the three-dimensional object exists has a constant frequency, and the frequency of the distance Z0 where the three-dimensional object exists has a peak. When there is a three-dimensional object in this way, since a peak occurs in the distance of the three-dimensional object having a height equal to or higher than a certain height in the mode value in the histogram, the distance of the three-dimensional object can be specified from the mode value.

After the distance to the three-dimensional object can be specified, the stereo detection unit 308 specifies the position of the three-dimensional object in the parallax images. Reference numeral 604 is an example of displaying the detection result of a three-dimensional object on the parallax image 500 . A rectangle 6022 indicates the range in which the three-dimensional object is detected. The stereo detection unit 308 performs labeling processing and clustering processing for identifying a set of parallax coordinates at the identified distance within the horizontal coordinate range of the three-dimensional object identified by the histogram. Note that if the parallax of a three-dimensional object contains invalid values of parallax due to invalid parallax or parallax error, or if there are many parallax errors, for example, clustering processing using Mean-Shift may be used to It is preferable to use a method that can extract a set of parallaxes at a constant distance from the parallaxes.

FIG. 7 is a flowchart showing speed calculation processing. First, in step S701, the stereo matching unit 301 performs stereo matching. In stereo matching, a parallax image 500 is generated. In subsequent step S702, the monocular detection unit 302 or the stereo detection unit 308 extracts a specific object, three-dimensional object, or moving object, and performs object detection processing for outputting the coordinates of any object on the image coordinates.

In subsequent step S703, the velocity calculation unit 309 calculates the distance of the object detected in immediately preceding step S702. In subsequent step S704, the velocity calculation unit 309 extracts the features of the object detected by the template creation unit 304 as a template. For object detection, the results obtained by the monocular detection unit 302 or the stereo detection unit 308 are used. The similar part searching unit 306 executes both the subsequent search range setting processing in step S705 and the similar part identifying process in step S706. An outline of steps S705 and S706 will be described here, and details will be described later with reference to FIG.

In step S705, a range of coordinates on the image for searching for similar portions is set. With this setting, you can also set the representative points of the template to be searched and the template to be queried, such as the range at the bottom center, the width and height of the image clipping range when queried with the template, and the width and height. Also decide the scale to be decided. This setting also sets the interval between points in the search range for query processing, depending on the query method for similar locations. The representative point is the time at which an image for searching for a similar location and an image for which object detection S702 and object ranging S703 are performed are captured based on the behavior information of the vehicle with respect to the distance measured in step S703. Set the position as the center based on the assumption of how close or far it would have been if it had been stationary.

In step S706, the search range set in S705 is searched for similar portions to the template created in S704. The image to be searched is an image captured at a time different from that of the images detected and distance-measured in S702 and S703. A speed calculation unit 309 searches and outputs the image coordinates of the most similar part. Furthermore, at the most similar location, the classifier is used to identify the type of the object to determine whether it matches the type of the detected object, or regression estimation is performed to estimate the coordinates of the detected object type to determine whether the coordinates match. We also do a recalculation of what we are doing. By performing this recalculation, it is possible to improve the accuracy of identifying similar portions. In addition, by this verification, in the image to search for similar locations, when the detected object cannot be seen in the camera image due to obstructions, light adjustment, and exposure problems, the speed of the object is calculated differently from the actual speed. prevent

In step S707, the view angle specifying unit 307 specifies the view angle of the similar portion specified in step S706. In this step S707, from the image coordinates of the similar part, the camera position of the object is specified using camera geometry, etc., and the angle of view is specified by a trigonometric function from the ratio of the depth distance and the horizontal distance. If it is difficult to identify changes in camera posture or height, and camera geometry errors are a concern, the angle of view can be calculated from the distance of the coordinates of similar points from the optical axis. The angle of view may be specified without specifying the posture change or height change. However, if you specify and calculate changes in the camera's posture and height, the accuracy of the speed calculation by camera geometric calculation will increase, so only when you fail to specify the camera's posture and height changes. The angle of view is calculated from the distance of the coordinates of the similar point from the optical axis, and in this case, the fact that the reliability of the speed is lowered is also output.

In subsequent step S708, the speed calculation unit 309 performs speed calculation processing. The speed calculation unit 309 calculates the speed from the angle of view specified in step S707 and the information on the object ranging and vehicle behavior in step S703. For the vehicle behavior, information on vehicle speed, vehicle steering angle, and yaw rate obtained by vehicle speed input section 310, vehicle steering angle input section 311, and yaw rate input section 312 is used.

FIG. 8 is a flowchart showing details of steps S705 and S706 in FIG. First, in step S801, the speed calculation unit 309 identifies the amount of movement between frames using vehicle behavior information. Specifically, the speed calculation unit 309 calculates the difference between the photographing time of the detection image for which the template is created and the photographing time of the image for searching for similar locations, and multiplies the difference by the speed of the vehicle included in the vehicle behavior information. Calculate the amount of movement. In subsequent step S802, the velocity calculation unit 309 identifies the number of pixels in the height and width of the detected object on the detected image.

In subsequent step S803, the velocity calculation unit 309 estimates the size of the object on the detection image in the search target image, that is, the number of pixels. The details of the processing in step S803 are as follows. First, in step S803, the camera geometry is used to identify the actual size of the object on the detected image. Next, with respect to the distance of the detected object on the detected image, the position of the object moved by the amount of movement between frames identified in step S801 is specified, and the position of the object on the search target image when this object is stationary is determined. Locate. A case where the object is moving will be described later. Next, the number of pixels in height and width on the image when the object having the specified actual size exists at the specified object position on the search image is specified using camera geometry or the like. As described above, in step S803, the number of pixels in the height and width of the object on the search image is estimated.

If the identified object is not stationary, proceed as follows. The moving speed may be assumed based on information about the type and direction of the object known at the time of detection, and the object may be in uniform linear motion. The assumed moving speed is about 30 to 80 km/h for a four-wheeled vehicle, about 10 to 20 km/h for a bicycle, and about 5 to 8 km/h for a pedestrian. Velocity may be assumed. By assuming the speed, it is possible to improve the possibility of searching for similar locations of objects with a high collision probability in a short time, and to increase the success rate of accident avoidance.

Alternatively, the moving speed of the object may be assumed from the detected direction of the object, and the object may be assumed to be in uniform linear motion. In that case, for example, eight directions are specified, and the depth or lateral velocity of the object is assumed for the process of specifying the depth and lateral distance according to the left-right direction, the front-back direction, and the oblique direction therebetween. In the front-rear direction, all velocities assumed for the object are assumed to be in the depth direction, and the sign of the velocity value is determined according to the orientation.

In the left-right direction, all the velocities of the object are assumed to be in the left-right direction, and the sign of the velocity value is determined according to the direction. In the oblique direction, the sign of the velocity value is determined according to the direction, with the velocity 1/√2 times the velocity being the magnitude of the depth and lateral velocity. Search time can be shortened by assuming the speed of an object based on its type and orientation. In addition, the accuracy of the calculated speed can be improved by taking the assumption of the speed of the object into consideration when calculating the speed. The above is the description of step S803.

In subsequent step S804, the template creation unit 304 determines the template size. The template size is the size to which the image is scaled when extracting features as a template. Excessive enlargement during template creation and search may reduce search accuracy. It is determined from variations of the template size designed in the process.

Specifically, the template creation unit 304 selects a template size variation prepared in advance for a smaller number of pixels from among the number of pixels specified in step S802 and the number of pixels estimated in step S803. Generally, a camera system installed facing the front of the vehicle 100 is installed for the purpose of avoiding a collision accident while moving forward. When moving forward, the search target image used in step S803 is an image earlier than the detected image used in step S802. Therefore, since the search target image is farther and has fewer pixels, the template size is determined using the number of pixels estimated in step S803.

In step S805, the template creation unit cuts out the periphery of the detection frame in the detection image. Specifically, the template creation unit widens the height or width according to the aspect ratio of the template size determined in step S804, and also increases the size of the margin around the object, which is assumed in the processing used in the search processing in advance. to cut out the expanded area.

In subsequent step S806, the template creation unit enlarges or reduces the image cut out in step S805 to the template size determined in step S804. When scaling, maintain the aspect ratio or do not change it significantly. In subsequent step S807, the template creation unit performs preprocessing such as brightness adjustment and noise removal processing for the image that has been enlarged or reduced in step S804. In subsequent step S808, the template creation unit extracts feature amounts to be used as a template from the image preprocessed in step S807, and the processing shown in FIG. 8 ends.

FIG. 9 is a flow chart showing details of the speed calculation process shown in step S708 of FIG. First, in step S901, the velocity calculation unit 309 uses the movement amount specified in step S801 to specify the depth distance of the detected object in the search target image, as in step S803. Specifically, using the amount of movement in the depth direction between frames identified in step S801, the position of the object moved by the amount of movement in step S801 with respect to the depth distance of the detection object on the detection image is identified. The object depth distance at the time when the search target image is captured when the object is stationary is specified.

In step S902, the speed calculation unit 309 identifies the horizontal distance of the similar portion of the search target image using the depth distance identified in step S901 and the angle of view identified in step S707. Specifically, when Z is the depth distance and θ is the angle of view, the horizontal distance X is calculated by the following equation (1).

X=Z·Tan(θ) (Formula 1)

In subsequent step S903, the velocity calculation unit 309 subtracts the depth distance and the lateral distance specified in steps S901 and S902 from the depth distance and lateral distance measured in step S703. By this calculation, the amount of movement between the photographing time of the detection image and the photographing time of the search target image is calculated.

In subsequent step S904, the speed calculation unit 309 differentiates the movement amount calculated in step S903 by the time difference between the detected image and the search target image to calculate the speed, and ends the processing shown in FIG. . Although not shown in the flow chart, depending on the type of detected object, the calculated speed is checked to see if it is within the expected speed range for the type, and if there is a large discrepancy may output that the velocity is unreliable. If the reliability is low, it is possible to suppress adverse effects due to erroneous control by suspending vehicle control or limiting the amount of control and the type of control.

Four specific examples will be described with reference to FIGS. 10 to 16. FIG. First, a first specific example will be described with reference to FIGS. 10 to 12. FIG. FIG. 10 is a diagram showing a state of collision between the vehicle 100 (hereinafter also referred to as "self-vehicle") and the colliding vehicle. In FIG. 10, the vehicle 100 runs from the bottom to the top, and the collided vehicle runs from the right to the left.

Reference numeral 1000 is the own vehicle at time T0, reference numeral 1001 is the own vehicle at time T1, reference numeral 1002 is the own vehicle at T2, reference numeral 1010 is the colliding vehicle at time T0, reference numeral 1011 is the colliding vehicle at time T1, and reference numeral 1012 is T2. is the colliding vehicle. In FIG. 10, both vehicles collide at time T2. The own vehicle and the colliding vehicle are in approximately uniform straight-ahead motion, and are not accelerating, decelerating, or turning.

FIG. 11 is a diagram of the relative position of the vehicle center, in which the example of FIG. 10 is replaced with the position of the collision-affected vehicle in the relative position with the position of the vehicle as the center. In the present embodiment, the position of the colliding vehicle with respect to the traveling direction of the own vehicle is called the "angle of view", like the angle of view of the camera. The angle of view 1120 is the angle formed by the straight line connecting the own vehicle and the collided vehicle 1010 at time T0 with the traveling direction of the own vehicle. The angle of view 1121 is the angle formed by the straight line connecting the own vehicle and the collided vehicle 1011 at time T1 with the traveling direction of the own vehicle. The angle of view 1122 is the angle formed by the straight line connecting the own vehicle and the collided vehicle 1012 at time T2 with the traveling direction of the own vehicle. In the present embodiment, since the trajectories of the own vehicle and the colliding vehicle are assumed to be perpendicular to each other, the lateral position of the colliding vehicle can be estimated using the angle of view and the depth distance.

FIG. 12 is a diagram for explaining distances in FIG. The position of each vehicle in FIG. 12 is the same as in FIG. However, the colliding vehicle at time T2 is omitted for convenience of drawing. The depth distance Z1 is the distance in the depth direction of the colliding vehicle 1011 from the own vehicle at time T1. The lateral distance X1 is the lateral distance of the colliding vehicle 1011 from the host vehicle at time T1. The depth distance Z0 is the distance in the depth direction from the own vehicle of the colliding vehicle 1010 at the time T0. The lateral distance X0 is the lateral distance of the collided vehicle 1010 from the host vehicle at time T0. The depth difference distance Z10 is the difference between the depth distance Z0 and the depth distance Z1.

The lateral distance X1 and the depth distance Z0 are calculated in the process of step S703. The depth difference distance Z10 is calculated from the vehicle behavior, and is calculated by the following equation 2 when the time between times T1 and T0 is T10 and the vehicle speed is V0.

Z10=V0·T10 (Formula 2)

The lateral distance X0 is calculated by the following equation 3, where the angle 1220 of the colliding vehicle at time T0 is set to "θ0".

X0=Z0·Tan(θ0)=(Z1+Z10)·Tan(θ0) (Formula 3)

A second specific example will be described with reference to FIGS. 13 and 14. FIG. In the second specific example, the own vehicle and the crossing vehicle do not collide. FIG. 13 is a diagram showing the travel trajectories of the own vehicle traveling straight ahead and the crossing vehicle crossing in front of the own vehicle. Reference numeral 1300 denotes the own vehicle at time T0, 1301 the own vehicle at time T1, 1302 the own vehicle at T2, 1310 the crossing vehicle at time T0, 1311 the crossing vehicle at time T1, and 1312 the crossing vehicle at T2. Since the crossing vehicle crosses the intersection before the own vehicle reaches the intersection, the crossing vehicle does not collide with the own vehicle.

FIG. 14 is a diagram of the relative positions of the own vehicle center, in which the example shown in FIG. 13 is replaced with the position seen by the crossing vehicle in the relative position with the position of the own vehicle as the center. The angle of view 1420 is the angle formed by the traveling direction of the vehicle and the straight line connecting the vehicle and the crossing vehicle 1310 at time T0. The angle of view 1421 is the angle formed by the straight line connecting the own vehicle and the crossing vehicle 1311 at time T1 and the traveling direction of the own vehicle.

The depth distance Z41 is the distance in the depth direction of the crossing vehicle 1311 from the own vehicle at time T1. The lateral distance X41 is the lateral distance of the crossing vehicle 1311 from the host vehicle at time T1. The depth distance Z40 is the depth direction distance of the crossing vehicle 1310 from the own vehicle at the time T0. The lateral distance X40 is the lateral distance of the crossing vehicle 1310 from the host vehicle at time T0. The depth difference distance Z410 is the difference between the depth distance Z0 and the depth distance Z1.

Lateral distance X41 and depth distance Z40 are calculated in the process of step S703. The depth difference distance Z410 is calculated from the vehicle behavior, and is calculated by the following equation 4 when the time between times T41 and T40 is T40 and the vehicle speed is V4.

Z410=V4·T40 (Formula 4)

The lateral distance X40 is calculated by the following equation 5, where the angle 1220 of the colliding vehicle at time T0 is set to "θ4".

X40=Z40·Tan(θ4)=(Z41+Z410)·Tan(θ4) (Formula 5)

A third specific example will be described with reference to FIG. In this third specific example, a situation will be described in which the accuracy of the calculated speed is improved or the speed is calculated in a short time. FIG. 15, like FIGS. 11 and 14, shows relative positional relationships with other vehicles centering on the position of own vehicle 100. FIG. Reference numeral 1510 is the relative position of the crossing vehicle at time T0, and reference numeral 1511 is the relative position of the crossing vehicle at time T1. A distance 1521 is the upper limit depth distance for ranging in the camera system 101 of the host vehicle.

Since the crossing vehicle 1511 at time T1 is in front of the depth distance 1521 that can be measured by the camera system 101, the camera system 101 calculates the speed as well as the distance measurement result. The depth distance of crossing vehicle 1510 at time T0 is slightly greater than distance 1521 described above, so even if it can be measured, the accuracy is low. is low and can result in erroneous brake control.

In the present embodiment, the speed is calculated using the distance measurement result of the crossing vehicle 1511 at time T1, the vehicle behavior information, and the angle of view information of the crossing vehicle 1510 at time T0. Accuracy can be improved. It can be said that the advantage of this embodiment is that the speed calculation can be made earlier than when the speed is calculated after the vehicle 1511 has come closer to the fusing vehicle 1511 .

When perspective projection of a captured image is performed on a plane with a constant depth distance, calibration is performed by arranging a calibration chart on a plane at a position with a constant depth distance. In this case, the accuracy of distortion correction of the captured image is often the same for each depth distance. If the geometrical accuracy of pixels is substantially the same for each depth distance, in principle, the farther the depth distance is, the worse the accuracy of distance measurement is in any of the distance measuring means in the present embodiment.

In the above description with reference to FIG. 15, the depth distance for ranging is constant at the distance indicated by reference numeral 1521. However, depending on the calibration method, optical system, and ranging principle, different ranging ranges and upper limit distances may be determined. good too. For example, when capturing a 360-degree image and processing it with an image projected onto a cylindrical or spherical surface, the distance measurement accuracy varies depending on the Euclidean distance from the camera. There are good cases.

A reference numeral 1522 indicates the range-finding limit distance on the near side. The monocular distance measurement unit 303 calculates the distance from the image coordinates of the contact point between the object and the road surface and the camera geometry. Can not. Therefore, distance measurement becomes impossible within a certain distance. A reference numeral 1520 indicates a range in which the camera system 101 can measure the distance. A range 1520 is an area sandwiched between an upper limit distance 1521 for distance measurement on the far side, a limit distance 1522 for distance measurement on the near side, and the upper limit of the angle of view of the sensor.

The camera system 101 uses the distance measurement result of the measurement object within the range 1520 and the angle of view of the measurement object outside the range 1520 to calculate the velocity when the measurement object exits or enters the range 1520. By doing so, it is possible to calculate the speed when the conventional technology cannot calculate the speed. In addition, when the object to be measured goes out of the range 1520, not only the speed but also the distance measurement result within the range 1520 and the amount of movement calculated when calculating the speed are added together to determine the position of the object to be measured at the time outside the range 1520. A distance may be calculated.

A fourth specific example will be described with reference to FIG. In the fourth specific example, similar to the third specific example, a situation will be described in which the accuracy of the calculated speed is improved or the speed is calculated in a short period of time.
Similar to FIG. 15 and the like, FIG. 16 shows relative positional relationships with other vehicles centering on the position of own vehicle 100 . Reference numeral 1610 is the relative position of the crossing vehicle at time T0, and reference numeral 1611 is the relative position of the crossing vehicle at time T1. A reference numeral 1620 indicates an angle of view of a monocular visual field, which is an angle of view that can be captured by at least one imaging unit. Reference numeral 1621 denotes a stereo viewing angle at which the fields of view of the left imaging unit 102 and the right imaging unit 103 overlap and stereo matching and stereo detection are performed.

The camera system 101 is capable of ranging not only within the range of the stereo viewing angle 1621 but also within the range of the monocular viewing angle 1620 . However, the distance measurement accuracy is higher within the range of the stereoscopic viewing angle 1621 than within the range of the monocular viewing angle 1620 . Therefore, the camera system 101 does not dare to use the distance measurement result at the monocular viewing angle 1620, but uses the distance measurement result at the stereo viewing angle 1621, which has a narrow range, to calculate the speed, thereby increasing the speed accuracy. can. Specifically, accuracy can be improved by calculating the speed using the angle of view of the position 1611 of the distance measurement result of stereo vision and the position 1610 of the detection result of monocular vision and the vehicle behavior.

Ranging errors include random errors, in which values are randomly distributed both positively and negatively, and systematic errors, in which average values are systematically offset positively or negatively. The tendency of the systematic error may change when the distance measurement means, sensor, imaging unit, and the like are switched. This may occur when there is misalignment between sensors or between imaging units. When the sensor or image pickup unit is switched, if the movement amount is obtained from the difference in the distance measurement results of different sensors or image pickup units to calculate the speed, accuracy may deteriorate. In the present embodiment, the accuracy can be improved by calculating the velocity using the distance measurement result using only one of the left imaging unit 102 and the right imaging unit 103 and the angle of view.

FIG. 17 is a flow chart showing the details of the template selection process in step S804 of FIG. First, in step S1701, the template creation unit 304 selects the smaller one of the number of pixels of the detection image and the number of pixels of the search target image calculated in steps S802 and S803, and defines this number of pixels as the "reference number of pixels". Determined as In subsequent step S1702, the template creation unit 304 determines whether or not the number of reference pixels exceeds the reference A, which is a predetermined threshold. If so, the process advances to step S1704. In step S1703, the template creation unit 304 determines template size 1, which is the first template size, as the template size.

In subsequent step S1704, the template creation unit 304 determines whether or not the number of reference pixels exceeds the reference B, which is a predetermined threshold. If so, the process advances to step S1706. In step S1705, the template creation unit 304 determines template size 2, which is the second template size, as the template size. In step S1706, the template creation unit 304 determines template size 3, which is the third template size, as the template size.

Criterion A is greater than Criterion B. The first template size is larger than the second template size and the second template size is larger than the third template size. The first template size and the reference A are the same or close values. The second template size and the reference B are the same or close values.

FIG. 18 is a diagram for explaining the sizes of templates. FIG. 18 is associated with the first specific example. At time T1, the distance between the own vehicle and the other vehicle is closer than at time T0, and the other vehicle is captured larger in the captured image. A reference numeral 1801 denotes an image of the detection location of the vehicle in the captured image captured at time T0. A reference numeral 1802 denotes an image of the detection location of the vehicle in the captured image captured at time T1. Reference numeral 1811 denotes an image that is referred to when searching for a similar part in the captured image captured at time T0. A reference numeral 1812 is an example of the number of pixels of the image when creating the template in the captured image captured at time T1.

In the example shown in FIG. 18, the number of pixels of the vehicle in the photographed image 1812 for which the template is to be created is smaller than the number of pixels of the vehicle in the photographed image 1811 to be searched for similar locations. Therefore, a template is created according to a predetermined template size close to the number of pixels of the vehicle in the photographed image 1812, and is also queried during the search.

According to the first embodiment described above, the following effects are obtained.
(1) The camera system 101 is mounted on the own vehicle 100 . The camera system 101 includes an intersection angle specifying unit 1901Z that assumes that the intersection angle between the traveling direction of the moving object and the traveling direction of the detected object is 90 degrees, and a photographed image obtained by the camera. A left imaging unit 102 and a right imaging unit 103 including sensing information acquisition units that acquire at least sensing results, and a monocular distance measurement unit 303 that is a distance measurement unit that calculates the distance to the detected object at the first time based on the sensing results. and a stereo ranging unit 316 . The camera system 101 further includes an angle-of-view specifying unit 307 that specifies an angle of view, which is the angle of the detected object with respect to the moving direction of the moving object, at a second time that is different from the first time based on the captured image; a speed calculation unit 309 that calculates the speed of the detected object based on the distance to the moving object at the first time, the angle to the moving object at the second time, the time difference between the first time and the second time, and the speed of the moving object; , provided. Therefore, the speed of the object to be measured can be calculated using the sensing result at the second time when the distance information cannot be directly calculated and the sensing result at the first time when the distance information can be directly calculated.

For example, regardless of the configuration of the present embodiment, it is possible to calculate the velocity of the measurement target using distance information measured by stereo distance measuring section 316 at different times. However, in this case, two distance measurements cannot be performed unless two measurement cycles are performed after the object to be measured enters the stereo field of view. In contrast, according to the configuration of the present embodiment, the angle of view is measured before the object to be measured enters the stereo field of view, and the velocity can be calculated by measuring the distance only once immediately after entering the stereo field of view. Speed can be calculated by timing.

(2) The captured images include a left image 202 captured by the left imaging unit 102 and a right image 203 captured by the right imaging unit 103 . The distance measurement unit is composed of a monocular distance measurement unit 303 and a stereo distance measurement unit 316 . A monocular distance measurement unit 303 calculates the distance to the detection object based on the shooting position of the detection object in the left image 202 or the right image 203 . A stereo ranging unit 316 calculates the distance to the detected object based on the parallax of the detected object in the left image 202 and the right image 203 . Therefore, it is possible to calculate the speed of the measurement object by using the information of the measurement object in the captured image for which the distance information cannot be directly calculated.

(3) The crossing angle specifying unit 1901Z assumes that the crossing angle is 90 degrees. At a crossroad where two roads intersect, the roads often cross each other at 90 degrees. Therefore, by limiting the case where the crossing angle is 90 degrees, which occurs frequently, the process of specifying the crossing angle can be omitted and the processing speed can be increased.

(4) The angle-of-view specifying unit 307 acquires image information of the moving object from the captured image at a certain time, and specifies the angle of view from the position of the moving object captured in the captured image at another time. Therefore, the speed of the object to be measured can be calculated without using the sensing result of a sensor that can directly measure the distance, such as a millimeter wave sensor.

(5) As described with reference to FIGS. 8 and 17, the camera system 101 calculates the number of pixels of the moving object in the captured image at a certain time and the movement assumed to be captured in the captured image at another time. A template creation unit 304 is provided for creating a matching template based on the smaller number of pixels of the body. The angle-of-view specifying unit 307 specifies the angle of view by searching the captured image using the template created by the template creating unit 304 . Therefore, erroneous matching in template matching can be reduced.

(6) The angle-of-view specifying unit 307 determines the angle of view based on the photographed image of the detected object existing outside the range-measurable area where the stereo range-finding unit 316, which is a range-finding unit, can perform range-finding. Identify. Therefore, in addition to the left image 202 and the right image 203 at a certain time, which are used by the stereo ranging unit 316, the left image 202 or the right image 203 captured at another time when the distance cannot be measured can be used for measurement. Can calculate target speed.

(7) The first time is later than the second time. Therefore, when the object to be measured approaches the vehicle 100 from infinity and the distance cannot be measured accurately, the angle of view is measured first, and then the object to be measured approaches the vehicle 100. By measuring the distance, the speed can be calculated with high accuracy and at an early timing.

(8) The first time is earlier than the second time. Therefore, by first measuring the distance and then measuring the angle of view, as in the case where the object to be measured moves away from the vehicle 100, even though the distance information cannot be obtained at the second time or later, the measurement Can calculate target speed.

(9) The captured images include a left image 202 captured by the left imaging unit 102 and a right image 203 captured by the right imaging unit 103 . As described with reference to FIG. 1, the field of view of the left imaging unit 102 and the field of view of the right imaging unit 103 overlap in the stereo field of view. The stereo ranging unit 316 calculates the distance to the detected object based on the parallax of the detected object existing in the stereo field of view in the left image 202 and the right image 203 . The angle-of-view specifying unit 307 specifies the angle of view based on the detected object present in the left image 202 or the right image 203, which is not a stereo field of view, that is, the left monocular region 204L and the right monocular region 204R in FIG. Therefore, by measuring the distance with high accuracy in the stereo area 204S and calculating the angle of view in the monocular area, the speed can be calculated with high accuracy and at an early timing.

(10) The captured image is the first image captured by the left imaging unit 102, and the stereo ranging unit 316 calculates the distance to the detected object based on the captured position of the detected object in the first image.

(Modification 1)
The camera system 101 in the first embodiment described above includes a left imaging section 102 and a right imaging section 103 . However, the camera system 101 may have no imaging unit, and may have a sensing result acquisition unit that acquires a captured image from an imaging unit that exists outside the camera system 101 .

FIG. 19 is a functional configuration diagram of a camera system 101A in Modification 1. As shown in FIG. A difference from FIG. 3 is that a sensing result acquisition unit 110 is provided instead of the left imaging unit 102 and the right imaging unit 103 . Sensing result acquisition section 110 acquires left image 202 and right image 203, which are sensing results, from left imaging section 102 and right imaging section 103, and outputs them to other functional blocks included in camera system 101A. Note that in the first embodiment, the function of the sensing result acquisition unit 110 is included in the left imaging unit 102 and the right imaging unit 103 . According to Modification 1, since the camera system 101A does not include an imaging unit, it can be used in combination with any of various imaging devices.

(Modification 2)
The camera system 101 in the first embodiment described above includes the left imaging unit 102 and the right imaging unit 103, but one imaging unit may be replaced with a millimeter wave sensor. In this case, the velocity of the object is calculated using the result of distance measurement by the millimeter wave sensor, and the search for similar locations and the specification of the angle of view by the monocular camera. At that time, it is necessary to consider the mounting positions of the monocular camera and the millimeter wave sensor on the vehicle 100. With the monocular camera for calculating the angle of view as a base point, the distance from the monocular camera to the millimeter wave sensor is subtracted from the distance measurement result. Convert with . This modification may be combined with modification 1 described above.

FIG. 20 is a functional configuration diagram of a camera system 101B in Modification 2. As shown in FIG. A difference from FIG. 3 is that a sensing result acquisition unit 110 is provided instead of the left imaging unit 102 and the right imaging unit 103 . Note that in this modification, the camera system 101B does not have to include the stereo matching unit 301, the template creation unit 304, the stereo detection unit 308, and the stereo ranging unit 316, but in FIG. These configurations are left in consideration of the case where the In this modification, an object assumed to be a moving object is detected from the sensing result of the millimeter wave sensor 103R, and matching is performed in the captured image based on the positional relationship between the vehicle 100 and the object.

According to this modified example, the following effects are obtained.
(11) The sensing result includes the captured image captured by the imaging unit and the distance information measured by the distance sensor. The distance measuring unit uses the distance to the detected object as the distance information. Therefore, the technique described in the first embodiment can also be combined with the combination of the millimeter wave sensor and the camera. This combination is effective when the camera can capture a range that cannot be measured by a millimeter wave sensor.

(Modification 3)
Although the camera system 101 in the first embodiment described above has two imaging units, the left imaging unit 102 and the right imaging unit 103, it may have only one imaging unit. In this case, the camera system 101 does not need to include the stereo matching unit 301, the stereo detection unit 308, and the stereo distance measurement unit 316, and the monocular distance measurement unit 303 measures the distance.

According to this modified example 3, the following effects are obtained.
(12) The captured image is the first image captured by the left imaging unit 102 . A monocular distance measurement unit 303 calculates the distance to the detection object based on the shooting position of the detection object in the first image. Therefore, by using the images captured by one camera at different times, the sensing result at the second time when the distance information cannot be directly calculated and the sensing result at the first time when the distance information can be directly calculated are used. can be calculated. In other words, even if the sensing information is a plurality of captured images captured by one camera, the speed of the object to be measured can be calculated early.

-Second Embodiment-
A second embodiment of the arithmetic device will be described with reference to FIGS. 21 and 22. FIG. In the following description, the same components as those in the first embodiment are assigned the same reference numerals, and differences are mainly described. Points that are not particularly described are the same as those in the first embodiment. This embodiment differs from the first embodiment mainly in that it considers the case where the travel paths of the own vehicle and the crossing vehicle are not orthogonal.

FIG. 21 is a functional block diagram showing functions of the camera system 101A according to the second embodiment. A difference from the camera system 101 in the first embodiment is that an intersection angle identification unit 1901 is provided instead of the type identification unit 317 . The vehicle 100 is also equipped with a GPS 1903 and map information 1904 , which are connected to the intersection angle identification unit 1901 .

Position specifying device 1903 receives radio waves from a plurality of satellites that constitute the satellite navigation system, and calculates the position of vehicle 100, that is, the latitude and longitude, by analyzing the signals contained in the radio waves. The position specifying device 1903 outputs the calculated latitude and longitude to the intersection angle specifying section 1901 .

The map information 1904 is a map database that includes information that enables calculation of directions along roads. The map information 1904 includes, for example, nodes indicating intersections and end points of roads, and information on links connecting the nodes. The node information includes information on the combination of latitude and longitude. The link information includes identification information of two nodes to be connected, and may further include a plurality of pieces of latitude and longitude information indicating inflection points and passing points. The link information may include information indicating the direction along the link, for example, information from 0 to 180 degrees where north is 0 degrees.

The intersection angle specifying unit 1901 specifies an intersection angle φ that is an angle at which the direction of travel of the own vehicle 100 intersects with the direction of travel of a vehicle traveling in the surrounding area. However, the crossing angle φ is a value indicating the amount of deviation from orthogonality, and the crossing angle φ is zero when the traveling directions of the two vehicles are orthogonal. For example, the crossing angle specifying unit 1901 first uses the position information acquired from the position specifying device 1903 to obtain the latest position information of the vehicle 100 and information on the traveling direction from differentiation of the position information. Next, the crossing angle specifying unit 1901 refers to the map information 1904 to specify the angle at which the link on which the vehicle 100 travels intersects with another link at the crossing existing in the traveling direction of the vehicle 100 .

A speed calculation unit 1904 calculates a speed based on the crossing angle specified by the crossing angle specifying unit 1901, the distance specified by the monocular ranging unit 303 or the stereo ranging unit 316, and the vehicle speed, steering angle, and yaw rate of the own vehicle. calculate. Details will be described later.

FIG. 22 is a diagram showing a speed calculation method by the speed calculation unit 1904. As shown in FIG. Reference numeral 2000 indicates own vehicle 100 at time T0, and reference numeral 2001 indicates own vehicle 100 at time T1. Reference numeral 2010 indicates the crossing vehicle at time T0, and reference numeral 2011 indicates the crossing vehicle at time T1. The angle of view θ is an angle formed by a straight line connecting host vehicle 100 and collided vehicle 2010 at time T0 with the traveling direction of the host vehicle. The definition of this angle of view is the same as in the first embodiment. The angle φ is the angle at which the own vehicle 100 and the crossing vehicle intersect. Note that the angle φ was fixed at zero degrees in the first embodiment.

The depth distance z9 is the distance traveled by the vehicle 100 from time T0 to time T1. The depth distance z8 is the distance from the own vehicle 2001 to the intersection at time T1. The sum of the depth distance z9 and the depth distance z8 is called "depth distance z98". Depth distance z5 is the depth distance from host vehicle 100 to crossing vehicle 2010 at time T0. Lateral distance x5 is the illustrated lateral distance between host vehicle 100 and crossing vehicle 2010 at time T0. The horizontal distance x5 is calculated using Equation 6 below, and the depth distance z5 is calculated using Equation 7 below.

However, "[theta]'" in Equations 6 and 7 has a relationship with [theta] described above and Equation 8 shown below.

θ′=90−θ (Formula 8)

The depth distance z9 is obtained by multiplying the speed of the vehicle 100 in the depth direction by the time difference from time T0 to time T1. The speed of the vehicle 100 in the depth direction is obtained from the vehicle speed, the steering angle, and the yaw rate. When the steering angle and yaw rate are 0 and the vehicle is traveling straight, the vehicle speed is the speed in the depth direction. The depth distance Z8 is the depth component of the distance specified by the monocular distance measurement unit 303 or the stereo distance measurement unit 316. FIG. The angle θ is the angle of view of the crossing vehicle seen from the camera of the own vehicle at time T<b>0 specified by the angle of view specifying unit 307 . By using Equations 6 and 7, the distance at time T0 can be specified without direct observation, and the speed can be calculated from the amount of movement.

According to the second embodiment described above, the following effects are obtained.
(13) The intersection angle identification unit 1901 calculates the intersection angle based on the position information of the moving object acquired by the position identification device 1903 and the map information 1904 . Therefore, a more reliable crossing angle can be calculated based on the position and traveling direction of the vehicle 100 .

(Modification of Second Embodiment)
The intersection angle identification unit 1901 may calculate the intersection angle by identifying the orientation of the crossing vehicle from the identification result of the object in the detected portion and the three-dimensional point group information from the stereo matching unit 301 . The intersection angle identification unit 1901 identifies the intersection angle, for example, from the angle of view on the image of the classifier and the crossing vehicle. If the roads intersect at an angle close to a right angle, the image will appear obliquely in front of the vehicle according to the angle at which the crossing vehicle appears. For example, if it is 45 degrees, the front and side surfaces are captured at angles of 315 degrees and 45 degrees, respectively. The degree of angle difference between the direction of the vehicle identified by the discriminator and the direction of crossing at an angle close to a right angle due to the angle of view is the angle at which the vehicle intersects with the crossing vehicle. Further, for example, the parallax obtained by stereo matching becomes a three-dimensional point group, and the front and side surfaces of the crossing vehicle can also be detected as planes from the point group. The direction of travel of the vehicle may be specified from the direction of this plane, and the crossing angle at which the own vehicle and the crossing vehicle intersect may be calculated.

In addition, the present invention is not limited to the above-described embodiments, and includes various modifications. For example, the above-described embodiments have been described in detail in order to explain the present invention in an easy-to-understand manner, and are not necessarily limited to those having all the described configurations. In addition, it is possible to replace part of the configuration of one embodiment with the configuration of another embodiment, and it is also possible to add the configuration of another embodiment to the configuration of one embodiment. Moreover, it is possible to add, delete, or replace a part of the configuration of each embodiment with another configuration.

Further, each of the above configurations, functions, processing units, processing means, and the like may be realized by hardware, for example, by designing a part or all of them using an integrated circuit. Moreover, each of the above configurations, functions, etc. may be realized by software by a processor interpreting and executing a program for realizing each function. Information such as programs, tables, and files that implement each function can be stored in recording devices such as memories, hard disks, SSDs (Solid State Drives), or recording media such as IC cards and SD cards.

Further, the control lines and information lines indicate those considered necessary for explanation, and not all control lines and information lines are necessarily indicated on the product. In practice, it may be considered that almost all configurations are interconnected.

In each of the embodiments and modifications described above, the configuration of the functional blocks is merely an example. Some functional configurations shown as separate functional blocks may be configured integrally, or a configuration represented by one functional block diagram may be divided into two or more functions. Further, a configuration may be adopted in which part of the functions of each functional block is provided in another functional block.

Each of the embodiments and modifications described above may be combined. Although various embodiments and modifications have been described above, the present invention is not limited to these contents. Other aspects conceivable within the scope of the technical idea of the present invention are also included in the scope of the present invention.

101, 101A... camera system 102... left imaging unit 103... right imaging unit 301... stereo matching unit 302... monocular detection unit 303... monocular distance measurement unit 304... template creation unit 306... similar part search unit 307... angle of view specification unit 308 .

Claims (13)

A computing device mounted on a mobile body,
an intersection angle specifying unit that specifies an intersection angle, which is an angle at which the traveling direction of the moving body and the traveling direction of the detected object intersect;
a sensing information acquisition unit that acquires a sensing result including at least an image captured by a camera;
a distance measuring unit that calculates a distance to the detected object at a first time based on the sensing result;
a view angle specifying unit that specifies a view angle, which is an angle with respect to the moving direction of the moving object at a second time, based on the captured image;
Based on the intersection angle, the distance to the moving body at the first time, the angle to the moving body at the second time, the time difference between the first time and the second time, and the speed of the moving body, A computing device comprising: a speed calculation unit that calculates the speed of a detected object. The arithmetic device according to claim 1,
The captured image includes a first image captured by a first imaging unit and a second image captured by a second imaging unit,
The distance measuring unit calculates a distance to the detection object based on a shooting position of the detection object in the first image or the second image, or a parallax of the detection object in the first image and the second image. A computing device that calculates the distance to the detected object based on The arithmetic device according to claim 1,
The sensing result includes the captured image captured by the imaging unit and the distance information measured by the distance sensor,
The distance measuring unit is an arithmetic device, wherein the distance information is a distance to the detected object. The arithmetic device according to claim 1,
The arithmetic device, wherein the intersection angle specifying unit assumes that the intersection angle is 90 degrees. The arithmetic device according to claim 1,
The intersection angle identification unit is a computing device that calculates the intersection angle based on position information and map information of the moving object. The arithmetic device according to claim 1,
The angle-of-view specifying unit acquires image information of the moving object from the captured image at the first time, and specifies the angle of view from the position of the moving object captured in the captured image at the second time. , arithmetic unit. In the arithmetic device according to claim 6,
Matching based on the number of pixels of the moving body assumed to be captured in the captured image at the first time and the pixel count of the moving body assumed to be captured in the captured image at the second time, whichever is smaller. It further comprises a template creation unit that creates a template for
The angle-of-view specifying unit specifies the angle of view by searching the captured image at the second time using the matching template. The arithmetic device according to claim 1,
The angle-of-view specifying unit specifies the angle of view based on the photographed image of the detected object existing outside a range-measurable area in which the range-finding unit can measure the range. In the arithmetic device according to claim 8,
The computing device, wherein the first time is later than the second time. In the arithmetic device according to claim 8,
The computing device, wherein the first time is earlier than the second time. In the arithmetic device according to claim 8,
The captured image includes a first image captured by a first imaging unit and a second image captured by a second imaging unit,
The field of view of the first imaging unit and the field of view of the second imaging unit overlap in a stereo field of view,
The distance measuring unit calculates a distance to the detection object based on parallax of the detection object existing in the stereo field of view in the first image and the second image,
The angle-of-view specifying unit specifies the angle of view based on the detected object existing in a region other than the stereo field of view in the first image. The arithmetic device according to claim 1,
The captured image includes a first image captured by a first imaging unit,
The arithmetic device, wherein the distance measuring unit calculates a distance to the detection object based on a photographing position of the detection object in the first image. A speed calculation method executed by an arithmetic device mounted on a moving object,
specifying an intersection angle, which is an angle at which the traveling direction of the moving body and the traveling direction of the detected object intersect;
Acquiring a sensing result including at least a photographed image obtained by photographing by a camera;
calculating a distance to the detected object at a first time based on the sensing result;
specifying an angle of view, which is an angle with respect to the traveling direction of the moving object at a second time, based on the captured image;
Based on the intersection angle, the distance to the moving body at the first time, the angle to the moving body at the second time, the time difference between the first time and the second time, and the speed of the moving body, calculating the velocity of the sensed object.

Images