WO2024204585A1

WO2024204585A1 - Recognition device, moving body control device, recognition method, and program

Info

Publication number: WO2024204585A1
Application number: PCT/JP2024/012743
Authority: WO
Inventors: 成光土屋; 修敏莊
Original assignee: 本田技研工業株式会社
Priority date: 2023-03-29
Filing date: 2024-03-28
Publication date: 2024-10-03
Also published as: WO2024201837A1

Abstract

A recognition device that recognizes the position and the speed of an object present around a moving body on the basis of the output of a detection device for detecting the conditions around the moving body, the recognition device comprising: a first recognition unit that repeatedly outputs the position of the object, which is a result of performing a process for recognizing the position of the object, at a first cycle; and a second recognition unit that repeatedly outputs the speed of the object, which is a result of performing a process for recognizing the speed of the object, at a second cycle shorter than the first cycle.

Description

Recognition device, mobile object control device, recognition method, and program

The present invention relates to a recognition device, a control device for a moving object, a recognition method, and a program.

In recent years, there has been much research and practical application into autonomously moving vehicles and other moving bodies. In most cases, this technology requires technology to recognize objects that exist around the moving body (hereinafter, surrounding recognition technology). One invention relating to surrounding recognition technology is disclosed in which, based on driving scene data and driving behavior data in the driving scene in which the vehicle is currently located, it is determined whether or not a certain stable driving condition is met, and if the certain stable driving condition is met, driving scene data and driving behavior data are collected at a frequency lower than a certain sampling frequency, thereby reducing data redundancy in similar scenes and similar driving modes (Patent Document 1).

Special Publication No. 2021-526699

In order to realize surrounding recognition technology, it is necessary to balance reducing the load on the processor with being able to quickly track sudden events. While the above conventional technology is able to comprehensively assess the environment of a moving object, there are concerns that it cannot quickly track local changes. Specifically, conventional technology may not be able to quickly track changes in the speed of an object.

The present invention was made in consideration of these circumstances, and one of its objectives is to provide a recognition device, a control device for a moving object, a recognition method, and a program that can quickly track changes in the speed of an object.

The recognition device, the control device for a moving object, the recognition method, and the program according to the present invention employ the following configuration.
(1): A recognition device according to one embodiment of the present invention is a recognition device that recognizes a position and speed of an object present in the vicinity of a moving body based on the output of a detection device for detecting the surrounding conditions of the moving body, and includes a first recognition unit that repeatedly outputs the position of the object, which is a result of processing for recognizing the position of the object, in a first period, and a second recognition unit that repeatedly outputs the speed of the object, which is a result of processing for recognizing the speed of the object, in a second period that is shorter than the first period.

(2): In the above aspect (1), the second recognition unit has a function of provisionally recognizing the position of an object whose position has not been output by the first recognition unit, and when a position is output from the first recognition unit and a velocity is output from the second recognition unit, the second recognition unit outputs the position output by the first recognition unit and the velocity output by the second recognition unit as the state of the object, and further includes a fusion unit that outputs the position and velocity output by the second recognition unit as the state of the object for an object whose position has not been output by the first recognition unit and whose position and velocity have been output by the second recognition unit.

(3): In the above aspect (2), the second recognition unit has a function of recognizing the presence of an object whose position has not been output by the first recognition unit.

(4): In the above aspect (1), the first recognition unit and the second recognition unit each perform processing by executing a processing procedure that is at least partially common to both units.

(5): In the above aspect (1), each of the first recognition unit and the second recognition unit is realized by one or more processors performing the processing as the first recognition unit and the processing as the second recognition unit in a time-division manner.

(6): In the above aspect (1), the first recognition unit and the second recognition unit are each realized by a separate processor performing processing.

(7): In the above aspect (1), the first recognition unit has a function of repeatedly outputting the position and velocity of the object in the first period, the second recognition unit has a function of repeatedly outputting the position and velocity of the object in the second period, and further includes a fusion unit that outputs the position output by the first recognition unit and the velocity output by the second recognition unit as the state of the object when the first recognition unit first outputs.

(8): In the above aspect (7), the fusion unit outputs the position and velocity output by the first recognition unit as the state of the object when the first recognition unit outputs for the second or subsequent time.

(9): In the above aspect (1), the first recognition unit has a function of repeatedly outputting the position and velocity of the object in the first period, and further includes a fusion unit that outputs the position and velocity output by the first recognition unit as the state of the object at the timing when the first recognition unit outputs the position and velocity.

(10): In the above aspect (1), the first recognition unit has a function of repeatedly outputting the position and velocity of the object in the first period, and the second recognition unit has a function of repeatedly outputting the position and velocity of the object in the second period, and further includes a fusion unit that, at the timing when the first recognition unit performs an output for the first time, outputs as the state of the object the position output by the first recognition unit and the velocity output by the first recognition unit based on the position output by the first recognition unit and the position output by the second recognition unit at the previous timing before the timing.

(11): In any of the above aspects (1) to (10), after the first recognition unit outputs the first position at a first timing, at a second timing when the first recognition unit does not output and the second recognition unit outputs, the second recognition unit updates the first position by performing linear interpolation based on historical information of the positions and velocities of the object that have been recognized in the past.

(12): A control device for a moving body according to another aspect of the present invention includes the recognition device according to aspect (1) above, and a driving control unit that moves the moving body so as to avoid approaching an object whose state has been output by the recognition device.

(13): A recognition method according to another aspect of the present invention is a recognition method executed by a recognition device that recognizes the position and speed of an object present around a moving body based on the output of a detection device for detecting the surrounding conditions of the moving body, and includes repeatedly outputting the position of the object, which is a result of a process for recognizing the position of the object, in a first period, and repeatedly outputting the speed of the object, which is a result of a process for recognizing the speed of the object, in a second period shorter than the first period.

(14): A program according to another aspect of the present invention is a program for causing a processor of a recognition device that recognizes the position and speed of an object present around a moving body based on the output of a detection device for detecting the surrounding conditions of the moving body to repeatedly output the position of the object, which is a result of processing for recognizing the position of the object, in a first period, and repeatedly output the speed of the object, which is a result of processing for recognizing the speed of the object, in a second period shorter than the first period.

According to aspects (1) to (14), it is possible to quickly track changes in the speed of an object.

1 is a configuration diagram of a vehicle system 1 that uses a recognition device and a control device for a moving object according to a first embodiment. FIG. 2 is a functional configuration diagram of a first control unit and a second control unit. FIG. 2 illustrates an example of a configuration of a recognition unit. FIG. 4 is a diagram illustrating an example of a relationship between a first recognition unit and a second recognition unit. FIG. 11 is a diagram illustrating an example of the operation of a first recognition unit and a second recognition unit in a certain scene. 11 is a diagram for explaining the processing of a future position prediction/risk setting unit. FIG. FIG. 11 is a diagram illustrating an example of the operation of a first recognition unit and a second recognition unit in a certain scene according to the second embodiment. FIG. 13 is a diagram illustrating an example of the operation of a first recognition unit and a second recognition unit in a certain scene according to the third embodiment. FIG. 13 is a diagram illustrating an example of the operation of a first recognition unit and a second recognition unit in a certain scene according to the fourth embodiment.

[overview]
Hereinafter, with reference to the drawings, an embodiment of a recognition device, a control device for a moving body, a recognition method, and a program of the present invention will be described. The recognition device recognizes at least the position and speed of an object present around the moving body. The recognition device is mounted on the moving body, for example, but may be installed outside the moving body. The control device for the moving body controls the drive device of the moving body to move the moving body. The moving body refers to any object that can move in space using mechanical power, such as a four-wheeled or two-wheeled vehicle, micromobility, or a drone, aircraft, or ship. The moving body may move with a person or animal on board, or may be an unmanned moving body. In the following description, the moving body is assumed to be a vehicle that moves on the road with a person on board, and an "automatic driving control device" is taken as an example of a control device.

First Embodiment
[Overall configuration]
1 is a configuration diagram of a vehicle system 1 that uses a recognition device and a mobile object control device according to the first embodiment. The vehicle on which the vehicle system 1 is mounted is, for example, a two-wheeled, three-wheeled, or four-wheeled vehicle, and its drive source is an internal combustion engine such as a diesel engine or a gasoline engine, an electric motor, or a combination of these. The electric motor operates using power generated by a generator connected to the internal combustion engine, or discharged power from a secondary battery or a fuel cell.

The vehicle system 1 includes, for example, a camera 10, a radar device 12, a LIDAR (Light Detection and Ranging) 14, an object recognition device 16, a communication device 20, an HMI (Human Machine Interface) 30, a vehicle sensor 40, a navigation device 50, an MPU (Map Positioning Unit) 60, a driving operator 80, an automatic driving control device 100, a driving force output device 200, a braking device 210, and a steering device 220. These devices and equipment are connected to each other by multiple communication lines such as a CAN (Controller Area Network) communication line, serial communication lines, a wireless communication network, etc. Note that the configuration shown in FIG. 1 is merely an example, and some of the configuration may be omitted, or other configurations may be added.

The camera 10 is, for example, a digital camera that uses a solid-state imaging element such as a CCD (Charge Coupled Device) or a CMOS (Complementary Metal Oxide Semiconductor). The camera 10 is attached at any location of the vehicle (hereinafter, vehicle M) in which the vehicle system 1 is mounted. When capturing an image of the front, the camera 10 is attached to the top of the front windshield, the back of the rearview mirror, or the like. The camera 10, for example, periodically and repeatedly captures images of the surroundings of the vehicle M. The camera 10 may be a stereo camera.

The radar device 12 emits radio waves such as millimeter waves around the vehicle M and detects radio waves reflected by objects (reflected waves) to detect at least the position (distance and direction) of the object. The radar device 12 is attached to any location on the vehicle M. The radar device 12 may detect the position and speed of an object using the FM-CW (Frequency Modulated Continuous Wave) method.

The LIDAR 14 irradiates light (or electromagnetic waves with a wavelength close to that of light) around the vehicle M and measures the scattered light. The LIDAR 14 detects the distance to the target based on the time between emitting and receiving the light. The irradiated light is, for example, a pulsed laser light. The LIDAR 14 can be attached to any location on the vehicle M.

The object recognition device 16 performs sensor fusion processing on the detection results from some or all of the camera 10, radar device 12, and LIDAR 14 to recognize the position, type, speed, etc. of the object. The object recognition device 16 outputs the recognition results to the autonomous driving control device 100. The object recognition device 16 may output the detection results from the camera 10, radar device 12, and LIDAR 14 directly to the autonomous driving control device 100. The object recognition device 16 may be omitted from the vehicle system 1.

The communication device 20 communicates with other vehicles in the vicinity of the vehicle M using, for example, a cellular network, a Wi-Fi network, Bluetooth (registered trademark), or DSRC (Dedicated Short Range Communication), or communicates with various server devices via a wireless base station.

The HMI 30 presents various information to the occupants of the vehicle M and accepts input operations by the occupants. The HMI 30 includes various display devices, speakers, buzzers, touch panels, switches, keys, etc.

The vehicle sensor 40 includes a vehicle speed sensor that detects the speed of the vehicle M, an acceleration sensor that detects the acceleration, a yaw rate sensor that detects the angular velocity around the vertical axis, a direction sensor that detects the direction of the vehicle M, etc.

The navigation device 50 includes, for example, a GNSS (Global Navigation Satellite System) receiver 51, a navigation HMI 52, and a route determination unit 53. The navigation device 50 stores first map information 54 in a storage device such as a HDD (Hard Disk Drive) or flash memory. The GNSS receiver 51 determines the position of the vehicle M based on signals received from GNSS satellites. The position of the vehicle M may be determined or supplemented by an INS (Inertial Navigation System) that uses the output of the vehicle sensor 40. The navigation HMI 52 includes a display device, a speaker, a touch panel, keys, etc. The navigation HMI 52 may be partially or completely shared with the HMI 30 described above. The route determination unit 53 determines a route (hereinafter, a route on a map) from the position of the vehicle M specified by the GNSS receiver 51 (or any input position) to a destination input by the occupant using the navigation HMI 52, for example, by referring to the first map information 54. The first map information 54 is, for example, information in which a road shape is expressed by links indicating roads and nodes connected by the links. The first map information 54 may include road curvature and POI (Point of Interest) information. The route on the map is output to the MPU 60. The navigation device 50 may perform route guidance using the navigation HMI 52 based on the route on the map. The navigation device 50 may be realized by the function of a terminal device such as a smartphone or tablet terminal owned by the occupant. The navigation device 50 may transmit the current position and the destination to a navigation server via the communication device 20, and obtain a route equivalent to the route on the map from the navigation server.

The MPU 60 includes, for example, a recommended lane determination unit 61, and stores second map information 62 in a storage device such as an HDD or flash memory. The recommended lane determination unit 61 divides the route on the map provided by the navigation device 50 into a number of blocks (for example, every 100 m in the vehicle travel direction), and determines a recommended lane for each block by referring to the second map information 62. The recommended lane determination unit 61 determines, for example, which lane from the left to use. When there is a branch on the route on the map, the recommended lane determination unit 61 determines a recommended lane so that the vehicle M can use a reasonable route to proceed to the branch destination.

The second map information 62 is map information with higher accuracy than the first map information 54. The second map information 62 includes, for example, information on the center of lanes or information on lane boundaries. The second map information 62 may also include road information, traffic regulation information, address information (address and postal code), facility information, telephone number information, information on prohibited sections where mode A or mode B described below is prohibited, and the like. The second map information 62 may be updated at any time by the communication device 20 communicating with other devices.

The driving operators 80 include, for example, a steering wheel, an accelerator pedal, a brake pedal, a shift lever, and other operators. The driving operators 80 are fitted with sensors that detect the amount of operation or the presence or absence of operation, and the detection results are output to the automatic driving control device 100, or some or all of the driving force output device 200, the brake device 210, and the steering device 220.

The automatic driving control device 100 includes, for example, a first control unit 120 and a second control unit 160. The first control unit 120 and the second control unit 160 are each realized by, for example, a hardware processor such as a CPU (Central Processing Unit) executing a program (software). Furthermore, some or all of these components may be realized by hardware (including circuitry) such as an LSI (Large Scale Integration), ASIC (Application Specific Integrated Circuit), FPGA (Field-Programmable Gate Array), or GPU (Graphics Processing Unit), or may be realized by collaboration between software and hardware. The program may be stored in advance in a storage device (a storage device with a non-transient storage medium) such as the HDD or flash memory of the autonomous driving control device 100, or may be stored in a removable storage medium such as a DVD or CD-ROM, and installed in the HDD or flash memory of the autonomous driving control device 100 by mounting the storage medium (non-transient storage medium) in a drive device.

The autonomous driving control device 100 is an example of a "control device for a moving body", the recognition unit 130 is an example of a "recognition device", and the combination of the action plan generation unit 140 and the second control unit 160 is an example of a "driving control unit".

FIG. 2 is a functional configuration diagram of the first control unit 120 and the second control unit 160. The first control unit 120 includes, for example, a recognition unit 130 and an action plan generation unit 140. The first control unit 120 realizes, for example, a function based on AI (Artificial Intelligence) and a function based on a pre-given model in parallel. For example, the "intersection recognition" function may be realized by executing in parallel the recognition of the intersection using deep learning or the like and the recognition based on pre-given conditions (such as traffic lights and road markings that can be pattern matched), and by scoring both and evaluating them comprehensively. This ensures the reliability of autonomous driving.

The recognition unit 130 recognizes the presence, position, and state of the speed, acceleration, etc. of objects around the vehicle M based on information input from the camera 10, the radar device 12, and the LIDAR 14 via the object recognition device 16. The position of the object is recognized as a position on an absolute coordinate system with a representative point of the vehicle M (such as the center of gravity or the center of the drive shaft) as the origin, and is used for control. The position of the object may be represented by a representative point such as the center of gravity or a corner of the object, or may be represented by an area. The "state" of the object may include the acceleration or jerk of the object, or the "behavioral state" (for example, whether or not the object is changing lanes or is about to change lanes).

The recognition unit 130 also recognizes, for example, the lane in which the vehicle M is traveling (the driving lane). For example, the recognition unit 130 recognizes the driving lane by comparing the pattern of road dividing lines (for example, an arrangement of solid and dashed lines) obtained from the second map information 62 with the pattern of road dividing lines around the vehicle M recognized from the image captured by the camera 10. Note that the recognition unit 130 may recognize the driving lane by recognizing road boundaries (road boundaries) including not only road dividing lines but also road dividing lines, shoulders, curbs, medians, guard rails, etc. In this recognition, the position of the vehicle M obtained from the navigation device 50 and the processing results by the INS may be taken into account. The recognition unit 130 also recognizes stop lines, obstacles, red lights, toll booths, and other road phenomena.

When recognizing the driving lane, the recognition unit 130 recognizes the position and attitude of the vehicle M with respect to the driving lane. For example, the recognition unit 130 may recognize the deviation of the reference point of the vehicle M from the center of the lane and the angle with respect to a line connecting the centers of the lanes in the direction of travel of the vehicle M as the relative position and attitude of the vehicle M with respect to the driving lane. Alternatively, the recognition unit 130 may recognize the position of the reference point of the vehicle M with respect to either side end of the driving lane (road division line or road boundary) as the relative position of the vehicle M with respect to the driving lane.

The behavior plan generating unit 140 generates a target trajectory for the vehicle M to automatically (without the driver's operation) travel in the future so that, in principle, the vehicle travels in the recommended lane determined by the recommended lane determining unit 61 and avoids approaching objects (excluding road division lines, road markings, manholes, and other objects that can be climbed over) recognized by the recognition unit 130. For example, the recognition unit 130 sets a risk area centered on the object whose state has been output, and within the risk area, the recognition unit 130 sets a risk as an index value indicating the degree to which the vehicle M should not approach. The behavior plan generating unit 140 generates a target trajectory so that the vehicle M does not pass through a point where the risk is equal to or greater than a predetermined value. Since the objects include moving objects, the distribution of risk is not one for each control cycle, but is set for multiple future time points taking into account the future position of the object predicted based on the speed of the object. The target trajectory includes, for example, a speed element. For example, the target trajectory is expressed as a sequence of points (trajectory points) to be reached by the vehicle M. A trajectory point is a point where the vehicle M should reach at each predetermined travel distance (e.g., about several meters) along a road, and separately, a target speed and target acceleration are generated as part of the target trajectory for each predetermined sampling time (e.g., about a few tenths of a second). A trajectory point may also be a position where the vehicle M should reach at each predetermined sampling time. In this case, the information on the target speed and target acceleration is expressed as the interval between trajectory points.

The second control unit 160 controls the driving force output device 200, the brake device 210, and the steering device 220 so that the vehicle M passes through the target trajectory generated by the action plan generation unit 140 at the scheduled time.

The second control unit 160 includes, for example, an acquisition unit 162, a speed control unit 164, and a steering control unit 166. The acquisition unit 162 acquires information on the target trajectory (trajectory points) generated by the action plan generation unit 140, and stores it in a memory (not shown). The speed control unit 164 controls the driving force output device 200 or the brake device 210 based on the speed element associated with the target trajectory stored in the memory. The steering control unit 166 controls the steering device 220 according to the curvature of the target trajectory stored in the memory. The processing of the speed control unit 164 and the steering control unit 166 is realized, for example, by a combination of feedforward control and feedback control. As an example, the steering control unit 166 executes a combination of feedforward control according to the curvature of the road ahead of the vehicle M and feedback control based on the deviation from the target trajectory.

The driving force output device 200 outputs a driving force (torque) to the drive wheels for the vehicle to travel. The driving force output device 200 comprises, for example, a combination of an internal combustion engine, an electric motor, and a transmission, and an ECU (Electronic Control Unit) that controls these. The ECU controls the above configuration according to information input from the second control unit 160 or information input from the driving operator 80.

The brake device 210 includes, for example, a brake caliper, a cylinder that transmits hydraulic pressure to the brake caliper, an electric motor that generates hydraulic pressure in the cylinder, and a brake ECU. The brake ECU controls the electric motor according to information input from the second control unit 160 or information input from the driving operator 80, so that a brake torque corresponding to the braking operation is output to each wheel. The brake device 210 may include a backup mechanism that transmits hydraulic pressure generated by operating the brake pedal included in the driving operator 80 to the cylinder via a master cylinder. Note that the brake device 210 is not limited to the configuration described above, and may be an electronically controlled hydraulic brake device that controls an actuator according to information input from the second control unit 160 to transmit hydraulic pressure from the master cylinder to the cylinder.

The steering device 220 includes, for example, a steering ECU and an electric motor. The electric motor changes the direction of the steered wheels by, for example, applying a force to a rack and pinion mechanism. The steering ECU drives the electric motor according to information input from the second control unit 160 or information input from the driving operator 80, to change the direction of the steered wheels.

[More detailed configuration of the recognition unit]
Here, a more detailed configuration of the recognition unit 130 will be described. In order to realize the above functions, the recognition unit 130 has the configuration described below. Fig. 3 is a diagram showing an example of the configuration of the recognition unit 130. The recognition unit 130 includes, for example, a first recognition unit 132, a second recognition unit 134, a fusion unit 136, and a future position prediction/risk setting unit 138.

The first recognition unit 132 repeatedly outputs the object's position, which is the result of processing to recognize the object's position, in a first period.

The second recognition unit 134 repeatedly outputs the object's speed, which is the result of processing to recognize the object's speed, in a second period shorter than the first period. In other words, the second recognition unit 134 performs processing at a higher speed than the first recognition unit 132. For example, the second recognition unit 134 performs processing at a frequency several times to several tens of times higher than the first recognition unit 132. The second recognition unit 134 has a function of provisionally recognizing the presence and position of an object whose position has not been output by the first recognition unit 132. There are no particular restrictions on this function, and it is sufficient that it is a process that is simpler and has a lower load than the process performed by the first recognition unit 132. As an example, this function is realized by a process that combines a contour extraction process and a size recognition process. This function may be stopped when the first recognition unit 132 starts to output the position, or may be continued even after the first recognition unit 132 starts to output the position.

When the first recognition unit 132 outputs a position and the second recognition unit 134 outputs a speed (and position), the fusion unit 136 outputs the position output by the first recognition unit 132 and the speed output by the second recognition unit 134 as the state of the object. When the first recognition unit 132 does not output a position, but the second recognition unit 134 outputs a position (provisionally recognized as described above) and speed, the fusion unit 136 outputs the position and speed output by the second recognition unit 134 as the state of the object.

The first recognition unit 132 and the second recognition unit 134 each perform processing by executing at least a part of a common processing procedure. For example, the first recognition unit 132 and the second recognition unit 134 each input an image captured by the camera 10 (or an image that has been preprocessed) into a trained model such as a DNN (Deep Neural Network) to recognize and output the position, speed, etc. of an object, and the second recognition unit 134 compresses the input image to a lower resolution than the image input to the first recognition unit 132, thereby performing processing faster than the first recognition unit 132. Figure 4 is a diagram showing an example of the relationship between the first recognition unit 132 and the second recognition unit 134. In this case, the input image input to the first recognition unit 132 is thinned out in time, resulting in a smaller number of frames than the input image input to the second recognition unit 134.

FIG. 5 is a diagram showing an example of the operation of the first recognition unit 132 and the second recognition unit 134 in a certain scene. In this diagram, the second recognition unit 134 performs processing at a frequency three times that of the first recognition unit 132. The control timing is a virtual time that arrives at a predetermined period (which is the same as the processing period of the second recognition unit 134 in this embodiment). At each control timing, an image captured by the camera 10 is repeatedly input. For convenience, in this diagram, the control timing is assumed to start from 1.

At control timing 1, the second recognition unit 134 provisionally recognizes the presence and position of the object for the first time. The second recognition unit 134 also recognizes the speed of the object. At this time, the first recognition unit 132 or the second recognition unit 134 (or another functional unit) predicts the position and speed of the object at the next control timing 2, for example, by calculating a covariance matrix generated by arranging the states calculated before that control timing, its eigenvalues, and an interpolated value for future expansion (next prediction). This next prediction process is repeatedly executed at each control timing.

At control timing 1, the state output by the recognition unit 130 is defined as an object whose position has been provisionally recognized and whose speed has been definitively recognized. The degree of control over the object is changed depending on whether it has been provisionally recognized or definitively recognized. For example, for an object whose position has been provisionally recognized, the risk is calculated to be smaller than for an object whose position has been definitively recognized, and the degree of control is relaxed.

At control timing 2, the second recognition unit 134 updates the object's position and speed based on the result of the next prediction at control timing 1 and the result of processing the input image.

At control timing 3, the first recognition unit 132 updates the object's position based on the result of the next prediction at control timing 2 and the result of processing the input image. The second recognition unit 134 updates the object's speed based on the result of the next prediction at control timing 2 and the result of processing the input image. At this point, it is defined that the presence of the object is confirmed, and the object's position is also confirmed to be recognized definitively.

At control timing 4, the second recognition unit 134 updates the object's speed based on the result of the next prediction at control timing 3 and the result of processing the input image.

At control timing 5, the second recognition unit 134 updates the object's speed based on the result of the next prediction at control timing 4 and the result of processing the input image.

At control timing 6, the first recognition unit 132 updates the object's position based on the result of the next prediction at control timing 5 and the result of processing the input image. The second recognition unit 134 updates the object's speed based on the result of the next prediction at control timing 5 and the result of processing the input image. After that, the same processing as at control timings 4 to 6 is repeatedly executed.

The future position prediction/risk setting unit 138 predicts the future position of the object based on the state (position, speed) of the object output by the fusion unit 136, and sets a risk for the object. FIG. 6 is a diagram for explaining the processing of the future position prediction/risk setting unit 138. The future position prediction/risk setting unit 138 sets risk, which is an index value indicating the degree to which the vehicle M should not enter or approach, on the assumed plane S, which is a virtual plane that represents the space around the vehicle M as a two-dimensional plane viewed from above. The higher the risk value, the more likely the vehicle M should not enter or approach, and the closer the value is to zero, the more favorable it is for the vehicle M to travel. However, this relationship may be reversed. If the moving object is a flying object such as a drone rather than a vehicle, the future position prediction/risk setting unit 138 may perform similar processing in three-dimensional space rather than on the assumed plane S.

The future position prediction/risk setting unit 138 sets the risk on the imaginary plane S not only for the current time, but also for the future positions of objects predicted in advance and specified at regular time intervals, such as the current time t, Δt later (time t + Δt), 2Δt later (time t + 2Δt), etc.

The future position prediction/risk setting unit 138 sets the risk of vehicles, pedestrians, bicycles, and other traffic participants (moving targets) on the assumed plane S, with ellipses or circles based on the direction of travel and speed as contour lines, and sets a fixed value of risk for impassable areas. In the figure, DM is the direction of travel of vehicle M. R(M1) is the risk of stopped vehicle M1, and R(P) is the risk of pedestrian P. Since pedestrian P is moving in a direction crossing the road, a risk is set at a position different from the current time for each future point in time. The same applies to moving vehicles, bicycles, and the like. R(BD) is the risk of impassable areas BD. In the figure, the darkness of the hatching indicates the risk value, and the darker the hatching, the greater the risk. The future position prediction/risk setting unit 138 may set the risk so that the value increases the further away from the center of the lane.

By performing processing in such a configuration, the recognition unit 130 and the automatic driving control device 100 can quickly follow the speed change of the object. As described in Figures 3 to 5, the second recognition unit 134 performs processing using a compressed input image, so the overall processing time can be shortened, and processing can be performed faster than the first recognition unit 132, which performs processing using a high-resolution image. The advantage of processing using a high-resolution image is that it can accurately detect objects that are mainly far away from the vehicle M, but it is known that there is not much performance difference for objects that are close to the vehicle M. On the other hand, when performing processing to set a risk based on the future position of an object, it is desirable to quickly follow events such as a sudden change in the speed of the object (for example, a pedestrian suddenly starts running, or another vehicle suddenly stops). In this regard, in the recognition unit 130 of the embodiment, the second recognition unit 134, which mainly recognizes the speed, processes at a high speed, so sudden changes in the speed of the object can be detected quickly. Therefore, it is possible to quickly follow the speed change of the object, and ultimately to quickly control the behavior of the vehicle M.

Each of the first recognition unit 132 and the second recognition unit 134 is realized, for example, by one or more processors performing processing as the first recognition unit 132 and processing as the second recognition unit 134 in a time-division manner. The same may be true for the fusion unit 136 and the future position prediction/risk setting unit 138. Alternatively, each of the first recognition unit 132 and the second recognition unit 134 may be realized by separate processors performing processing. The same may be true for the fusion unit 136 and the future position prediction/risk setting unit 138.

Second Embodiment
Next, the second embodiment will be described. The second embodiment differs from the first embodiment in that the first recognition unit 132 starts outputting the position when it is able to recognize the position of the object, and outputs both the position and the velocity after it recognizes the velocity of the object. Furthermore, the second embodiment differs from the first embodiment in that after the position of the object is recognized by the first recognition unit 132, the position of the object is updated based on the velocity recognized by the second recognition unit 134 at a control timing when the first recognition unit 132 does not operate. In the following description, the same reference numerals and names are used for configurations having the same functions as those in the first embodiment, and a detailed description thereof will be omitted.

FIG. 7 is a diagram showing an example of the operation of the first recognition unit 132 and the second recognition unit 134 in a certain scene according to the second embodiment. In this diagram, the second recognition unit 134 performs processing at a frequency three times that of the first recognition unit 132.

At control timing 1, the second recognition unit 134 provisionally recognizes the presence and position of an object for the first time. The second recognition unit 134 also recognizes the speed of the object. That is, the position and speed are recognized based on the object recognition result ("high-speed object recognition result") by the second recognition unit 134, which performs object recognition at high speed. At this time, the second recognition unit 134 (or another functional unit) predicts the position and speed of the object at the next control timing 2, for example, by calculating a covariance matrix generated by arranging the states calculated before the control timing in question, its eigenvalues, and an interpolated value for future expansion (next prediction). This next prediction process is repeatedly executed at each control timing. At control timing 1, the state output by the recognition unit 130 is defined as tentatively recognized for the object position and definitively recognized for the speed.

Control timing 3 is the timing when the first recognition unit 132 outputs for the first time. At this control timing 3, the first recognition unit 132 updates the object position based on the result of the next prediction at control timing 2 and the result of processing the input image. That is, the object position is updated based on the object recognition result by the first recognition unit 132 performing object recognition at normal speed (the "normal object recognition result"). In addition, the second recognition unit 134 updates the object speed based on the result of the next prediction at control timing 2 and the result of processing the input image. That is, the object speed is updated based on the high-speed object recognition result by the second recognition unit 134. At this time, it is defined that the existence of the object is confirmed, and the object position is also confirmed to be recognized definitively.

At control timing 4, the second recognition unit 134 updates the object's speed based on the result of the next prediction at control timing 3 and the result of processing the input image. Furthermore, the second recognition unit 134 updates the object's position based on the speed. For example, the second recognition unit 134 updates the object's position based on linear interpolation. The second recognition unit 134 estimates the current object's position by taking the object's position recognized by the first recognition unit 132 at control timing 3 as a reference position and adding the object's movement amount calculated using linear interpolation (the movement amount from control timing 3 to control timing 4) to this reference position, and updates the object's position to the estimated position. For example, the linear interpolation uses a covariance matrix (position and speed history information) generated by arranging the states calculated before the control timing in question, and its eigenvalues, etc.

At control timing 5, the second recognition unit 134 updates the object's speed based on the result of the next prediction at control timing 4 and the result of processing the input image. Furthermore, the second recognition unit 134 updates the object's position based on the speed. For example, the second recognition unit 134 updates the object's position by linear interpolation. The second recognition unit 134 estimates the current object's position by taking the object's position updated at control timing 3 as a reference position and adding the object's movement amount calculated using linear interpolation (movement amount from control timing 3 to control timing 5) to this reference position, and updates the object's position at the estimated position. Alternatively, the second recognition unit 134 estimates the current object's position by taking the object's position updated at control timing 4 as a reference position and adding the object's movement amount calculated using linear interpolation (movement amount from control timing 4 to control timing 5) to this reference position, and updates the object's position at the estimated position.

In other words, after the first recognition unit 132 outputs the first position at a first timing, at a second timing when the first recognition unit 132 does not output and the second recognition unit 134 outputs, the second recognition unit 134 updates the first position by performing linear interpolation based on the historical information of the positions and velocities of objects that have been recognized in the past.

Control timing 6 is the timing at which the first recognition unit 132 performs the second or subsequent output. At this control timing 6, the first recognition unit 132 updates the object's position based on the result of the next prediction at control timing 5 and the result of processing the input image. Also, at this control timing 6, information on the object's position recognized by the first recognition unit 132 is accumulated, making it possible to estimate the speed based on the position information. Therefore, the first recognition unit 132 updates the object's speed based on the recognition result of the object's position previously recognized (for example, the recognition result at control timing 3) and the result of processing the input image. Thereafter, the same processing as at control timings 4 to 6 is repeatedly executed.

In other words, the first recognition unit 132 has a function of repeatedly outputting the position and velocity of an object in a first period. The second recognition unit 134 has a function of repeatedly outputting the position and velocity of an object in a second period. When the first recognition unit 132 performs an output for the first time, the fusion unit 136 outputs the position output by the first recognition unit 132 and the velocity output by the second recognition unit 134 as the state of the object. Furthermore, when the first recognition unit 132 performs an output for the second time or later, the fusion unit 136 outputs the position and velocity output by the first recognition unit 132 as the state of the object.

According to the second embodiment described above, it is possible to quickly track changes in the speed of an object, and therefore to quickly control the behavior of the moving object. Furthermore, after the speed of an object is recognized by the first recognition unit 132, the position and speed recognized by the first recognition unit 132 are output as the state of the object, thereby improving the accuracy of estimating the position and speed of the object. Furthermore, after the position of an object is recognized by the first recognition unit 132, at a control timing when the first recognition unit 132 is not operating, the second recognition unit 134 updates the object's position based on the speed, thereby improving the accuracy of estimating the object's position, and effectively reducing the impact of position errors on the fusion unit to improve fusion performance.

Third Embodiment
Next, a third embodiment will be described. The third embodiment differs from the first and second embodiments in that the first recognition unit 132 does not output anything until it recognizes the speed, and outputs both the speed and the position after it recognizes the speed. In the following description, components having the same functions as those in the first and second embodiments are given the same reference numerals and names, and a detailed description thereof will be omitted.

FIG. 8 is a diagram showing an example of the operation of the first recognition unit 132 and the second recognition unit 134 in a certain scene according to the third embodiment. In this diagram, the second recognition unit 134 performs processing at a frequency three times that of the first recognition unit 132.

At control timing 1, the second recognition unit 134 provisionally recognizes the presence and position of an object for the first time. The second recognition unit 134 also recognizes the speed of the object. At this time, the second recognition unit 134 (or another functional unit) predicts the position and speed of the object at the next control timing 2, for example, by calculating a covariance matrix generated by arranging the states calculated before that control timing, its eigenvalues, and an interpolated value for future expansion (next prediction). This next prediction process is repeatedly executed at each control timing. At control timing 1, the state output by the recognition unit 130 is defined as tentatively recognized for the object's position and definitively recognized for its speed.

At control timing 3, the second recognition unit 134 updates the object's position and speed based on the result of the next prediction at control timing 2 and the result of processing the input image. Control timing 3 is also the timing at which the first recognition unit 132 operates for the first time. At this control timing 3, the first recognition unit 132 recognizes the object's position based on the result of the next prediction at control timing 2 and the result of processing the input image, but does not output the recognized position, but stores it in memory (not shown).

At control timing 4, the second recognition unit 134 updates the object's position and speed based on the result of the next prediction at control timing 3 and the result of processing the input image.

At control timing 5, the second recognition unit 134 updates the object's position and speed based on the result of the next prediction at control timing 4 and the result of processing the input image.

Control timing 6 is the timing when the first recognition unit 132 outputs for the first time. At this control timing 6, the first recognition unit 132 updates the object's position based on the result of the next prediction at control timing 5 and the result of processing the input image. At this time, the existence of the object is confirmed, and it is defined that the object's position has also been confirmed as being confirmed. Furthermore, at this control timing 6, information on the object's position recognized by the first recognition unit 132 is accumulated, making it possible to estimate the speed based on the information on that position. Therefore, the object's speed is updated based on the recognition result of the object's position previously recognized by the first recognition unit 132 (for example, the recognition result at control timing 3) and the result of processing the input image.

At control timing 7, the second recognition unit 134 updates the object's speed based on the result of the next prediction at control timing 6 and the result of processing the input image. Furthermore, the second recognition unit 134 updates the object's position based on the speed. For example, the second recognition unit 134 updates the object's position by linear interpolation. The second recognition unit 134 estimates the current object's position by setting the object's position updated at control timing 6 as a reference position and adding the object's movement amount calculated using linear interpolation (the movement amount from control timing 6 to control timing 7) to this reference position, and updates the object's position to the estimated position.

At control timing 8, the second recognition unit 134 updates the object's speed based on the result of the next prediction at control timing 7 and the result of processing the input image. Furthermore, the second recognition unit 134 updates the object's position based on the speed. For example, the second recognition unit 134 updates the object's position by linear interpolation. The second recognition unit 134 estimates the current object's position by taking the object's position updated at control timing 6 as a reference position and adding the object's movement amount calculated using linear interpolation (the movement amount from control timing 6 to control timing 8) to this reference position, and updates the object's position at the estimated position. Alternatively, the second recognition unit 134 estimates the current object's position by taking the object's position updated at control timing 7 as a reference position and adding the object's movement amount calculated using linear interpolation (the movement amount from control timing 7 to control timing 8) to this reference position, and updates the object's position at the estimated position.

Control timing 9 is the timing at which the first recognition unit 132 performs the second output. At this control timing 9, the first recognition unit 132 updates the object's position based on the result of the next prediction at control timing 8 and the result of processing the input image. The first recognition unit 132 also updates the object's speed based on the recognition result of the object's position previously recognized by the first recognition unit 132 (for example, the recognition result at control timings 3 and 6) and the result of processing the input image. As a result, at the timing at which the first recognition unit 132 performs an output, the fusion unit 136 outputs the position and speed output by the first recognition unit 132 as the object's state. Thereafter, the same processes as those at control timings 7 to 9 are repeatedly executed.

According to the third embodiment described above, it is possible to quickly track changes in the speed of an object, and therefore to quickly control the behavior of the moving body. Furthermore, after information on the object's position recognized by the first recognition unit 132 is accumulated and it becomes possible to estimate the speed, at the control timing when the first recognition unit 132 operates, the speed of the object is updated based on the speed recognized by the first recognition unit 132. This makes it possible to improve the accuracy of estimating the object's position and speed. Furthermore, it is possible to maintain design diversity in cases such as when the second recognition unit 134 is incorporated into an existing system that only has the first recognition unit 132.

Fourth Embodiment
Next, a fourth embodiment will be described. The fourth embodiment differs from the first to third embodiments in that, at the control timing when the first recognition unit 132 operates for the first time, the speed of the object is updated based on the recognition result by the second recognition unit 134 at the previous control timing immediately preceding the control timing in question and the recognition result by the first recognition unit 132 at the control timing in question. In the following description, components having the same functions as those in the first to third embodiments are given the same reference numerals and names, and a detailed description thereof will be omitted.

FIG. 9 is a diagram showing an example of the operation of the first recognition unit 132 and the second recognition unit 134 in a certain scene according to the fourth embodiment. In this figure, the second recognition unit 134 performs processing at a frequency three times that of the first recognition unit 132.

Control timing 3 is the timing when the first recognition unit 132 outputs for the first time. At this control timing 3, the first recognition unit 132 updates the object's position based on the result of the next prediction at control timing 2 and the result of processing the input image. At this time, it is defined that the existence of the object is confirmed and the object's position is also confirmed. The first recognition unit 132 updates the object's speed based on the recognition result by the second recognition unit 134 at control timing 2 (the position of the previous high-speed object recognition result) and the recognition result by the first recognition unit 132 at control timing 3 (the position of the current normal object recognition result). As a result, at the timing when the first recognition unit 132 outputs for the first time, the fusion unit 136 outputs, as the state of the object, the position output by the first recognition unit 132 and the speed output by the first recognition unit 132 based on the position output by the first recognition unit 132 and the position output by the second recognition unit 134 at the previous timing before that timing.

At control timing 4, the second recognition unit 134 updates the object's speed based on the result of the next prediction at control timing 3 and the result of processing the input image. Furthermore, the second recognition unit 134 updates the object's position based on the speed. For example, the second recognition unit 134 updates the object's position by linear interpolation. The second recognition unit 134 estimates the current object's position by setting the object's position recognized by the first recognition unit 132 at control timing 3 as a reference position and adding the object's movement amount calculated using linear interpolation (the movement amount from control timing 3 to control timing 4) to this reference position, and updates the object's position to the estimated position.

According to the fourth embodiment described above, it is possible to quickly track changes in the speed of an object, and therefore to quickly control the behavior of the moving object. Furthermore, at the control timing when the first recognition unit 132 operates for the first time, the speed of the object is updated based on the recognition result by the second recognition unit 134 at the previous timing and the recognition result by the first recognition unit 132 at the current control timing. This makes it possible to maintain design diversity when, for example, incorporating the second recognition unit 134 into an existing system equipped only with the first recognition unit 132.

In the above description, the control device is described as being mounted on the vehicle M (i.e., the moving body), but this is not limited thereto. The control device may be installed at a location away from the moving body, and may acquire output data from the camera 10, radar device 12, etc. through communication and transmit drive instruction signals to the moving body, i.e., may remotely control the moving body.

Furthermore, the embodiments described above are merely examples, and the present invention is not limited to the configurations of these embodiments. It is also possible to combine the functions or configurations included in each embodiment as appropriate. For example, as described in the second to fourth embodiments, it is also possible to incorporate into the first embodiment a configuration in which, after the position of an object is recognized by the first recognition unit 132, the position of the object is updated based on the speed recognized by the second recognition unit 134 at a control timing when the first recognition unit 132 is not operating.

The above-described embodiment can be expressed as follows.
A recognition device that recognizes a position and a speed of an object present around a moving body based on an output of a detection device for detecting a surrounding situation of the moving body,
one or more storage media storing computer-readable instructions;
a processor coupled to the one or more storage media;
The processor executes the computer-readable instructions to:
repeatedly outputting the position of the object, which is a result of performing processing for recognizing the position of the object, in a first period;
repeatedly outputting the velocity of the object, which is a result of performing a process for recognizing the velocity of the object, at a second period shorter than the first period;
Control device.

　Although the above describes the form for carrying out the present invention using an embodiment, the present invention is in no way limited to such an embodiment, and various modifications and substitutions can be made without departing from the spirit of the present invention.

10 Camera 12 Radar device 14 LIDAR
16 Object recognition device 100 Automatic driving control device 120 First control unit 130 Recognition unit 132 First recognition unit 134 Second recognition unit 136 Fusion unit 138 Future position prediction/risk setting unit 140 Action plan generation unit 160 Second control unit

Claims

A recognition device that recognizes a position and a speed of an object present around a moving body based on an output of a detection device for detecting a surrounding situation of the moving body,
a first recognition unit that repeatedly outputs the position of the object, which is a result of performing a process for recognizing the position of the object, in a first period;
a second recognition unit that repeatedly outputs the velocity of the object, which is a result of performing a process for recognizing the velocity of the object, at a second period that is shorter than the first period;
A recognition device comprising:
the second recognition unit has a function of provisionally recognizing a position of an object whose position has not been output by the first recognition unit,
When a position is output from the first recognition unit and a velocity is output from the second recognition unit, the position output from the first recognition unit and the velocity output from the second recognition unit are output as a state of the object,
a fusion unit that outputs the position and velocity output by the second recognition unit as a state of an object for which the position is not output from the first recognition unit and the position and velocity are output from the second recognition unit,
The recognition device according to claim 1.
The second recognition unit has a function of recognizing the presence of an object whose position has not been output by the first recognition unit.
The recognition device according to claim 2.
The first recognition unit and the second recognition unit each perform a process by executing a process procedure that is at least partially common to each other.
The recognition device according to claim 1.
Each of the first recognition unit and the second recognition unit is realized by one or a plurality of processors performing a process as the first recognition unit and a process as the second recognition unit in a time-division manner.
The recognition device according to claim 1.
The first recognition unit and the second recognition unit are each realized by a separate processor performing processing.
The recognition device according to claim 1.
the first recognition unit has a function of repeatedly outputting a position and a velocity of the object in the first period;
the second recognition unit has a function of repeatedly outputting the position and velocity of the object in the second period;
and a fusion unit that outputs the position output by the first recognition unit and the velocity output by the second recognition unit as a state of the object at a timing when the first recognition unit performs an output for the first time.
The recognition device according to claim 1.
The fusion unit outputs the position and velocity output by the first recognition unit as the state of the object at a timing when the first recognition unit performs a second or subsequent output.
The recognition device according to claim 7.
the first recognition unit has a function of repeatedly outputting a position and a velocity of the object in the first period;
and a fusion unit that outputs the position and velocity output by the first recognition unit as a state of the object at a timing when the first recognition unit outputs the position and velocity.
The recognition device according to claim 1.
the first recognition unit has a function of repeatedly outputting a position and a velocity of the object in the first period;
the second recognition unit has a function of repeatedly outputting the position and velocity of the object in the second period;
a fusion unit that outputs, at a timing when the first recognition unit performs an output for the first time, a position output by the first recognition unit, and a velocity output by the first recognition unit based on the position output by the first recognition unit and the position output by the second recognition unit at a previous timing before the timing, as a state of the object.
The recognition device according to claim 1.
After the first recognition unit outputs the first position at a first timing, at a second timing when the first recognition unit does not output and the second recognition unit outputs, the second recognition unit updates the first position by performing linear interpolation based on history information of positions and velocities of the object recognized in the past.
A recognition device according to any one of claims 1 to 10.
A recognition device according to claim 1;
a driving control unit that moves the moving body so as to avoid approaching an object whose state is output by the recognition device;
A control device for a moving object comprising:
A recognition method executed by a recognition device that recognizes positions and velocities of objects present around a moving body based on an output of a detection device for detecting a surrounding situation of the moving body, the method comprising:
repeatedly outputting the position of the object, which is a result of performing processing for recognizing the position of the object, in a first period;
repeatedly outputting the velocity of the object, which is a result of performing a process for recognizing the velocity of the object, at a second period shorter than the first period;
The recognition method includes:
A processor of a recognition device that recognizes positions and speeds of objects present around a moving object based on an output of a detection device for detecting a surrounding situation of the moving object,
repeatedly outputting the position of the object, which is a result of performing processing for recognizing the position of the object, in a first period;
repeatedly outputting the velocity of the object, which is a result of performing processing for recognizing the velocity of the object, at a second period shorter than the first period;
A program for executing.