[go: up one dir, main page]
More Web Proxy on the site http://driver.im/

EP4015336A1 - Systèmes, dispositifs et procédés impliquant des systèmes de commande - Google Patents

Systèmes, dispositifs et procédés impliquant des systèmes de commande Download PDF

Info

Publication number
EP4015336A1
EP4015336A1 EP21195534.9A EP21195534A EP4015336A1 EP 4015336 A1 EP4015336 A1 EP 4015336A1 EP 21195534 A EP21195534 A EP 21195534A EP 4015336 A1 EP4015336 A1 EP 4015336A1
Authority
EP
European Patent Office
Prior art keywords
vehicle
driver
data
risk
odd
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
EP21195534.9A
Other languages
German (de)
English (en)
Inventor
Florian Geissler
Rafael Rosales
Fabian Israel OBORIL
Cornelius Buerkle
Michael Paulitsch
Ignacio J. ALVAREZ
David Israel González Aguirre
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Intel Corp
Original Assignee
Intel Corp
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Intel Corp filed Critical Intel Corp
Publication of EP4015336A1 publication Critical patent/EP4015336A1/fr
Pending legal-status Critical Current

Links

Images

Classifications

    • B—PERFORMING OPERATIONS; TRANSPORTING
    • B60—VEHICLES IN GENERAL
    • B60W—CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION; CONTROL SYSTEMS SPECIALLY ADAPTED FOR HYBRID VEHICLES; ROAD VEHICLE DRIVE CONTROL SYSTEMS FOR PURPOSES NOT RELATED TO THE CONTROL OF A PARTICULAR SUB-UNIT
    • B60W50/00—Details of control systems for road vehicle drive control not related to the control of a particular sub-unit, e.g. process diagnostic or vehicle driver interfaces
    • B—PERFORMING OPERATIONS; TRANSPORTING
    • B60—VEHICLES IN GENERAL
    • B60W—CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION; CONTROL SYSTEMS SPECIALLY ADAPTED FOR HYBRID VEHICLES; ROAD VEHICLE DRIVE CONTROL SYSTEMS FOR PURPOSES NOT RELATED TO THE CONTROL OF A PARTICULAR SUB-UNIT
    • B60W60/00—Drive control systems specially adapted for autonomous road vehicles
    • B60W60/005—Handover processes
    • B60W60/0053—Handover processes from vehicle to occupant
    • B—PERFORMING OPERATIONS; TRANSPORTING
    • B60—VEHICLES IN GENERAL
    • B60K—ARRANGEMENT OR MOUNTING OF PROPULSION UNITS OR OF TRANSMISSIONS IN VEHICLES; ARRANGEMENT OR MOUNTING OF PLURAL DIVERSE PRIME-MOVERS IN VEHICLES; AUXILIARY DRIVES FOR VEHICLES; INSTRUMENTATION OR DASHBOARDS FOR VEHICLES; ARRANGEMENTS IN CONNECTION WITH COOLING, AIR INTAKE, GAS EXHAUST OR FUEL SUPPLY OF PROPULSION UNITS IN VEHICLES
    • B60K35/00—Instruments specially adapted for vehicles; Arrangement of instruments in or on vehicles
    • B—PERFORMING OPERATIONS; TRANSPORTING
    • B60—VEHICLES IN GENERAL
    • B60K—ARRANGEMENT OR MOUNTING OF PROPULSION UNITS OR OF TRANSMISSIONS IN VEHICLES; ARRANGEMENT OR MOUNTING OF PLURAL DIVERSE PRIME-MOVERS IN VEHICLES; AUXILIARY DRIVES FOR VEHICLES; INSTRUMENTATION OR DASHBOARDS FOR VEHICLES; ARRANGEMENTS IN CONNECTION WITH COOLING, AIR INTAKE, GAS EXHAUST OR FUEL SUPPLY OF PROPULSION UNITS IN VEHICLES
    • B60K35/00—Instruments specially adapted for vehicles; Arrangement of instruments in or on vehicles
    • B60K35/10—Input arrangements, i.e. from user to vehicle, associated with vehicle functions or specially adapted therefor
    • B—PERFORMING OPERATIONS; TRANSPORTING
    • B60—VEHICLES IN GENERAL
    • B60W—CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION; CONTROL SYSTEMS SPECIALLY ADAPTED FOR HYBRID VEHICLES; ROAD VEHICLE DRIVE CONTROL SYSTEMS FOR PURPOSES NOT RELATED TO THE CONTROL OF A PARTICULAR SUB-UNIT
    • B60W30/00—Purposes of road vehicle drive control systems not related to the control of a particular sub-unit, e.g. of systems using conjoint control of vehicle sub-units
    • B60W30/08—Active safety systems predicting or avoiding probable or impending collision or attempting to minimise its consequences
    • B60W30/095—Predicting travel path or likelihood of collision
    • B60W30/0956—Predicting travel path or likelihood of collision the prediction being responsive to traffic or environmental parameters
    • B—PERFORMING OPERATIONS; TRANSPORTING
    • B60—VEHICLES IN GENERAL
    • B60W—CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION; CONTROL SYSTEMS SPECIALLY ADAPTED FOR HYBRID VEHICLES; ROAD VEHICLE DRIVE CONTROL SYSTEMS FOR PURPOSES NOT RELATED TO THE CONTROL OF A PARTICULAR SUB-UNIT
    • B60W40/00—Estimation or calculation of non-directly measurable driving parameters for road vehicle drive control systems not related to the control of a particular sub unit, e.g. by using mathematical models
    • B60W40/02—Estimation or calculation of non-directly measurable driving parameters for road vehicle drive control systems not related to the control of a particular sub unit, e.g. by using mathematical models related to ambient conditions
    • B—PERFORMING OPERATIONS; TRANSPORTING
    • B60—VEHICLES IN GENERAL
    • B60W—CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION; CONTROL SYSTEMS SPECIALLY ADAPTED FOR HYBRID VEHICLES; ROAD VEHICLE DRIVE CONTROL SYSTEMS FOR PURPOSES NOT RELATED TO THE CONTROL OF A PARTICULAR SUB-UNIT
    • B60W40/00—Estimation or calculation of non-directly measurable driving parameters for road vehicle drive control systems not related to the control of a particular sub unit, e.g. by using mathematical models
    • B60W40/08—Estimation or calculation of non-directly measurable driving parameters for road vehicle drive control systems not related to the control of a particular sub unit, e.g. by using mathematical models related to drivers or passengers
    • B—PERFORMING OPERATIONS; TRANSPORTING
    • B60—VEHICLES IN GENERAL
    • B60W—CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION; CONTROL SYSTEMS SPECIALLY ADAPTED FOR HYBRID VEHICLES; ROAD VEHICLE DRIVE CONTROL SYSTEMS FOR PURPOSES NOT RELATED TO THE CONTROL OF A PARTICULAR SUB-UNIT
    • B60W50/00—Details of control systems for road vehicle drive control not related to the control of a particular sub-unit, e.g. process diagnostic or vehicle driver interfaces
    • B60W50/0098—Details of control systems ensuring comfort, safety or stability not otherwise provided for
    • B—PERFORMING OPERATIONS; TRANSPORTING
    • B60—VEHICLES IN GENERAL
    • B60W—CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION; CONTROL SYSTEMS SPECIALLY ADAPTED FOR HYBRID VEHICLES; ROAD VEHICLE DRIVE CONTROL SYSTEMS FOR PURPOSES NOT RELATED TO THE CONTROL OF A PARTICULAR SUB-UNIT
    • B60W60/00—Drive control systems specially adapted for autonomous road vehicles
    • B60W60/005—Handover processes
    • B60W60/0059—Estimation of the risk associated with autonomous or manual driving, e.g. situation too complex, sensor failure or driver incapacity
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V20/00—Scenes; Scene-specific elements
    • G06V20/50—Context or environment of the image
    • G06V20/59—Context or environment of the image inside of a vehicle, e.g. relating to seat occupancy, driver state or inner lighting conditions
    • G06V20/597—Recognising the driver's state or behaviour, e.g. attention or drowsiness
    • B—PERFORMING OPERATIONS; TRANSPORTING
    • B60—VEHICLES IN GENERAL
    • B60K—ARRANGEMENT OR MOUNTING OF PROPULSION UNITS OR OF TRANSMISSIONS IN VEHICLES; ARRANGEMENT OR MOUNTING OF PLURAL DIVERSE PRIME-MOVERS IN VEHICLES; AUXILIARY DRIVES FOR VEHICLES; INSTRUMENTATION OR DASHBOARDS FOR VEHICLES; ARRANGEMENTS IN CONNECTION WITH COOLING, AIR INTAKE, GAS EXHAUST OR FUEL SUPPLY OF PROPULSION UNITS IN VEHICLES
    • B60K2360/00—Indexing scheme associated with groups B60K35/00 or B60K37/00 relating to details of instruments or dashboards
    • B60K2360/146—Instrument input by gesture
    • B—PERFORMING OPERATIONS; TRANSPORTING
    • B60—VEHICLES IN GENERAL
    • B60K—ARRANGEMENT OR MOUNTING OF PROPULSION UNITS OR OF TRANSMISSIONS IN VEHICLES; ARRANGEMENT OR MOUNTING OF PLURAL DIVERSE PRIME-MOVERS IN VEHICLES; AUXILIARY DRIVES FOR VEHICLES; INSTRUMENTATION OR DASHBOARDS FOR VEHICLES; ARRANGEMENTS IN CONNECTION WITH COOLING, AIR INTAKE, GAS EXHAUST OR FUEL SUPPLY OF PROPULSION UNITS IN VEHICLES
    • B60K2360/00—Indexing scheme associated with groups B60K35/00 or B60K37/00 relating to details of instruments or dashboards
    • B60K2360/148—Instrument input by voice
    • B—PERFORMING OPERATIONS; TRANSPORTING
    • B60—VEHICLES IN GENERAL
    • B60K—ARRANGEMENT OR MOUNTING OF PROPULSION UNITS OR OF TRANSMISSIONS IN VEHICLES; ARRANGEMENT OR MOUNTING OF PLURAL DIVERSE PRIME-MOVERS IN VEHICLES; AUXILIARY DRIVES FOR VEHICLES; INSTRUMENTATION OR DASHBOARDS FOR VEHICLES; ARRANGEMENTS IN CONNECTION WITH COOLING, AIR INTAKE, GAS EXHAUST OR FUEL SUPPLY OF PROPULSION UNITS IN VEHICLES
    • B60K2360/00—Indexing scheme associated with groups B60K35/00 or B60K37/00 relating to details of instruments or dashboards
    • B60K2360/149—Instrument input by detecting viewing direction not otherwise provided for
    • B—PERFORMING OPERATIONS; TRANSPORTING
    • B60—VEHICLES IN GENERAL
    • B60W—CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION; CONTROL SYSTEMS SPECIALLY ADAPTED FOR HYBRID VEHICLES; ROAD VEHICLE DRIVE CONTROL SYSTEMS FOR PURPOSES NOT RELATED TO THE CONTROL OF A PARTICULAR SUB-UNIT
    • B60W40/00—Estimation or calculation of non-directly measurable driving parameters for road vehicle drive control systems not related to the control of a particular sub unit, e.g. by using mathematical models
    • B60W40/08—Estimation or calculation of non-directly measurable driving parameters for road vehicle drive control systems not related to the control of a particular sub unit, e.g. by using mathematical models related to drivers or passengers
    • B60W2040/089—Driver voice
    • B—PERFORMING OPERATIONS; TRANSPORTING
    • B60—VEHICLES IN GENERAL
    • B60W—CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION; CONTROL SYSTEMS SPECIALLY ADAPTED FOR HYBRID VEHICLES; ROAD VEHICLE DRIVE CONTROL SYSTEMS FOR PURPOSES NOT RELATED TO THE CONTROL OF A PARTICULAR SUB-UNIT
    • B60W50/00—Details of control systems for road vehicle drive control not related to the control of a particular sub-unit, e.g. process diagnostic or vehicle driver interfaces
    • B60W2050/0001—Details of the control system
    • B60W2050/0019—Control system elements or transfer functions
    • B60W2050/0028—Mathematical models, e.g. for simulation
    • B60W2050/0031—Mathematical model of the vehicle
    • B—PERFORMING OPERATIONS; TRANSPORTING
    • B60—VEHICLES IN GENERAL
    • B60W—CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION; CONTROL SYSTEMS SPECIALLY ADAPTED FOR HYBRID VEHICLES; ROAD VEHICLE DRIVE CONTROL SYSTEMS FOR PURPOSES NOT RELATED TO THE CONTROL OF A PARTICULAR SUB-UNIT
    • B60W50/00—Details of control systems for road vehicle drive control not related to the control of a particular sub-unit, e.g. process diagnostic or vehicle driver interfaces
    • B60W2050/0001—Details of the control system
    • B60W2050/0043—Signal treatments, identification of variables or parameters, parameter estimation or state estimation
    • B—PERFORMING OPERATIONS; TRANSPORTING
    • B60—VEHICLES IN GENERAL
    • B60W—CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION; CONTROL SYSTEMS SPECIALLY ADAPTED FOR HYBRID VEHICLES; ROAD VEHICLE DRIVE CONTROL SYSTEMS FOR PURPOSES NOT RELATED TO THE CONTROL OF A PARTICULAR SUB-UNIT
    • B60W2540/00—Input parameters relating to occupants
    • B60W2540/21—Voice
    • B—PERFORMING OPERATIONS; TRANSPORTING
    • B60—VEHICLES IN GENERAL
    • B60W—CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION; CONTROL SYSTEMS SPECIALLY ADAPTED FOR HYBRID VEHICLES; ROAD VEHICLE DRIVE CONTROL SYSTEMS FOR PURPOSES NOT RELATED TO THE CONTROL OF A PARTICULAR SUB-UNIT
    • B60W2540/00—Input parameters relating to occupants
    • B60W2540/22—Psychological state; Stress level or workload
    • B—PERFORMING OPERATIONS; TRANSPORTING
    • B60—VEHICLES IN GENERAL
    • B60W—CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION; CONTROL SYSTEMS SPECIALLY ADAPTED FOR HYBRID VEHICLES; ROAD VEHICLE DRIVE CONTROL SYSTEMS FOR PURPOSES NOT RELATED TO THE CONTROL OF A PARTICULAR SUB-UNIT
    • B60W2540/00—Input parameters relating to occupants
    • B60W2540/221—Physiology, e.g. weight, heartbeat, health or special needs
    • B—PERFORMING OPERATIONS; TRANSPORTING
    • B60—VEHICLES IN GENERAL
    • B60W—CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION; CONTROL SYSTEMS SPECIALLY ADAPTED FOR HYBRID VEHICLES; ROAD VEHICLE DRIVE CONTROL SYSTEMS FOR PURPOSES NOT RELATED TO THE CONTROL OF A PARTICULAR SUB-UNIT
    • B60W2540/00—Input parameters relating to occupants
    • B60W2540/223—Posture, e.g. hand, foot, or seat position, turned or inclined
    • B—PERFORMING OPERATIONS; TRANSPORTING
    • B60—VEHICLES IN GENERAL
    • B60W—CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION; CONTROL SYSTEMS SPECIALLY ADAPTED FOR HYBRID VEHICLES; ROAD VEHICLE DRIVE CONTROL SYSTEMS FOR PURPOSES NOT RELATED TO THE CONTROL OF A PARTICULAR SUB-UNIT
    • B60W2540/00—Input parameters relating to occupants
    • B60W2540/225—Direction of gaze
    • B—PERFORMING OPERATIONS; TRANSPORTING
    • B60—VEHICLES IN GENERAL
    • B60W—CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION; CONTROL SYSTEMS SPECIALLY ADAPTED FOR HYBRID VEHICLES; ROAD VEHICLE DRIVE CONTROL SYSTEMS FOR PURPOSES NOT RELATED TO THE CONTROL OF A PARTICULAR SUB-UNIT
    • B60W2540/00—Input parameters relating to occupants
    • B60W2540/229—Attention level, e.g. attentive to driving, reading or sleeping
    • B—PERFORMING OPERATIONS; TRANSPORTING
    • B60—VEHICLES IN GENERAL
    • B60W—CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION; CONTROL SYSTEMS SPECIALLY ADAPTED FOR HYBRID VEHICLES; ROAD VEHICLE DRIVE CONTROL SYSTEMS FOR PURPOSES NOT RELATED TO THE CONTROL OF A PARTICULAR SUB-UNIT
    • B60W2556/00—Input parameters relating to data
    • B60W2556/45—External transmission of data to or from the vehicle
    • B—PERFORMING OPERATIONS; TRANSPORTING
    • B60—VEHICLES IN GENERAL
    • B60W—CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION; CONTROL SYSTEMS SPECIALLY ADAPTED FOR HYBRID VEHICLES; ROAD VEHICLE DRIVE CONTROL SYSTEMS FOR PURPOSES NOT RELATED TO THE CONTROL OF A PARTICULAR SUB-UNIT
    • B60W2556/00—Input parameters relating to data
    • B60W2556/45—External transmission of data to or from the vehicle
    • B60W2556/50—External transmission of data to or from the vehicle of positioning data, e.g. GPS [Global Positioning System] data

Definitions

  • Various aspects of this disclosure generally relate to driving systems.
  • ADS automated driving systems
  • ODD Operational Design Domain
  • DMS Driver Monitoring Systems
  • a key challenge is the assessment of the ODD compliance, which means that the ADS has to identify whether or not it is currently in a state that belongs to the intended ODD for the L3 system. This can be defined as the "operating conditions under which a given driving automation system or feature thereof is specifically designed to function, including, but not limited to, environmental, geographical, and time-of-day restrictions, and/or the requisite presence or absence of certain traffic or roadway characteristics.”
  • This assessment can be performed in a separate component called the ODD monitor.
  • a major safety risk occurs if this assessment is false positive, i.e. if the monitor assesses to be in the ODD, while in reality it is not. False negative ODD compliance assessments on the other hand lead to unnecessary handovers which reduce availability of the L3 system and can be potentially unsafe as well during the transition phase.
  • At least one and “one or more” may be understood to include a numerical quantity greater than or equal to one (e.g., one, two, three, four,tinct, etc.).
  • a plurality may be understood to include a numerical quantity greater than or equal to two (e.g., two, three, four, five,tinct, etc.).
  • any phrases explicitly invoking the aforementioned words expressly refers to more than one of the said elements.
  • phrases "at least one of' with regard to a group of elements may be used herein to mean at least one element from the group including the elements.
  • the phrase "at least one of' with regard to a group of elements may be used herein to mean a selection of: one of the listed elements, a plurality of one of the listed elements, a plurality of individual listed elements, or a plurality of a multiple of individual listed elements.
  • data may be understood to include information in any suitable analog or digital form, e.g., provided as a file, a portion of a file, a set of files, a signal or stream, a portion of a signal or stream, a set of signals or streams, and the like. Further, the term “data” may also be used to mean a reference to information, e.g., in the form of a pointer. However, the term “data” is not limited to the aforementioned examples and may take various forms and represent any information as understood in the art.
  • processor or “controller” as, for example, used herein may be understood as any kind of technological entity that allows handling of data.
  • the data may be handled according to one or more specific functions executed by the processor or controller.
  • a processor or controller as used herein may be understood as any kind of circuit, e.g., any kind of analog or digital circuit, and may also be referred to as a "processing circuit,” “processing circuitry,” among others.
  • a processor or a controller may thus be or include an analog circuit, digital circuit, mixed-signal circuit, logic circuit, processor, microprocessor, Central Processing Unit (CPU), Graphics Processing Unit (GPU), Digital Signal Processor (DSP), Field Programmable Gate Array (FPGA), integrated circuit, Application Specific Integrated Circuit (ASIC), etc., or any combination thereof. Any other kind of implementation of the respective functions, which will be described below in further detail, may also be understood as a processor, controller, or logic circuit.
  • CPU Central Processing Unit
  • GPU Graphics Processing Unit
  • DSP Digital Signal Processor
  • FPGA Field Programmable Gate Array
  • ASIC Application Specific Integrated Circuit
  • any two (or more) of the processors, controllers, or logic circuits detailed herein may be realized as a single entity with equivalent functionality, among others, and conversely that any single processor, controller, or logic circuit detailed herein may be realized as two (or more) separate entities with equivalent functionality, among others.
  • circuitry As utilized herein, terms “module”, “component,” “system,” “circuit,” “element,” “slice,” “circuitry,” and the like are intended to refer to a set of one or more electronic components, a computer-related entity, hardware, software (e.g., in execution), and/or firmware.
  • circuitry or a similar term can be a processor, a process running on a processor, a controller, an object, an executable program, a storage device, and/or a computer with a processing device.
  • an application running on a server and the server can also be circuitry.
  • One or more circuits can reside within the same circuitry, and circuitry can be localized on one computer and/or distributed between two or more computers.
  • a set of elements or a set of other circuits can be described herein, in which the term “set” can be interpreted as "one or more.”
  • memory is understood as a computer-readable medium in which data or information can be stored for retrieval. References to “memory” included herein may thus be understood as referring to volatile or non-volatile memory, including random access memory (RAM), read-only memory (ROM), flash memory, solid-state storage, magnetic tape, hard disk drive, optical drive, among others, or any combination thereof. Registers, shift registers, processor registers, data buffers, among others, are also embraced herein by the term memory.
  • software refers to any type of executable instruction, including firmware.
  • the term “transmit” encompasses both direct (point-to-point) and indirect transmission (via one or more intermediary points).
  • the term “receive” encompasses both direct and indirect reception.
  • the terms “transmit,” “receive,” “communicate,” and other similar terms encompass both physical transmission (e.g., the transmission of radio signals) and logical transmission (e.g., the transmission of digital data over a logical software-level connection).
  • a processor or controller may transmit or receive data over a software-level connection with another processor or controller in the form of radio signals, where the physical transmission and reception is handled by radio-layer components such as RF transceivers and antennas, and the logical transmission and reception over the software-level connection is performed by the processors or controllers.
  • the term "communicate” encompasses one or both of transmitting and receiving, i.e., unidirectional or bidirectional communication in one or both of the incoming and outgoing directions.
  • the term “calculate” encompasses both 'direct' calculations via a mathematical expression/formula/relationship and 'indirect' calculations via lookup or hash tables and other array indexing or searching operations.
  • a “vehicle” may be understood to include any type of driven or drivable object.
  • a vehicle may be a driven object with a combustion engine, a reaction engine, an electrically driven object, a hybrid driven object, or a combination thereof.
  • a vehicle may be or may include an automobile, a bus, a mini bus, a van, a truck, a mobile home, a vehicle trailer, a motorcycle, a bicycle, a tricycle, a train locomotive, a train wagon, a moving robot, a personal transporter, a boat, a ship, a submersible, a submarine, a drone, an aircraft, a rocket, and the like.
  • a "ground vehicle” may be understood to include any type of vehicle, as described above, which is configured to traverse or be driven on the ground, e.g., on a street, on a road, on a track, on one or more rails, off-road, etc.
  • An “aerial vehicle” may be understood to be any type of vehicle, as described above, which is capable of being maneuvered above the ground for any duration of time, e.g., a drone. Similar to a ground vehicle having wheels, belts, etc., for providing mobility on terrain, an “aerial vehicle” may have one or more propellers, wings, fans, among others, for providing the ability to maneuver in the air.
  • An “aquatic vehicle” may be understood to be any type of vehicle, as described above, which is capable of being maneuvers on or below the surface of a liquid, e.g., a boat on the surface of water or a submarine below the surface. It is appreciated that some vehicles may be configured to operate as one or more of a ground, an aerial, and/or an aquatic vehicle.
  • autonomous vehicle may describe a vehicle capable of implementing at least one navigational change without driver input.
  • a navigational change may describe or include a change in one or more of steering, braking, or acceleration/deceleration of the vehicle.
  • a vehicle may be described as autonomous even in case the vehicle is not fully automatic (e.g., fully operational with driver or without driver input).
  • Autonomous vehicles may include those vehicles that can operate under driver control during certain time periods and without driver control during other time periods.
  • Autonomous vehicles may also include vehicles that control only some aspects of vehicle navigation, such as steering (e.g., maintaining a vehicle course between vehicle lane constraints) or some steering operations under certain circumstances (but not under all circumstances).
  • Autonomous vehicles may also include vehicles that share the control of one or more aspects of vehicle navigation under certain circumstances (e.g., hands-on, such as responsive to a driver input) and vehicles that control one or more aspects of vehicle navigation under certain circumstances (e.g., hands-off, such as independent of driver input).
  • Autonomous vehicles may also include vehicles that control one or more vehicle navigation aspects under certain circumstances, such as under certain environmental conditions (e.g., spatial areas, roadway conditions).
  • autonomous vehicles may handle some or all aspects of braking, speed control, velocity control, and/or steering of the vehicle.
  • An autonomous vehicle may include those vehicles that can operate without a driver.
  • the level of autonomy of a vehicle may be described or determined by the Society of Automotive Engineers (SAE) level of the vehicle (e.g., as defined by the SAE, for example in SAE J3016 2018: Taxonomy and definitions for terms related to driving automation systems for on-road motor vehicles) or by other relevant professional organizations.
  • SAE level may have a value ranging from a minimum level, e.g., level 0 (illustratively, substantially no driving automation), to a maximum level, e.g., level 5 (illustratively, full driving automation).
  • vehicle operation data may be understood to describe any type of feature related to the operation of a vehicle.
  • vehicle operation data may describe the vehicle's status, such as the type of propulsion unit(s), types of tires or propellers of the vehicle, the type of vehicle, and/or the age of the manufacturing of the vehicle.
  • vehicle operation data may describe or include static features or static vehicle operation data (illustratively, features or data not changing over time).
  • vehicle operation data may describe or include features changing during the operation of the vehicle, for example, environmental conditions, such as weather conditions or road conditions during the operation of the vehicle, fuel levels, fluid levels, operational parameters of the driving source of the vehicle, etc. More generally, “vehicle operation data” may describe or include varying features or varying vehicle operation data (illustratively, time-varying features or data).
  • model as, for example, used herein may be understood as any kind of algorithm, which provides output data from input data (e.g., any kind of algorithm generating or calculating output data from input data).
  • a computing system may execute a machine learning model to improve a specific task's performance progressively.
  • a machine learning model's parameters may be adjusted during a training phase based on training data.
  • a trained machine learning model may be used during an inference phase to make predictions or decisions based on input data.
  • the trained machine learning model may be used to generate additional training data.
  • An additional machine learning model may be adjusted during a second training phase based on the generated additional training data.
  • a trained additional machine learning model may be used during an inference phase to make predictions or decisions based on input data.
  • the machine learning models described herein may take any suitable form or utilize any suitable technique (e.g., for training purposes).
  • any machine learning models may utilize supervised learning, semi-supervised learning, unsupervised learning, or reinforcement learning techniques.
  • the model may be built using a training set of data including both the inputs and the corresponding desired outputs (illustratively, each input may be associated with a desired or expected output for that input).
  • Each training instance may include one or more inputs and a desired output.
  • Training may include iterating through training instances and using an objective function to teach the model to predict the output for new inputs (illustratively, for inputs not included in the training set).
  • a portion of the inputs in the training set may be missing the respective desired outputs (e.g., one or more inputs may not be associated with any desired or expected output).
  • the model may be built from a training set of data including only inputs and no desired outputs.
  • the unsupervised model may be used to find structure in the data (e.g., grouping or clustering of data points), illustratively, by discovering patterns in the data.
  • Techniques that may be implemented in an unsupervised learning model may include, e.g., self-organizing maps, nearest-neighbor mapping, k-means clustering, and singular value decomposition.
  • Reinforcement learning models may include positive or negative feedback to improve accuracy.
  • a reinforcement learning model may attempt to maximize one or more objectives/rewards.
  • Techniques that may be implemented in a reinforcement learning model may include, e.g., Q-learning, temporal difference (TD), and deep adversarial networks.
  • a classification model may be restricted to a limited set of values (e.g., one or more classes).
  • the classification model may output a class for an input set of one or more input values.
  • An input set may include sensor data, such as image data, radar data, LIDAR data, and the like.
  • a classification model may classify certain driving conditions and/or environmental conditions, such as weather conditions, road conditions, and the like.
  • references herein to classification models may contemplate a model that implements, e.g., any one or more of the following techniques: linear classifiers (e.g., logistic regression or naive Bayes classifier), support vector machines, decision trees, boosted trees, random forest, neural networks, or nearest neighbor.
  • linear classifiers e.g., logistic regression or naive Bayes classifier
  • support vector machines decision trees, boosted trees, random forest, neural networks, or nearest neighbor.
  • a regression model may output a numerical value from a continuous range based on an input set of one or more values (illustratively, starting from or using an input set of one or more values).
  • References herein to regression models may contemplate a model that implements, e.g., any one or more of the following techniques (or other suitable techniques): linear regression, decision trees, random forest, or neural networks.
  • a machine learning model described herein may be or may include a neural network.
  • the neural network may be any kind of neural network, such as a convolutional neural network, an autoencoder network, a variational autoencoder network, a sparse autoencoder network, a recurrent neural network, a deconvolutional network, a generative adversarial network, a forward-thinking neural network, a sum-product neural network, and the like.
  • the neural network may include any number of layers.
  • the training of the neural network (e.g., adapting the layers of the neural network) may use or may be based on any kind of training principle, such as backpropagation (e.g., using the backpropagation algorithm).
  • driving parameter set may be used as synonyms: driving parameter set, driving model parameters, driving model parameter set, safety layer parameter set, driver assistance, automated driving model parameter set, and/or the like (e.g., driving safety parameter set). These terms may correspond to groups of values used to implement one or more models for directing a vehicle to operate according to the manners described herein.
  • driving parameter e.g., driving model parameter, safety layer parameter, driver assistance and/or automated driving model parameter, and/or the like (e.g., driving safety parameter), and may correspond to specific values within the previously described sets.
  • FIG. 1 shows a vehicle 100, including a mobility system 120 and a control system 200 (see also FIG. 2 ) in accordance with various aspects.
  • vehicle 100 and control system 200 are exemplary in nature and may thus be simplified for explanatory purposes.
  • vehicle 100 is depicted as a ground vehicle, aspects of this disclosure may be equally or analogously applied to aerial vehicles such as drones or aquatic vehicles such as boats.
  • aerial vehicles such as drones or aquatic vehicles such as boats.
  • the quantities and locations of elements, as well as relational distances are provided as examples and are not limited thereto.
  • the components of vehicle 100 may be arranged around a vehicular housing of vehicle 100, mounted on or outside of the vehicular housing, enclosed within the vehicular housing, or any other arrangement relative to the vehicular housing where the components move with vehicle 100 as it travels.
  • the vehicular housing such as an automobile body, drone body, plane or helicopter fuselage, boat hull, or similar type of vehicular body dependent on the type of vehicle that vehicle 100 is.
  • vehicle 100 may also include a mobility system 120.
  • Mobility system 120 may include components of vehicle 100 related to steering and movement of vehicle 100.
  • vehicle 100 is an automobile
  • mobility system 120 may include wheels and axles, a suspension, an engine, a transmission, brakes, a steering wheel, associated electrical circuitry and wiring, and any other components used in the driving of an automobile.
  • vehicle 100 is an aerial vehicle
  • mobility system 120 may include one or more of rotors, propellers, jet engines, wings, rudders or wing flaps, air brakes, a yoke or cyclic, associated electrical circuitry and wiring, and any other components used in the flying of an aerial vehicle.
  • mobility system 120 may include any one or more of rudders, engines, propellers, a steering wheel, associated electrical circuitry and wiring, and any other components used in the steering or movement of an aquatic vehicle.
  • mobility system 120 may also include autonomous driving functionality, and accordingly may include an interface with one or more processors 102 configured to perform autonomous driving computations and decisions and an array of sensors for movement and obstacle sensing. In this sense, the mobility system 120 may be provided with instructions to direct the navigation and/or mobility of vehicle 100 from one or more components of the control system 200.
  • the autonomous driving components of mobility system 120 may also interface with one or more radio frequency (RF) transceivers 108 to facilitate mobility coordination with other nearby vehicular communication devices and/or central networking components.
  • the devices or components can perform decisions and/or computations related to autonomous driving.
  • RF radio frequency
  • the control system 200 may include various components depending on the requirements of a particular implementation. As shown in FIG. 1 and FIG. 2 , the control system 200 may include one or more processors 102, one or more memories 104, an antenna system 106 which may include one or more antenna arrays at different locations on the vehicle for radio frequency (RF) coverage, one or more radio frequency (RF) transceivers 108, one or more data acquisition devices 112, one or more position devices 114 which may include components and circuitry for receiving and determining a position based on a Global Navigation Satellite System (GNSS) and/or a Global Positioning System (GPS), and one or more measurement sensors 116, e.g., speedometer, altimeter, gyroscope, velocity sensors, etc.
  • GNSS Global Navigation Satellite System
  • GPS Global Positioning System
  • the control system 200 may be configured to control the vehicle's 100 mobility via mobility system 120 and/or interactions with its environment, e.g., communications with other devices or network infrastructure elements (NIEs) such as base stations, via data acquisition devices 112 and the radio frequency communication arrangement including the one or more RF transceivers 108 and antenna system 106.
  • NNEs network infrastructure elements
  • the one or more processors 102 may include a data acquisition processor 214, an application processor 216, a communication processor 218, and/or any other suitable processing device.
  • Each processor 214, 216, 218 of the one or more processors 102 may include various types of hardware-based processing devices.
  • each processor 214, 216, 218 may include a microprocessor, pre-processors (such as an image pre-processor), graphics processors, a central processing unit (CPU), support circuits, digital signal processors, integrated circuits, memory, or any other types of devices suitable for running applications and for image processing and analysis.
  • each processor 214, 216, 218 may include any type of single or multi-core processor, mobile device microcontroller, central processing unit, etc. These processor types may each include multiple processing units with local memory and instruction sets.
  • Such processors may include video inputs for receiving image data from multiple image sensors and may also include video out capabilities.
  • processors 214, 216, 218 disclosed herein may be configured to perform certain functions according to program instructions that may be stored in a memory of the one or more memories 104.
  • a memory of the one or more memories 104 may store software that, when executed by a processor (e.g., by the one or more processors 102), controls the operation of the system, e.g., a driving and/or safety system.
  • a memory of the one or more memories 104 may store one or more databases and image processing software, as well as a trained system, such as a neural network, or a deep neural network, for example.
  • the one or more memories 104 may include any number of random-access memories, read-only memories, flash memories, disk drives, optical storage, tape storage, removable storage, and other storage types.
  • each of processors 214, 216, 218 may include an internal memory for such storage.
  • the data acquisition processor 216 may include processing circuity, such as a CPU, for processing data acquired by data acquisition units 112. For example, suppose one or more data acquisition units are image acquisition units, e.g., one or more cameras. In that case, the data acquisition processor may include image processors for processing image data using the information obtained from the image acquisition units as an input. The data acquisition processor 216 may therefore be configured to create voxel maps detailing the surrounding of the vehicle 100 based on the data input from the data acquisition units 112, i.e., cameras in this example.
  • Application processor 216 may be a CPU, and may be configured to handle the layers above the protocol stack, including the transport and application layers. Application processor 216 may be configured to execute various applications and/or programs of vehicle 100 at an application layer of vehicle 100, such as an operating system (OS), a user interfaces (UI) 206 for supporting user interaction with vehicle 100, and/or various user applications. Application processor 216 may interface with communication processor 218 and act as a source (in the transmit path) and a sink (in the receive path) for user data, such as voice data, audio/video/image data, messaging data, application data, basic Internet/web access data, etc.
  • OS operating system
  • UI user interfaces
  • Application processor 216 may interface with communication processor 218 and act as a source (in the transmit path) and a sink (in the receive path) for user data, such as voice data, audio/video/image data, messaging data, application data, basic Internet/web access data, etc.
  • communication processor 218 may therefore receive and process outgoing data provided by application processor 216 according to the layer-specific functions of the protocol stack, and provide the resulting data to digital signal processor 208. Communication processor 218 may then perform physical layer processing on the received data to produce digital baseband samples, which digital signal processor may provide to RF transceiver(s) 108. RF transceiver(s) 108 may then process the digital baseband samples to convert the digital baseband samples to analog RF signals, which RF transceiver(s) 108 may wirelessly transmit via antenna system 106. In the receive path, RF transceiver(s) 108 may receive analog RF signals from antenna system 106 and process the analog RF signals to obtain digital baseband samples.
  • RF transceiver(s) 108 may provide the digital baseband samples to communication processor 218, which may perform physical layer processing on the digital baseband samples.
  • Communication processor 218 may then provide the resulting data to other processors of the one or more processors 102, which may process the resulting data according to the layer-specific functions of the protocol stack and provide the resulting incoming data to application processor 216.
  • Application processor 216 may then handle the incoming data at the application layer, which can include execution of one or more application programs with the data and/or presentation of the data to a user via one or more user interfaces 206.
  • User interfaces 206 may include one or more screens, microphones, mice, touchpads, keyboards, or any other interface providing a mechanism for user input.
  • the communication processor 218 may include a digital signal processor and/or a controller which may direct such communication functionality of vehicle 100 according to the communication protocols associated with one or more radio access networks, and may execute control over antenna system 106 and RF transceiver(s) 108 to transmit and receive radio signals according to the formatting and scheduling parameters defined by each communication protocol.
  • a digital signal processor and/or a controller which may direct such communication functionality of vehicle 100 according to the communication protocols associated with one or more radio access networks, and may execute control over antenna system 106 and RF transceiver(s) 108 to transmit and receive radio signals according to the formatting and scheduling parameters defined by each communication protocol.
  • Vehicle 100 may transmit and receive wireless signals with antenna system 106, which may be a single antenna or an antenna array that includes multiple antenna elements.
  • antenna system 202 may additionally include analog antenna combination and/or beamforming circuitry.
  • RF transceiver(s) 108 may receive analog radio frequency signals from antenna system 106 and perform analog and digital RF front-end processing on the analog radio frequency signals to produce digital baseband samples (e.g., In-Phase/Quadrature (IQ) samples) to provide to communication processor 218.
  • digital baseband samples e.g., In-Phase/Quadrature (IQ) samples
  • RF transceiver(s) 108 may include analog and digital reception components including amplifiers (e.g., Low Noise Amplifiers (LNAs)), filters, RF demodulators (e.g., RF IQ demodulators)), and analog-to-digital converters (ADCs), which RF transceiver(s) 108 may utilize to convert the received radio frequency signals to digital baseband samples.
  • LNAs Low Noise Amplifiers
  • ADCs analog-to-digital converters
  • RF transceiver(s) 108 may receive digital baseband samples from communication processor 218 and perform analog and digital RF front-end processing on the digital baseband samples to produce analog radio frequency signals to provide to antenna system 106 for wireless transmission.
  • RF transceiver(s) 108 may thus include analog and digital transmission components including amplifiers (e.g., Power Amplifiers (PAs), filters, RF modulators (e.g., RF IQ modulators), and digital-to-analog converters (DACs), which RF transceiver(s) 108 may utilize to mix the digital baseband samples received from communication processor 218 and produce the analog radio frequency signals for wireless transmission by antenna system 106.
  • communication processor 218 may control the radio transmission and reception of RF transceiver(s) 108, including specifying the transmit and receive radio frequencies for the operation of RF transceiver(s) 108.
  • the communication processor 218 includes a baseband modem configured to perform physical layer (PHY, Layer 1) transmission and reception processing to, in the transmit path, prepare outgoing transmit data provided by communication processor 218 for transmission via RF transceiver(s) 108, and, in the receive path, prepare incoming received data provided by RF transceiver(s) 108 for processing by communication processor 218.
  • the baseband modem may include a digital signal processor and/or a controller.
  • the digital signal processor may be configured to perform one or more of error detection, forward error correction encoding/decoding, channel coding and interleaving, channel modulation/demodulation, physical channel mapping, radio measurement and search, frequency and time synchronization, antenna diversity processing, power control, and weighting, rate matching/de-matching, retransmission processing, interference cancelation, and any other physical layer processing functions.
  • the digital signal processor may be structurally realized as hardware components (e.g., as one or more digitally-configured hardware circuits or FPGAs), software-defined components (e.g., one or more processors configured to execute program code defining arithmetic, control, and I/O instructions (e.g., software and/or firmware) stored in a non-transitory computer-readable storage medium), or as a combination of hardware and software components.
  • the digital signal processor may include one or more processors configured to retrieve and execute program code that defines control and processing logic for physical layer processing operations.
  • the digital signal processor may execute processing functions with software via the execution of executable instructions.
  • the digital signal processor may include one or more dedicated hardware circuits (e.g., ASICs, FPGAs, and other hardware) that are digitally configured to specific execute processing functions, where the one or more processors of digital signal processor may offload specific processing tasks to these dedicated hardware circuits, which are known as hardware accelerators.
  • exemplary hardware accelerators can include Fast Fourier Transform (FFT) circuits and encoder/decoder circuits.
  • FFT Fast Fourier Transform
  • the digital signal processor's processor and hardware accelerator components may be realized as a coupled integrated circuit in some aspects.
  • Vehicle 100 may be configured to operate according to one or more radio communication technologies.
  • the digital signal processor of the communication processor 218 may be responsible for lower-layer processing functions (e.g., Layer 1/PHY) of the radio communication technologies.
  • a controller of the communication processor 218 may be responsible for upper-layer protocol stack functions (e.g., Data Link Layer/Layer 2 and/or Network Layer/Layer 3).
  • the controller may thus be responsible for controlling the radio communication components of vehicle 100 (antenna system 106, RF transceiver(s) 108, position device 114, etc.) in accordance with the communication protocols of each supported radio communication technology, and accordingly may represent the Access Stratum and Non-Access Stratum (NAS) (also encompassing Layer 2 and Layer 3) of each supported radio communication technology.
  • NAS Access Stratum and Non-Access Stratum
  • the controller may be structurally embodied as a protocol processor configured to execute protocol stack software (retrieved from a controller memory) and subsequently control the radio communication components of vehicle 100 to transmit and receive communication signals in accordance with the corresponding protocol stack control logic defined in the protocol stack software.
  • the controller may include one or more processors configured to retrieve and execute program code that defines the upper-layer protocol stack logic for one or more radio communication technologies, which can include Data Link Layer/Layer 2 and Network Layer/Layer 3 functions.
  • the controller may be configured to perform both user-plane and control-plane functions to facilitate the transfer of application layer data to and from vehicle 100 according to the specific protocols of the supported radio communication technology.
  • User-plane functions can include header compression and encapsulation, security, error checking and correction, channel multiplexing, scheduling, and priority, while control-plane functions may include setup and maintenance of radio bearers.
  • the program code retrieved and executed by the controller of communication processor 218 may include executable instructions that define the logic of such functions.
  • vehicle 100 may be configured to transmit and receive data according to multiple radio communication technologies.
  • one or more of antenna system 106, RF transceiver(s) 108, and communication processor 218 may include separate components or instances dedicated to different radio communication technologies and/or unified components that are shared between different radio communication technologies.
  • multiple controllers of communication processor 218 may be configured to execute multiple protocol stacks, each dedicated to a different radio communication technology and either at the same processor or different processors.
  • multiple digital signal processors of communication processor 218 may include separate processors and/or hardware accelerators that are dedicated to different respective radio communication technologies, and/or one or more processors and/or hardware accelerators that are shared between multiple radio communication technologies.
  • RF transceiver(s) 108 may include separate RF circuitry sections dedicated to different respective radio communication technologies, and/or RF circuitry sections shared between multiple radio communication technologies.
  • antenna system 106 may include separate antennas dedicated to different respective radio communication technologies, and/or antennas shared between multiple radio communication technologies. Accordingly, antenna system 106, RF transceiver(s) 108, and communication processor 218 can encompass separate and/or shared components dedicated to multiple radio communication technologies.
  • Communication processor 218 may be configured to implement one or more vehicle-to-everything (V2X) communication protocols, which may include vehicle-to-vehicle (V2V), vehicle-to-infrastructure (V2I), vehicle-to-network (V2N), vehicle-to-pedestrian (V2P), vehicle-to-device (V2D), vehicle-to-grid (V2G), and other protocols.
  • V2X vehicle-to-everything
  • Communication processor 218 may be configured to transmit communications including communications (one-way or two-way) between the vehicle 100 and one or more other (target) vehicles in an environment of the vehicle 100 (e.g., to facilitate coordination of navigation of the vehicle 100 in view of or together with other (target) vehicles in the environment of the vehicle 100), or even a broadcast transmission to unspecified recipients in a vicinity of the transmitting vehicle 100.
  • communications including communications (one-way or two-way) between the vehicle 100 and one or more other (target) vehicles in an environment of the vehicle 100 (e.g., to facilitate coordination of navigation of the vehicle 100 in view of or together with other (target) vehicles in the environment of the vehicle 100), or even a broadcast transmission to unspecified recipients in a vicinity of the transmitting vehicle 100.
  • Communication processor 218 may be configured to operate via a first RF transceiver of the one or more RF transceivers(s) 108 according to different desired radio communication protocols or standards.
  • communication processor 218 may be configured according to a Short-Range mobile radio communication standard such as, e.g., Bluetooth, Zigbee, and the like first RF transceiver may correspond to the corresponding Short-Range mobile radio communication standard.
  • communication processor 218 may be configured to operate via a second RF transceiver of the one or more RF transceivers(s) 108 in accordance with a Medium or Wide Range mobile radio communication standard such as, e.g., a 3G (e.g., Universal Mobile Telecommunications System - UMTS), a 4G (e.g., Long Term Evolution - LTE), or a 5G mobile radio communication standard in accordance with corresponding 3GPP (3 rd Generation Partnership Project) standards.
  • 3G e.g., Universal Mobile Telecommunications System - UMTS
  • 4G e.g., Long Term Evolution - LTE
  • 5G mobile radio communication standard in accordance with corresponding 3GPP (3 rd Generation Partnership Project) standards.
  • communication processor 218 may be configured to operate via a third RF transceiver of the one or more RF transceivers(s) 108 in accordance with a Wireless Local Area Network communication protocol or standard such as, e.g., in accordance with IEEE 802.11 (e.g., 802.11, 802.11a, 802.11b, 802.11g, 802.11n, 802.11p, 802.11-12, 802.11ac, 802.11ad, 802.11ah, and the like).
  • the one or more RF transceiver(s) 108 may be configured to transmit signals via antenna system 106 over an air interface.
  • the RF transceivers 108 may each have a corresponding antenna element of antenna system 106, or may share an antenna element of the antenna system 106.
  • Memory 214 may embody a memory component of vehicle 100, such as a hard drive or another such permanent memory device.
  • vehicle 100 such as a hard drive or another such permanent memory device.
  • the various other components of vehicle 100 e.g., one or more processors 102, are shown in FIGS. 1 and 2 may additionally each include integrated permanent and non-permanent memory components, such as for storing software program code, buffering data, etc.
  • the antenna system 106 may include a single antenna or multiple antennas. In some aspects, each of the one or more antennas of antenna system 106 may be placed at a plurality of locations on the vehicle 100 in order to ensure maximum RF coverage.
  • the antennas may include a phased antenna array, a switch-beam antenna array with multiple antenna elements, etc.
  • Antenna system 106 may be configured to operate according to analog and/or digital beamforming schemes in order to maximize signal gains and/or provide levels of information privacy.
  • Antenna system 106 may include separate antennas dedicated to different respective radio communication technologies, and/or antennas shared between multiple radio communication technologies. While shown as a single element in FIG.
  • antenna system 106 may include a plurality of antenna elements (e.g., antenna arrays) positioned at different locations on vehicle 100.
  • the placement of the plurality of antenna elements may be strategically chosen in order to ensure a desired degree of RF coverage.
  • additional antennas may be placed at the front, back, corner(s), and/or on the side(s) of the vehicle 100.
  • Data acquisition devices 112 may include any number of data acquisition devices and components depending on the requirements of a particular application. This may include: image acquisition devices, proximity detectors, acoustic sensors, infrared sensors, piezoelectric sensors, etc., for providing data about the vehicle's environment.
  • Image acquisition devices may include cameras (e.g., standard cameras, digital cameras, video cameras, single-lens reflex cameras, infrared cameras, stereo cameras, etc.), charge coupling devices (CCDs) or any type of image sensor.
  • Proximity detectors may include radar sensors, light detection and ranging (LIDAR) sensors, mmWave radar sensors, etc.
  • Acoustic sensors may include: microphones, sonar sensors, ultrasonic sensors, etc.
  • each of the data acquisition units may be configured to observe a particular type of data of the vehicle's 100 environment and forward the data to the data acquisition processor 214 in order to provide the vehicle with an accurate portrayal of the vehicle's environment.
  • the data acquisition devices 112 may be configured to implement preprocessed sensor data, such as radar target lists or LIDAR target lists, in conjunction with acquired data.
  • Measurement devices 116 may include other devices for measuring vehicle-state parameters, such as a velocity sensor (e.g., a speedometer) for measuring a velocity of the vehicle 100, one or more accelerometers (either single-axis or multi-axis) for measuring accelerations of the vehicle 100 along one or more axes, a gyroscope for measuring orientation and/or angular velocity, odometers, altimeters, thermometers, etc. It is appreciated that vehicle 100 may have different measurement devices 116 depending on the type of vehicle it is, e.g., car vs. drone vs. boat.
  • Position devices 114 may include components for determining a position of the vehicle 100. For example, this may include a global position system (GPS) or other global navigation satellite system (GNSS) circuitry configured to receive signals from a satellite system and determine the vehicle 100. Position devices 114, accordingly, may provide vehicle 100 with satellite navigation features.
  • GPS global position system
  • GNSS global navigation satellite system
  • the one or more memories 104 may store data, e.g., in a database or in any different format, that may correspond to a map.
  • the map may indicate a location of known landmarks, roads, paths, network infrastructure elements, or other elements of the vehicle's 100 environment.
  • the one or more processors 102 may process sensory information (such as images, radar signals, depth information from LIDAR, or stereo processing of two or more images) of the environment of the vehicle 100 together with position information, such as a GPS coordinate, a vehicle's ego-motion, etc., to determine a current location of the vehicle 100 relative to the known landmarks, and refine the determination of the vehicle's location. Certain aspects of this technology may be included in a localization technology, such as a mapping and routing model.
  • the map database (DB) 204 may include any type of database storing (digital) map data for the vehicle 100, e.g., for the control system 200.
  • the map database 204 may include data relating to the position, in a reference coordinate system, of various items, including roads, water features, geographic features, businesses, points of interest, restaurants, gas stations, etc.
  • the map database 204 may store the locations of such items and descriptors relating to those items, including, for example, names associated with any of the stored features.
  • a processor of the one or more processors 102 may download information from the map database 204 over a wired or wireless data connection to a communication network (e.g., over a cellular network and/or the Internet, etc.).
  • the map database 204 may store a sparse data model including polynomial representations of certain road features (e.g., lane markings) or target trajectories for the vehicle 100.
  • the map database 204 may also include stored representations of various recognized landmarks that may be provided to determine or update a known position of the vehicle 100 with respect to a target trajectory.
  • the landmark representations may include data fields such as landmark type, landmark location, among other potential identifiers.
  • control system 200 may include a driving model, e.g., implemented in an advanced driving assistance system (ADAS) and/or a driving assistance and automated driving system.
  • ADAS advanced driving assistance system
  • control system 200 may include (e.g., as part of the driving model) a computer implementation of a formal model such as a safety driving model.
  • a safety driving model or driving model may be or include a mathematical model formalizing an interpretation of applicable laws, standards, policies, etc. that are applicable to self-driving vehicles.
  • a safety driving model may be designed to achieve, e.g., three goals: first, the interpretation of the law should be sound in the sense that it complies with how humans interpret the law; second, the interpretation should lead to a useful driving policy, meaning it will lead to an agile driving policy rather than an overly-defensive driving which inevitably would confuse other human drivers and will block traffic and in turn limit the scalability of system deployment; and third, the interpretation should be efficiently verifiable in the sense that it can be rigorously proven that the self-driving (autonomous) vehicle correctly implements the interpretation of the law.
  • a safety driving model may be or include a mathematical model for safety assurance that enables identification and performance of proper responses to dangerous situations such that self-perpetuated accidents can be avoided.
  • the vehicle 100 may include the control system 200 and described with reference to FIG. 2 .
  • the vehicle 100 may include the one or more processors 102 integrated with or separate from an engine control unit (ECU), which may be included in the mobility system 120 of the vehicle 100.
  • the control system 200 may, in general, generate data to control or assist to control the ECU and/or other components of the vehicle 100 to directly or indirectly control the movement of the vehicle 100 via mobility system 120.
  • the one or more processors 102 of the vehicle 100 may be configured to implement the aspects and methods described herein.
  • FIGS. 1 and 2 may be operatively connected to one another via any appropriate interfaces. Furthermore, it is appreciated that not all the connections between the components are explicitly shown, and other interfaces between components may be covered within the scope of this disclosure.
  • FIG. 3 shows an exemplary network area 300 according to some aspects.
  • Network area 300 may include a plurality of vehicles 100, which may include, for example, drones and ground vehicles. Any one of these vehicles may communicate with one or more other vehicles 100 and/or with network infrastructure element (NIE) 310.
  • NIE 310 may be a base station (e.g., an eNodeB, a gNodeB, etc.), a road side unit (RSU), a road sign configured to wirelessly communicate with vehicles and/or a mobile radio communication network, etc., and serve as an interface between one or more of vehicles 100 and a mobile radio communications network, e.g., an LTE network or a 5G network.
  • a base station e.g., an eNodeB, a gNodeB, etc.
  • RSU road side unit
  • a road sign configured to wirelessly communicate with vehicles and/or a mobile radio communication network, etc., and serve as an interface between one or more of vehicles 100 and a
  • NIE 310 may include, among other components, at least one of an antenna system 312, an RF transceiver 314, and a baseband circuit 316 with appropriate interfaces between each of them.
  • NIE 310 may transmit and receive wireless signals via antenna system 312, which may be an antenna array including multiple antenna arrays.
  • Antenna system 312 may include multiple antenna elements (e.g., multiple antenna arrays) in order to employ multiple-input and multiple-output (MIMO) methods and schemes.
  • MIMO multiple-input and multiple-output
  • RF transceiver 314 may perform transmit and receive RF processing to convert outgoing baseband samples from baseband circuit 316 into analog radio signals to provide to antenna system 312 for radio transmission and to convert incoming analog radio signals received from antenna system 312 into baseband samples to provide to baseband circuit 316. Accordingly, RF transceiver 314 may be configured to operate similarly to the RF transceiver(s) described in FIGS. 1 and 2 , albeit perhaps on a much larger scale (e.g., amplifiers to transmit higher power signals, etc.).
  • Baseband circuit 316 may include a controller 310 and a physical layer processor 318 which may be configured to perform transmit and receive PHY processing on baseband samples received from RF transceiver 314 to provide to a controller 310 and on baseband samples received from controller 310 to provide to RF transceiver 314.
  • the baseband modem 316 may be located external to the NIE 310, e.g., at a centralized location of a mobile radio communication network.
  • Controller 310 may control the communication functionality of NIE 310 according to the corresponding radio communication technology protocols, which may include exercising control over antenna system 312, RF transceiver 314, and physical layer processor 318.
  • Each of RF transceiver 314, physical layer processor 318, and controller 310 may be structurally realized with hardware (e.g., with one or more digitally-configured hardware circuits or FPGAs), as software (e.g., as one or more processors executing program code defining arithmetic, control, and I/O instructions stored in a non-transitory computer-readable storage medium), or as a mixed combination of hardware and software.
  • NIE 310 may also include an interface 320 for communicating with (e.g., receiving instructions from, providing data to, etc.) with a core network according to some aspects.
  • NIE 310 may include a memory 330, which may be internal to NIE 310 (as shown in FIG. 3 ) or external to NIE 310 (not shown).
  • Memory 330 may store one or more maps of the coverage area of NIE 310 among other types of information.
  • Each of the one or more maps may include a static layer depicting environmental elements that remain largely unchanged over longer periods of time (e.g., roads, structures, trees, etc.) and/or a dynamic layer with more frequent changes (e.g., vehicles, detected obstacles, construction, etc.).
  • memory 330 may also store maps corresponding to one or more neighboring areas of NIE 310 so as to provide vehicles within its coverage area with information of neighboring coverage areas (e.g., to facilitate the process when a vehicle moves to the coverage of the neighboring NIE).
  • FIG. 4 is a diagram that shows various components related to driver monitoring. Some of the features or components may be implemented or integrated into a vehicle 405. The components illustrated in FIG. 4 may be operatively connected to one another via any appropriate interfaces. Furthermore, it is appreciated that not all the connections between the components are explicitly shown, and other interfaces between components may be covered within the scope of this disclosure.
  • Vehicle 405 may be any suitable type of vehicle described herein, e.g., vehicle 100 described in connection with FIG. 1 .
  • the vehicle 405 can include automated driving systems (ADS) 410 or, in other cases, may be or include an advanced driving assistance system (ADAS).
  • the ADS 410 may include a control system (not shown), e.g., the control system 200 described in connection with FIG. 2 .
  • the vehicle 405 (e.g., through a control system) may be configured to operate at one or more different levels of driving automation.
  • Table 500 of FIG. 5 describes various known automated driving levels. According to various aspects of the disclosure, the vehicle 405 may operate at L3, L4, and/or L5 automation level.
  • the ADS component 410 is responsible for determining vehicle data, including environmental perception data.
  • the ADS 410 can be configured to evaluate the environment surrounding the vehicle 405 to produce environmental perception data.
  • the environmental perception data may include a risk assessment or data indicating a safety risk concerning (perceived) features or elements external to the vehicle 405.
  • the risk assessment may indicate a collision risk involving one or more perceived or detected elements in the vehicle's environment or vicinity 405.
  • the ADS 410 may generate the vehicle data (e.g., environmental perception data) from sensor data obtained from one or more sensors, e.g., sensors of the vehicle 405 or other external sources.
  • vehicle data e.g., environmental perception data
  • the detected or perceived elements can include, for example, vehicles, pedestrians, bicyclists, animals, road obstructions, or any other type of road actor.
  • the vehicle 405 may further include an Operational Design Domain monitor 415 and a Driver Monitoring System (DMS) 420.
  • the Operational Design Domain (ODD) of the vehicle 405 may be the operating conditions under which a given driving system/vehicle is designed explicitly to or properly function or operate.
  • the ODD monitor 415 may determine ODD compliance for the vehicle 405. For example, the ODD monitor 415 can determine whether the vehicle 405 is operating with the proper operating conditions for a current operation mode. If the ODD monitor determines that vehicle 405 is not operating under the proper conditions, then the vehicle 405 is determined to be out of the ODD and not in ODD compliance. For example, the ODD monitor 415 may determine or detect when the vehicle 405 is not in ODD compliance in response to determining that a safety risk to vehicle 405 exceeds a threshold.
  • the ODD monitor 415 can inform the ADS 410 (e.g., send a data signal) to indicate the ODD compliance status of the vehicle 405.
  • the ADS 410 can be configured in response to take one or more actions to resolve the ODD non-compliance.
  • the ODD monitor 415 can inform or provide the ODD compliance assessment to the ADS 410, which may take one or more actions in response.
  • the ADS 410 may take one or more actions such as a handover in which automated control of the vehicle by the control system of the vehicle 405 transitions to driver/manual control of the vehicle.
  • the ADS 410 may take one more actions even when the ODD compliance is reached, but a determined safety risk of the vehicle is too high.
  • the ADS may initiate a "reverse transition".
  • the ADS may take control from the driver or prevent the driver from controlling the vehicle, e.g., the ADS may maintain automated control.
  • the driving monitoring system or DMS 420 may be a component of the vehicle 405 that monitors the driver 412 of the vehicle 405.
  • the DMS 420 may monitor and interpret driver data (e.g., driver feedback). Further, the DMS 420 can generate or produce driver perception data based on the monitoring and interpretation of driver data.
  • the DMS 420 can ascertain an awareness/attention status or level of the driver 412.
  • the DMS 420 may obtain and analyze and interpret sensor data, e.g., image, video, audio, concerning the driver to determine the awareness/attention status.
  • the DMS 420 can determine how attentive the driver 412 in one or more situations or contexts. This awareness or attention level may be in the form of a probabilistic risk assessment.
  • Interpreting driver feedback includes the DMS 420 configured to interpret signals from the driver 412, e.g., as safety-enhancing feedback.
  • the signals may be an audio and/or visual signal.
  • the driver can be in the form of proactive feedback.
  • the DMS 420 interprets driver feedback (e.g., signals) to determine or estimate information regarding one or more objects or elements in the vehicle's environment or vicinity.
  • the driver 412 can provide one or more signals, that if correctly interpreted, indicate the existence (or potential existence) of one or more objects in the vehicle's vicinity.
  • the feedback information may indicate a level of risk or threat regarding such an object or objects.
  • the driver perception data can indicate and be used to indicate or infer a safety risk (e.g., collision risk) for one or more elements in the vehicle's vicinity or environment.
  • the DMS 420 can provide information that can be used directly or indirectly by the ODD monitor 415 for assessing ODD compliance.
  • the vehicle 405 of FIG. 4 includes a risk estimator 425, a component configured to make or produce risk assessments for the vehicle 405 regarding current or (immediate) upcoming situations/scenarios.
  • the risk assessment or risk assessment data can be sent to and used by the ODD monitor 415 to determine ODD compliance, e.g., the ODD compliance concerning a current driving or automation mode of the vehicle 405 for a current or upcoming scenario.
  • the DMS 420 provides driver data or driver perception data to the risk estimator 425.
  • the risk estimator 425 can be a component for evaluating scenarios involving the vehicle 405, e.g., situations regarding the vehicle 405 and the vehicle's surrounding environment to generate or produce a risk assessment.
  • the risk estimator 425 can generate a risk assessment that includes data indicating the vehicle's risk, e.g., a risk of collision.
  • the risk estimator 425 may generate or provide a risk assessment by assimilating or integrating information from different sources, e.g., using driver perception data and environmental perception data.
  • the environmental perception data (which can be obtained and/or determined from sensor data of the vehicle's external environment) can include data indicating a safety risk (e.g., collision risk) for the vehicle 405.
  • the safety risk may be specified with respect to one or more perceived elements or objects in the vehicle's vicinity.
  • the risk estimator 425 can determine a combined or integrated risk assessment regarding the vehicle.
  • the risk estimator 425 can integrate the driver feedback or the driver perception data with the ADS risk assessment (from the vehicle or environmental perception data). The result produces an integration of risk assessment from the driver perception data and the environmental perception data.
  • the integrated risk assessment or risk assessment may be a probabilistic estimation.
  • the risk estimator 425 determines the existence or likelihood of elements in the vehicle's environment and one or more (potential) situations/scenarios involving the vehicle 405. Further, the risk estimator 425 may determine a safety risk such as a collision risk between the vehicle 405 and such elements.
  • the ODD monitor can use the integrated risk assessment to determine whether the vehicle is currently in or out of the vehicle's ODD.
  • the ODD monitor 415 may evaluate ODD compliance by comparing the determined integrated risk assessment with one or more risk thresholds.
  • the particular threshold(s) used for comparison may be selected based on the driver perception data, indicating the driver's attentional awareness.
  • the risk estimator 420 can provide the risk assessment to the ADS 410.
  • the ADS 410 may take one or more actions, even if the ODD assessment provided by the ODD monitor 415 does not indicate out of the ODD. For example, if the integrated risk determined by the risk estimator 425 is higher than the level of risk determined by the ADS 410, then the ADS may initiate action(s) to change the driving behavior even if the vehicle is in ODD compliance. That is, automated control of the vehicle 405 may be maintained, but the driving behavior may be altered by ADS 410, e.g., the ADS 410 may cause a change in driving model parameters to drive with more caution or safety due to the risk assessment provided by the risk estimator 425.
  • the vehicle 405 may systematically collect ODD compliance assessments.
  • ODD compliance assessments can be collected and stored, for example, in a database or other storage 435 that may be part of the vehicle, or in other cases, may be a remote database.
  • the ODD compliance assessments may be stored along with the corresponding scenarios, labelling (e.g., see below with respect to FIG. 6 ) which may be obtained from the vehicle data and the driver perception data.
  • labelling e.g., see below with respect to FIG. 6
  • not every ODD compliance assessment may be stored.
  • a subset of the ODD compliance assessments e.g., corner cases such as those in which the driver perception data and the vehicle data are discordant, may be stored.
  • a ground-truth label e.g., interpreted driver feedback
  • a snapshot e.g., sensor image data
  • a driver will be active mostly in critical situations, such data will contain an above-average proportion of corner cases, e.g., challenging cases for the DMS and ADS in the form of high perception uncertainty perception errors (misclassifications, missed objects).
  • Detected corner cases and associated ground-truth labels can be collected systematically and are forwarded by the ODD monitor to a database, e.g., database 435.
  • each pair of corner case and ground-truth data may be later verified by an operator (e.g., certified operator) first before it is shared or used by other users. That is, the data may be updated with the verification information by certified or legitimate operators.
  • This verification may be in the form or similar to labeling tasks during any dataset generation, and no significant training is necessary for such an operator.
  • a database of corner cases can be of great value for other ADS perception systems because critical situations do not frequently occur during normal operations.
  • Traditional dataset collection techniques have a notorious lack of such corner cases, which makes, for example, neural networks trained with such a database can provide informed results, e.g., regarding risk, ODD compliance, and overall provide superior to results for driver systems compared to others.
  • the database information may be used for a fleet with the information being transmitted or downloaded to one more vehicles for use in their driving and perception system.
  • the ODD compliance assessment may fall into certain categories as determined by the ODD monitor 415 and its compliance assessment.
  • the ODD compliance assessments may be labeled, e.g., by the ODD monitor 415 or another suitable component according to its category.
  • the ODD compliance assessments may be annotated or labeled "exit", "save", or "support”.
  • FIG. 6 shows an exemplary diagram illustrating the ODD compliance assessment using driver perception data (e.g., proactive driver signal(s)) and the ADS risk assessment.
  • driver feedback 610 which may be in the form of a proactive signal, may be consistent with the ADS data at 620 to produce an ODD assessment at 630.
  • exit may indicate situations where the perception data of the vehicle system and the driver feedback disagree, and this discrepancy suggests that the current ODD compliance assessment is incorrect and needs to be changed to maintain safety. For example, such situations appear if the driver points out elements of high estimated risk that the vehicle perception system has missed and cannot handle. In that case, an exit from the current ODD to another, safe ODD is being forced.
  • the label "support” may be used for ODD assessments where the driver perception data agrees or supports the environmental perception data of the vehicle data agrees with.
  • the “support” label means the risk estimation determines that the vehicle is either in or out of the defined ODD, and the driver perception data (e.g., driver feedback) supports the current ODD compliance determination.
  • the "save" label may be used to indicate situations in which the environmental perception data of the vehicle data disagrees with the driver perception data but does not alter or appreciably alter the relevant safety risk, e.g., safety risk and the vehicle remains in ODD compliance.
  • the risk indicated in the driver perception data is within a specific range of the risk indicated by the environmental perception data.
  • a driver can be proactive and provide feedback and notification regarding possible or pending hazards or indicate their attention status during automated vehicle control. Therefore, the vehicle's mechanism can confirm (ODD support) or correct (ODD save, exit) the ODD assessment and thus reduce the critical time for performing a handover operation or avoiding a handover operation altogether.
  • the safety analysis or risk estimation in aspects of the present disclosure relates to assimilating information from the inside (driver perception) and outside (vehicle environment perception).
  • a driver 412 can proactively signal his attention status.
  • the DMS 420 e.g., through sensors connected (e.g., through an interface), can interpret and determine attention status or awareness level from the driver 412. This interpreted information can be passed on to the risk estimator module 425 for risk estimation.
  • the risk estimator 425 can receive the sensing input (e.g., risk estimation of the vehicle's environment) from the ADS 410.
  • the risk estimator 425 can combine or integrate both types of data to determine or calculate a quantitative risk, or a combined risk.
  • the combined risk determined by the risk estimator 425 may be determined based on the consistency between the types of data, e.g., the vehicle data and the driver perception data.
  • the risk estimator 425 may determine a combined risk from the vehicle data and the driver perception data based on a consistency between risk indicated, inferred, and/or determined from the vehicle data and risk indicated, inferred, and/or determined from the driver perception data.
  • the combined risk may be produced regarding or be associated with imminent actions regarding features or elements external to the vehicle 405.
  • the vehicle data's risk estimate will not affect or appreciably affect the ADS risk estimate (indicated in the vehicle data) up to an uncertainty correction.
  • determining the integrated risk estimate may include incorporating extra precautions to ensure safety. For example, if the proactive driver signals attention but the DMS recognizes unawareness, the combined estimate will output a high risk. Further, ins such instances, the ADS 410 may provide a driver alert depending on the current environment and the determined risk.
  • the risk estimator 425 may use one or more awareness or attention thresholds and compare the determined awareness level from the DMS 420 with one or more awareness/attention thresholds.
  • the surpassing of the one or more attention/awareness threshold level can be used to calculate the integrated risk. For example, greater levels of detected driver awareness can produce a smaller integrated risk assessment. In comparison, low or lower driver awareness levels can produce a more significant integrated risk assessment by the risk estimator 425.
  • the driver perception data may be interpreted when the driver 412 proactively signals or refers to the vehicle's external environment. That is, the DMS 420 may interpret the driver perception data (e.g., from sensors) and determine a driver's indication regarding an element or object in the vehicle's vicinity.
  • a fused environment may be created, e.g., by the risk estimator 425. All elements detected by both data sources (e.g., vehicle data from ADS 410 and the driver perception data from the DMS 420) can be aligned temporally and spatially in a common coordinate system, using standard sensor fusion techniques.
  • the fused environment can be a temporal and/or spatial representation of the environment surrounding the vehicle, including one or more elements in the vehicle's environment.
  • the generated or created fused environment can contain the same number or more elements than each of the individual sources. Any additional elements can be correct observations enhancing the overall perception completeness or false positives.
  • the fused environment can include a safety risk for each of the elements or objects detected.
  • the risk estimator 425 to ensure safety, can account for all observations from both sources, and the ADS 410 can then evaluate the resulting combined/integrated risk for the planned driving strategy. For example, if the data sources are inconsistent or discordant (e.g., observations by driver and ADS conflict or the risks indicated from the sources are in conflict), then the ADS 410 may be configured to choose a more cautious option.
  • This situation may occur when a possible object in the vehicle's vicinity is seen or detected by one of the sources but is not seen or detected by the other source. In such a case, observation of the object or element is added to the combined environment with a safety risk (e.g., collision risk) determined for the object/element.
  • a safety risk e.g., collision risk
  • an ODD exit can be triggered, or the respective driving task can be adapted if an ODD save with reduced risk is possible.
  • the ADS 410 can adopt a driving strategy that results in safer driving behavior to avoid accidents.
  • sensor data captured of the driver e.g., sensor data of the driver's feedback
  • the DMS 420 and/or any other suitable component may be configured to interpret driver feedback.
  • the DMS 420 (or another component), for example, may include logic (interpretation logic) for understanding or interpreting the driver data captured from one or more sensors of the vehicle.
  • the interpretation logic may, in some instances, rely on a hierarchical signal and interpretation structure.
  • the signals may be a combination of audiovisual signals. Since the signals are audiovisual, e.g., they are any suitable combination of audio and visual signals, this reduces the incidence of false positives and provides enhanced robustness.
  • Any suitable signal detection techniques known in the art may be used, including, for example, speech detection, physical feedback (e.g., from buttons such as on steering or console), gaze estimation or head pose estimation of the driver, hand gesture recognition (e.g., swipe left/right, circle clockwise/counterclockwise actions), finger gestures (e.g., any of one, two, & three fingers up), pointing gestures (e.g., including with full arm), and the like.
  • the interpretation logic can include or use a mapping of signals to communicated information can be implemented in various ways.
  • Table 700 of FIG. 7 shows one exemplary type of mapping; however, other implementations and variants can be realized and used.
  • a neural network or machine learning logic trained for driver signal interpretation may be used.
  • the interpretation logic may be trained for application-specific audiovisual signals and can be configured to determine or calculate a probabilistic risk assessment of the driver's monitoring ability and compute feedback of the driver's risk assessment ((e.g., driver risk assessment regarding possible elements in vehicle's environment).
  • in-vehicle for the driver may include one or multiple of the following: RGB cameras, Infrared (IR) cameras, IR LED, Time-of-flight (ToF) camera, dynamic vision sensor (event camera), Structured Light at diverse wavelengths, microphones (e.g., placed in the driver's cabin for audio input and output), physical buttons (e.g., on the steering wheel), interactive displays or other interfaces.
  • RGB cameras Infrared (IR) cameras
  • IR LED Time-of-flight (ToF) camera
  • ToF Time-of-flight
  • vent camera dynamic vision sensor
  • Structured Light at diverse wavelengths e.g., microphones (e.g., placed in the driver's cabin for audio input and output), physical buttons (e.g., on the steering wheel), interactive displays or other interfaces.
  • FIG. 8 shows an exemplary method 800 that may be performed in accordance with aspects of the present disclosure.
  • the method 800 may be performed by one or more components of a vehicle.
  • the vehicles may be ones that support or include autonomous-type control (e.g., L3).
  • the method may be embodied as instructions contained (non-transitory) computer-readable medium with the method being performed by one or more processors executing the instructions.
  • he method 800 includes at 805, obtaining vehicle data comprising environmental perception data indicating a risk assessment regarding one or more perceived elements of an environment surrounding a vehicle.
  • the method 800 includes obtaining driver perception data regarding a driver inside the vehicle.
  • The further includes at 815 determining an integrated risk assessment based on the vehicle data and the driver perception data.
  • the method includes determining an Operational Design Doman (ODD) compliance assessment of the vehicle at least based on the determined integrated risk assessment.
  • ODD Operational Design Doman
  • the data used for ODD assessment may include uncertainties. That is, uncertainties in the perception can arise due to imperfections or noise on sensor information and algorithms. Uncertainties may take the form of probability of existence, which expresses how likely it is that an object that has been detected is a real object. This uncertainty can be expressed as a probabilistic value of existence. Further, uncertainties may take the form of properties of an object/element, e.g., an exact velocity or position. This uncertainty is usually expressed by a distribution (e.g., Gaussian). The vehicles or components described herein may be configured to process and make decisions with such types of uncertainties.
  • uncertainties in the perception can arise due to imperfections or noise on sensor information and algorithms. Uncertainties may take the form of probability of existence, which expresses how likely it is that an object that has been detected is a real object. This uncertainty can be expressed as a probabilistic value of existence. Further, uncertainties may take the form of properties of an object/element, e.g., an exact velocity or position. This uncertainty is usually expressed by a distribution (e.g.
  • a vehicle has some threshold that influences whether or how information is used.
  • a probability value could be used. If the object probability is higher than the threshold, then the object is considered existing; otherwise, no object is considered.
  • a threshold could be, e.g., defined by a multiple sigma quantile of the distribution.
  • components of the vehicles described herein may calculate a risk based on the given information and uncertainties.
  • risk may be defined as the "probability of something happening multiplied by the resulting cost or benefit it does". Any suitable or appropriate risk estimation methods known in the art may be used. If collision severity is used for risk, a simple inelastic collision model can be applied. With a risk calculation, it is possible to calculate for each given object in the environment a probabilistic risk value, referring to harmful collision with this object, given its probability of existence and a distribution of the information. However, for the vehicle's decision-making, the use of a threshold or thresholds may be used or required in which the threshold(s) defines the acceptable risk.
  • a device corresponding to a method detailed herein may include one or more components configured to perform each aspect of the related method.

Landscapes

  • Engineering & Computer Science (AREA)
  • Transportation (AREA)
  • Mechanical Engineering (AREA)
  • Automation & Control Theory (AREA)
  • Physics & Mathematics (AREA)
  • Human Computer Interaction (AREA)
  • Mathematical Physics (AREA)
  • Chemical & Material Sciences (AREA)
  • Combustion & Propulsion (AREA)
  • General Physics & Mathematics (AREA)
  • Multimedia (AREA)
  • Theoretical Computer Science (AREA)
  • Traffic Control Systems (AREA)
EP21195534.9A 2020-12-18 2021-09-08 Systèmes, dispositifs et procédés impliquant des systèmes de commande Pending EP4015336A1 (fr)

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
US17/126,089 US20220194385A1 (en) 2020-12-18 2020-12-18 Systems, devices, and methods involving driving systems

Publications (1)

Publication Number Publication Date
EP4015336A1 true EP4015336A1 (fr) 2022-06-22

Family

ID=77666376

Family Applications (1)

Application Number Title Priority Date Filing Date
EP21195534.9A Pending EP4015336A1 (fr) 2020-12-18 2021-09-08 Systèmes, dispositifs et procédés impliquant des systèmes de commande

Country Status (3)

Country Link
US (1) US20220194385A1 (fr)
EP (1) EP4015336A1 (fr)
CN (1) CN114715164A (fr)

Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20070131468A1 (en) * 2003-08-23 2007-06-14 Daimlerchrysler Ag Motor vehicle provided with a pre-safe system
US20110169625A1 (en) * 2010-01-14 2011-07-14 Toyota Motor Engineering & Manufacturing North America, Inc. Combining driver and environment sensing for vehicular safety systems
DE102012112802A1 (de) * 2012-12-20 2014-06-26 Conti Temic Microelectronic Gmbh Verfahren zur Steuerung eines Fahrzeugs mit einem ein automatisiertes, teilautomatisiertes und ein manuelles Fahren ermöglichenden Fahrerassistenzsystem
EP3109118A2 (fr) * 2015-06-24 2016-12-28 Audi Ag Procédé destiné au fonctionnement d'un système d'assistance du conducteur d'un véhicule automobile et véhicule automobile
EP3252432A1 (fr) * 2016-06-03 2017-12-06 Toyota Motor Sales, U.S.A., Inc. Système d'obtention d'informations basé sur la surveillance d'un occupant
US20180046851A1 (en) * 2016-08-15 2018-02-15 Apple Inc. Command processing using multimodal signal analysis

Family Cites Families (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2013074899A1 (fr) * 2011-11-16 2013-05-23 Flextronics Ap, Llc Dispositif d'affichage de tableau de bord configurable
US10620627B2 (en) * 2017-12-19 2020-04-14 PlusAI Corp Method and system for risk control in switching driving mode
US10762785B1 (en) * 2018-01-09 2020-09-01 State Farm Mutual Automobile Insurance Company Vehicle collision alert system and method
JP7151393B2 (ja) * 2018-11-08 2022-10-12 トヨタ自動車株式会社 車両情報提供装置、車両情報提供方法及びプログラム
BR112021020043A2 (pt) * 2019-04-12 2021-12-14 Stoneridge Electronics Ab Monitoramento de uso de dispositivo móvel para gestão da frota de veículos comerciais
US11150665B2 (en) * 2019-09-17 2021-10-19 Ha Q Tran Smart vehicle

Patent Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20070131468A1 (en) * 2003-08-23 2007-06-14 Daimlerchrysler Ag Motor vehicle provided with a pre-safe system
US20110169625A1 (en) * 2010-01-14 2011-07-14 Toyota Motor Engineering & Manufacturing North America, Inc. Combining driver and environment sensing for vehicular safety systems
DE102012112802A1 (de) * 2012-12-20 2014-06-26 Conti Temic Microelectronic Gmbh Verfahren zur Steuerung eines Fahrzeugs mit einem ein automatisiertes, teilautomatisiertes und ein manuelles Fahren ermöglichenden Fahrerassistenzsystem
EP3109118A2 (fr) * 2015-06-24 2016-12-28 Audi Ag Procédé destiné au fonctionnement d'un système d'assistance du conducteur d'un véhicule automobile et véhicule automobile
EP3252432A1 (fr) * 2016-06-03 2017-12-06 Toyota Motor Sales, U.S.A., Inc. Système d'obtention d'informations basé sur la surveillance d'un occupant
US20180046851A1 (en) * 2016-08-15 2018-02-15 Apple Inc. Command processing using multimodal signal analysis

Also Published As

Publication number Publication date
CN114715164A (zh) 2022-07-08
US20220194385A1 (en) 2022-06-23

Similar Documents

Publication Publication Date Title
EP3916623B1 (fr) Dispositifs et procédés d'identification précise d'objets dans l'environnement d'un véhicule
US11932249B2 (en) Methods and devices for triggering vehicular actions based on passenger actions
US11597393B2 (en) Systems, methods, and devices for driving control
US11886968B2 (en) Methods and devices for detecting objects and calculating a time to contact in autonomous driving systems
US12134379B2 (en) Systems, devices, and methods for predictive risk-aware driving
US11814052B2 (en) Safety system for a vehicle
US11308363B2 (en) Device and method for training an object detection model
EP3907722A1 (fr) Systèmes, dispositifs et procédés de conduite consciente des risques
EP4015335B1 (fr) Dispositif pour un véhicule
US20220348217A1 (en) Electronic apparatus for vehicles and operation method thereof
CN111373458B (zh) 用于自主车辆的轨迹规划的基于预测的系统和方法
US11815908B2 (en) Enhanced operational domain monitoring and visualization systems and methods
EP4019352A1 (fr) Système de sécurité pour un véhicule
US20210101620A1 (en) Systems, methods, and devices for generating and using safety threat maps
EP4015336A1 (fr) Systèmes, dispositifs et procédés impliquant des systèmes de commande
KR20190115435A (ko) 차량용 전자 장치 및 차량용 전자 장치의 동작 방법

Legal Events

Date Code Title Description
PUAI Public reference made under article 153(3) epc to a published international application that has entered the european phase

Free format text: ORIGINAL CODE: 0009012

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: THE APPLICATION HAS BEEN PUBLISHED

AK Designated contracting states

Kind code of ref document: A1

Designated state(s): AL AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MK MT NL NO PL PT RO RS SE SI SK SM TR

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: REQUEST FOR EXAMINATION WAS MADE

17P Request for examination filed

Effective date: 20221216

RBV Designated contracting states (corrected)

Designated state(s): AL AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MK MT NL NO PL PT RO RS SE SI SK SM TR

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: EXAMINATION IS IN PROGRESS

17Q First examination report despatched

Effective date: 20240131

GRAP Despatch of communication of intention to grant a patent

Free format text: ORIGINAL CODE: EPIDOSNIGR1

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: GRANT OF PATENT IS INTENDED

INTG Intention to grant announced

Effective date: 20240919

RAP3 Party data changed (applicant data changed or rights of an application transferred)

Owner name: INTEL CORPORATION

RIN1 Information on inventor provided before grant (corrected)

Inventor name: GONZALEZ AGUIRRE, DAVID ISRAEL

Inventor name: ALVAREZ, IGNACIO J.

Inventor name: PAULITSCH, MICHAEL

Inventor name: BUERKLE, CORNELIUS

Inventor name: OBORIL, FABIAN ISRAEL

Inventor name: ROSALES, RAFAEL

Inventor name: GEISSLER, FLORIAN