DE102019108075A1 - Method and control device for operating a sensor system of a device and a motor vehicle - Google Patents

Abstract

The invention relates to a method for operating a sensor system (13) of a device (10), wherein the sensor system (13) comprises at least one sensor unit (M, K, T, F) which has an adaptive signal filtering device (31) and where at the method by a control device (15) from at least one object data source object data (27) are received, based on the object data (27) it is recognized that there is a candidate object (18) under the at least one object (17), the sensor system (13) is adapted for a detection of the recognized candidate object (18), the recognized candidate object (18) is detected by means of the adapted sensor system (13) and detection data (14) relating to the detected candidate object (18) are generated as a result, depending on the detection data (14 ) it is recognized on the basis of a predetermined relevance criterion (20) whether there is a need for signaling with regard to the recognized candidate object (18), and if so If the signaling requirement is recognized, a notification signal (21) describing the recognized candidate object (18) is generated.

Description

The invention relates to a method for operating a sensor system in a device. The device can be, for example, a motor vehicle or a portable assistance device. An assistance device can, for example, be worn by a person in a danger area for their protection. The sensor system of the device has at least one sensor unit which has an adaptive signal filtering device.

When using a device, a person or a user can be supported in detecting objects in the vicinity of the device and taking these into account during use. In the case of a motor vehicle, this can mean that the motor vehicle monitors the surroundings for at least one object and, for example, in the event of a risk of collision with such an object, a protective measure against the collision is triggered. In the case of a portable assistance device, for example, a source of danger in the vicinity can be detected and the user of the assistance device can be made aware of this source of danger.

To detect at least one object in the vicinity of a device, the device can be equipped with a sensor system that provides at least one sensor unit for generating detection data. These detection data describe the at least one detected object in the vicinity. For the detection of an object in the environment by a sensor unit, processing of the raw sensor data of the respective sensor unit by means of a signal filtering device is usually provided. The processing of the raw sensor data provided for an object type corresponds to an object-type-specific filter characteristic. According to the filter characteristic, the signal filtering device highlights those signal components in the sensor data that are necessary or advantageous for the detection of a certain object type (for example for the detection of a person or a vehicle or a predetermined source of danger). However, if objects of different object types in the environment are to be detected by means of a sensor unit, the sensor data of the sensor unit must be processed accordingly with different filter characteristics in order to enable optimal or adapted detection for each object type. In order to be able to operate a signal filtering device of a sensor unit with a plurality of filter characteristics, however, corresponding computing resources are necessary, which makes the implementation of such a sensor system in a device technically complex.

A monitoring system which detects and maps objects of different object types in the surroundings of a motor vehicle by means of a sensor system is, for example, from the DE 10 2017 203 785 A1 known.

The invention is based on the object of detecting objects of different object types from a device by means of its sensor system and of reacting to the detected objects.

The object is achieved by the subjects of the independent claims. Advantageous embodiments of the invention are described by the dependent claims, the following description and the figures.

The invention provides a method for operating a sensor system of a device. In the manner described, the device can be, for example, a motor vehicle or a mobile terminal device (for example a smartphone or a tablet PC) or a portable assistance device (for example to protect a person in a danger area). The method assumes that the sensor system comprises at least one sensor unit which has an adaptive signal filtering device. The term “adaptive” means that a filter characteristic of the signal filtering device can be set. An example of such an adaptive signal filtering device is an adaptive directional microphone in which a detection direction for sound can be set or aligned. It is then a matter of spatial filtering of the recorded sound. By setting a specific filter characteristic of the signal filtering device, the sensor unit is able to detect a specific object type, for example an object in a specific relative direction with respect to the device, while an object of another object type is only detected suboptimally, for example an object in another Direction. By setting a filter characteristic in the signal filtering device, the detection of a specific object type can be promoted or improved.

A person skilled in the art can provide an object-type-specific filter characteristic for each object type.

The method according to the invention provides that the following steps are carried out by a control device of the device. From at least one object data source, object data describing at least one object or potentially arranged object arranged in the environment or in the device are received. In other words, the at least one object data source supplies information about which at least one object is or could potentially be in the vicinity or in the device (i.e. a device component). The object data do not have to describe the respective object exactly, but the object data can, for example, only be an indication that the respective object is located in the vicinity or in the device or is potentially located (for example with a predetermined minimum probability and / or due to a nature of the Surroundings). An object recognition device then uses the object data to recognize that there is a candidate object of a predetermined object type, that is to say with a predetermined object property, under the at least one object, as described or announced by the object data. It is thus recognized whether one of the at least one object is a candidate object because it has a certain object type. The object recognition device thus establishes or detects that at least one object is to be treated as a candidate object because it corresponds to a predetermined object type. In connection with the present invention, a candidate object is an object to which a user of the device is to be made aware and / or which is to be taken into account when the device is operated. As an object type it can be provided, for example, that the object is a person or a vehicle or a stationary object or a source of danger (for example a fire or a source of heat). The object recognition device thus interprets the object data to the effect that it is recognized whether a candidate object is located or is arranged in the vicinity.

The control device then adapts the sensor system of the device for a detection of the recognized candidate object, in that such a filter characteristic is selected from several possible filter characteristics in the respective signal filter device of the at least one sensor unit of the sensor system that is provided for the object type of the recognized candidate object. The selected filter characteristic is set accordingly in the respective signal filter device. In other words, the at least one sensor unit is sensitized or specialized or configured in the sensor system for sensing or detecting the candidate object by setting the filter characteristic provided for the corresponding object type of the candidate object in the signal filter device of the sensor unit.

In a next step, the recognized candidate object is then recorded using the adapted sensor system. In other words, the at least one sensor unit of the sensor system now generates sensor data that are processed or prepared by means of the signal filter device of the respective sensor unit in accordance with the set filter characteristic. As a result, the at least one sensor unit of the sensor system delivers corresponding acquisition data for the acquired candidate object (sensor data processed by means of the set filter characteristic), which is generated on the basis of the respectively set filter characteristic.

By means of an evaluation device, it is then checked or recognized, as a function of the detection data on the basis of a predetermined relevance criterion, whether there is a need for signaling with regard to the recognized candidate object. The candidate object is therefore a candidate for whether there is a need to react to this candidate object. This is checked by the evaluation unit by checking, on the basis of the detection data relating to the recognized candidate object, whether the candidate object is relevant for signaling, the relevance being able to be determined by a person skilled in the art using a relevance criterion. In other words, in the method, by specifying a relevance criterion, it can be determined when the control device determines a need for signaling. In the case of a recognized need for signaling, a notification signal describing the recognized candidate object is generated. Otherwise, if the relevance criterion is not met, no notification signal is generated.

As a specific example, it can be assumed that the device is a motor vehicle and object data relating to other road users are received from the object data source and represent objects in the vicinity of the motor vehicle. The object recognition device can then recognize, for example, that a pedestrian is located near the motor vehicle as a candidate object. Whether it is necessary to react to the pedestrian depends on its relevance for the motor vehicle, for example on the risk of a collision. For this, the behavior of the pedestrian must be recognized. In order to be able to evaluate or recognize the behavior of the pedestrian, the sensor system can be adapted to detect people by setting a corresponding filter characteristic in the at least one sensor unit, which can be specialized, for example, in sensor data for a shape and / or movement of a pedestrian to recognize or emphasize or highlight the at least one sensor unit. If the candidate object, i.e. the pedestrian, is then detected or sensed with this adaptive sensor system, the said evaluation device can then use the detection data generated in this way to detect whether the pedestrian fulfills a relevance criterion, which can for example say that the pedestrian is moving along a movement path, which will cross the planned path of movement of the motor vehicle that so there is a risk of collision. In this case, a need for signaling can then be recognized and a notification signal can be generated which, for example, warns the user of the device, that is to say here the driver of the motor vehicle, of the pedestrian.

The invention has the advantage that the at least one sensor unit in the sensor system does not always have to operate several different filter characteristics in parallel or keep them activated, but that each sensor unit is only operated with an adapted or suitable filter characteristic for a currently required object type when required. Computing resources can thus be saved and / or real-time capability of the sensor system can be guaranteed.

It is particularly preferred in the method according to the invention to detect a direction of incidence of a sound by means of acoustic localization. This can be done by evaluating a phase offset between the microphone signals of two microphones of a sensor unit or more than two microphones. In this way, a relative position of a sound source that is causing the noise can be recognized. Unambiguous directional recognition is possible, in particular when using more than two microphones. A filter characteristic is then set in the sensor unit with the microphones and / or in another sensor unit in such a way that the acquisition of the acquisition data is oriented in the direction of the sound source. As an exemplary embodiment, it can be seen that the object data describe different sound sources (in the device or in its surroundings) as objects and that a sound source with a predetermined sound characteristic of a predetermined object type, for example “traffic accident” or “defective device component of the device”, is recognized as the candidate object becomes. The acquisition data can then be acquired by means of at least one sensor unit with a set filter characteristic in order, for example, to determine the relevance of the sound source, i.e. for example the traffic accident or the defect of the device component, using the relevance criterion.

If the filter characteristic is, for example, a directional characteristic of a directional sensor, for example a directional microphone or a radar, then there is no need to permanently scan or record the entire environment with the directional sensor, but rather the directional sensor is only aimed at a recognized candidate object when required aligned and otherwise the directional sensor can be aligned in a different direction.

The invention also comprises embodiments which result in additional advantages.

As already stated, the filter characteristic can be a directional characteristic of the respective sensor unit. In one embodiment, a direction-selective detection area is correspondingly provided in at least one of the at least one sensor unit of the sensor system by means of the signal filter device. The directional characteristic is therefore not omnidirectional, but direction-selective. By setting the filter characteristic, the detection area of the sensor unit is aligned with the candidate object. The signal filter device thus results in spatial filtering, which is also referred to here as a filter characteristic. An example of such a sensor unit is a directional microphone, which can be formed from a plurality of microphones on the basis of a microphone array, for example. By superimposing the microphone signals of several microphones, direction-selective sound detection can take place according to the principle of so-called beamforming and / or zero steering. In the case of a radar system, the emission of radar radiation can also be controlled in relation to the direction, for example by means of a so-called patch antenna, so that an emission direction can be determined, which also results in a filter characteristic in the form of spatial filtering, since only objects along the emission direction are recorded. In the case of a camera, an image capturing area limited to a solid angle of less than 360 ° likewise results in a spatial selectivity which can be spatially aligned by pivoting an optical system of the camera. The object type of an object here includes in particular the relative direction or position of the respective object with respect to the device. This can be provided as an angle specification or as an index specification of a predefined circle sector (e.g. "front left"). Aligning the detection area to the candidate object has the advantage that the detection data contain more information and / or details on the candidate object. The respective sensor unit is completely or at least predominantly available for detecting the candidate object. A disadvantageous alternative on the other hand would be to change the filter characteristic cyclically and thus to scan the surroundings by means of a raster method so that the candidate object would only be in the detection area of the at least one sensor unit at times.

Spatial filtering is only one option. In one embodiment, a zoom (magnification) and / or signal processing is set in at least one sensor unit of the sensor system by means of the signal filter device by setting the filter characteristic. Examples of signal processing are each the following: Edge detection, segmentation, optical flow analysis. It can therefore be provided, for example, that the filter characteristic for a specific object type is that the candidate object is zoomed in, that is, an optical system or generally a lens is set or changed in such a way that an image of the candidate object is shown enlarged on a sensor of the sensor unit. Signal processing can also be adapted by setting the filter characteristics so that, depending on the object type, those characteristics of the candidate object that are necessary or relevant for checking the relevance criterion are highlighted or emphasized compared to other characteristics (compared to the raw sensor data).

So that the correct filter characteristic can be found for an object type, the object data of the at least one object data source are used in the manner described. In one embodiment, at least one sensor unit of the sensor system itself is used as an object data source, which sensor unit provides sensor data as object data even before the sensor system is adapted (that is, before an object-type-specific filter characteristic is set). In other words, at least some of the object data can be generated by means of the sensor system in the non-adapted state. For example, a directional microphone can be used as a sensor unit, which is initially operated omnidirectionally, that is, has not set a preferred detection direction. If a noise is then recognized in the sensor data by the object recognition device, which indicates a candidate object, for example a bang which indicates an accident, a filter characteristic can then be selected that is aligned with the sound source of the bang or the sound in general. For this purpose, the object recognition device can determine a direction of incidence of the noise in a manner known per se from a phase relationship of microphone signals from several microphones. In addition, a filter characteristic can also be set for at least one further sensor unit, for example a pivotable camera, which is aligned with the sound source and thus with the candidate object. The detection of a candidate object by means of a sensor unit can thus be used to align at least one other sensor unit with the candidate object. The use of the sensor system as an object data source has the advantage that the object data itself can be derived or generated in the device.

In one embodiment, a stationary server device of the Internet is used as an object data source. Such a server device is also referred to as a backend for the device. The device can be coupled to the server device for data exchange, for example by means of a radio link. The server device can be based on a computer or a computer network. The control device can send position data to the server device which describes a current position of the device in the vicinity. Database data of a database in which the at least one object is registered are then received from the server device as object data. In other words, the position data of the server device can be used to signal where the device is currently located. In the server device, such database data can then be searched out from a database and provided to the control device of the device, which data indicate which at least one object is or could be located in the vicinity of the device. This means that a database of registered objects can be used.

In one embodiment, a digital map of the surroundings is used as an object data source, in which the surroundings are mapped. Map entries relating to the at least one object are read out from the map of the surroundings as object data. Thus, previously known stationary objects, each of which is registered in the map of the surroundings as a map entry, can be recognized and checked to determine whether they represent a candidate object and / or meet the relevance criterion.

In one embodiment, a respective portable transmitter which is arranged on one of the at least one object is used as an object data source. Such a transmitter is therefore a signal transmitter which is attached to an object and which transmits a radio signal describing the respective object as object data. The transmitter thus represents a radio-based marker to mark the respective object. This advantageously makes it possible to localize and / or recognize a respective object by means of radio location or radio direction finding. In addition, the radio signal can also be used to signal the object type of the respective object, so that there is no need for a complex evaluation of image data, for example.

The object recognition device described, which recognizes on the basis of the object data whether a candidate object is located in the environment, can be implemented on the basis of software or program code for a computing device of the control device. In one embodiment, the object recognition device is implemented using a machine learning method, so that the candidate object is detected and / or its object type is recognized by the object recognition device using the machine learning method. An example of a method of machine learning is the operation of an artificial neural network. By using A method of machine learning has the advantage that even if there is a discrepancy between the appearance or the object data in general from training object data, on the basis of which the object recognition device is trained according to the machine learning method, a candidate object and / or its object type are still reliably recognized can. The object recognition device can additionally or alternatively also be based on a method of signal processing that uses, for example, a Hidden Markov Model (HMM) for recognition and / or an image processing algorithm and / or a so-called expert system.

The evaluation device described, which checks the relevance criterion on the basis of the detection data, can be implemented on the basis of software or a program code for a computing device of the control device. In connection with the evaluation device, too, one embodiment provides that the relevance criterion is implemented at least partially in the evaluation device by a method of machine learning. In other words, the evaluation of a situation in which the device is located and for which a decision must be made as to whether the candidate object is relevant can be carried out using a machine learning method, for example on the basis of an artificial neural network. This advantageously makes it possible to also check a situation with several objects in the vicinity of the device to determine whether an object recognized as a candidate object is relevant for the device in terms of the relevance criterion. A combination of several parameters of the candidate object, for example a movement direction and a movement speed and the object type, can be linked by means of a method of machine learning in order to define a relevance criterion. A relevance criterion based on a method of machine learning also has the advantage that it is possible to specify when a candidate object is relevant by means of training acquisition data with example situations.

As already stated, in one embodiment a motor vehicle is provided or operated as a device. As an alternative to this, in one embodiment, a mobile terminal device, for example a smartphone or a tablet PC, is provided or operated as the device. As an alternative to this, in one embodiment, a portable assistance device is provided or operated as a device, that is to say, for example, a device that a person can wear to recognize a danger when the person enters a danger area. For example, a firefighter can wear such an assistance device when entering a burning building in order to warn the firefighter of a danger that is recognized in the form of a candidate object.

If a candidate object meets the relevance criterion, the notification signal is generated in the manner described. This can be important in the event that the user of the device has not yet recognized the candidate object independently or on his own. In order to check this, one embodiment provides that the signaling requirement for generating the notification signal is additionally determined as a function of a recognition signal of an alertness recognition of the device. The attention recognition uses the recognition signal to signal whether a user of the device perceives the surroundings and / or in which direction the user is looking. For example, the attention recognition can signal whether the user has opened his eyes and / or whether the user is looking at the candidate object himself. As attention recognition, for example, the per se known line of sight recognition can be provided, which determines, for example, on the basis of image data from a video camera and / or an infrared camera by evaluating at least one camera image showing the user's eyes, in which direction the user is looking and / or whether the user has closed their eyes. In addition or as an alternative to this, the attention recognition can take place on the basis of an evaluation of at least one image of a head of the user, the spatial orientation of which also provides an indication of the current direction of view of the user.

The process can be used for different purposes. The intended use can be set by specifying the object type. In one embodiment, the object type indicates that the candidate object is a road user. The method can thus advantageously be upgraded for a motor vehicle, for example. As already stated, the object type can specify a relative position or direction of an object, ie it is an object with a predetermined relative position or in a predetermined relative direction with respect to the device. In this way, an alignment of a directional sensor that is adapted to requirements can advantageously take place. In one embodiment, the object type indicates that it is an object with a predetermined relative direction of movement with respect to the device. This enables an object that is on a collision course with the device to be recognized as a candidate object. In one embodiment, the object type indicates that the candidate object is an object with a potential anomaly, that is to say, for example, a defective object. This allows in advantageously, a defective component of the device can be automatically recognized. By means of the object recognition device, for example, a predetermined sound or a predetermined noise or a predetermined appearance can be recognized, which gives an indication of a defect. By adapting the sensor system, detection data on the candidate object, that is to say the potentially defective component, can then be determined. The evaluation device can then use the relevance criterion to verify whether the candidate object, that is to say the potentially defective component, is actually defective. In one embodiment, the object type indicates that the candidate object is an object with a predetermined quality and / or with a predetermined movement behavior. If there are several people in the vicinity, for example, a predetermined movement behavior can indicate, for example, that the person is waving. This can be recognized, for example, by the object recognition device on the basis of camera images which depict the surroundings. Then, by setting a corresponding filter characteristic, the person can be zoomed, for example, in order to then use the evaluation device to check the acquisition data obtained in this way to determine whether the waving person fulfills the relevance criterion, i.e. whether the warning signal for this person should be generated.

With regard to the notification signal, one embodiment provides that an output device for outputting an acoustic and / or visual and / or haptic notification is controlled by means of the notification signal. In other words, a user of the device can be informed of the recognized candidate object by means of the notification signal via the output device, if this fulfills the relevance criterion. Additionally or alternatively, one embodiment provides that a control device for user-independent setting of an operating behavior of the device is controlled by means of the notification signal. This means that the device automatically reacts to the candidate object as a function of the notification signal, that is to say that the user of the device does not have to make a contribution, but rather changes the control device of the device automatically. In the case of a motor vehicle, for example, a driving trajectory or a driving direction can be changed as operating behavior in order to avoid a candidate object.

In order to carry out the method according to the invention in a device, the invention also provides a control device for a device. The device is one with a sensor system, which in turn has at least one sensor unit which comprises an adaptive signal filter device in the manner described. The control device according to the invention has a computing device which is set up to carry out an embodiment of the method according to the invention. For this purpose, the computing device can have at least one microprocessor. The method can be implemented as program code or software that has program instructions which are set up to carry out the embodiment of the method according to the invention when executed by the computing device. The program code or the software can be stored in a data memory of the computing device, which can be coupled to the at least one microprocessor.

With regard to the implementation of the method in connection with a device which is designed as a motor vehicle, the invention provides a motor vehicle which comprises a sensor system which has at least one sensor unit with an adaptive signal filter device. The motor vehicle according to the invention is characterized in that it has an embodiment of the described control device according to the invention. The motor vehicle can be configured, for example, as a passenger vehicle or truck or as a passenger bus.

Further features of the invention emerge from the claims, the figures and the description of the figures. The features and combinations of features mentioned above in the description as well as the features and combinations of features mentioned below in the description of the figures and / or shown alone in the figures can be used not only in the respectively specified combination, but also in other combinations or on their own.

The invention will now be explained in more detail using a preferred exemplary embodiment and with reference to the drawings. Show it:

• 1 eine schematische Darstellung einer Ausführungsform eines erfindungsgemäßen Kraftfahrzeugs, das eine Steuervorrichtung aufweist, die eine Ausführungsform des erfindungsgemäßen Verfahrens durchführen kann;
• 2 eine schematische Darstellung eines Geräts mit einem Sensorsystem, das als Sensoreinheit mit adaptiver Signalfilterungseinrichtung eine Richtmikrofoneinrichtung aufweist;
• 3 eine schematische Darstellung eines Geräts mit einem Sensorsystem, das als Sensoreinheit eine stereoskopische Kameraanordnung aufweist;
• 4 eine schematische Darstellung eines Geräts, welches ein Sensorsystem mit einer Sensoreinheit aufweist, die eine Wärmestrahlung erfassen kann;
• 5 eine schematische Darstellung eines Geräts mit einem Sensorsystem, das eine Sensoreinheit aufweist, die einen gerichteten Funkempfang vorsieht;
• 6 eine schematische Darstellung eines Geräts, bei welchem eine geräteexterne Servereinrichtung als Objektdatenquelle für Objektdaten genutzt wird; und
• 7 ein Flussschaudiagramm zur Veranschaulichung einer Ausführungsform des erfindungsgemäßen Verfahrens.

Embodiments of the invention are described below. This shows:

• 1 a schematic representation of an embodiment of a motor vehicle according to the invention which has a control device which can carry out an embodiment of the method according to the invention;
• 2 a schematic representation of a device with a sensor system which has a directional microphone device as a sensor unit with an adaptive signal filtering device;
• 3 a schematic representation of a device with a sensor system which has a stereoscopic camera arrangement as a sensor unit;
• 4th a schematic representation of a device that has a sensor system with a sensor unit that can detect thermal radiation;
• 5 a schematic representation of a device with a sensor system having a sensor unit that provides a directional radio reception;
• 6th a schematic representation of a device in which a device-external server device is used as an object data source for object data; and
• 7th a flowchart to illustrate an embodiment of the method according to the invention.

In the figures, functionally identical elements are each provided with the same reference symbols.

1 shows a device 10 , which is, for example, a motor vehicle 11 can act as if through a steering wheel 12 is illustrated symbolically. In the device 10 can be a sensor system 13 be provided by which acquisition data 14th can be generated by a control device 15th received by the device. By means of the sensor system 13 can through the device 10 an environment 16 of the device are detected or sensed, so that objects 17th in the neighborhood 16 can be recognized or mapped. The acquisition data 14th can refer to an object 17th that as a candidate object 18th was recognized, focus or be limited to it. Depending on the acquisition data 14th can the control device 15th by means of an evaluation device 19th determine whether the candidate object 18th a relevance criterion 20th Fulfills. For this case it can be provided that by the control device 15th a warning signal 21st is produced. The control device 15th can for example be used as a control device or a control circuit of the device 10 be designed. The control device 15th can be a computing device 22nd have, by means of which the method steps described here can be implemented in the form of software or program code.

By means of the notification signal 21st can for example be an output device 23 and / or a control device 24 be controlled. By means of the output device 23 can for example be a user of the device 10 Be advised that in the area 16 a candidate object 18th which is the relevance criterion 20th is located. In the case of a motor vehicle 11 can be used as a candidate object 18th for example, another road user can be recognized by an object recognition device 25th the control device 15th it can be recognized that it is of a predetermined object type 26th is. This can be done by the object recognition device 25th based on property data 27 be recognized that the environment 16 or at least the objects 17th or the candidate object 18th describe. Used by the object recognition device 25th recognized that in the area 16 an object 17th a candidate object 18th of the predetermined object type 26th is arranged or is located therein, for example by an actuating device 28 the sensor system 13 be adapted to the effect that a detection area 29 of the sensor system 13 on the candidate object 18th is aligned. The adaptation is described in more detail below. Then the sensor system 13 the acquisition data 14th generate, on the basis of which the evaluation device 19th can check whether the candidate object 18th also the relevance criterion 20th Fulfills. In this case, preferably only in this case, the notification signal is activated 21st generated.

The control device 24 can be designed for an operating behavior of the device 10 also independent of the user depending on the notification signal 21st set or change. In the case of a motor vehicle 11 For example, an evasive maneuver can be carried out in an automated manner, which then applies to the control device 24 can for example be a driver assistance system.

To the sensor system 13 to adapt to the effect that the detection area 29 on the candidate object 18th can be adjusted as follows.

The sensor system 13 can at least one sensor unit M. , K , T , F. have, where in 1 for example four sensor units M. , K , T , F. are shown. It is also shown as an example that each sensor unit M. , K , T , F. each may have several sensors, namely the sensor unit M. the sensors M1 , M2 , the sensor unit K the sensors K1 , K2 , the sensor unit T the sensors T1 , T2 and the sensor unit F. the sensors F1 , F2 that each have a sensor signal or sensor data 30th which each can produce the environment 16 describe or depict. Any sensor unit M. , K , T , F. can be a signal filter device 31 have which the respective sensor data 30th the sensors of the respective sensor unit M. , K , T , F. can receive and this according to a currently set filter characteristic 32 from the sensor data 30th the acquisition data 14th can generate. The filter characteristic 32 is at the signal filter devices 31 adjustable, i.e. there are several possible filter characteristics 33 provided, one of which is the current filter characteristic 32 can be set or selected. It can also have a filter characteristic 33 be provided, which an omnidirectional reception or an omnidirectional sensing provides, that is, it will be the sensor data 30th one of the sensors of the respective sensor unit M. , K , T , F. passed through or forwarded as acquisition data.

Because of the multiple sensors M1 , M2 , K1 , K2 , T1 , T2 , F1 , F2 can through the respective sensor unit M. , K , T , F. directional sensing or detection of the surroundings also take place, which increases the detection area 29 in the neighborhood 16 can pivot or align or deflect. This allows the sensor system in the manner described 13 be adapted to the effect that the detection of the candidate object 18th takes place, i.e. the acquisition data 14th the candidate object 18th but not the rest of the objects 17th describe or record.

To illustrate, it is assumed that the device 10 around the motor vehicle described 11 acts which by the environment 16 along a direction of movement 34 moves. The objects 17th in the neighborhood 16 also indicate a direction of movement 35 on. Through the property data 27 can through the object recognition device 25th can be recognized that the direction of movement 35 of the candidate object 18th a collision course with the direction of movement 34 of the motor vehicle 11 forms, while the other directions of movement 35 the remaining objects 17th present as safe. The object type 26th thus represents a possible collision object that is detected by the object detection device 25th in the candidate object 18th is recognized. So by an object 17th as an object type 26th is recognized in the form of a collision object, this object becomes the candidate object 18th . Alignment data or direction data 37 for example to the control device 28 by the object recognition device 25th given, which indicate in which position relative to the device 10 the candidate object 18th is located. Then, for example, the adjusting device 28 one in the signal filter devices 31 of the sensor units M. , K , T , F. as a filter characteristic 32 set a detection direction aimed at the candidate object 18th is aligned. This results in the operation of the sensor system 13 Acquisition data 14th that the candidate object 18th describe more precisely or in more detail than the other objects 17th . Thus, the evaluation device 19th based on the acquisition data 14th decide whether the candidate object 18th the relevance criterion 20th fulfilled, so for example represents an actual collision object, or whether the collision is impossible, for example, because the candidate object 18th moves slowly and the way of the motor vehicle 11 will only cross behind the vehicle. On the other hand, it is the criterion of relevance 20th fulfilled, so there is, for example, a risk of collision, then by means of the notification signal 21st the driver of the motor vehicle 11 by the display device 23 be warned and / or by means of the control device 24 an evasive maneuver can be initiated.

The following are based on 2 , 3 , 4th , 5 and 6th possible sensor units described in the sensor system 13 can be provided individually or in a combination to enable directed detection.

2 shows the device 10 with one embodiment of the sensor unit M. , in which four sensors M1 , M2 , M3 , M4 are provided. At the sensor unit M. it can be a directional microphone unit, which acts as sensors M1 , M2 , M3 , M4 Microphones can be provided which are arranged to form a microphone array. On the one hand, it can be provided by means of the sensor unit M. Object data 27 to generate by one of an object 17th in the neighborhood 16 outgoing sound 40 that from the object 17th along a direction of incidence 41 on the sensor unit M. hits, it is recognized that the noise 40 on an object type 26th indicates who the object 17th as a candidate object 18th identified. For generating the direction data 37 can be evaluated with what time difference a wave front 42 a sound of noise 40 on the individual sensors M1 , M2 , M3 , M4 meets. The time difference corresponds to a phase shift 43 which depends on the direction of incidence 41 . This allows a relative horizontal position or direction of the candidate object 18th regarding the device 10 , so for example the motor vehicle 11 , by means of the sensor unit M. be recognized. Should then also acquisition data 14th by means of the sensor unit M. can be generated by setting a directional characteristic as a filter characteristic 32 processing of the sensor signals or sensor data 30th be provided in which, in a manner known per se, a so-called beam (main beam) or detection beam as the detection area by means of beamforming 44 can be formed on the candidate object 18th is aligned. This is done by means of a weighted superimposition of the sensor data 30th , whereby by providing a time offset between the sensor data 30th of the different sensors M1 , M2 , M3 , M4 a spatial alignment or a pivoting direction of the beam is set. The microphones, that is, the sensors M1 , M2 , M3 , M4 can be designed to produce an infrasound and / or a sound of the audible spectrum ( 5 Hertz to 20 kilohertz) and / or an ultrasound. The direction recognition can for example be recognized on the basis of a correlation. The sensors M1 , M2 , M3 , M4 can be arranged spatially spaced apart.

3 illustrates how a sensor unit K as sensors K1 , K2 can each have a camera to thereby an object 17th in the Surroundings 16 to detect optically, that is, electromagnetic waves 45 by the object 17th go out to capture. This can be visible light and / or infrared light and / or ultraviolet radiation emitted by the sensors K1 , K2 can be captured. The sensors K1 , K2 can be arranged spatially spaced apart. This enables stereoscopic or, in general, spatial recognition, that is to say on the basis of the sensor data 30th the sensors can also be a distance 47 of the object 17th to the device 10 , for example the motor vehicle 11 , be determined. A corresponding filter characteristic can be used for this 33 be provided, with different filter characteristics 33 different overlays of the sensor data 30th of the sensors K1 , K2 can be provided in order to enable a distance measurement for different distance ranges. Another adaptation of the sensor unit K by means of different filter characteristics 33 can be achieved, for example, that the sensors K1 , K2 , that is to say the respective camera, is arranged to be spatially pivotable, that is to say it can be rotated and / or the zoom or the magnification can be adjusted by means of an optical system of the respective camera. Thus, through a current filter characteristic 32 a direction-related evaluation and / or an enlarged image section and / or an increased optical resolution and / or oversampling take place, for example for gesture recognition. An image repetition rate can also be variable, so that a currently set image repetition rate results in a filter characteristic, namely temporal filtering.

To by means of the sensor unit K also general object data 27 to generate, it can be provided that the largest possible detection area for the sensor unit K and then in the sensor data 30th first a candidate object 18th to be recognized then by means of the setting of the filter characteristic 32 the detection area 44 on the candidate object 18th align.

4th illustrates a sensor unit T with heat-sensitive sensors T1 , T2 which can each be designed as an infrared camera, for example. One from an object 17th outgoing thermal radiation 48 can through the sensors T1 , T2 as object data 27 to be provided. By aligning the detection area 44 the sensor unit T in the way as it relates to the sensor unit K ( 3 ), acquisition data can also be used 14th with a limited detection area 44 pointing to a candidate object 18th is aligned.

5 illustrates how with a sensor unit F. as sensors F1 , F2 Radio receivers can be provided through which a radio radiation 49 that can be received by an object 17th can go out. For this purpose, as in 1 and 5 is illustrated, a transmitter on the object 50 be arranged, which the radio radiation 49 can generate. By evaluating a direction of incidence 41 , for example based on the phase shift of a wavefront 42 the radio radiation 49 can be the direction of incidence 41 can be determined and interpreted in the same way as it is related to 2 has been described.

The radio signal 51 which the radio radiation 49 can be based on WLAN (Wireless Local Area Network), 5G (cellular network), UV (ultraviolet light) or a radar signal. The sensor unit F. can also be a transmitter 52 have which the radio radiation 49 generated so the sensors F1 , F2 a reflected radio signal from the direction of incidence 41 receive.

By the radio signal 51 can also be a source for object data 27 be formed by adding in the radio signal 51 the object type 26th is specified so that the object 17th it can be decided whether it is a candidate object 18th acts. The multiple sensors can be used to generate the detection data F1 , F2 a detection area 44 on the candidate object 18th restricted or aligned. This can be done in the manner described by means of beamforming for the radio signal or the radio waves.

Regarding the 1 one hand and the 3 to 5 on the other hand, it should be noted here that the respective detection area 44 the detection area of the individual sensor units 29 can represent or a combination of several of the detection area 44 can change the detection area 29 of the sensor system 13 represent.

6th illustrates how object data 27 from a server facility 53 of the internet 54 can be received. The server facility 53 can be a database 55 have from which the object data 27 can be read out depending on position data 56 by the control device 15th to the server facility 53 can be transmitted. The position data 56 can for example be generated by means of a receiver for a GNSS position signal, for example a GPS signal. 1 illustrates how communication between the server device 53 and the control device 15th can be provided. 1 shows that a communication link 57 based on a communication facility 58 can be generated by means of a radio link 59 the control device 15th with the Internet 54 can connect. The communication facility 58 can do this based on a WLAN technology and / or a cellular technology.

7th illustrates an embodiment of a method as it is overall performed by the control device 15th can be executed. In one step S10 can initialize the sensor units M. , K , T , F. if they are in the device 10 available. The initialization enables, for example, a non-directional detection or a detection with the largest possible detection area 44 be provided.

By means of the sensor system 13 object data themselves can thus 27 are generated, for example raw sensor data in the form of to the control device 15th transmitted sensor data 30th to get out of objects 17th the environment 16 a candidate object 18th to investigate. From the objects 17th can be done in one step at a time S11 an acoustic identifier as part of the object data 27 , in one step S12 an optical identifier as part of the object data 27 , in one step S13 a thermal identifier as part of the object data 27 , in one step S14 at least one further identifier as part of the object data 27 will be generated. In one step S15 can be used as further object data 27 the database data from the server device 53 be determined. In one step S16 can through the object recognition device 25th determine whether a candidate object 18th in the neighborhood 16 is available. If this is the case, it can be done in one step S17 by adapting the sensor system 13 the detection area 29 , i.e. at least one of the detection areas 44 , on the candidate object 18th aligned so that collection data 14th are generated, on the basis of which the evaluation device 19th can decide whether the warning signal 21st should be generated (relevance criterion 20th Fulfills). Provides a one-step review S18 that the operation of the device 10 is not finished, so for example a motor vehicle 11 continues, can with the step S11 can be continued (indicated by a "-" symbol). Results in step S18 however, an end to the operation of the device 10 , so for example the end of a journey of a motor vehicle 11 so can in one step S19 the sensing can be ended (symbolized by a "+" symbol).

Directional sensing of the environment or surroundings is preferred 16 provided by means of a combination of optical, acoustic and / or informal data acquisition, that is, an alignment of at least one detection area 44 .

A directional sensor system is therefore proposed (see 1 , Sensor system 13 ), which, based on directional and in particular acoustic sensing and the additional information relating to a defined relevance or object type from camera detection / image object detection or based on other wavelengths, detects and signals a relevant situation.

An essential feature is the adaptation by filtering the relevance. I.e. the signal filtering is adapted in its characteristics according to the situation to be expected or the candidate object to be expected. In other words, this corresponds to an automated “sharpening of the senses”.

The relevance / relevance class or the object type relates, for example, to people (waving, shouting, gestures), situations (moving objects, accidents), circumstances (danger, shop object, "geocash").

In addition, a request can be sent to a backend system for the object or the direction (see 6th ) take place (e.g. unrestricted level crossing / timetable / train on the route, opening times of the shop / workshop / dealer or generally geo-referenced knowledge) in order to concretize the sensing or information evaluation.

• - Mikrofon M1/M2: Infraschall + hörbarer Schall + Ultraschall.
• - Mehr als zwei, örtlich getrennte Sensoren zur Richtungserkennung.
• - Die Filterung der Information (z.B. spezifisches Geräusch / akustische Kennung aus Abstrahlung, Spektrum, Korrelation, Reflektion)
• - Option: örtlich ausrichtbares Richtmikrofon.
• - Schärfung der Filterung - Laufzeitunterschied: temporär höhere Abtastung in definiertem Zeitfenster entsprechend der gewünschten Richtung.

A specific acoustic identifier or recognition is in 2 illustrates:

• - microphone M1 / M2 : Infrasound + audible sound + ultrasound.
• - More than two locally separated sensors for direction detection.
• - The filtering of information (e.g. specific noise / acoustic identification from radiation, spectrum, correlation, reflection)
• - Option: locally adjustable directional microphone.
• - Sharpening of the filtering - Runtime difference: temporarily higher sampling in a defined time window according to the desired direction.

• - Die optische Erkennung basiert auf der Erkennung mittels Kamerasystemen. Mit zwei versetzten Kameras ist eine räumliche Erkennung möglich. Diese kann mit Methoden der Bildverarbeitung erfolgen. Die Bildverarbeitung kann weiterhin Objekte erkennen.
• - Option: örtlich ausrichtbare Richtkamera.
• - Schärfung der Filterung: Richtungsbezogene Auswertung des Bildsensors reduziert auf Bildausschnitt aber mit erhöhter Auflösung / Überabtastung, z.B. zur Gestenerkennung bei erhöhter Entfernung

A specific optical identifier is in 3 illustrates:

• - Optical recognition is based on recognition using camera systems. Spatial recognition is possible with two offset cameras. This can be done using image processing methods. The image processing can still recognize objects.
• - Option: locally adjustable directional camera.
• - Sharpening of the filtering: Direction-related evaluation of the image sensor reduced to image section but with increased resolution / oversampling, e.g. for gesture recognition at a greater distance

A specific heat identifier is in 4th illustrates: Usage in a comparable way for an extended optical spectrum such as thermal radiation.

Another specific identifier is in 5 illustrates: Use in a comparable way for WLAN, 5G, UV, passive radars.

The respective overall filter characteristics of the sensor system 13 is based on the "sum" of the information.

• • Ein bestimmtes Objekt hat eine definierte akustische Ausstrahlung, Geräusch (Bahn, Martinshorn, ...)
• • Ein bestimmtes Objekt hat eine definierte optische Ausstrahlung (Lichtstärke, Farben)
• • Ein bestimmtes Objekt hat eine definiertes Wärmebild
• • Ein bestimmtes Objekt hat eine definierte Position

A decision system or an object recognition device is used for the selection of the respective characteristic or detection data 25th , for example with a Kl system, advantageous. Examples:

• • A certain object has a defined acoustic charisma, noise (train, siren, ...)
• • A certain object has a defined optical appearance (light intensity, colors)
• • A certain object has a defined thermal image
• • A certain object has a defined position

An evaluation device is used to characterize a situation 19th Helpful and beneficial with a Kl system.

The situation interpreted in this way can, in sum, be, for example: a calling person with certain clothes and luggage who wants to be picked up at a certain location. Does this correspond to the relevance criterion 20th , can be the warning signal 21st be generated.

The 7th shows an exemplary flowchart of a sequence of a software program for the control device 15th .

• • Synchronisation mit der Aufmerksamkeitserkennung Fahrer zur besseren Kommunikation der Situation
• • Erkennung von Anomalien / Defekten eines Objektes (eigenes Fahrzeug, Motor, im Prinzip auch für Kaffeemaschine, Waschmaschine oder anderes Gerät)
• • Erkennung der Beschaffenheit eines Fahrbahnbelags.

More options:

• • Synchronization with the driver's attention recognition for better communication of the situation
• • Detection of anomalies / defects of an object (own vehicle, engine, in principle also for coffee machine, washing machine or other device)
• • Detection of the condition of a road surface.

Variant portable / mobile: Portable assistance system or assistance device, e.g. helps a jogger or hearing impaired person while listening to music or other things. nevertheless to recognize dangers or approaching objects or the like.

Example of a communication scenario between vehicle and jogger: The vehicle detects the jogger and communicates its position and acoustic identifier (e.g. engine noise or coded acoustic signal) via data transmission. The described transmitter can do this 50 to be provided. The jogger's system determines the appropriate signal filtering and signal processing, it derives any danger signal / warning that may be present.

A similar assist scenario is conceivable for craftsmen or in general for people in danger areas (track workers, traffic regulating police officers, firefighters, technical aid organizations).

• - Verbesserung der Umwelt-/Objekterkennung inkl. Informationsverarbeitung mittels Backend- / Objektvernetzung,
• - Verbesserung der Risikobewertung, vor allem im Bereich Fahrerassistenz und automatisiertes bzw. autonomes Fahren,
• - Ableitung weiterer kundenwerter Funktionen (Objekterkennung),
• - Kostenneutral durch Nutzung vorhandener Sensorik.

The following advantages result:

• - Improvement of the environment / object recognition including information processing by means of backend / object networking,
• - Improvement of the risk assessment, especially in the area of driver assistance and automated or autonomous driving,
• - Deriving further customer-worthy functions (object recognition),
• - Cost-neutral through the use of existing sensors.

Overall, the examples thus show a directed, sensor-wide or cross-domain environmental assessment, as made possible by the invention.

List of reference symbols

10

device

11

Motor vehicle

12th

steering wheel

13

Display device

13

Sensor system

14th

Acquisition data

15th

Control device

16

Surroundings

17th

object

18th

Candidate object

19th

Evaluation device

20th

Relevance criterion

21st

Warning signal

22nd

Computing device

23

Output device

24

Control device

25th

Object recognition device

26th

Object type

27

Object data

28

Adjusting device

29

Detection area

30th

Sensor data

31

Signal filter device

32

Filter characteristic

33

Filter characteristic

34

Direction of movement

35

Direction of movement

37

Direction data

40

noise

41

Direction of incidence

42

Wavefront

43

Phase shift

44

Detection area

45

waves

47

distance

48

Thermal radiation

49

Radio radiation

50

Channel

51

Radio signal

52

Channel

53

Server setup

54

Internet

55

Database

56

Position data

57

Communication link

58

Communication facility

59

Radio link

QUOTES INCLUDED IN THE DESCRIPTION

This list of the documents listed by the applicant was generated automatically and is included solely for the better information of the reader. The list is not part of the German patent or utility model application. The DPMA assumes no liability for any errors or omissions.

Patent literature cited

• DE 102017203785 A1 [0004]

Claims (15)

Method for operating a sensor system (13) of a device (10), wherein the sensor system (13) comprises at least one sensor unit (M, K, T, F) which has an adaptive signal filtering device (31) and wherein in the method by a Control device (15): - Object data (27) which describe at least one object (17) arranged or potentially arranged in an environment (16) of the device (10) or in the device (10) are received from at least one object data source, - it is recognized by means of an object recognition device (25) based on the object data (27) that a candidate object (18) of a predetermined object type (26) is located under the at least one object (17), - The sensor system (13) is adapted for detecting the recognized candidate object (18) by a filter characteristic (32) from several possible filter characteristics (33) in the respective signal filtering device (31) of the at least one sensor unit (M, K, T, F) ), which is provided for the object type (26) of the recognized candidate object (18), is selected and set in the respective signal filtering device (31), - the recognized candidate object (18) is detected by means of the adapted sensor system (13) and, as a result, detection data (14) relating to the recognized candidate object (18) are generated, - by means of an evaluation device (19) as a function of the detection data (14) on the basis of a predetermined relevance criterion (20) it is recognized whether there is a need for signaling relating to the recognized candidate object (18), and - In the case of a recognized need for signaling, a notification signal (21) describing the recognized candidate object (18) is generated. Procedure according to Claim 1 , with at least one of the at least one sensor unit (M, K, T, F) of the sensor system (13) providing a direction-selective detection area (44) by means of the signal filter device (31), and by setting the filter characteristic (32) the detection area ( 44) of the respective sensor unit (M, K, T, F) is aligned with the candidate object (18). Method according to one of the preceding claims, wherein in at least one sensor unit (M, K, T, F) of the sensor system (13) a zoom and / or signal processing is set by means of the signal filter device (13) by setting the filter characteristic (32). Method according to one of the preceding claims, wherein at least one sensor unit (M, K, T, F) of the sensor system (13) is used as the object data source, which provides sensor data (30) as object data (27) even before the sensor system (13) is adapted . Method according to one of the preceding claims, wherein a stationary server device (53) of the Internet (54) is used as the object data source, to which position data (56) describing a current position of the device (10) in the environment (16) are sent and from which database data from a database (55) in which the at least one object (17) is registered is received as object data (27). Method according to one of the preceding claims, wherein a digital environment map in which the environment (16) is mapped is used as the object data source and from which map entries relating to the at least one object (17) are read as object data (27). Method according to one of the preceding claims, wherein a respective portable transmitter (50) is used as the object data source, which is respectively arranged on one of the at least one object (17) and which as object data (27) is a radio signal (51) which the respective object (17) describes, sends out. Method according to one of the preceding claims, wherein the candidate object (18) is detected and / or its object type (26) is recognized by the object recognition device (25) by means of a machine learning method. Method according to one of the preceding claims, wherein in the evaluation device (19) the relevance criterion (20) is at least partially implemented by a method of machine learning. Method according to one of the preceding claims, wherein a motor vehicle (11) or a mobile terminal or a portable assistance device is operated as the device (10). Method according to one of the preceding claims, wherein the signaling requirement is additionally determined as a function of a recognition signal of an attention recognition of the device (10), the attention recognition signaling by means of the recognition signal whether a user of the device (10) perceives the surroundings (16) and / or in which direction the user is looking. Method according to one of the preceding claims, wherein the object type (26) indicates that the candidate object (18) is - an object with a predetermined relative position or in a predetermined relative direction with respect to the device and / or - an object with a predetermined relative direction of movement with respect to the device and / or - a road user and / or - an object (17) with a potential anomaly and / or - an object (17) is of a predetermined nature and / or a predetermined movement behavior. Method according to one of the preceding claims, wherein by means of the notification signal (21) an output device (23) for outputting an acoustic and / or visual and / or haptic notification and / or a control device (24) for user-independent setting of an operating behavior of the device (10) is controlled. Control device (15) for a device (10) with a sensor system (13) which comprises at least one sensor unit (M, K, T, F) which has an adaptive signal filtering device (31), characterized in that the control device (15 ) has a computing device (22) which is set up to carry out a method according to one of the preceding claims. Motor vehicle (11) with a sensor system (13) which comprises at least one sensor unit (M, K, T, F) which has an adaptive signal filtering device (31), characterized by a control device (15) according to Claim 14 .

Images