DE102019113592A1 - Vehicle user assistant based on an analysis of an attention zone - Google Patents

Abstract

The present invention relates to a method and a system for answering a user query which is related to an object (O1, O2, O3) or an area outside the vehicle. In particular, the invention relates to a method and a system for expanding in-vehicle voice assistants by providing them with additional context information that is learned from visual information whenever a voice-based request is made and when this request is open or partially open. The method describes how this visual information is extracted using the gaze vector of the driver's gaze, the scene recorded by the front camera of the vehicle and the use of image recognition solutions, and how this process is integrated into conventional voice assistant solutions. The present invention also relates to a computer program containing program code instructions for carrying out the steps of the above-mentioned method.

Description

The present invention relates to a method and a system for responding to a user query which is related to an object or an area outside the vehicle.

The present invention also relates to a computer program which contains program code instructions for carrying out the steps of the above-mentioned method.

Voice assistants typically rely only on natural language analysis of language queries to provide meaningful answers. Solutions have been proposed to provide context information for voice assistants in addition to voice requests for a more natural user experience.

DE 10 2016 221 983 A1 publishes, for example, a method in which an object or an area outside of a vehicle is determined to which at least one user's attention is directed for a period of time. It is then checked whether information related to the detected object or the detected area is stored in a memory of the vehicle in order to obtain a test result. If the test result is positive, the stored information is output. In addition, published DE 10 2016 221 983 A1 For example, in section [0065], a car 400 with a storage device 440 that stores information about objects 110 located outside the car. The car has a device 410 that can detect which object the driver of the car is looking at.

The above method, however, has the disadvantage that it is sometimes difficult to identify which object or which area is actually requested by the user, as a result of which the system has only limited possibilities for identifying and answering the request.

The present invention seeks to provide a better way of implementing similar tasks.

The problem is solved by the independent claims. Advantageous refinements are given in the subclaims.

• Erfassen einer Benutzeranfrage bezüglich eines Objekts oder eines Bereichs durch ein Assistentenmodul zu einem Zeitpunkt,
• Prüfen, ob mehr Kontextinformation über das angefragte Objekt oder den angefragten Bereich im Vergleich zu vorhandener Information benötigt wird, um ein Prüfergebnis zu erhalten, und
• wenn ein Prüfergebnis „Falsch“ ist, vorzugsweise Verarbeiten der Benutzeranfrage und Ausgeben von Information,
• wenn ein Prüfergebnis „Wahr“ ist, Prüfen, ob die Benutzeranfrage eine offene oder eine teilweise offene Anfrage ist, wobei eine offene Anfrage kein Referenzschlüsselwort aufweist und eine teilweise offene Anfrage mindestens ein Referenzschlüsselwort aufweist,
• im Falle einer offenen Anfrage, Erfassen der Blickrichtung des Benutzers und Abrufen von Information über ein Objekt am Schnittpunkt des Blickvektors mit mindestens einem Sichtfeld einer Frontkamera eines Fahrzeugs bezüglich des Zeitpunkts durch ein blickgesteuertes Interaktionsmodul, und Zuschneiden eines Bildes des Sichtfeldes der Frontkamera des Fahrzeugs durch das blickgesteuerte Interaktionsmodul,
• Anfordern eines allgemeinen Bilderkennungsprozesses bezüglich des Bildes durch ein Bilderkennungsmodul, nachdem ein Bild des Sichtfelds der Frontkamera des Fahrzeugs zugeschnitten wurde,
• im Falle einer teilweise offenen Anfrage, Erfassen der Blickrichtung des Benutzers und Abrufen von Information über ein Objekt am Schnittpunkt des Blickvektors mit mindestens einem Sichtfeld einer Frontkamera eines Fahrzeugs bezüglich des Zeitpunkts durch ein blickgesteuertes Interaktionsmodul, und Zuschneiden eines Bildes des Sichtfelds der Frontkamera des Fahrzeugs durch das blickgesteuerte Interaktionsmodul.

In particular, the present invention specifies a method for answering a user request that is related to an object or an area outside the vehicle, comprising the following steps:

• Recording of a user request regarding an object or an area by an assistant module at a time
• Check whether more context information about the requested object or the requested area is required compared to the information available in order to obtain a check result, and
• if a test result is "false", preferably processing the user request and outputting information,
• if a test result is "true", checking whether the user request is an open or a partially open request, an open request having no reference keyword and a partially open request having at least one reference keyword,
• In the case of an open request, recording the direction of view of the user and retrieving information about an object at the intersection of the gaze vector with at least one field of view of a front camera of a vehicle with respect to the time by a gaze-controlled interaction module, and cropping an image of the field of view of the front camera of the vehicle by the eye-controlled interaction module,
• Requesting a general image recognition process relating to the image by an image recognition module after an image of the field of view of the vehicle's front camera has been cropped,
• in the case of a partially open request, recording the direction of view of the user and retrieving information about an object at the intersection of the gaze vector with at least one field of view of a front camera of a vehicle with respect to the time by a gaze-controlled interaction module, and cropping an image of the field of view of the front camera of the vehicle the eye-controlled interaction module.

Requesting a general image recognition process relating to the image by an image recognition module after an image of the field of view of the vehicle's front camera has been cropped,
wherein information of the at least one reference keyword is extracted in order to request the general image recognition process relating to the image by the image recognition module.

At least one field of view of the front camera of the vehicle means that at least one camera from possibly many cameras is able to record the front of the vehicle.

The present invention also provides a system for responding to a user query that is related to an object or area outside the vehicle.

The basic idea of the invention is that an analysis process is triggered on the basis of the user request, which in particular determines the direction of view of the user and, with a suitably equipped system and vehicle, tracks which object or which area is in the user's field of vision. Based on this identification of the field of view, an image is recorded and automatically compared with a database system. Depending on the complexity of the queries and the completeness of the database, the user can receive differently detailed answers to his queries. The user can advantageously keep his hands on the steering wheel and at the same time obtain a more precise result about the object actually viewed or the area actually viewed. This invention suggests a better way of using gaze and image recognition to implement similar tasks and more without relying on gestures, GPS coordinates, or pre-saved locations of interest.

According to a modified embodiment of the invention, in the case of receiving an image-related proposal from any recognition process, merging any image-related proposals,
wherein after the merging of any image-related suggestions for new context information, the new context information is added to the existing information according to the user request by the assistant module, whereupon the user request is processed and information is output,
Processing the user request and outputting information if no image-related proposal is received after the general image recognition process request.

• wenn das Abrufen von Information über ein Objekt am Schnittpunkt des Blickvektors mit dem Sichtfeld der Frontkamera des Fahrzeugs bezüglich des Zeitpunkts nicht erfolgreich ist, wird ein Bildausschnitt mit einer Standardgröße zugeschnitten,
• wenn das Abrufen von Information über ein Objekt am Schnittpunkt des Blickvektors mit dem Sichtfeld der Frontkamera des Fahrzeugs bezüglich des Zeitpunkts erfolgreich war, wird ein Bildausschnitt zugeschnitten, dessen Größe aus dem scheinbaren Abstand und der scheinbaren Größe des Objekts am Schnittpunkt berechnet wird, insbesondere berechnet wird von Sensoren wie LiDAR oder Radar.
• wobei der allgemeine Bilderkennungsprozess bezüglich des Bildes durch das Bilderkennungsmodul nach dem Zuschneiden eines Bildausschnitts mit der Standardgröße oder nach dem Zuschneiden eines Bildausschnitts angefordert wird, dessen Größe aus dem scheinbaren Abstand und der scheinbaren Größe des Objekts am Schnittpunkt berechnet wird.

According to a modified embodiment of the invention, cropping an image of the field of view of the front camera of the vehicle in the case of an open request includes two options, namely:

• if the retrieval of information about an object at the point of intersection of the gaze vector with the field of view of the front camera of the vehicle is unsuccessful with regard to the point in time, an image section is cropped with a standard size,
• If the retrieval of information about an object at the point of intersection of the gaze vector with the field of view of the front camera of the vehicle was successful with respect to the point in time, an image section is cropped, the size of which is calculated from the apparent distance and the apparent size of the object at the point of intersection, in particular is calculated from sensors such as LiDAR or radar.
• wherein the general image recognition process relating to the image is requested by the image recognition module after cropping an image section with the standard size or after cropping an image section, the size of which is calculated from the apparent distance and the apparent size of the object at the point of intersection.

In the case of an open request, this measure enables better property reception if possible. The cropping of an image section with the standard size enables a comprehensive image analysis by a qualified downstream image analysis system. This is particularly advantageous if no object or area can be identified more precisely when checking the field of view. In contrast to this, the cropping of an image section, the size of which is calculated from the apparent distance and the apparent size of the cropped object, enables faster image evaluation since a specific object or a specific area is analyzed. No subsequent object or area recognition is required, so that the image recognition module can immediately start with a database comparison for more precise determination.

• Zuschneiden eines Bildausschnitts mit einer Standardgröße, wenn das Abrufen von Information über ein Objekt am Schnittpunkt des Blickvektors mit dem Sichtfeld der Frontkamera des Fahrzeugs bezüglich des Zeitpunkts nicht erfolgreich ist,
• Zuschneiden eines Bildausschnitts, dessen Größe aus dem scheinbaren Abstand und der scheinbaren Größe des zugeschnittenen Objekts berechnet wird, wenn das Abrufen von Information über ein Objekt am Schnittpunkt des Blickvektors mit dem Sichtfeld der Frontkamera des Fahrzeugs bezüglich des Zeitpunkts erfolgreich war,
• wobei der allgemeine Bilderkennungsprozess bezüglich des Bildes durch das Bilderkennungsmodul nach dem Zuschneiden eines Bildausschnitts mit der Standardgröße oder nach dem Zuschneiden eines Bildausschnitts angefordert wird, dessen Größe aus dem scheinbaren Abstand und der scheinbaren Größe des Objekts am Schnittpunkt berechnet wird.

According to a modified embodiment of the invention, the cropping of an image of the field of view of the front camera of the vehicle in the case of a partially open request includes two options, namely:

• Cropping an image section with a standard size if the retrieval of information about an object at the point of intersection of the gaze vector with the field of view of the front camera of the vehicle is unsuccessful with regard to the point in time,
• Cropping of an image section, the size of which is calculated from the apparent distance and the apparent size of the cropped object, if the retrieval of information about an object at the intersection of the gaze vector with the field of view of the front camera of the vehicle was successful with regard to the point in time,
• wherein the general image recognition process with respect to the image by the image recognition module after cropping an image section with the standard size or after A cropping of an image section is requested, the size of which is calculated from the apparent distance and the apparent size of the object at the point of intersection.

In the case of a partially open request, this measure enables better object reception if possible. The cropping of an image section with the standard size enables a comprehensive image analysis by a qualified downstream image analysis system. This is particularly advantageous if no object or area can be identified more precisely when checking the field of view. In contrast to this, the cropping of an image section, the size of which is calculated from the apparent distance and the apparent size of the object at the intersection point, enables faster image evaluation, since a specific object or a specific area is analyzed. No subsequent object or area recognition is required, so that the image recognition module can immediately start with a database comparison for more precise determination.

• nach dem Zuschneiden eines Bildes des Sichtfelds der Frontkamera des Fahrzeugs ein allgemeiner Bilderkennungsprozess bezüglich des Bildes angefordert, und, falls dies möglich ist, außerdem ein spezieller Objekterkennungsprozess für das mindestens eine Referenzschlüsselwort durch ein Bilderkennungsmodul angefordert,
• wobei das Extrahieren von Information des mindestens einen Referenzschlüsselworts ausgeführt wird, um den allgemeinen Bilderkennungsprozess bezüglich des Bildes anzufordern und, falls dies möglich ist, auch um einen spezialisierten Objekterkennungsprozess für das mindestens eine Referenzschlüsselwort durch das Bilderkennungsmodul anzufordern.

According to a modified embodiment of the invention, in the case of a partially open request

• after cropping an image of the field of view of the front camera of the vehicle, a general image recognition process with regard to the image is requested and, if this is possible, a special object recognition process for the at least one reference key word is also requested by an image recognition module,
• wherein the extraction of information of the at least one reference keyword is carried out in order to request the general image recognition process relating to the image and, if this is possible, also to request a specialized object recognition process for the at least one reference keyword by the image recognition module.

According to this exemplary measure, it is therefore always provided that the step which requests a general image recognition process with regard to the image is carried out first. Only if a sufficiently suitable keyword is available, for example, is it preferred to request a specialized object recognition process for the at least one reference keyword. In this case, the database analysis is performed on the existing image using the reference keyword. This reference keyword can restrict the database area so that the image-related suggestion can be output in a short time and with a high degree of matching.

According to a modified embodiment of the invention, requesting the general image recognition process with respect to the image by the image recognition module can trigger at least one reference keyword, so that requesting a specialized object recognition process for the at least one reference keyword is the next step.

Since a reference key word can be found by requesting the general image recognition process with respect to the image by the image recognition module, it is possible to subsequently change an open request to a partially open request without precise knowledge of the user. For example, if a user asks what kind of object the viewed object is, a car can be identified by requesting the general image recognition process related to the image. In a next step, the system can use this information internally to request a specialized object recognition process for the at least one reference keyword in order to find out which model this car is.

According to a modified embodiment of the invention, the request for the general image recognition process with regard to the image and / or the request for a specialized object recognition process for the at least one reference keyword are carried out externally, the image recognition module being a remote server or a cloud server or at least using such a server.

A data feed from an external source ensures that the latest data is always available. For example, the opening times of a place of interest may change or a car may be a new model. These data can preferably be processed on a remote server or a cloud server. In particular, a remote server provides information. This information can be evaluated in the vehicle. This can be advantageous in areas with poor reception in order to keep the data flow volume as low as possible and thus ensure a short response time for the system. On the other hand, a cloud server is configured in such a way that the actual data analysis also takes place on the cloud server. The use of a remote server or a cloud server reduces the computing capacity required in the car. It should be noted that a car usually has a significantly longer lifespan than a computer device. Outsourcing the computing unit to a cloud server prevents a car (which, as previously noted, can have a useful life of several years) or its Computing unit is too old or too slow to process the required data stream.

According to a modified embodiment of the invention, the request for the general image recognition process with regard to the image is primarily carried out in the vehicle and only externally if there is a further request, with the request for a specialized object recognition process for the at least one reference keyword always being carried out externally.

This can represent a meaningful distribution of resources in that the step of requesting the general image recognition process relating to the image can be used for an initial query of simple terms or objects. For example, a car can be relatively easily identified on the basis of certain patterns, whereas the requested model of a car can have been manufactured according to the year of construction of the ego vehicle, so that the step of requesting a specialized object recognition method for the at least one reference key word can be processed by an external database should, which is up to date.

According to a modified embodiment of the invention, the user request is a voice-based request from the user.
This can increase driving safety because, for example, the driver can keep his hands on the steering wheel at all times. This also leads to an intuitive operation of the entire system.

According to a modified embodiment of the invention, the user request is a gesture-based user request.
This can be advantageous if, for example, a hands-free call is being made in the vehicle. For example, the user may want to receive information about the environment for orientation purposes. For a voice-based request from the user (i.e. a request spoken by the user), the phone call would have to be abandoned because if the request were made while on the phone, the person on the other end of the line the user is talking to could be confused what does not have to be done by the proposed measure.

The invention also relates to a system with a device for applying one of the preceding methods.
Such a system enables a safe driving experience for the user, who can be provided with precise information at any time if necessary.

According to a modified embodiment of the invention, the gaze-controlled interaction module has an eye tracker and / or a gesture tracker. This enables an analysis of an attention zone of the user on the basis of his gaze and / or his gestures. The user therefore does not have to invest a great deal of effort to make his inquiries and can concentrate better on the traffic.

According to a modified embodiment of the invention, the assistant module is a voice assistant module. This can increase driving safety because, for example, the driver can keep his hands on the steering wheel at all times. This also leads to an intuitive operation of the entire system.

According to a modified embodiment of the invention, the assistant module is a gesture assistant module. This can be advantageous if, for example, a hands-free call is being made in the vehicle. For example, the user may want to receive information about the environment for orientation purposes. Otherwise the phone call would have to be broken off, which does not have to take place due to the proposed measure.

According to a modified embodiment of the invention, an internal memory unit of a vehicle is used to call up information offline, this information in particular containing places of interest. This can be advantageous in areas with poor reception in order to keep the data flow volume as low as possible and thus ensure a short response time for the system. The internal memory can optionally be updated in order to receive current information even after a longer period of time.

According to a modified embodiment of the invention, the image recognition module is an external server, in particular a remote server or a cloud server, preferably using a database, preferably based on neural network technology. The data supply from an external source ensures that the latest data is always available. For example, the opening hours of a place of interest may change, or a car may be a new model. These data can preferably be processed on a remote server or a cloud server. In particular, a remote server provides information. This information can be evaluated in the vehicle. This can be advantageous in areas with poor reception in order to keep the data flow volume as low as possible and thus ensure a short response time for the system. On the other hand, a cloud server is designed in such a way that the actual Data analysis also takes place on the cloud server. This reduces the computational capacity of the car. It should be noted that a car usually has a significantly longer lifespan than a computer device. By outsourcing the processing unit to a cloud server, it can be prevented in the case of a car that has been in operation for several years that the car or its processing unit is too old or too slow for the required data flow. Neural network technology has the advantage that it can adapt. For example, cars can have different contours due to fashion. With a recurrent analysis, the neural network technology can identify these contours faster than vehicle contours.

According to a modified embodiment of the invention, the field of view of the front camera of the vehicle is supplemented by a sensor system using radar, LIDAR, ultrasound, stereo vision and / or infrared technology. This increases the measurement accuracy. These sensors have been found to be useful for measuring distances to objects and cropping the correct area.

The invention also relates to a computer program with program code instructions for executing the steps of the method according to one of the above measures.

These and other aspects of the invention will be apparent and illustrated in the embodiments described below. Individual features that are shown in the embodiments can form an aspect of the present invention on their own or in combination. Features of the various embodiments can be transferred from one embodiment to another embodiment.

• 1 ein Ablaufdiagramm gemäß einer bevorzugten Ausführungsform der Erfindung; und
• 2 beispielhaft, wie die Blickrichtung eines Benutzers bezüglich Objekten digitalisiert wird, um ein Bild des Sichtfelds der Frontkamera des Fahrzeugs zuzuschneiden, gemäß der Ausführungsform von 1.

Show it:

• 1 a flow chart according to a preferred embodiment of the invention; and
• 2 an example of how the direction of view of a user with respect to objects is digitized in order to crop an image of the field of view of the front camera of the vehicle, according to the embodiment of FIG 1 .

1 shows a flow chart according to a preferred embodiment of the invention.

In particular shows 1 a preferred method of responding to a user query involving an object O1 , O2 , O3 or an area outside the vehicle, with the following steps: First, a user request 100 according to an object O1 , O2 , O3 or an area through an assistant module 1 recorded at a time t. Here, time t is the time at which the user request is recorded.

As a next step, it is checked 200 according to the requested object or area, whether more context information is required compared to the existing information in order to obtain a check result.

If a check result is false F, the user request is processed 300 and information output 400 . This is done according to the state of the art.

If a test result is true T is, it is checked whether the user request is an open or a partially open request. An open request has no reference keyword. A partially open request contains at least one reference keyword.

In the case of an open request, a gaze-controlled interaction module 2 the direction of view G of the user captures and information about an object O1 at the point of intersection of the gaze vector with at least one field of view of a front camera of a vehicle with respect to the point in time t retrieved 500 . In general, after this step, an image of the field of view of the front camera of the vehicle is cropped 600 .

Capturing the direction of view G of the user can be performed with an eye-tracker system which is part of the eye-controlled interaction module 2 can be. When getting information about the object O1 at the intersection of the gaze vector with at least the field of view of the front camera of the vehicle at time t 500 is unsuccessful, which means that the object recognition result is false F, a standard-sized portion of the image is cropped 600a . This ensures that a wide image is recorded on which the object seen and requested by the user is somewhere O1 should be present. So this object stands O1 available in a further analysis step despite the lack of identification. In contrast, when getting information about the object O1 at the intersection of the gaze vector with at least the field of view of the front camera of the vehicle with respect to time t 500 is successful, which means that the object recognition result is true T, an image section, its size from the apparent distance and the apparent size of the cropped object O1 is calculated, tailored 600b . It can thus be ensured that an image of the object viewed by the user O1 is recorded as accurately and comprehensively as possible. Due to the elimination of unnecessary image information, further analysis steps can be carried out with less Resources are executed, ie faster, more energy-saving and, in particular, more precisely.

After cropping a portion of the image at the standard size 600a or after cropping an image section, its size from the apparent distance and the apparent size of the object O1 is calculated at the intersection 600b , the next step is a general image recognition process with regard to the image by an image recognition module 3 requested 900 .

In the case of a partially open request, a gaze-controlled interaction module 2 the direction of view G of the user captures and information about an object O1 at the point of intersection of the gaze vector with at least the field of view of a front camera of a vehicle with respect to the point in time t retrieved 500 . In general, after this step, an image of the field of view of the front camera of the vehicle is cropped 600 .

Capturing the direction of view G of the user can be executed through an eye-tracker system which is part of the eye-controlled interaction module 2 can be. When getting information about the object O1 at the intersection of the gaze vector with at least the field of view of the front camera of the vehicle with respect to time t 500 is unsuccessful, which means that the object recognition result is false F, a portion of the image is cropped with the standard size 600a . This ensures that a wide image is recorded on which the object seen and requested by the user is somewhere O1 should be present. So this object stands O1 available in a further analysis step despite the lack of identification. In contrast, when getting information about the object O1 at the intersection of the gaze vector with at least the field of view of the front camera of the vehicle with respect to time t 500 is successful, which means that the object recognition result is true T, an image section cropped, its size from the apparent distance and the apparent size of the object O1 is calculated at the intersection 600b . It can thus be ensured that an image of the object viewed by the user O1 is recorded as accurately and comprehensively as possible. Because unnecessary image information is omitted, further analysis steps can be carried out with fewer resources, ie faster, more energy-saving and, in particular, more precisely.

A situation for detecting the line of sight G of the user and to get information about an object O1 at the point of intersection of the gaze vector with at least the field of view of a front camera of a vehicle with respect to time t 500 through the gaze-controlled interaction module 2 is in 2 shown. Accordingly, there is a traffic situation in which there are at least three objects O1 , O2 , O3 outside of your own vehicle. When the user has made an inquiry and the next step is the above-mentioned step, it is recognized that the line of sight G of the user to the object O1 leads. The other objects O2 and O3 are irrelevant. Dashboard objects too DO are irrelevant and not part of the regular detection range of the viewing direction G of the user. As a next step, a section of the image is cut, its size based on the apparent distance and the apparent size of the object O1 is calculated at the intersection 600b to get a detailed picture of the object O1 to obtain. The capture of the object O1 can be simplified if it is a moving object and the background is stationary. Capturing could be more difficult if the object O1 is in a traffic jam and motionless with respect to the background, or when it is foggy or rainy.

Furthermore, it is in accordance 1 provided that after cropping an image section of the standard size 600a or after cropping an image section, its size from the apparent distance and the apparent size of the object O1 is calculated at the intersection 600b , the general image recognition process regarding the image is requested 900 and, if possible, also a specialized object recognition process for the at least one reference key word by an image recognition module 3 is requested 800 . The last step mentioned above goes at least one level deeper and tries to get precise information about the object. An example could be that not only the object class, for example car, is requested, but also the model of the car considered, for example manufacturer or color of the car. However, this query can only be answered with a reference keyword that is somehow available, especially with regard to the superordinate category.

For this reason, information of the at least one reference keyword is extracted in this case 700 to request the general image recognition process regarding the image 900 and in particular a specialized object recognition process for the at least one reference keyword by the image recognition module 3 to request 800 .

If after requesting the specialized object recognition process for the at least one reference keyword 800 or after requesting the general image recognition process on the image 900 an image-related suggestion P will be any image related proposals 1000 merged. In this step, all new context information is combined into groups.

After merging any image related suggestions 1000 the new context information of the existing information according to the user request by the assistant module becomes the new context information 1 added 1100 . In this step, all information to be output is grouped together. Finally, the user request is processed 300 and output the information 400 .

In the event that no image-related suggestion NP is obtained after the general image recognition process 900 was requested, the user request 300 processed and the information 400 issued. In this case, the main assumption may be that the information is that no information can be output.

According to a modified embodiment of the invention, the request for the general image recognition process relating to the image by the image recognition module 3 trigger at least one reference keyword, so that requesting a specialized object recognition process for the at least one reference keyword is the next step 800 is. Because by requesting the general image recognition process regarding the image 900 through the image recognition module 3 If a reference keyword can be found, it is possible to subsequently change an open request to a partially open request without the user having precise knowledge. For example, when a user asks what kind of object the viewed object is O1 can by requesting the general image recognition process on the image 900 a car on a tow truck can be identified by an external database. The system can use the information “car on a tow truck” in a next step for requesting a special object recognition process internally as the at least one reference keyword 800 to find out exactly what model this car or tow truck is.

Requesting the general image recognition process on the image 900 and / or requesting a specialized object recognition process for the at least one reference keyword 800 are executed externally, with the in 1 image recognition module shown as an example 3 a remote server or a cloud server, preferably using a database, preferably based on neural network technology.

The data supply from an external source ensures that the latest data is always available. For example, the opening hours of a place of interest may change, or a car may be a new model. These data can preferably be processed on a remote server or a cloud server. In particular, a remote server provides information. This information can be evaluated in the vehicle. This can be advantageous in areas with poor reception in order to keep the data flow volume as low as possible and thus ensure a short response time for the system. On the other hand, a cloud server is configured in such a way that the actual data analysis also takes place on the cloud server. This reduces the computational capacity of the car. It should be noted that a car usually has a significantly longer lifespan than a computer device. By outsourcing the computing unit to a cloud server, it can be prevented in a car that has been in operation for several years that the car or its computing unit is too old or too slow for the required data flow.

Neural network technology has the advantage that it can adapt. For example, cars can have different contours due to fashion. With a recurrent analysis, the neural network technology can identify these contours faster than vehicle contours.

It is possible that requesting the general image recognition process regarding the image 900 is carried out primarily in the vehicle and only externally when there is further demand, with the request of a specialized object recognition process for the at least one reference keyword 800 is always executed externally.

This can represent a meaningful distribution of resources insofar as the step of requesting the general image recognition process with respect to the image 900 can be used for an initial query of simple terms or objects. For example, a car can be relatively easily identified on the basis of certain patterns, whereas the requested model of a car can have been manufactured according to the year of construction of the ego vehicle, so that the step of requesting a specialized object recognition process for the at least one reference keyword 800 should be processed by an external database that is up-to-date. Instead of a car, other examples of data in a database can be brands or logos, buildings or special architectures. Other equivalent solutions are also envisaged.

According to a modified embodiment of the invention, as in 1 is shown, the user request is a voice-based user request.

This can increase driving safety because, for example, the driver can keep his hands on the steering wheel at all times. This also leads to an intuitive operation of the entire system.

Additionally or as an alternative to the in 1 The illustrated embodiment of the invention, the user request is a gesture-based user request. This can be advantageous if, for example, a hands-free call is being made in the vehicle. For example, the user might wish to receive information about the environment for orientation purposes. Otherwise the phone call would have to be broken off, which does not have to be done with the proposed measure.

In other words, it is possible for the system to be configured to capture voice-based or gesture-based user requests.

The invention also relates to a system for applying the above-mentioned method according to one of the above measures, with a device according to one of the above features. Such a system enables the user to have a safe driving experience, and precise information can be made available to the user at any time if necessary.

According to a modified embodiment of the invention, the interaction module 2 an eye tracker and / or a gesture tracker. This enables an analysis of the user based on his gaze and / or his gestures. The user therefore does not have to make a great effort to make his inquiries and can concentrate better on the traffic.

According to a modified embodiment of the invention, the assistant module 1 a voice assistant module. This can increase driving safety because, for example, the driver can keep his hands on the steering wheel at all times. This also leads to an intuitive operation of the entire system.

According to a modified embodiment of the invention, the assistant module 1 a gesture assistant module.

This can be advantageous if, for example, a hands-free call is being made in the vehicle. For example, the user may wish to receive information about the environment for orientation purposes. Otherwise the phone call would have to be broken off, which does not have to be done with the proposed measure.

According to a modified embodiment of the invention, an internal memory unit of a vehicle is used to call up information offline, this information in particular containing places of interest. This can be advantageous in areas with poor reception in order to keep the data flow volume as low as possible and thus ensure a short response time for the system. The internal memory can optionally be updated in order to receive current information even after a longer period of time.

According to a modified embodiment of the invention, the field of view of the front camera of the vehicle is supplemented by a sensor system using radar, LIDAR, ultrasound, stereo vision and / or infrared technology. This increases the measurement accuracy. For example, it can be difficult at night to evaluate an image that was recorded exclusively by a camera. Other sensory systems can help here.

List of reference symbols

1

Assistant module

2

Eye-controlled interaction module

3

Image recognition module

100

Capture a user request

200

Check if more information is required

300

Process the user request

400

Output of information

500

Detecting the direction of view of the user and retrieving information about an object at an intersection of the gaze vector with at least the field of view of the front camera of a vehicle with regard to the time of the user request

600

Cropping an image of the field of view of the vehicle's front camera

600a

Cropping a section of an image in a standard size

600b

Cropping a section of an image, the size of which is calculated from the apparent distance and the apparent size of the object at the point of intersection

700

Extracting information from the at least one reference keyword

800

Requesting a specialized object recognition process for the at least one reference keyword

900

Requesting a general image recognition process regarding the image

1000

Merge any image-related suggestions

1100

Adding the new context information

F.

Not correct

T

True

t

Time at which a user request is recorded

G

Direction of view of the user

P

Image-related suggestion

NP

No image related suggestion

O1

Object 1

O2

Object 2

O3

Object 3

DO

Dashboard object

QUOTES INCLUDED IN THE DESCRIPTION

This list of the documents listed by the applicant was generated automatically and is included solely for the better information of the reader. The list is not part of the German patent or utility model application. The DPMA assumes no liability for any errors or omissions.

Patent literature cited

• DE 102016221983 A1 [0004]

Claims (10)

Method for answering a user request which is related to an object (O1, O2, O3) or an area outside the vehicle, with the following steps: - Detection of a user request (100) by an assistant module (1) at a point in time (t ); - Checking whether more context information relating to the requested object or area is required compared to the existing information (200) in order to obtain a checking result; and if a test result is true (T), checking whether the user request is an open or a partially open request, an open request having no reference keyword and a partially open request having at least one reference keyword; - In the case of an open request, or a partially open request, recording the direction of view (G) of the user and retrieving information about an object (O1) at the intersection of the view vector with the field of view of the front camera of a vehicle with respect to a point in time (t) (500 ) through a gaze-controlled interaction module (2) and cropping an image of the field of view of the front camera of the vehicle (600) by the gaze-controlled interaction module (2); Requesting a general image recognition process relating to the image (900) by an image recognition module (3) after cropping an image of the field of view of the front camera of the vehicle (600); characterized by , in the case of a partially open request, extracting information from the at least one reference keyword (700) in order to request (900) the general image recognition process relating to the image by the image recognition module (3). Procedure according to Claim 1 , characterized by - merging any image-related suggestions (P) (1000) in the case of receipt of an image-related suggestion (P) from some recognition process (800, 900), - after merging any image-related suggestions (1000) to new context information: adding the new Context information (1100) according to the user request by the assistant module (1) for the existing information, whereupon the user request (300) is processed and information (400) is output; and - processing the user request (300) and outputting information (400) if no image-related proposal is received (NP) after the general image recognition process (900) has been requested. Procedure according to Claim 1 or 2 , characterized by - in the case of an open request the cropping of an image of the field of view of the front camera of the vehicle (600) has two options, namely - if the retrieval of information about the object (O1) at the intersection of the gaze vector with at least the field of view of the front camera of the Vehicle is unsuccessful (F) with respect to time (t) (500), an image section is cropped with a standard size (600a); - If the retrieval of information about the object (O1) at the intersection of the gaze vector with at least the field of view of the front camera of the vehicle with respect to the time (t) (500) is successful (T), an image section is cut, the size of which is based on the apparent distance and the apparent size of the object (O1) at the intersection is calculated (600b); - wherein after cropping an image section with the standard size (600a) or after cropping an image section, the size of which is calculated from the apparent distance and the apparent size of the object (O1) at the intersection point (600b), the general image recognition process with regard to the image ( 900) is requested by the image recognition module (3). Method according to one of the preceding claims, characterized by - in the case of a partially open request, the cropping of an image of the field of view of the front camera of the vehicle (600) has two options, namely - if information about the object (O1) is called up at the intersection the gaze vector is unsuccessful (F) with at least the field of view of the front camera of the vehicle with respect to the point in time (t) (500), an image section with a standard size is cropped (600a); - If the retrieval of information about the object (O1) at the intersection of the gaze vector with at least the field of view of the front camera of the vehicle with respect to the time (t) (500) is successful (T), an image section is cut, the size of which is based on the apparent distance and the apparent size of the object (O1) at the intersection is calculated (600b); - after cropping an image section with the standard size (600a) or after cropping an image section, the size of which is calculated from the apparent distance and the apparent size of the object (O1) at the intersection point (600b), the general image recognition process relating to the image ( 900) requested by the image recognition module (3). Method according to one of the preceding claims, characterized in that in the case of a partially open request - After cropping an image of the field of view of the front camera of the vehicle (600) by an image recognition module (3), a general image recognition process is requested with regard to the image (900) and, if this is possible, a special object recognition process for the at least one reference key word (800) is requested; - wherein the extraction of information of the at least one reference keyword is carried out (700) in order to request (900) the general image recognition process relating to the image by the image recognition module (3) and, if this is possible, also to carry out a specialized object recognition process for the at least one reference keyword request (800). Method according to one of the preceding claims, characterized in that the requesting of the general image recognition process with regard to the image (900) by the image recognition module (3) can trigger at least one reference keyword, so that the request of a specialized object recognition process for the at least one reference keyword is the next step ( 800) is. Method according to one of the preceding claims, characterized in that the requesting of the general image recognition process with regard to the image (900) and / or the requesting of a specialized object recognition process for the at least one reference key word (800) are carried out externally, the image recognition module (3) being a separate one Server or a cloud server or at least one such server is used. Method according to one of the preceding claims, characterized in that the requesting of the general image recognition process with respect to the image (900) is primarily carried out in the vehicle and only when there is a further request externally, with the requesting of a specialized object recognition process for the at least one reference keyword (800) always is executed externally. Method according to one of the preceding claims, characterized in that the user request is spoken by a user or is a gesture-based user request. Computer program with program code instructions for carrying out the steps of the method according to one of the Claims 1 to 9 .

Images