CN109733285B

CN109733285B - Vehicle driving state display method, device and system

Info

Publication number: CN109733285B
Application number: CN201910145641.9A
Authority: CN
Inventors: 霍建宇; 饶思粤; 宫博
Original assignee: Beijing Baidu Netcom Science and Technology Co Ltd
Current assignee: Beijing Baidu Netcom Science and Technology Co Ltd
Priority date: 2019-02-27
Filing date: 2019-02-27
Publication date: 2021-05-07
Anticipated expiration: 2039-02-27
Also published as: CN113246858A; CN113246858B; CN109733285A

Abstract

The invention provides a vehicle driving state display method, device and system, wherein the method comprises the following steps: receiving running environment information of a vehicle from a vehicle controller, wherein the running environment information comprises information which is received by the vehicle controller from a first image acquisition device and is obtained according to a first running image; receiving a second driving image of the vehicle from a second image capturing device; and generating a driving state image of the vehicle according to the second driving image and the driving environment information. The invention can be beneficial to visually presenting the driving state in the driving process of the vehicle; the driving environment information of the vehicle controller is visually compared with the second driving image, so that debugging personnel can find problems conveniently.

Description

Vehicle driving state display method, device and system

Technical Field

The invention relates to the field of automobile driving, in particular to a method, equipment and a system for displaying a vehicle running state.

Background

The current scheme of image analysis for vehicle driving is to use an intelligent camera to read self-perceived environmental information, such as an equation of a lane line and coordinates of an obstacle, and then use a low-performance processor to make subsequent decision planning and control an automobile according to the environmental information. However, the inventor discovers that the low-performance processor lacks of graphic rendering capability, debugging personnel cannot visually acquire driving environment information, visual verification cannot be achieved, and debugging difficulty of research personnel is increased in the prior art.

Disclosure of Invention

The embodiment of the invention provides a vehicle driving state display method, device and system, and aims to at least solve the technical problems in the prior art.

In a first aspect, an embodiment of the present invention provides a vehicle driving state display method, including:

receiving running environment information of a vehicle from a vehicle controller, wherein the running environment information comprises information which is received by the vehicle controller from a first image acquisition device and is obtained according to a first running image;

receiving a second driving image of the vehicle from a second image capturing device;

generating a driving state image of the vehicle according to the second driving image and the driving environment information;

and displaying the driving state image.

In one embodiment, the driving environment information includes digital information obtained by processing a first driving image of the vehicle by the first image acquisition device;

the driving environment information further includes control information obtained by the vehicle controller performing vehicle route planning using the digital information.

In one embodiment, the digital information includes at least one of lane line information, obstacle information, indicator light information, and sign information;

the control information includes at least one of an accelerator, a brake, a steering, a gear, and a light.

In one embodiment, the method further comprises:

recording the running environment information and the second running image after time alignment to obtain a running state video;

and in the case of receiving a playback instruction, playing back the recorded running state video.

In a second aspect, an embodiment of the present invention provides a vehicle running state display apparatus including:

the driving environment information acquisition module is used for receiving driving environment information of a vehicle from a vehicle controller, wherein the driving environment information comprises information which is received by the vehicle controller from a first image acquisition device and is obtained according to a first driving image;

the second driving image acquisition module is used for receiving a second driving image of the vehicle from a second image acquisition device;

a driving state image generation module, configured to generate a driving state image of the vehicle according to the second driving image and the driving environment information;

and the display module is used for displaying the driving state image.

In one embodiment, the driving state image generation module includes:

a time alignment sub-module for time-aligning the driving environment information and the second driving image;

and the generation submodule is used for generating the driving state image according to the driving environment information and the second driving image after time alignment, and the set area of the driving state image comprises the driving environment information after time alignment.

In one embodiment, the method further comprises:

the recording module is used for recording the running environment information and the second running image after time alignment to obtain a running state video;

and the playback module is used for playing back the recorded running state video under the condition of receiving a playback instruction.

In a third aspect, embodiments of the present invention provide a vehicle driving state display device, where functions of the vehicle driving state display device may be implemented by hardware, or may be implemented by hardware executing corresponding software. The hardware or software includes one or more modules corresponding to the above-described functions.

In one possible design, the vehicle driving state display device is structured to include a processor and a memory, the memory is used for storing a program for supporting the vehicle driving state display device to execute the vehicle driving state display method in the first aspect, and the processor is configured to execute the program stored in the memory. The vehicle running state display device may further include a communication interface for the vehicle running state display device to communicate with other devices or a communication network.

In a fourth aspect, an embodiment of the present invention provides a vehicle driving state display system, including a computing device, a vehicle controller, a first image capturing device, and a second image capturing device; wherein,

the computing device includes the vehicle travel state display device according to any one of the above embodiments;

the first image acquisition equipment is connected with a vehicle controller, and the second image acquisition equipment and the vehicle controller are respectively connected with the computing equipment;

the first image acquisition equipment is used for acquiring a first running image of a vehicle and sending information obtained according to the first running image to the vehicle controller;

the second image acquisition device is used for acquiring a second driving image of the vehicle and sending the second driving image to the computing device;

the vehicle controller is used for generating the running environment information from the information received by the first image acquisition device and sending the running environment information to the computing device.

In one embodiment, the first image acquisition device is further configured to process the first driving image to obtain digital information, and send the digital information to the vehicle controller;

the vehicle controller is further configured to perform vehicle route planning using the digital information to obtain control information, and send the driving environment information including the digital information and the control information to the computing device.

In a fifth aspect, the embodiment of the present invention further provides a computer-readable storage medium for storing computer software instructions for displaying the vehicle driving state of the device, which includes a program for executing the vehicle driving state displaying method.

One of the above technical solutions has the following advantages or beneficial effects: visually presenting the driving state in the driving process of the vehicle for visual presentation; the running environment information of the vehicle controller is compared with the second running image, so that debugging personnel can find problems conveniently.

The foregoing summary is provided for the purpose of description only and is not intended to be limiting in any way. In addition to the illustrative aspects, embodiments, and features described above, further aspects, embodiments, and features of the present invention will be readily apparent by reference to the drawings and following detailed description.

Drawings

In the drawings, like reference numerals refer to the same or similar parts or elements throughout the several views unless otherwise specified. The figures are not necessarily to scale. It is appreciated that these drawings depict only some embodiments in accordance with the disclosure and are therefore not to be considered limiting of its scope.

Fig. 1 is a flowchart of a vehicle driving state display method according to an embodiment of the present invention;

fig. 2 is an exemplary view of a running state image displayed in a running state display method of a vehicle according to an embodiment of the present invention;

fig. 3 is a flowchart of step S3 in a vehicle driving state display method according to an embodiment of the present invention;

fig. 4 is a block connection diagram of a vehicle driving state display apparatus according to an embodiment of the present invention;

fig. 5 is a structural view of a vehicle running state display apparatus according to an embodiment of the present invention;

fig. 6 is a structural diagram of a vehicle driving state display system according to an embodiment of the present invention.

Detailed Description

In the following, only certain exemplary embodiments are briefly described. As those skilled in the art will recognize, the described embodiments may be modified in various different ways, all without departing from the spirit or scope of the present invention. Accordingly, the drawings and description are to be regarded as illustrative in nature, and not as restrictive.

The embodiment of the invention mainly provides a method, equipment and a system for displaying a vehicle running state, and the technical scheme is developed and described through the following embodiments respectively.

An embodiment of the present invention provides a method for displaying a vehicle driving state, referring to fig. 1, where fig. 1 is a flowchart of this embodiment, including:

s1, receiving running environment information of the vehicle from the vehicle controller, wherein the running environment information comprises information which is received by the vehicle controller from the first image acquisition equipment and is obtained according to the first running image;

s2, receiving a second driving image of the vehicle from a second image acquisition device;

s3, generating a driving state image of the vehicle according to the second driving image and the driving environment information;

and S4, displaying the driving state image.

The present embodiment may be applied to an implementation of a computing device, such as a Personal Computer (PC). In the process of vehicle driving debugging, a debugging person can directly connect a computing device, such as a PC, to a vehicle controller of a vehicle and a second image capturing device through a communication bus, and the vehicle controller is connected to a first image capturing device, referring to fig. 6.

In this embodiment, the PC has a display function, can record communication bus data by time, and has a playback function. The communication Bus includes, but is not limited to, Message Queue Telemetry Transport (MQTT), Universal Serial Bus (USB), User Datagram Protocol (UDP), and the like. The vehicle controller may include a low performance processor without image rendering capability, including but not limited to a Micro Control Unit (MCU) and a Field-Programmable Gate Array (FPGA).

The first image capturing device and the second image capturing device may each include one or more cameras disposed on the vehicle. The camera of the first image acquisition device can shoot a first running image comprising a running environment outside the vehicle, and the first running image is processed to obtain corresponding information and sent to the PC; the camera included in the second image acquisition device can shoot a second driving image including a driving environment outside the vehicle and send the second driving image to the PC; the first traveling image and the second traveling image may include a road on which the vehicle is located, an obstacle on the road, a lane line, an indicator lamp, a sign, and the like. Obstacles include, but are not limited to, pedestrians, animals, other vehicles, and the like.

In one embodiment, the driving environment information comprises digital information obtained by processing a first driving image of the vehicle by a first image acquisition device;

the driving environment information also includes control information obtained by the vehicle controller using the digital information to perform vehicle route planning.

In one example, the camera included in the first image capturing device may be an intelligent camera, and the camera included in the second image capturing device may be a general camera. Generally, after the smart camera acquires the second driving image, the processing generates and outputs digital information, which may include but is not limited to: at least one of lane line information, obstacle information, indicator light information, and sign information; the common camera can collect and output a second driving image.

In one example, a method for obtaining digital information by a smart camera includes: various environmental elements included in the running environment of the vehicle, such as lane lines, obstacles, indicator lights, and signs, are recognized from the first running image. Then, based on modeling of the vehicle running environment, the image information is expressed as digital information, for example, the lane line information can be expressed by a lane line equation (y ═ ax ^3+ bx ^2+ cx + d), the equation refers to an equation of a lane line with the vehicle body as an origin and the straight ahead as an x axis, a, b, c and d are unknown parameters, and the unknown parameters are solved by an algorithm; the obstacle information may be represented by a coordinate position (x, y).

The smart camera may send this digital information to the vehicle controller. The vehicle controller utilizes the received digital information to perform vehicle route planning to obtain control information. The control information may include, but is not limited to, at least one of throttle, brake, steering, gear, and light.

Then, the vehicle controller may transmit the running environment information including the digital information and the control information to the PC. The driving environment information of the vehicle and the second driving image can be synthesized by using the graphic rendering capability of the PC, and the synthesized driving state image is displayed through the display of the PC.

In an example, the vehicle controller may also perform some processing on the received digital information, such as denoising, error correction, data format conversion, and the like, to obtain the processed digital information, so as to perform vehicle route planning using the processed digital information, thereby implementing more accurate driving control. At this time, the running environment information may further include final digital information that the vehicle controller derives based on the digital information acquired from the first image pickup device. As shown in fig. 2, the upper left corner of fig. 2 presents digital information in an image, including a second lane line 23 and a second obstacle 24, which are further processed by the vehicle controller on the basis of the information output by the image capturing device, in addition to the first lane line 21 and the first obstacle 22 processed by the image capturing device.

The algorithm adopted by the first image acquisition device is generally established by a camera manufacturer and is not published to the outside, and the algorithm of the vehicle controller can be designed by vehicle developers.

In one example, the vehicle controller stores digital information acquired from the first image acquisition device as a message in a specified format; storing all information generated in the information processing process of the vehicle controller, including processed digital information and control information generated by vehicle route planning, as a log according to a specified format; and send the message and log to the executing computing device.

In the embodiment of the invention, the driving state in the driving process of the vehicle can be intuitively presented by utilizing the graphic rendering capability of the PC without the image rendering capability of the vehicle controller. And the algorithm result of the vehicle controller, such as the driving environment information, is compared with the second driving image, so that debugging personnel can find the driving problem of the vehicle conveniently, and the safety of scenes such as automatic driving and the like is improved.

In one embodiment, the present embodiment may be applied to an autonomous driving backup system as a backup system different from the autonomous driving main system scheme. The scheme of combining the image acquisition equipment, the vehicle controller and the computing equipment is adopted, the running state is visually presented, and the debugging is facilitated. Meanwhile, a backup system is designed to meet the requirement of safety of the automatic driving function.

In one embodiment, referring to fig. 3, step S3 includes:

s31, time alignment is carried out on the driving environment information and the second driving image;

s32, a driving state image is generated based on the time-aligned driving environment information and the second driving image, and the set area of the driving state image includes the time-aligned driving environment information.

In one embodiment, referring to the example diagram of the driving state image of fig. 2, in step S32, the driving state image is obtained by drawing the second driving image into the background area 202 of the driving state image and drawing the driving environment information onto the setting area 201 of the second driving image.

Specifically, the drawing process of step S32 may include writing the data of the running environment information and the second running image into a buffer. In step S4, the displaying the driving state image on the screen includes: rendering the driving state image written in the buffer.

In one embodiment, the present embodiment further comprises the following steps:

s5, recording the running environment information and the second running image after time alignment to obtain a running state video;

s6, when a playback instruction is received, the recorded running state video is played back.

The embodiment provides recording and playback functions, can reflect the processing result of the algorithm at that time, and can reproduce when a research and development personnel encounters a problem. Moreover, the computing device to which the present embodiment is applied is connected to the image capturing device and the vehicle controller via the communication bus, respectively. The different communication buses are encapsulated into an agreed protocol that the image capture device and the vehicle controller can use to transmit data to the computing device. The data adopted by the recording of the embodiment can be directly played back without considering the problem of format conversion, and the debugging difficulty is reduced.

Referring to fig. 4, an embodiment of the present invention provides a vehicle driving state display apparatus including:

a driving environment information obtaining module 41, configured to receive driving environment information of the vehicle from the vehicle controller, where the driving environment information includes information obtained according to the first driving image and received by the vehicle controller from the first image capturing device;

a second travel image acquisition module 42 for receiving a second travel image of the vehicle from the second image capturing device;

a driving state image generation module 43 for generating a driving state image of the vehicle based on the second driving image and the driving environment information;

and the display module 44 is used for displaying the running state image.

In one embodiment, the digital information includes: at least one of lane line information, obstacle information, indicator light information, and sign information;

the control information includes at least one of throttle, brake, steering, gear, and light.

In one embodiment, the driving state image generation module 43 includes:

the time alignment submodule is used for performing time alignment on the driving environment information and the second driving image;

and the generation submodule is used for generating a driving state image according to the time-aligned driving environment information and the second driving image, and the set area of the driving state image comprises the time-aligned driving environment information.

In one embodiment, the method further comprises:

The functions of the modules in the embodiments of the present invention may refer to the corresponding descriptions in the above methods, and are not described herein again.

Referring to fig. 5, an embodiment of the present invention also provides a vehicle driving state display apparatus including:

a memory 11 and a processor 12, the memory 11 storing a computer program operable on the processor 12. The processor 12, when executing the computer program, implements the account classification model training method or the account classification method in the above embodiments. The number of the memory 11 and the processor 12 may be one or more.

The vehicle running state display apparatus may further include:

and the communication interface 13 is used for communicating with external equipment and exchanging and transmitting data.

The memory 11 may comprise a high-speed RAM memory, and may also include a non-volatile memory (non-volatile memory), such as at least one disk memory.

If the memory 11, the processor 12 and the communication interface 13 are implemented independently, the memory 11, the processor 12 and the communication interface 13 may be connected to each other through a bus and perform communication with each other. The bus may be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, an Extended ISA (EISA), or the like. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of illustration, only one thick line is shown in FIG. 5, and does not indicate only one bus or one type of bus.

Optionally, in a specific implementation, if the memory 11, the processor 12 and the communication interface 13 are integrated on a chip, the memory 11, the processor 12 and the communication interface 13 may complete communication with each other through an internal interface.

In the description herein, references to the description of the term "one embodiment," "some embodiments," "an example," "a specific example," or "some examples," etc., mean that a particular feature, structure, material, or characteristic described in connection with the embodiment or example is included in at least one embodiment or example of the invention. Furthermore, the particular features, structures, materials, or characteristics described may be combined in any suitable manner in any one or more embodiments or examples. Furthermore, various embodiments or examples and features of different embodiments or examples described in this specification can be combined and combined by one skilled in the art without contradiction.

Furthermore, the terms "first", "second" and "first" are used for descriptive purposes only and are not to be construed as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one such feature. In the description of the present invention, "a plurality" means two or more unless specifically defined otherwise.

Any process or method descriptions in flow charts or otherwise described herein may be understood as representing modules, segments, or portions of code which include one or more executable instructions for implementing specific logical functions or steps of the process, and alternate implementations are included within the scope of the preferred embodiment of the present invention in which functions may be executed out of order from that shown or discussed, including substantially concurrently or in reverse order, depending on the functionality involved, as would be understood by those reasonably skilled in the art of the present invention.

The logic and/or steps represented in the flowcharts or otherwise described herein, e.g., an ordered listing of executable instructions that can be considered to implement logical functions, can be embodied in any computer-readable medium for use by or in connection with an instruction execution system, apparatus, or device, such as a computer-based system, processor-containing system, or other system that can fetch the instructions from the instruction execution system, apparatus, or device and execute the instructions. For the purposes of this description, a "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transport the program for use by or in connection with the instruction execution system, apparatus, or device. More specific examples (a non-exhaustive list) of the computer-readable medium would include the following: an electrical connection (electronic device) having one or more wires, a portable computer diskette (magnetic device), a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber device, and a portable read-only memory (CDROM). Additionally, the computer-readable medium could even be paper or another suitable medium upon which the program is printed, as the program can be electronically captured, via for instance optical scanning of the paper or other medium, then compiled, interpreted or otherwise processed in a suitable manner if necessary, and then stored in a computer memory.

It should be understood that portions of the present invention may be implemented in hardware, software, firmware, or a combination thereof. In the above embodiments, the various steps or methods may be implemented in software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, any one or combination of the following techniques, which are known in the art, may be used: a discrete logic circuit having a logic gate circuit for implementing a logic function on a data signal, an application specific integrated circuit having an appropriate combinational logic gate circuit, a Programmable Gate Array (PGA), a Field Programmable Gate Array (FPGA), or the like.

It will be understood by those skilled in the art that all or part of the steps carried by the method for implementing the above embodiments may be implemented by hardware that is related to instructions of a program, and the program may be stored in a computer-readable storage medium, and when executed, the program includes one or a combination of the steps of the method embodiments.

In addition, functional units in the embodiments of the present invention may be integrated into one processing module, or each unit may exist alone physically, or two or more units are integrated into one module. The integrated module can be realized in a hardware mode, and can also be realized in a software functional module mode. The integrated module, if implemented in the form of a software functional module and sold or used as a separate product, may also be stored in a computer readable storage medium. The storage medium may be a read-only memory, a magnetic or optical disk, or the like.

Referring to fig. 6, an embodiment of the present invention provides a vehicle driving state display system, which includes a computing device 61, a vehicle controller 62, a first image capturing device 631, and a second image capturing device 632; wherein,

the computing device 61 includes the vehicle travel state display device provided in any of the embodiments described above;

the first image acquisition device 631 is connected with the vehicle controller 62, and the second image acquisition device 632 and the vehicle controller 62 are respectively connected with the computing device 61;

the first image capturing device 631 is configured to capture a first travel image of the vehicle, and transmit information obtained from the first travel image to the vehicle controller 62;

the second image capturing device 632 is configured to capture a second driving image of the vehicle, and send the second driving image to the computing device 61;

the vehicle controller 62 is configured to generate the running environment information from the information received by the first image capturing device 631, and transmit the running environment information to the computing device 61.

In one embodiment, the first image capturing device 631 is further configured to process the first driving image to obtain digital information, and send the digital information to the vehicle controller 62;

the vehicle controller 62 is also configured to perform vehicle route planning using the digital information to obtain control information, and to transmit the running environment information including the digital information and the control information to the computing device 61.

For details which are not described in the embodiment of the present invention, reference may be made to corresponding descriptions in the method and the device for displaying a driving state of a vehicle provided in the embodiment of the present invention, and details are not described herein again.

In summary, the vehicle driving state display method, device and system provided by the embodiments of the present invention; in the process of vehicle driving debugging, the driving state in the vehicle driving process can be visually presented without the need that a vehicle controller has image rendering capability; and moreover, the running environment information of the vehicle controller is compared with the second running image, so that debugging personnel can find problems conveniently.

The above description is only for the specific embodiment of the present invention, but the scope of the present invention is not limited thereto, and any person skilled in the art can easily conceive various changes or substitutions within the technical scope of the present invention, and these should be covered by the scope of the present invention. Therefore, the protection scope of the present invention shall be subject to the protection scope of the appended claims.

Claims

1. A vehicle driving state image display method characterized by comprising:

displaying the driving state image;

wherein the generating of the driving state image of the vehicle according to the second driving image and the driving environment information includes: time-aligning the driving environment information and the second driving image; and generating the driving state image according to the driving environment information and the second driving image after time alignment.

2. The method according to claim 1, wherein the driving environment information includes digital information obtained by processing a first driving image of the vehicle by the first image capturing device;

3. The method of claim 2, wherein the digital information includes at least one of lane line information, obstacle information, indicator light information, and sign information;

4. The method according to claim 1, characterized in that the set region of the travel state image includes the travel environment information after time alignment.

5. The method of claim 4, further comprising:

6. A vehicle running state image display apparatus characterized by comprising:

the display module is used for displaying the driving state image;

the driving state image generation module includes:

and the generation submodule is used for generating the driving state image according to the driving environment information and the second driving image after time alignment.

7. The apparatus according to claim 6, wherein the running environment information includes digital information obtained by the first image capturing apparatus processing a first running image of the vehicle;

8. The apparatus of claim 7, wherein the digital information includes at least one of lane line information, obstacle information, indicator light information, and sign information;

9. The apparatus according to claim 6, characterized in that the set region of the travel state image includes the travel environment information after time alignment.

10. The apparatus of claim 9, further comprising:

11. A vehicle running state image display apparatus, characterized by comprising:

one or more processors;

storage means for storing one or more programs;

the one or more programs, when executed by the one or more processors, cause the one or more processors to implement the method recited in any of claims 1-5.

12. A vehicle driving state image display system is characterized in that the system comprises a computing device, a vehicle controller, a first image acquisition device and a second image acquisition device; wherein,

the computing device includes the vehicle running state image display device according to any one of claims 6 to 11;

13. The system of claim 12, wherein the first image capture device is further configured to process the first travel image into digital information, send the digital information to the vehicle controller;

14. A computer-readable storage medium, in which a computer program is stored which, when being executed by a processor, carries out the method according to any one of claims 1-5.