JP6526696B2 - System and method for transmitting camera based parameters without using dedicated back channel - Google Patents

Description

This application claims priority to US Provisional Application No. 61 / 951,597, filed March 2014, the entire contents of which are incorporated by reference. Incorporated herein.

BACKGROUND The present invention relates to vehicle systems and methods for transmitting camera based parameters to an end point for display in a vehicle along with video images from a video camera mounted on the vehicle.

  In a state-of-the-art camera system, the endpoint needs to receive more than the image data from the camera. In the driver assistance device, the camera scans the surrounding environment to detect potential hazards and warns the driver in advance. In this device, it is required that the relationship between the distance in the real world and the coordinates of the pixels in the image be sufficiently known and accurate. In such an apparatus, it is required to calibrate each camera individually and store the parameters of these cameras directly in the non-volatile memory of each camera.

  In the case of cameras used in driver assistance applications, functional safety is even more important. In order to verify the functionality of the image sensor, specific camera information is sent to the remote processing unit of the vehicle, which verifies the functional performance of the image sensor.

  Several important issues arise when implementing driver assistance functions on low cost video camera hardware. First, low-cost video camera systems are currently being developed around VGA sensors, which are based on the National Television System Committee (NTSC) method of analog Transmit video Because NTSC video transmission was developed in the mid 20th century, it has not traditionally been permitted to embed metadata in video streams (a feature that is often permitted in digital video transmission). In the case of the NTSC system, the standard line resolution for one frame is 525. One frame can be represented as 720 horizontal pixels × 486 vertical pixels. Compression algorithms, including digital video compression, allow for improved resolution.

  Another analog video transmission scheme used in various countries is the phase reversal line (PAL) scheme. The standard vertical line resolution for the PAL system is 576 for one frame. One frame is 720 × 576 pixels.

  In automotive systems, most electronic control units (ECUs) are connected to the mobile machine bus (i.e. SPI5, DSI, SENT, CAN, FLEXRAY, LIN). However, in some multi-camera systems such as surround view video systems, bus connection to each video camera is not desirable due to the increase in the number of components and the increase in wiring cost. Thus, another approach is contemplated.

Overview In one embodiment, in accordance with the present invention, data associated with a video camera is embedded in a dynamic video stream to overcome the limitations associated with low cost video camera systems and the video stream at the remote ECU It is separated from Data stored in non-volatile memory can be embedded and encoded on the delivered video stream.

  In one embodiment of the invention, camera-based parameters and a video stream from a video camera arranged to observe a given area of the vehicle are transmitted to an electronic control unit mounted on said vehicle In the method, generating a video stream using an image sensor mounted on the vehicle, and transmitting camera-based parameters on the video stream to convey the video stream and the camera-based parameters A method is presented that includes providing a signal, transmitting the analog signal to the electronic control unit, and recovering the camera-based parameters from the analog signal.

  In another embodiment, the area includes an area outside the vehicle observed from the vehicle, and the video camera controls the image sensor mounted on the vehicle and the camera-based parameters. An apparatus, and an image signal processing apparatus for transmitting the camera-based parameters on the video stream to provide the analog signal, the method further comprising, in part, the camera-based parameters. Displaying the video stream on a vehicle display.

  In one embodiment, the analog signal includes either an NTSC analog signal or a PAL analog signal. In another embodiment, the analog signal comprises an NTSC analog signal, and the camera based parameter comprises: a pixel provided in a first horizontal line at the beginning of a frame of the NTSC analog signal; It is supplied as one or both of the pixels provided on the last horizontal line at the end of the frame of the NTSC analog signal. This pixel data is decoded downstream for automotive applications. In another embodiment, the analog signal is a PAL analog signal, and the camera-based parameter is a pixel provided on a first horizontal line at the beginning of a frame of the PAL analog signal, and / Alternatively, it may be supplied as one or both of the pixels provided on the last horizontal line at the end of the frame of the PAL analog signal.

  In another embodiment, dynamic information is read from an image sensor and converted into pixels, which are encoded on the video stream to be delivered. Pixels with non-video parameters are encoded in the corner region of a frame of an NTSC analog signal, which is a 180 degree video image obtained from the driver's field of view. These pixels are decoded at the downstream ECU for automotive applications. In one embodiment, the analog output video camera is a PAL analog output video camera, and non-video pixels are encoded in the corner region of a frame of the PAL analog signal.

  In another embodiment of the present invention, a sequence of data is provided which is a non-video data identifier. The non-video data identifier is to signal that camera-based parameters follow in sequence.

  In one embodiment, the camera-based parameters include at least one of camera calibration parameters and camera self-monitoring parameters. In one embodiment, the camera calibration parameters include lens characteristics and the self-monitoring parameters include at least one of a frame counter and a temperature value for the video camera, in addition to the video The camera includes one of a plurality of video cameras connected to the electronic control unit.

  In another embodiment, the video camera includes an image sensor, a control device having a non-volatile memory, and an image signal processing device for supplying the analog signal to the electronic control unit. The said area | region which is said is an indoor area | region of the said vehicle.

  In one embodiment, the camera based parameters retrieved from the analog signal control the display of the video stream on a vehicle display. In another embodiment, the analog signal is transmitted on a single channel to the electronic control unit and does not use a dedicated back channel.

  In one embodiment of the present invention, in a vehicle system transmitting camera-based parameters and a video stream, the vehicle system includes a video camera disposed in the vehicle for observing a predetermined area of the vehicle. The video camera includes an image sensor for generating a video image of the area of the vehicle and acquiring a video stream, a controller for providing camera-based parameters, and an image signal processor. The image signal processing apparatus receives the video stream from the image sensor, receives camera-based parameters from the controller, transmits the camera-based parameters on the video stream, and the video stream and the video stream Configured to output an analog signal that includes camera-based parameters and The vehicle system further includes an electronic control unit including a video receiver, the electronic control unit configured to receive the analog signal from the video camera and recover the camera based parameters from the analog signal. It is done.

  In one embodiment, the vehicle display is configured to receive the video stream provided by the electronic control unit and to display the video stream to a vehicle operator in the vehicle, the vehicle display comprising: The area includes an area outside the vehicle observed from the vehicle.

  In one embodiment, the camera-based parameters are: a pixel provided in the first horizontal line at the beginning of the frame of the analog signal and a pixel provided in the last horizontal line at the end of the frame of the analog signal , Or as one or both.

  In another embodiment, the camera-based parameters are embedded as pixels at corners of frames of video images of the video stream carried by the analog signal.

  In one embodiment, the camera based parameters are transmitted on the video stream following the non-video data identifier transmitted on the video stream.

  In one embodiment, the camera-based parameter includes at least one of a camera calibration parameter and a camera self-monitoring parameter, and the video camera is one of a plurality of video cameras connected to the electronic control unit. including. In one embodiment, the video stream is output based on the camera parameters.

  In another embodiment, the control device includes non-volatile memory, and the video camera is configured to transmit the analog signal to the electronic control unit on a single channel, the vehicle The system does not use a dedicated back channel.

  Other aspects of the invention will become apparent upon consideration of the detailed description and accompanying drawings.

1 is a block diagram of a vehicle having a plurality of video cameras. It is a block diagram of a video camera and an electronic control unit. FIG. 7 shows a 180 degree video image containing transmitted non-video data to be displayed. FIG. 7 shows a 180 degree video image including non-video data encoded on one line of the image.

DETAILED DESCRIPTION Before describing in detail every embodiment of the present invention, the present invention will be limited in its application to the details of construction and the arrangement of the components described in the following description or illustrated in the following drawings. It should be understood that it is not The invention is capable of other embodiments and of being practiced or of being carried out in various ways.

  As shown in the block diagram of FIG. 1, the vehicle 18 includes an electronic control unit (ECU) 20, which has a front view video camera 22 disposed around the vehicle and a left side The respective video streams are received from the view video camera 24, the right side view video camera 26, and the rear view video camera 28. Each video camera acquires a video stream from a respective out-of-vehicle area of the vehicle 18. The ECU 20 receives video images from the respective video cameras, processes these video images to obtain one surround view image of the vehicle, and this surround view image is output on the vehicle display 30. In this manner, the vehicle system provides the vehicle display 30 with an image of the entire vehicle 18. In some embodiments, a single video camera provides the video stream to the ECU 20, which outputs a single video camera view on the vehicle display 30 for viewing by the driver at the driver's seat. In another embodiment, a video camera is arranged to observe the interior area of the vehicle 18. It is contemplated to provide any number of video cameras.

  FIG. 2 shows a block diagram of a rear view video camera 28 for providing a video image embedded with camera based parameters to a video receiver of the ECU 20. The rear view video camera 28 includes an image sensor 40, a controller 42 connected to the non-volatile memory 44, and a video or image signal processor 46. The video or image signal processor 46 transmits camera-based parameters to the ECU 20 in combination with the video stream. FIG. 2 illustrates a rear view video camera 28 in which an image sensor 40, a control unit 42, a non-volatile memory 44 and an image signal processing unit 46 are individually provided in the video camera 28. In some embodiments, these devices are configured as a single component.

  In operation, controller 42 obtains camera-based parameters, such as metadata or performance metrics, from non-volatile memory 44 and provides them to image signal processor 46. Camera-based parameters may be camera calibration parameters and / or dynamic camera self-monitoring parameters in addition to metadata and measurement performance criteria for video camera 28. In some embodiments, the calibration parameters include lens characteristics of the lens of video camera 28. In one embodiment, the self-monitoring parameters include a frame counter. In one embodiment, the self-monitoring parameters include the temperature detection value of the image sensor 40 and / or the voltage value of each component of the video camera 28. These parameters can indicate the camera status of the video camera 28 and potential failures. The image signal processor 46 is configured to encode or embed the data parameters supplied by the controller 42 onto the video stream from the image sensor 40. The image signal processing device 46 is configured to transmit the encoded analog video stream or analog signal to the ECU 20. In some embodiments, dynamic information from the image sensor 40 is converted to a live or near real time delivery video stream to form an analog signal, or a live or near real time delivery video stream Embedded / encoded on top.

  Non-video data representing camera-based parameters supplied to the ongoing video stream is separated from the analog signal at the ECU 20. The ECU 20 recovers camera-based parameters from the analog signal to obtain a video stream. More specifically, non-video data representing camera-based parameters derived from the non-volatile memory 44 or the sensor of the image sensor 40 is decoded. The ECU 20 includes a video receiver and a processor, which is configured to separate camera-based non-video data and video streams from the analog signal. In some embodiments, the ECU 20 is also configured to control the vehicle display 30 with partial consideration of non-video data, or other information related to camera-based parameters, etc. Supply.

  The transmitted or embedded signal can be visualized in multiple ways, but there are different specific ways that may prove to be the most appropriate. In one method, the transmitted or embedded pixels are placed in a region of the video image of the video stream that does not degrade system functionality. For example, as shown in FIG. 3, in a panoramic camera of 180 degrees, corresponding data is provided in vignettes present at each corner of the image. While some may consider this data, shown in the corner of FIG. 3, to be an overlay, such an interpretation can be misleading. Although non-video data is shown in FIG. 3, this data is not displayed on the vehicle display 30 provided in the driver's field of view and therefore does not function as an overlay. Thus, data can be overwritten without causing noticeable degradation of the video system.

  In any NTSC video camera, data regarding camera-based parameters can be encoded as pixels on a single line of an NTSC analog signal. The embedded pixels mask a single image line and use known NTSC levels to encode data for associated camera parameters. This arrangement is illustrated in FIG. 4, where the first horizontal line at the top of the image is the encoded data. Therefore, the horizontal line at the top of FIG. 4 looks black. At the end of the frame, which represents the beginning of the frame and / or the last horizontal line, the vehicle display 30 displays a line of non-video data if a horizontal line for camera-based parameter data is selected. The ECU 20 is programmed to prevent that. Thus, as in the embodiment of FIG. 3, encoded non-video data is not displayed on the vehicle display 30.

  In another embodiment, a PAL video camera that outputs a PAL analog signal is provided. Camera-based parameters can be encoded as pixels on one line of the PAL analog signal or as pixels on both the first and last lines of the PAL frame. The embedded pixels mask a single image line and use known PAL levels to encode the associated camera parameter data. Thus, an embodiment using a PAL video camera operates in the same manner as an NTSC video camera.

  In another embodiment, a sequence of data, which is a non-video data identifier, provided in the analog signal from video camera 28 is embedded in the video stream by image signal processor 46. Thereafter, the image signal processor 46 similarly transmits camera-based parameters on the video stream. The non-video data identifier acts as a flag / marker to signal that a camera based parameter will follow. The processing unit of electronic control unit 20 is programmed to retrieve non-video data identifiers during processing of the analog signal. Once the non-video data identifier is identified, a predetermined number of subsequent pixels are not displayed either, but instead are treated as camera-based parameters. Non-video data is removed from the analog signal such that the video stream remains for display.

  By combining the camera parameters with the video stream to form an analog signal, video camera 28 transmits the resulting analog signal to ECU 20 on a single channel. Thus, the system does not use a dedicated back channel, as it does not require a dedicated back channel.

  Thus, the present invention specifically provides camera-based non-video parameters as "pixels" in the analog video stream, in addition to encoding camera-based parameters onto the downstream decoded analog video stream. Vehicle system and method. Various features and advantages of the present invention are set forth in the appended claims.

Claims (17)

A method of transmitting camera based parameters and a video stream from a video camera arranged to observe a predetermined area of a vehicle to an electronic control unit mounted on said vehicle,
Generating a video stream using an image sensor mounted on the vehicle;
Transmitting camera-based parameters over the video stream to provide an analog signal carrying the video stream and the camera-based parameters;
Transmitting the analog signal to the electronic control unit;
Recovering the camera based parameters from the analog signal;
Displaying the video stream supplied by the electronic control unit on a vehicle display;
Including
The camera-based parameters include camera self-monitoring parameters
The camera self-monitoring parameters include a frame counter
It said video camera is viewing contains one of a plurality of video cameras connected to the electronic control unit,
The analog signal includes an analog signal of NTSC system,
The camera-based parameters are: a pixel provided in the first horizontal line at the beginning of the frame of the NTSC analog signal, and a pixel provided in the last horizontal line at the end of the frame of the NTSC analog signal, Supplied as one or both of
The electronic control unit is programmed to block the vehicle display from displaying non-video data lines.
A method characterized by Before SL region comprises a vehicle exterior area observed from the vehicle,
The video camera transmits the camera-based parameters on the video stream to provide the analog signal, with the image sensor mounted on the vehicle, a controller for receiving the camera-based parameters, and And an image signal control device for
The method further includes the camera-based and parameters based in part on, displays the video stream on the vehicle display,
The method of claim 1. Before Symbol camera-based parameters are embedded as a pixel at a corner of the frame of the image carried by the analog signal of the NTSC system,
The method of claim 1. The camera based parameters are transmitted on the video stream following the non-video data identifier transmitted on the video stream
The method of claim 1. The camera based parameters further include camera calibration parameters
The method of claim 1. The camera calibration parameters include lens characteristics
The method of claim 5 . The camera self-monitoring parameters further include a temperature value for the video camera
The method of claim 1. The video camera includes the image sensor, a control device having a non-volatile memory, and an image signal processing device for supplying the analog signal to the electronic control unit.
The area to be observed is an indoor area of the vehicle,
The method of claim 1. The method includes providing the camera based parameters retrieved from the analog signal to control the display of the video stream on a vehicle display.
The method of claim 1. Transmitting the analog signal to the electronic control unit includes transmitting the analog signal on a single channel,
The method does not use a dedicated back channel,
The method of claim 1. In a vehicle system transmitting camera-based parameters and a video stream,
The vehicle system includes a video camera disposed on the vehicle for observing a predetermined area of the vehicle, the video camera comprising
An image sensor for generating a video image of the area of the vehicle and acquiring a video stream;
A controller for supplying camera based parameters;
An image signal processing device;
And the image signal processing device
Receiving the video stream from the image sensor;
Receive camera-based parameters from the controller;
Transmitting the camera-based parameters over the video stream to output an analog signal comprising the video stream and the camera-based parameters;
The vehicle system further includes an electronic control unit including a video receiver, the electronic control unit including:
Receiving the analog signal from the video camera;
Configured to recover the camera based parameters from the analog signal;
The vehicle system further includes a vehicle display, the vehicle display comprising:
Receiving the video stream provided by the electronic control unit and displaying the video stream
Is configured as
The camera-based parameters include camera self-monitoring parameters
The camera self-monitoring parameters include a frame counter
It said video camera is viewing contains one of a plurality of video cameras connected to the electronic control unit,
The analog signal includes an analog signal of NTSC system,
The camera-based parameters are: a pixel provided in the first horizontal line at the beginning of the frame of the NTSC analog signal, and a pixel provided in the last horizontal line at the end of the frame of the NTSC analog signal, Supplied as one or both of
The electronic control unit is programmed to block the vehicle display from displaying non-video data lines.
Vehicle system characterized by The area before Symbol vehicle includes a vehicle exterior area observed from the vehicle,
A vehicle system according to claim 11 . The camera based parameters are embedded as pixels at the corners of the frames of the video image of the video stream carried by the analog signal.
A vehicle system according to claim 11 . The camera based parameters are transmitted on the video stream following the non-video data identifier transmitted on the video stream
A vehicle system according to claim 11 . The camera based parameters further include camera calibration parameters
A vehicle system according to claim 11 . The electronic control unit is configured to output the video stream based at least in part on the camera based parameters.
A vehicle system according to claim 15 . The controller includes non-volatile memory,
The video camera is configured to transmit the analog signal to the electronic control unit on a single channel,
The vehicle system does not use a dedicated back channel,
A vehicle system according to claim 11 .

Images