WO2009119337A1 - Image processing device, image processing program, image processing system and image processing method - Google Patents

Abstract

Plural captured images are respectively converted into bird's eye view images, a synthetic bird's eye view image is generated by synthesizing the respective converted bird's eye view images, obstacle information relating to an obstacle is extracted on the basis of the bird's eye view image, an obstacle original image that is an image including the obstacle is extracted from an image captured by an imaging device on the basis of the extracted obstacle information, an obstacle bird's eye view image corresponding to the obstacle in the synthetic bird's eye view image is detected from the synthetic bird's eye view image on the basis of the extracted obstacle information, an image obtained by substituting the obstacle original image extracted by an obstacle original image extracting unit for the obstacle bird's eye view image detected by an obstacle bird's eye view image detecting unit is outputted as a substituted synthetic bird's eye view image, and video data to display the outputted substituted synthetic bird's eye view image on a display device is generated. Thus, the visibility of the synthetic bird's eye view image is improved.

Description

Image processing apparatus, image processing program, image processing system, and image processing method

The present invention relates to an image processing system and an image processing method, and also relates to an image processing apparatus and an image processing program used in the image processing system. In particular, the present invention relates to a technique for generating bird's-eye view images from captured images of a plurality of cameras installed in a vehicle and synthesizing the bird's-eye view images.

It has been considered to provide a driver of a vehicle such as an automobile with visibility information of a blind spot during driving. When the vehicle backs, a blind spot occurs behind the vehicle, making it difficult for the driver to confirm the rear. Therefore, a system has been developed in which an in-vehicle camera (hereinafter simply referred to as a camera), which is an imaging device that monitors the rear of a vehicle that is likely to become a blind spot, is mounted on a car and the captured image is displayed on a screen of a car navigation device or the like. It is coming.

Also, instead of simply displaying the image captured by the camera, research is being conducted to show an unreasonable image that can be seen and understood by humans using image processing technology. One of them is a technique for generating and displaying an image called a bird's-eye view as seen from a virtual viewpoint such as above the ground by performing geometric transformation, that is, coordinate transformation, on a captured image. By displaying this bird's-eye view image on the screen, the driver can grasp the situation behind the vehicle as a blind spot in detail and more comfortably by looking at the bird's-eye view image.

Furthermore, a composite bird's-eye view image (hereinafter, referred to as an image of the situation of the entire circumference of the vehicle seen from above by combining the bird's-eye view images obtained by coordinate transformation of images obtained from a plurality of cameras as described above. An image processing system has been developed in which an all-around bird's-eye view image) is created and this all-around bird's-eye view image is displayed on the screen. In this image processing system, since the situation of the entire circumference of the vehicle can be presented to the driver as an image viewed from above, there is an advantage that the periphery of the vehicle can be grasped more easily by looking at the 360 ° blind spot. The term “composite” as used herein means that adjacent bird's-eye view images are moved and rotated with respect to each other so that the bird's-eye images can be consistent with each other and bonded together to form one bird's-eye view. When adjacent to each other with good consistency, an overlapping portion is generated between the bird's-eye views.

Patent Document 1 describes a vehicle periphery monitoring device that uses a plurality of bird's-eye images (bird's-eye images) and displays a bird's-eye image that is synthesized.
JP 2007-27948 A

However, according to the technique described in Patent Document 1, in the bird's eye view, the captured three-dimensional obstacle appears to have fallen in a plane on the ground, and the shape of the obstacle is actually May feel like a different distorted shape, making it difficult for the driver who is driving the vehicle to figure out what the obstacle is when looking at the synthesized bird's-eye view There are many cases. Since the obstacle is often an obstacle in progress, it should be grasped by the driver without a sense of incongruity.

In view of this, the present invention provides a visibility such that when a bird's-eye view image is synthesized as described above to obtain a combined bird's-eye view image, an obstacle is displayed in the synthesized bird's-eye view image when the obstacle is displayed. In view of the above problems, an object of the present invention is to provide an “image processing apparatus” and an “image processing program” that improve the visibility of a composite bird's-eye view image. It is another object of the present invention to provide an “image processing system” and an “image processing method” using them.

An image processing apparatus according to the present invention includes a viewpoint conversion unit that converts each of images captured by a plurality of imaging devices into a bird's-eye view image viewed from a virtual viewpoint, and each bird's-eye view image converted by the viewpoint conversion unit. An image processing apparatus including an image composition unit that synthesizes and generates a composite bird's-eye view image, the obstacle information extraction unit extracting obstacle information related to the obstacle based on the bird's-eye view image, and the obstacle information extraction An obstacle original image extraction unit that extracts an obstacle original image, which is an image including the obstacle, from an image captured by the imaging device based on the obstacle information extracted by a part, and the obstacle information Based on the obstacle information extracted by the extraction unit, an obstacle bird's-eye image detection unit that detects an obstacle bird's-eye image corresponding to the obstacle in the synthetic bird's-eye view image from the synthetic bird's-eye view image; A replacement processing unit that outputs an image obtained by replacing the obstacle bird's-eye image detected by the obstacle bird's-eye image detection unit with the obstacle original image extracted by the obstacle original image extraction unit as a replacement synthesized bird's-eye view image; A video data generation unit configured to generate video data for causing the display device to display the replacement composite bird's-eye view image output from the replacement processing unit.

Note that the obstacle information extraction unit is an image of an overlapping region of the target bird's-eye view image that is the bird's-eye view image captured by the target imaging device that is at least two of the plurality of imaging devices. The obstacle information may be extracted by performing a difference process between the target bird's-eye view images.

An image processing apparatus according to the present invention includes a viewpoint conversion unit that converts each of images captured by a plurality of imaging devices into a bird's-eye view image viewed from a virtual viewpoint, and each bird's-eye view image converted by the viewpoint conversion unit. An image processing apparatus including an image composition unit that synthesizes and generates a composite bird's-eye view image, the obstacle information extraction unit extracting obstacle information related to the obstacle based on the bird's-eye view image, and the obstacle information extraction An obstacle bird's-eye image detection unit that detects an obstacle bird's-eye image corresponding to the obstacle in the synthetic bird's-eye view image from the synthetic bird's-eye view image based on the obstacle information extracted by a unit; Based on the information on the obstacle bird's-eye image detected by the bird's-eye image detection unit, an obstacle original image that is an image including the obstacle is extracted from the image captured by the imaging device, and the obstacle A replacement processing unit that outputs, as a replacement composite bird's-eye view image, an image obtained by replacing the obstacle bird's-eye image detected by the object bird's-eye image detection unit with the obstacle original image, and the replacement composite bird's-eye view image output from the replacement processing unit And a video data generation unit for generating video data for displaying on the display device.

An image processing program according to the present invention includes a viewpoint conversion step of converting each of images captured by a plurality of imaging devices into a bird's eye view image viewed from a virtual viewpoint, and each bird's eye view image converted in the viewpoint conversion step. An image processing program for operating a computer as an image processing apparatus, including an image synthesis step of generating a synthesized bird's-eye view image by combining, and extracting obstacle information relating to an obstacle based on the bird's-eye view image An obstacle original image that extracts an obstacle original image that is an image including the obstacle from the image captured by the imaging device based on the obstacle information extracted in the step and the obstacle information extraction step Based on the obstacle information extracted in the extraction step and the obstacle information extraction step, from the synthetic bird's-eye view image, The obstacle bird's-eye image detection step for detecting an obstacle bird's-eye image corresponding to the obstacle in the synthetic bird's-eye view image, and the obstacle bird's-eye image detected by the obstacle bird's-eye image detection unit is the obstacle original image. A replacement processing step for outputting an image replaced with the obstacle original image extracted by the extraction unit as a replacement composite bird's-eye view image, and a video for displaying the replacement composite bird's-eye view image output from the replacement processing step on a display device And a video data generation step for generating data.

An image processing system according to the present invention includes the above-described image processing device, the plurality of imaging devices, and the display device.

The image processing method according to the present invention converts each of images captured by a plurality of imaging devices into a bird's-eye view image viewed from a virtual viewpoint, and generates a combined bird's-eye view image by combining the converted bird's-eye view images. An image processing method, wherein obstacle information related to an obstacle is extracted based on the bird's eye view image, and an image including the obstacle is extracted from an image captured by the imaging device based on the extracted obstacle information. The obstacle original image is extracted, and the obstacle bird's-eye image corresponding to the obstacle in the synthetic bird's-eye view image is detected from the synthesized bird's-eye view image based on the extracted obstacle information, and the obstacle An image obtained by replacing the obstacle bird's eye image detected by the object bird's eye image detection unit with the obstacle original image extracted by the obstacle original image extraction unit is output as a replacement synthesized bird's eye view image and output. It generates video data for displaying the replacement composite bird's eye view image on the display device.

The imaging device corresponds to a camera that captures a still image, a camera that captures a moving image, or the like.

An obstacle refers to a three-dimensional object existing on the road surface or the ground, not a flat character or figure drawn on the road surface or the ground. Therefore, for example, creatures such as people, plants, and animals, vehicles such as cars, motorcycles, and bicycles, and artificial objects such as signs, fences, and posts are applicable as obstacles.

The image is not limited to the meaning of the image in which the image of the captured subject is visualized. In the present invention, the image includes data of an image to be subjected to image processing such as viewpoint conversion or replacement. Is to be interpreted.

The obstacle bird's-eye view image is not limited to the entire image as long as the obstacle is reflected, and may be a partial image.

The obstacle information extraction unit extracts the obstacle information by an optical flow technique when each of images taken by the plurality of imaging devices includes temporally continuous images. May be.

The image processing apparatus according to the present invention is an image processing apparatus including a viewpoint conversion unit that converts each temporally continuous image captured by one imaging apparatus into a bird's eye view image viewed from a virtual viewpoint. An obstacle information extracting unit for extracting obstacle information of an area where the obstacle is imaged by the imaging device by an optical flow technique, and the obstacle information extracted by the obstacle information extracting unit. Based on the obstacle original image extraction unit that extracts an obstacle original image that is an image including the obstacle from the image captured by the imaging device, and the obstacle information extracted by the obstacle information extraction unit Based on the above, the obstacle bird's-eye image detection unit for detecting an obstacle bird's-eye image corresponding to the obstacle in the bird's-eye view image, and the obstacle detected by the obstacle bird's-eye image detection unit A replacement processing unit that outputs, as a replacement bird's-eye view image, an image obtained by replacing an obstacle bird's-eye image with the obstacle original image extracted by the obstacle original image extraction unit, and the replacement bird's-eye view image output from the replacement processing unit. A video data generation unit that generates video data to be displayed on the display device may be provided.

The image processing apparatus of the present invention includes a viewpoint conversion unit that converts each of images captured by a plurality of imaging devices into a bird's eye view image viewed from a virtual viewpoint, and each bird's eye view image converted by the viewpoint conversion unit. An image processing apparatus including an image synthesis unit that synthesizes and generates a synthesized bird's-eye view image, wherein the obstacle is imaged by the imaging device using information from a sensor that detects the obstacle An obstacle information extracting unit for extracting the obstacle information, and an image including the obstacle from an image captured by the imaging device based on the obstacle information extracted by the obstacle information extracting unit. Based on the obstacle information extracted by the obstacle original image extracting unit that extracts the obstacle original image and the obstacle information extracting unit, the obstacle in the synthetic bird's eye view image is changed from the synthetic bird's eye view image to the obstacle in the synthetic bird's eye view image. versus An obstacle bird's-eye image detection unit that detects an obstacle bird's-eye image, and the obstacle bird's-eye image detected by the obstacle bird's-eye image detection unit into the obstacle original image extracted by the obstacle original image extraction unit A replacement processing unit that outputs the replaced image as a replacement composite bird's-eye view image; and a video data generation unit that generates video data for displaying the replacement composite bird's-eye view image output from the replacement processing unit on a display device. May be.

The image processing apparatus according to the present invention is an image processing apparatus including a viewpoint conversion unit that converts each temporally continuous image captured by one imaging apparatus into a bird's eye view image viewed from a virtual viewpoint. An obstacle information extraction unit that extracts information on an area in which the obstacle is imaged by the imaging device using information from a sensor that detects the obstacle, and the obstacle information extraction unit An obstacle original image extraction unit that extracts an obstacle original image, which is an image including the obstacle, from the image captured by the imaging device based on the obstacle information extracted in Step 1, and the obstacle information extraction An obstacle bird's-eye image detection unit that detects an obstacle bird's-eye image corresponding to the obstacle in the bird's-eye view image from the bird's-eye view image based on the obstacle information extracted by a part, and the obstacle bird's-eye image Detection unit A replacement processing unit that outputs an image obtained by replacing the detected obstacle bird's-eye image with the obstacle original image extracted by the obstacle original image extraction unit as a replacement bird's-eye view image, and the output from the replacement processing unit You may provide the video data production | generation part which produces | generates the video data for displaying a replacement bird's-eye view image on a display apparatus.

The obstacle information extraction unit may extract obstacle information related to the obstacle from each of the images captured by the imaging device.

According to the present invention, it is possible to provide an “image processing apparatus” and an “image processing program” that improve the visibility of a composite bird's-eye view image. In addition, an “image processing system” and an “image processing method” using them can be provided.

The significance or effect of the present invention will be further clarified by the following description of embodiments.

However, the following embodiment is merely one embodiment of the present invention, and the meaning of the term of the present invention or each constituent element is not limited to that described in the following embodiment. Absent.

1 is an overall configuration diagram of an image processing system according to an embodiment of the present invention. It is the figure which looked at the vehicle based on embodiment of this invention from upper direction. It is a figure showing each bird's-eye view image based on embodiment of this invention. It is the figure which converted each bird's-eye view image on the coordinate of a perimeter bird's-eye view image based on embodiment of this invention. It is the figure which looked at the vehicle based on embodiment of this invention from diagonally left front. It is a figure explaining a part of all-around bird's-eye view image based on embodiment of this invention. It is a figure explaining a part of all-around bird's-eye view image based on embodiment of this invention. It is a figure explaining a part of all-around bird's-eye view image based on embodiment of this invention. It is a figure explaining the image processing method based on embodiment of this invention. It is a figure explaining the memory access operation | movement in the image processing method based on embodiment of this invention. It is a figure explaining a part of all-around bird's-eye view image based on embodiment of this invention. It is a figure showing the all-around bird's-eye view image based on embodiment of this invention. 1 is an overall configuration diagram of an image processing system according to an embodiment of the present invention. It is a figure explaining installation of the camera mounted in the vehicle based on embodiment of this invention. According to an embodiment of the present invention, a camera coordinate system XYZ, a coordinate system X _bu Y _bu of the imaging surface of the camera, and a world coordinate system X _w Y _w Z _w including a two-dimensional ground coordinate system X _w Z _w It is a figure showing a relationship. It is a whole block diagram of the image processing system based on the other form of this invention.

Explanation of symbols

DESCRIPTION OF SYMBOLS 1, 1F, 1B, 1L, 1R Camera 2 Viewpoint conversion part 3 Image composition part 4, 4a Obstacle information extraction part 5 Obstacle original image extraction part 6 Obstacle bird's-eye image detection part 7 Replacement process part 8 Video data generation part 1000 Image processing system 2000 Image processing device 3000 Display device 10F, 10B, 10L, 10R Bird's eye view image 12F, 12B, 12L, 12R Field of view 13 Common imaging space 14 Obstacle 14a Area where obstacle is imaged 14b Obstacle is imaged 14c Image including area where obstacle is imaged 14d Image including area where obstacle is imaged in bird's-eye view image C _FL overlap portion 100 Vehicle 121, 122 (bird's-eye conversion) image 123 Difference area 400 Sensor

Hereinafter, embodiments of an image processing apparatus, an image processing program, an image processing method using them, and an image processing system according to the present invention will be specifically described with reference to the drawings. In each of the drawings to be referred to, the same part is denoted by the same reference numeral, and redundant description regarding the same part is omitted.
(First embodiment)
First, a method for creating an all-around bird's-eye view image will be described.
[Configuration of image processing system and operation of image processing apparatus]
FIG. 1 is an overall configuration diagram of an image processing system according to the present embodiment. An image processing apparatus, an image processing system using the same, and an image processing method used in the image processing system will be described with reference to FIG. The image processing system 1000 according to the present embodiment generates all-round bird's-eye view images from the cameras 1F, 1B, 1L, and 1R attached to the vehicle 100 and captured images obtained by these cameras, and displays the video data on the display device 3000. And an image processing device 2000 that outputs the image data of the all-around bird's-eye view image.

As the cameras 1F, 1B, 1L and 1R, for example, a camera using a CCD (Charge Coupled Devices) or a camera using a CMOS (Complementary Metal Oxide Semiconductor) image sensor is used.

FIG. 2 is a plan view of the camera 100 installed on the vehicle 100 as viewed from above. In the figure, the vehicle 100 is illustrated as an example of a truck formed of a cab and a luggage compartment that is higher than the cab. As shown in the figure, the front part of the vehicle 100 (for example, the upper part of the front mirror of the vehicle 100), the rear part (for example, the uppermost part of the rear part of the vehicle 100), the left side part (for example, the uppermost part of the left side surface of the vehicle 100) and the right side part. Cameras 1 </ b> F, 1 </ b> B, 1 </ b> L, and 1 </ b> R, which are imaging devices, are attached to (for example, the uppermost portion of the right side surface of vehicle 100). The cameras 1F, 1B, 1L, and 1R may be referred to as a front camera, a rear camera, a left side camera, and a right side camera, respectively.

The optical axis of the camera 1F is obliquely downward toward the front of the vehicle 100, the optical axis of the camera 1B is obliquely downward to the rear of the vehicle 100, and the optical axis of the camera 1L is oblique to the left of the vehicle 100. Each camera is installed in the vehicle 100 so that the optical axis of the camera 1 </ b> R is obliquely downward to the right of the vehicle 100. The heights of the cameras 1L and 1R are assumed to be higher than the height of the camera 1F. Moreover, the vehicle 100 shall be located on the ground.

The image processing apparatus 2000 will be described.

The image processing apparatus 2000 includes, for example, hardware formed from an integrated circuit that executes image processing in the present embodiment, and a CPU (central processing unit) that performs software processing of an image processing program that executes image processing in the present embodiment. And memory. The display device 3000 is formed from a liquid crystal display panel or the like. A display device included in a car navigation system or the like may be used as the display device 3000 in the image processing system.

In the image processing apparatus 2000, the viewpoint conversion unit 2 receives the captured image data from the cameras 1F, 1B, 1L, and 1R, and creates a bird's eye view image corresponding to each captured image data. The image synthesis unit 3 synthesizes the bird's-eye view images to generate an all-around bird's-eye view image.

Here, the relationship between each bird's-eye view image and the all-around bird's-eye view image will be described in order to grasp the operations of the viewpoint conversion unit 2 and the image composition unit 3 regarding the all-around bird's-eye view creation method. A method for creating a bird's eye view image corresponding to each captured image data will be described last.

As shown in FIG. 3, from the captured images obtained from the cameras 1F, 1B, 1L, and 1R, bird's-eye view images 10F (coordinate systems X _F Y _F ), 10B (coordinate systems X _B Y _B ), 10L ( A coordinate system X _L Y _L ) and 10R (coordinate system X _R Y _R ) are shown. Next, by rotating and / or translating the other three bird's- eye view images 10F, 10L, and 10R with reference to the bird's-eye view image 10B corresponding to the camera 1B, they (10F, 10L, and 10R) are converted into the bird's-eye view image 10B. Convert to coordinate system X _B Y _B Thereby, the coordinates of each bird's-eye view image are converted into coordinates in the all-around bird's-eye view (synthetic bird's-eye view referred to in the present invention) image. Hereinafter, the coordinates in the all-around bird's-eye view image are referred to as “all-around bird's-eye view coordinates”.

FIG. 4 shows the bird's- eye view images 10F, 10B, 10L, and 10R represented on the all-around bird's-eye view and the coordinate system X _B Y _B. The all-around bird's-eye view is composed of at least two adjacent bird's- eye view images 10F, 10B, 10L, and 10R. In the all-around bird's-eye view in FIG. Composed. Hereafter, the all-around bird's-eye view will be described using an example including all bird's-eye view images of the bird's- eye view images 10F, 10B, 10L, and 10R. In the all-around bird's-eye view coordinates, there is a portion where two adjacent bird's-eye view images overlap as shown in FIG. In the figure, the hatched area to which C _FL is attached is the portion where the bird's eye view images 10F and 10L overlap on the all-around bird's eye view coordinates.

に つ い て Explain the overlapping part.

FIG. 5 is a view of the vehicle 100 as viewed obliquely from the left front.

In the figure, the imaging space of each camera is schematically represented. Each camera generates a captured image obtained by capturing an image of a subject in its field of view (that is, an imaging space). The fields of view of the cameras 1F, 1B, 1L and 1R are represented by 12F, 12B, 12L and 12R, respectively. Note that only a part of the visual fields 12R and 12B is shown in FIG.

The visual field 12F of the camera 1F includes an obstacle located within a predetermined range in front of the vehicle 100 and the ground in front of the vehicle 100 with respect to the installation position of the camera 1F. This predetermined range is determined from the angle of view of the camera.

The visual field 12B of the camera 1B includes an obstacle positioned within a predetermined range behind the vehicle 100 and the ground behind the vehicle 100 with respect to the installation position of the camera 1B.

The visual field 12L of the camera 1L includes an obstacle located within a predetermined range on the left side of the vehicle 100 with respect to the installation position of the camera 1L and the ground on the left side of the vehicle 100.

The visual field 12R of the camera 1R includes an obstacle located within a predetermined range on the right side of the vehicle 100 and the ground on the right side of the vehicle 100 with respect to the installation position of the camera 1L.

As described above, the viewpoint of each camera is different, that is, the subject that is within the field of view of each camera is different. The obstacle is a three-dimensional object such as a person, in other words, an object having a height. The road surface that forms the ground is not an obstacle because it has no height.

Now, according to the figure, the cameras 1F and 1L commonly image a predetermined area in front of the vehicle 100 diagonally to the left. That is, the visual fields 12F and 12L intersect in a predetermined region diagonally to the left of the vehicle 100, and a common visual field exists. Hereinafter, the common part common visual field of the visual field of each camera, which is a space captured by each camera in common, is referred to as the above-described common imaging space.

Similarly, the visual fields 12F and 12R intersect with each other in a predetermined area on the right front side of the vehicle 100 to form a common imaging space, and the visual fields 12B and 12L intersect with each other in a predetermined area on the left rear side of the vehicle 100. These common imaging spaces are formed, and the visual fields 12B and 12R intersect with each other in a predetermined area obliquely rearward to the right of the vehicle 100 to form the common imaging space.

Hereinafter, description will be made by paying attention to the common imaging space 13 between the visual fields 12F and 12L. The same description applies to the common imaging space other than the common imaging space 13.

In FIG. 5 (b), the common imaging space 13 is shown using a bold line. The common imaging space 13 is a space that forms a cone whose bottom surface is the ground diagonally left front of the vehicle 100. In the figure, an obstacle 14 such as a person is shown in the common imaging space 13.

Overlapping portions of among a plurality of bird's-eye images, namely a C _FL common imaging volume 13 is converted on the bird's eye view portion of the supra (hereinafter, referred to as overlapping portion).

Bird's eye view images 10F, image appears in the subject in a common imaging space 13 in the range of overlapping portion C _FL viewed from the camera 1F (obstacle 14 such as in FIG. 5 (b)), the bird's-eye view image 10L, the overlap portion C An image of the subject in the common imaging space 13 viewed from the camera 1L appears in the _FL range.

Processing when there is an obstacle in the overlapping portion will be described later in the processing at the overlapping portion. The overlapped portions other than C _FL, the overlapping portion C _FR bird's 10F and 10R overlap, and the overlap portion C _BL bird's 10B and 10L overlap, but there are a overlapping portion C _BR bird's 10B and 10R overlap, Description will be made by paying attention to the overlapping portion _CFL corresponding to the common imaging space 13.

3 and 4, the XF axis and the YF axis are coordinate axes of the coordinate system of the bird's eye view image 10F, and they are the above-described _Xau axis and _Yau axis. Similarly, the XR axis and the YR axis are coordinate axes of the coordinate system of the bird's eye view image 10R, and they are the _Xau axis and the _Yau axis. Similarly, the XL axis and the YL axis are coordinate axes of the coordinate system of the bird's eye view image 10L, and they are the X _au axis and the Y _au axis. Similarly, the XB axis and the YB axis are coordinate axes of the coordinate system of the bird's eye view image 10B, and they are the X _au axis and the Y _au axis.

3 and 4, for the sake of explanation, the overlapping portion C _FL is indicated by a rectangle, but the overlapping portion C _FL is not actually a rectangle. In addition, each bird's eye view image is not necessarily rectangular. FIG. 6 shows an example in which each bird's-eye view image appears and the overlapping portion _CFL is not rectangular. FIGS. 9A and 9B show bird's- eye view images 10L and 10F in all-around bird's-eye view coordinates, respectively. In FIG. 10C, their overlapping portions C _FL are indicated by hatched areas. ing. In addition, in the same figure (a) and (c), illustration of the image from the rear part of a vehicle is abbreviate | omitted.

Referring back to FIG. In the image processing apparatus 2000 of FIG. 1, the obstacle information extraction unit 4 extracts obstacle information related to the obstacle based on the bird's eye view image from the viewpoint conversion unit 2. For example, the bird's-eye view image data corresponding to each captured image data from the viewpoint conversion unit 2 is received, and each of the bird's-eye view images in the bird's-eye view image corresponding to the portion where the obstacle is imaged in the captured image is received. Obstacle information including position information related to the corresponding area, which is an area, is extracted and the obstacle information is output.

The position information, for example, the coordinates of the camera coordinate system XYZ, a coordinate of the coordinate system X _bu Y _bu of the imaging surface S, the coordinates of the two-dimensional ground surface coordinate system X _w Z _w, the world coordinate system X _w Y _w Z _w coordinates Information such as coordinates of other coordinate systems is included. In addition, the obstacle information includes vertex information. Vertex information is the coordinates of vertices in a set of pixels (ie, an image) indicated by coordinate data. By knowing the coordinates of the vertices, it is possible to assume the figure by connecting the vertices with lines. Information such as the range, size, and area of the image can also be calculated. Hereinafter, it is assumed that the consideration of vertex information includes the assumption of information such as the range, size, and area of the image.

The obstacle information extraction unit 4 according to the present embodiment is a target bird's-eye view image in an overlapping portion image that is commonly captured in a target bird's-eye view image that is a bird's-eye view image captured by a target imaging device including at least two cameras. Obstacles can be detected by performing difference processing between them, and the obstacle information is extracted using the results of the difference processing. Note that the overlapping portion is not limited to a portion where two bird's-eye view images overlap as shown in FIG. 4, but is a portion where bird's-eye view images calculated from captured images of three or more imaging devices, that is, three or more bird's-eye view images overlap. Also good.

The obstacle information extraction unit 4 receives the captured image data from the cameras 1F, 1B, 1L, and 1R, which are a plurality of imaging devices, and the obstacle in the area where the obstacle is captured from each of the captured images. Information may be extracted and obstacle information may be output. An image processing system 1000 in this case is shown in FIG. In this case, instead of the above-described difference processing between the bird's-eye view images, difference processing is performed between portions corresponding to the respective common imaging spaces of the captured image data from the cameras.

The method of extracting the obstacle information by the obstacle information extracting unit 4 will be described later in the process of the overlapping part. In that case, description will be made on the assumption of the overlapping portion _{CFL and the} like in FIG.

If no obstacle is detected in the obstacle information extraction unit 4, the obstacle information extraction unit 4 does not extract the obstacle information and does not output the obstacle information. In the following, the description will be continued on the assumption that the obstacle information extracting unit 4 detects an obstacle unless otherwise specified.

The captured image data from the cameras 1F, 1B, 1L, and 1R is received by the obstacle original image extraction unit 5, and based on the obstacle information extracted by the obstacle information extraction unit 4, the imaging including the obstacle is included. An obstacle original image which is an image including an obstacle (for example, an image including a part where the obstacle is imaged) is extracted from the image, and the information is output. When no obstacle is detected in the obstacle information extraction unit 4, the obstacle information extraction unit 4 does not extract the obstacle information and does not output the obstacle information to the obstacle original image extraction unit 5. Therefore, the obstacle original image extraction unit 5 does not extract the obstacle original image.

Based on the obstacle information extracted by the obstacle information extraction unit 4, the obstacle bird's-eye image detection unit 6 obtains an obstacle bird's-eye image corresponding to the obstacle in the synthetic bird's-eye view image from the all-around bird's-eye view image. To detect. For example, a bird's-eye image obstacle region (an obstacle bird's-eye image) corresponding to the corresponding region in the synthesized bird's-eye view image is detected, and information related thereto is output. When no obstacle is detected by the obstacle information extraction unit 4, the obstacle information extraction unit 4 does not extract the obstacle information and does not output the obstacle information to the obstacle bird's-eye image detection unit 6. Therefore, the obstacle bird's-eye image detection unit 6 does not detect the obstacle bird's-eye image.

The replacement processing unit 7 extracts the obstacle bird's-eye image detected by the obstacle bird's-eye image detection unit 6 by the obstacle original image extraction unit 5 based on the information about the bird's-eye image obstacle region from the obstacle bird's-eye image detection unit 6. The image replaced with the obstacle original image is output as a replacement synthesized bird's-eye view image. For example, in the all-round bird's-eye view image, replacement is performed using the obstacle original image extracted by the obstacle original image extraction unit 5 instead of the image including the obstacle bird's-eye image detected by the obstacle bird's-eye image detection unit 6. Then, a new all-round bird's-eye view image (hereinafter referred to as a replacement combined bird's-eye view image) is generated and output. If no obstacle is detected in the obstacle information extraction unit 4, the obstacle original image extraction unit 5 does not extract the obstacle original image, but sends the information of the obstacle original image to the replacement processing unit 7. Since the obstacle bird's-eye image detection unit 6 does not output the obstacle bird's-eye image and does not output the information of the bird's-eye image obstacle region to the replacement processing unit 7, the replacement processing unit 7 does not output the information. Without generating an image, the all-around bird's-eye view image from the image composition unit 3 is output as it is.

The operation of the replacement processing unit 7 will be described later.

The video data generation unit 8 generates video data for causing the display device 9 to display the all-around bird's-eye view image or the replacement synthesized bird's-eye view image output from the replacement processing unit. That is, the data is converted into a data structure or pixel information that matches the display format or display standard of the display device 9 and output.

As described above, the visibility problem that the driver feels uncomfortable when the obstacle is displayed in the all-around bird's-eye view image can be solved in the replacement synthetic bird's-eye view image, and the visibility is improved. It is an all-around bird's-eye view image. This point will be clarified in the explanation of visibility in an embodiment described later.
[Processing at overlapping part]
A process for each obstacle image captured by the camera 1F and the camera 1L in the overlapping part _CFL and a method for extracting the obstacle information by using the result of the difference process in the obstacle information extraction unit 4 will be described. .

For example, as shown in FIG. 5B, when the obstacle 14 exists in the common imaging space 13 where the imaging area of the camera 1F and the imaging area of the camera 1L overlap, the obstacle is detected by the camera 1F. When a bird's-eye view image is generated from an image obtained by capturing 14, as shown in FIG. 7B, an image 122 in which the obstacle 14 falls in the left direction of the vehicle in the overlapping portion C _FL of the all-around bird's-eye view image. Converted. On the other hand, when generating the bird's-eye view image obstacle 14 by the camera 1L from the image captured, as shown in FIG. 6 (a), the obstacle 14 at the overlapping portion C _FL all-round bird's eye view image collapse in front of the vehicle The image 121 is converted. As is often seen when converting to a bird's eye view image, the image 122 is not only tilted leftward but also distorted and / or extended in the imaging direction depth direction. It is converted not only to fall forward but also to be distorted and / or extended in the imaging direction depth direction, which is schematically shown in the figure. The obstacle bird's-eye view image is not limited to the whole image as shown in the figure, and corresponds to the obstacle bird's-eye image even if only a part is shown. This is because even a partial image is later replaced with an obstacle original image, so there is no problem even if the image before replacement is partial.

In the obstacle information extraction section 4, a portion corresponding to the overlapping portion C _FL in the bird's-eye view portion 112 corresponding to the overlapping portion C _FL in bird's-eye view according to the captured image (see FIG. (B)) by the captured image of the camera 1L camera 1F A difference calculation from 111 (see FIG. 11A) is performed. The information of the overlapping portion is obtained from the rotation angle when the coordinate conversion is performed when creating FIG. 4, the translational mobility, the vertex coordinates of the regions 10F, 10L, 10R, and 10B.

It will be further described with reference to FIG. By performing a difference calculation between the portion 111 in FIG. 11A and the portion 112 in FIG. 10B, a portion 123 shown in FIG. 10C is extracted.

In FIG. (A), part of the area 121a of the portion 111 corresponding to the overlapping portion C _FL in bird's-eye view according to the captured image of the camera 1L is a portion corresponding to the image 122. In FIG. (B), part of the region 122a of the portion 112 corresponding to the overlapping portion C _FL in bird's-eye view according to the captured image of the camera 1F is a portion corresponding to the image 121. Usually, the part excluding the image 121 and the area 121a of the part 111 and the part excluding the image 122 and the area 122a of the part 112 are bird's-eye view conversion images of the image of the ground. Since it is converted, a portion 123 corresponding to the image obtained by combining the image 121 and the image 122 shown in FIG. As the part 123, for example, a region whose difference value is not 0 is extracted. In the difference calculation, pixel values (luminance and color difference, red green blue value, etc.) between corresponding pixels are subtracted, and when the absolute value is compared with a predetermined threshold value, it can be regarded as zero. An arithmetic method for setting to zero is used.

By extracting the part 123, it can be detected that an obstacle exists in the common imaging space 13, and the coordinates of the part 123 in the coordinate system X _B Y _B (all-around bird's-eye view image) shown in FIG. And the range thereof are obstacle information such as position information in the coordinate system X _B Y _B included in the obstacle information and vertex information. The obstacle information includes positional information, vertex information, and the like in the captured image (specifically, in the case of the portion 123, the visual field 12F and the visual field 12L including the obstacle) in order to extract the obstacle original image. It is. An imaging surface of the operation, using the inverse of equation (7) described in method of generating bird's-eye view image after e.g., infra coordinates (in the coordinate system X _B Y field 12F and viewing 12L from coordinates in _B in composite bird's eye view Coordinate system).

As described above, the obstacle information is mainly used by the coordinate system X _B Y _B used in the obstacle bird's-eye image detection unit 6, that is, the coordinates in the bird's-eye view image, and mainly in the obstacle original image extraction unit 5. The coordinates in the visual field 12F and the visual field 12L, that is, the coordinates in the captured image are included, and are output from the obstacle information extraction unit 4. The coordinates in the captured image are calculated from the coordinates in the bird's-eye view image, but since the bird's-eye view image is originally obtained by converting from the captured image, all the coordinates of the obstacle in the captured image are calculated from all the coordinates of the obstacle in the bird's-eye view. It can be easily understood that it can be calculated.

It can also be extracted from the shape of the portion 123 that the image 121 and the image 122 are two-dimensional obstacle images. For example, when the shape of the portion 123 is a V-shape formed of the image 121 and the image 122 as shown in FIG. 8C, the image 121 and the image 122 are two-dimensional obstacle images captured. It can be. Obstacle information can be obtained, for example, by storing the direction, position, vertex information, and the like of the portion 123 as table information in the image processing apparatus 2000. In the following description, the obstacle information is described as an example of coordinates as position information.
[Method of extracting obstacle original image]
A flow in which the all-around bird's-eye view image is replaced with the replacement composite bird's-eye view image when the bird's-eye view corresponding to the captured image of the camera 1F is used in the overlapping portion _CFL will be described with reference to FIG.

(A) in the figure is an image taken by the camera 1F. The obstacle 14 is imaged at the left end of the image. The area 14a is an area where the obstacle 14 extracted by the obstacle original image extraction unit 5 is imaged, and is the obstacle 14 shown in the captured image. A region 14b in an image captured by the camera 1L, which will be described later, is a region in which the obstacle 14 extracted by the obstacle original image extraction unit 5 is imaged, and is an obstacle 14 imaged in the captured image. is there.

The image 14c is an image (an obstacle original image) including a region 14a where an obstacle is imaged. The range of the image 14c is arbitrary as long as it includes the image 14a. For example, a circumscribed ellipse or a circumscribed rectangle of the image 14a, the entire overlapping portion _CFL , or a shape obtained by enlarging the image 14a by 50% can be taken. In the figure, a circumscribed ellipse is described as an example.

An example of how to calculate the ellipse will be described.

The obstacle original image extraction unit 5 uses the position information included in the obstacle information (specifically, all coordinates included in the image 14a) to determine the length in the long direction of the region 14a and the length in the short direction perpendicular thereto. The ellipse is calculated and the major and minor lengths in the major and minor directions are calculated. After that, the ellipse of the image 14d is obtained by calculating an ellipse obtained by enlarging the ellipse about 2 to 3 times in the major axis direction and about twice in the minor axis direction. The reason why the enlargement width in the major axis direction is large is that the image 14c replaces the image 121, so the size of the image 14c must exceed the image 121, and the characteristics of conversion to a bird's eye view must be taken into account. It is. When converting to a bird's-eye view, for example, the captured image has a characteristic of being distorted and extended radially from the vertical direction of the captured image or from the center of the captured image. In addition, the characteristic that the captured image is distorted and stretched mainly depends on the height of the obstacle and the mounting height of the camera. If the height of the obstacle is relatively high and the mounting height of the camera is low, the distortion and elongation are large. . In the present embodiment, since the vehicle 100 is an example of a truck, the above-mentioned enlarged value is used on the assumption that the image is distorted and extended by about 2 to 3 times at the time of conversion to a bird's eye view from the knowledge of experiments and the like.
[Method of detecting bird's-eye view of obstacles]
FIG. 9B is an enlarged view of the overlapping portion _CFL portion of the bird's-eye view image of the captured image of FIG. As described above, the obstacle 14 is an image 121 that has been transformed as if it has fallen forward of the vehicle, or has been stretched and distorted unlike the actual vehicle.

The image 14d is an image including an image 121 that is an area (the obstacle bird's-eye view image) in which the obstacle 14 detected by the obstacle bird's-eye image detection unit 6 is imaged. The range of the image 14d is arbitrary as long as it includes the image 121. For example, the circumscribed ellipse or circumscribed rectangle of the image 121, the entire overlapping portion _CFL , or a shape obtained by enlarging the image 121 by 10% can be taken. In the figure, a circumscribed ellipse is described as an example.

An example of how to calculate the ellipse will be described.

The obstacle bird's-eye image detection unit 6 uses the position information included in the obstacle information (specifically, all coordinates included in the image 121) to determine the length in the long direction of the image 121 and the length in the short direction perpendicular thereto. The ellipse is calculated and the major and minor lengths in the major and minor directions are calculated. Thereafter, an ellipse obtained by enlarging the ellipse by 10% with respect to the center is obtained to obtain the ellipse of the image 14d.
[Operation of replacement processing unit]
FIG. 9C is an enlarged view of the overlapping portion C _FL portion of the replacement synthesized bird's-eye view image generated by replacing the image 14c in place of the image 14d in the all-round bird's-eye view image by the replacement processing unit 7. . Since the image 14c is used for replacement instead of the image 14d, the shape, size, etc. of the image 14c may be any shape, size, etc. including the image 14d. In the present embodiment, the case where the size of the image 14c is larger than the size of the image 14d will be described.

When the shape, size, etc. of the image 14c and the shape, size, etc. of the image 14d are matched, the obstacle bird's-eye image detection unit 6 to the obstacle original image extraction unit according to a data flow not shown in FIG. Information such as position information and vertex information of the image 14d may be sent to 5, or information such as position information and vertex information of the image 14c is sent from the obstacle original image extraction unit 5 to the obstacle bird's-eye view image detection unit 6. May be. In this case, the obstacle original image extraction unit 5 extracts the obstacle original image by using the obstacle information and / or information such as the position information and vertex information of the image 14d from the obstacle bird's-eye image detection unit 6. The obstacle bird's-eye image detection unit 6 detects the obstacle bird's-eye image using information such as obstacle information and / or position information of the image 14c from the obstacle original image extraction unit 5 and vertex information.

Referring to FIG. 10, a description will be given of a method in which the image processing apparatus or the image processing program uses the image 14c instead of the image 14d.

In the figure, a memory area A in which a bird's eye view of the entire circumference in the memory space is stored and a memory area B in which a captured image captured by the camera 1F is stored are depicted. An image 14d is stored in the memory area A, and an image 14c is stored in the memory area B. In the figure, the image data stored in the memory is schematically drawn on the memory area. This figure also shows a case where the shape and size of the image 14c match the shape and size of the image 14d.

In the following, two examples of the method of replacing using the image 14c instead of the image 14d are given.

First example: In FIG. 5A, the image 14c is read from the memory area B (read), and the image 14c is written to the memory area A so as to fill the image 14d in the memory area A (write). Thereafter, by reading out the all-around bird's-eye view image generated from the memory area A, the replacement synthesized bird's-eye view image can be output.

Second example: In FIG. 5B, the all-round bird's-eye view image other than the image 14d is read from the memory area A, and the image 14c is read from the memory area B. As described above, the replacement composite bird's-eye view image can be output.

Note that the storage destination address of the image 14c and the storage destination address of the image 14d include at least position information of the area where the obstacle is imaged in the obstacle information output from the obstacle information extraction unit 4. It is calculated with reference to the obstacle information.

This can be considered, for example, as follows. When one pixel is represented by 1 byte, and a bird's eye view image of the entire circumference is divided and captured with a block of 8 pixels × 8 pixels as one unit, data of 64 bytes in which 8 pixels × 8 pixels = 64 pixels in one block are continuous. By assigning addresses so as to be, giving the raster scan order for the blocks constituting the all-around bird's-eye view image, and assigning addresses to each block in that order (in this case, the addresses are every 64), An address can be continuously given to each pixel constituting the all-around bird's-eye view image. Which block the image 14c and the image 14d are included in is determined from obstacle information such as position information (ie, image coordinates) in the coordinate system X _B Y _B , vertex information (ie, an area included in the image), and the like. Thus, the storage destination address can be determined. Without being limited to this example, if each pixel is given an address with a certain regularity, it is easy to calculate which address is the pixel constituting the image 14c or 14d from obstacle information such as coordinates. I can do things.

The replacement processing unit 7 may perform the above-described processing of the obstacle original image extraction unit 5 by the replacement processing unit 7 performing the above address calculation and reading from the memory. That is, the circumscribed ellipse of the image 14c is calculated by a replacement process including the function of the obstacle original image extraction unit 5 by the inverse operation of the expression (7) based on the coordinate information in the captured image in the obstacle information and the information on the image 14d. This is done in part 7.

When the image 14c and the image 14d have the same shape, the replacement processing unit 7 detects the obstacle from the captured image by the camera based on the information related to the obstacle bird's-eye image detected by the obstacle bird's-eye image detection unit 6. An original image is extracted, and an image obtained by replacing the obstacle bird's-eye image detected by the obstacle bird's-eye image detection unit 6 with the obstacle original image can be output as a replacement synthesized bird's-eye view image. That is, since the image 14c and the image 14d have the same shape from the obstacle information related to the image 14d, the length of the major axis and the length of the minor axis of the circumscribed ellipse of the image 14c are found, and the inverse calculation of (Equation (7)) Since the position information of the image 14c is known, the replacement processing unit 7 can extract the obstacle original image from the captured image like the obstacle original image extracting unit 5, and replace the obstacle bird's-eye view image with the obstacle. It can be replaced using the original image.
[Visibility in Examples]
Next, the point that the visibility is improved in the replacement synthesized bird's-eye view image will be described with reference to FIG. 9.

As described above, when the obstacle is converted into a bird's-eye view image, the image is like a fallen image like the image 121 or the image 122, and in addition, as if it extends in the imaging direction depth direction, And / or it is often seen that it is transformed in a distorted manner.

Therefore, in this overlapping part _CFL , in the conventional all-around bird's-eye view image using either the image 122 or the image 121 as the imaged obstacle 14, or the conventional all-around bird's-eye view using the image in consideration of both of them. In the image, for the driver who has seen the display device 3000, the object representing the obstacle 14 is inferior in visibility and uncomfortable.

However, the replacement composite bird's-eye view image using the obstacle image extracted from the image captured by each camera without using the obstacle image converted into the bird's-eye view image as in the present embodiment, for the driver. An image of an obstacle without a sense of incongruity is displayed on the display device 3000, and the visibility of the driver who has seen the obstacle 14 is improved.

Figure 11 shows what has been the expansion of the overlap portion C _FL displayed on the display unit of the display device 3000. As shown in (c) or (d) of the figure, compared with the image 121 or image 122 converted into a bird's eye view as shown in (a) or (b) of the figure. Although the obstacle image before being converted into a bird's eye view imaged by a simple camera was used, it can be said that the visibility is improved in (14a, 14b). Note that which of (c) and (d) is used is determined by an arbitrary method from the viewpoint of visibility. For example, the figure (c) may be selected from the point that the obstacle is completely displayed. Whether or not the image is completely displayed can be determined from the degree of overlap with the boundary of the overlap portion _CFL . The present invention is not limited to this, and even if the same figure (c) is not completely displayed and is a partial case, the obstacle is viewed in a larger degree, that is, the entire obstacle is displayed. The one closer to the image may be selected.

Near come was overlapped portion C _FL that used in the above description, an end portion of the imaging area, because it is distorted portion easily imaging material in an image after converting the captured image also to the bird's-eye view, viewing the synthesized bird's This is a part where the effect of improving the property is great.

FIG. 12 shows an example of the all-around bird's-eye view image displayed on the display device 3000 which is a liquid crystal display panel or the like included in a car navigation system or the like. A picture corresponding to the vehicle 100 is displayed in the center, and video data generation units are respectively provided in portions corresponding to the front, rear, left and right sides of the vehicle (that is, the upper, lower left and right portions of the vehicle on the display screen, respectively). 8 is displayed according to the video data of the bird's eye view image obtained from the cameras 1F, 1B, 1L and 1R.

As described above, it is possible to provide the image processing apparatus 2000 that improves the visibility in the synthetic bird's-eye view image, the image processing method using the image processing apparatus 2000, and the image processing system 1000.
(Second Embodiment)
The overall configuration of this embodiment is shown in FIG. 1 as in the first embodiment described above, but the configuration or operation of the obstacle information extraction unit 4 in the same drawing is different from that of the first embodiment. In the figure, since the configuration other than the obstacle information extraction unit 4 is the same as that of the first embodiment, the description thereof is omitted.
[Method of extracting obstacle information in the obstacle information extraction unit]
In contrast to the first embodiment, in this embodiment, the obstacle information extraction unit 4 detects an obstacle using an optical flow technique, and extracts obstacle information using the detection result obtained by the optical flow technique. To do. That is, it is possible to detect an obstacle without using a common imaging space of the imaging spaces of the two cameras or an overlapping portion where it is converted on the bird's eye view. Therefore, the application range of the method of the present embodiment is not limited to the overlapping portion. Further, no separate mechanism or parts for detecting an obstacle are required. It is assumed that each of the images captured by the plurality of cameras includes images that are temporally continuous.

In the optical flow method, for example, in an image captured every time, a corresponding pixel group is specified, the amount of movement is calculated, and an imaging target is detected. Therefore, it is possible to detect from the amount of movement whether it is a part different from the background in the captured image or even an obstacle. For example, if there are two large amounts of movement, the larger object can be regarded as an imaging object in front, so that it can be regarded as an obstacle.

Specifically, when there is movement in a captured image (moving image) of a certain camera by the optical flow technique, a portion with a different movement is detected, and an obstacle exists in the captured image from the behavior. Detect that Obstacle information can be extracted from the position of the different part, vertex information, and the like. The obstacle information can be obtained, for example, by providing the image processing apparatus 2000 with table information corresponding to the position of different parts, vertex information, and the like.

As described above, it is possible to provide the image processing apparatus 2000 that improves the visibility in the synthetic bird's-eye view image, the image processing method using the image processing apparatus 2000, and the image processing system 1000.
(Third embodiment)
In the present embodiment, parts different from the first embodiment will be described. In addition, description is abbreviate | omitted about the same part as 1st Embodiment.
[Method of extracting obstacle information in the obstacle information extraction unit]
In contrast to the first embodiment and the second embodiment, in this embodiment, the obstacle information extraction unit detects an obstacle using information from the sensor that detects the obstacle, and uses the information from the sensor. The obstacle information including at least position information of the area where the obstacle is imaged by the camera is extracted. That is, it is possible to detect an obstacle without using a common imaging space of the imaging spaces of the two cameras or an overlapping portion where it is converted on the bird's eye view. Therefore, the application range of the method of the present embodiment is not limited to the overlapping portion.

FIG. 13 shows an overall configuration diagram of the image processing system according to the present embodiment.

The difference from FIG. 1 is that a sensor 400 is added, and that the obstacle information extraction unit 4 uses the information from the sensor 400 to detect an obstacle and uses the information from the sensor to obtain obstacle information. It is the point which becomes the obstruction information extraction part 4a to extract.

Specifically, the obstacle information extraction unit 4a detects the position of the obstacle, vertex information, and the like based on the detection result of the sensor 400, and detects the presence of the obstacle in the common imaging space 13. When the sensor 400 is an ultrasonic sensor, the position of an object that reflects ultrasonic waves, such as a sonar (SONAR: sound navigation ranging) used in a fish finder or an acoustic sounder using ultrasonic waves, It is possible to grasp vertex information and the like. Obstacle information can be extracted from the position, vertex information, and the like based on the detection result. The obstacle information can be obtained, for example, by providing the image processing apparatus 2000 with table information corresponding to the position of different parts, vertex information, and the like.

As described above, it is possible to provide the image processing apparatus 2000 that improves the visibility in the composite bird's-eye view image, the image processing method using the image processing apparatus 2000, and the image processing system 1000.

In the second embodiment and the third embodiment, the description has been made on the premise that an all-around bird's-eye view is created, but the content of the present invention is not limited to the bird's-eye view image covering the entire periphery of the vehicle. . For example, the present invention can also be applied to a bird's-eye view image that is a half circle around the vehicle or three-quarters around the vehicle. In the case of a bird's-eye view only in front of the vehicle, a single front camera is sufficient.

Also, other than the first embodiment using the image difference calculation, it can be realized using only one monocular camera. That is, when a monocular camera is used, the original image of the detected area is cut out using the optical flow and sensor information, and an image obtained by replacing the obstacle bird's-eye view image with the obstacle original image is output as a replacement composite bird's-eye view image. can do. Thereby, a bird's eye view image with high visibility can be displayed.

Thus, the technical idea disclosed in the second and third embodiments can be applied even when only one camera is used.

Note that the image processing apparatus according to the present embodiment can be realized in hardware by a CPU, memory, or other LSI of an arbitrary computer. In terms of software, it is realized by a program having a view support function loaded in a memory. 1 and 13 show functional blocks of a visual field support function realized by hardware and software. However, it goes without saying that these functional blocks can be realized in various forms such as hardware only, software only, or a combination thereof. Although FIGS. 1 and 13 are expressed in terms of functional units, they tend to be understood in terms of hardware, but can be easily understood in terms of software by replacing the functional units with functional steps.

That is, the image processing program that causes the general-purpose computer used in the image processing system 1000 to operate as the image processing apparatus 2000 performs the operations of the flowcharts shown in FIGS. That is, referring to FIG. 1, each of images captured by a plurality of cameras 1 </ b> F, 1 </ b> B, 1 </ b> L, 1 </ b> R is converted in a viewpoint conversion step through a viewpoint conversion step of converting a bird's-eye view image viewed from a virtual viewpoint. An obstacle that extracts the obstacle information of the area where the obstacle is imaged from each of the images captured by the camera while executing an image synthesis step of generating a synthesized bird's eye view image by synthesizing each bird's eye view image An object information extraction step is executed.

Based on the obstacle information extracted in the obstacle information extraction step, an obstacle original image extraction step for extracting an obstacle original image from an image captured by the camera, and a synthetic bird's-eye view based on the obstacle information An obstacle bird's-eye image detection step for detecting an obstacle bird's-eye image from the image is executed, and instead of the image including the obstacle bird's-eye image detected in the obstacle bird's-eye image detection step in the synthesized bird's-eye view image, the obstacle A replacement composition bird's eye view image is output by executing a change composition step for replacement using the obstacle original image extracted in the original image extraction step.

The replacement synthesized bird's-eye view image output from the change synthesis step generates video data to be displayed on the display device by executing the video data generation step.

Since the image processing program has the same effect as the image processing apparatus 2000 shown in FIG. 1, the image processing program that improves the visibility in the composite bird's-eye view image can be provided.
[Method for generating bird's-eye view image]
First, a method for generating a bird's eye view image from a captured image captured by one camera will be described. The generation of the bird's eye view image is performed by the viewpoint conversion unit 2. In the following description, the ground is assumed to be on a horizontal plane, and “height” represents the height with respect to the ground.

As shown in FIG. 14, consider a case in which the camera 1 that is an imaging device is disposed at the rear part of the vehicle 100 with its optical axis obliquely rearward and downward. The vehicle 100 is, for example, a truck. In the figure, the vehicle 100 is illustrated as an example of a truck formed of a cab and a luggage compartment that is higher than the cab. There are two types of angles between the horizontal plane and the optical axis of the camera 1, the angle represented by θ and the angle represented by θ ₂ in FIG. The angle θ ₂ is generally called a look-down angle or a depression angle. Hereinafter, the angle θ is referred to as the tilt angle of the camera 1 with respect to the horizontal plane, and description will be made using this θ. Note that 90 ° <θ <180 ° and θ = 180 ° −θ ₂ .

FIG. 15 shows the relationship between the camera coordinate system XYZ, the coordinate system X _bu Y _bu of the imaging surface S of the camera 1, and the world coordinate system X _w Y _w Z _w including the two-dimensional ground coordinate system X _w Z _w. ing. The camera coordinate system XYZ is a three-dimensional coordinate system as shown in the figure with the X axis, the Y axis, and the Z axis orthogonal to each other as coordinate axes. The coordinate system X _bu Y _bu is a two-dimensional coordinate system represented in the figure with the X _bu axis and the Y _bu axis orthogonal to each other as coordinate axes. The two-dimensional ground coordinate system X _w Z _w is a two-dimensional coordinate system represented in the figure with the X _w axis and the Z _w axis orthogonal to each other as coordinate axes. World coordinate system X _w Y _w Z _w is, X _w axis and Z _w axes, a three-dimensional coordinate system with the coordinate axes 3 axes Y _w axis perpendicular to these two axes.

In the camera coordinate system XYZ, with the optical center of the camera 1 as the origin O, the Z axis in the optical axis direction, the X axis in the direction perpendicular to the Z axis and parallel to the ground, and the direction perpendicular to the Z axis and the X axis The Y axis is taken. In the coordinate system X _bu Y _bu of the imaging surface S, taking the origin at the center of the imaging surface S, X _bu axis in the lateral direction of the imaging surface S, Y _bu axes are taken in the longitudinal direction of the imaging surface S.

In the world coordinate system X _w Y _w Z _w, the intersection of the vertical line and the ground passing through the origin O of the camera coordinate system XYZ with the origin O _w, Y _w axis to the ground and perpendicular directions, X camera coordinate system XYZ X _w axis in the axial direction parallel, Z _w axis is taken in a direction orthogonal to the X _w axis and Y _w axis.

The _Xw axis is at a position translated from the X axis, the direction of movement is the vertical line direction, and the amount of translation is h. The obtuse angle formed by the Z _w axis and Z-axis, coincides with the inclination angle theta.

The coordinates in the camera coordinate system XYZ are expressed as (x, y, z). x, y, and z are an X-axis component, a Y-axis component, and a Z-axis component, respectively, in the camera coordinate system XYZ.

The coordinates in the world coordinate system X _w Y _w Z _w are expressed as (x _w , y _w , z _w ). x _w , y _w, and z _w are an X _w axis component, a Y _w axis component, and a Z _w axis component in the world coordinate system X _w Y _w Z _w , respectively.

The coordinates in the two-dimensional ground coordinate system X _w Z _w are expressed as (x _w , z _w ). x _w and z _w are, respectively, in the two-dimensional ground surface coordinate system X _w Z _w, a X _W-axis component and Z _W-axis component, they X _W axis component and in the world coordinate system based X _w Y _w Z _w It matches the Z _W axis component.

The coordinates of the imaging surface S in the coordinate system X _bu Y _bu are expressed as (x _bu , y _bu ). x _bu and y _bu are an X _bu axis component and a Y _bu axis component in the coordinate system X _bu Y _bu of the imaging surface S, respectively.

A conversion formula between the coordinates (x, y, z) of the camera coordinate system XYZ and the coordinates (x _w , y _w , z _w ) of the world coordinate system X _w Y _w Z _w is expressed by the following formula (1). Is done.

Here, let the focal length of the camera 1 be f. Then, the conversion formula between the coordinates (x _bu , y _bu ) of the coordinate system X _bu Y _bu of the imaging surface S and the coordinates (x, y, z) of the camera coordinate system XYZ is expressed by the following formula (2). expressed.

From the above equations (1) and (2), the coordinates (x _bu , y _bu ) of the coordinate system X _bu Y _bu of the imaging surface S and the coordinates (x _w , z _w ) of the two-dimensional ground coordinate system X _w Z _w (3) is obtained.

Next, a bird's eye view coordinate system X _au Y _au that is a coordinate system for the bird's eye view image is defined. The bird's eye view coordinate system X _au Y _au is a two-dimensional coordinate system having the X _au axis and the Y _au axis as coordinate axes. The coordinates in the bird's eye view coordinate system X _au Y _au are expressed as (x _au , y _au ). The bird's-eye view image is represented by pixel signals of a plurality of pixels arranged two-dimensionally, and the position of each pixel on the bird's-eye view image is represented by coordinates (x _au , y _au ). x _au and y _au are an X _au axis component and a Y _au axis component in the bird's eye view coordinate system X _au Y _au , respectively.

In the following, the relational expression of the coordinates (x _au , y _au ) in the bird's eye view coordinate system X _au Y _{au and} the coordinates (x _bu , y _bu ) in the coordinate system X _bu Y _{bu on the} imaging surface S Will be described.

The bird's-eye view image is obtained by converting an image captured by an actual camera into an image viewed from the viewpoint of a virtual camera (hereinafter referred to as a virtual camera) existing at the position of the virtual viewpoint. More specifically, it is assumed that the virtual camera is located above the ground and the imaging direction is vertically downward, and the bird's-eye view image is converted into an image in which the actual captured image of the camera looks down on the ground surface in the vertical direction. Is. The conversion of the viewpoint when generating the bird's-eye view image from the captured image is generally called viewpoint conversion.

Projection from the two-dimensional ground coordinate system X _w Z _w to the bird's eye view coordinate system X _au Y _{au of the} virtual camera is performed by parallel projection. The height of the virtual camera (i.e., the height of the virtual viewpoint) When the H, the two-dimensional ground surface coordinate system X _w Z _w of the coordinate (x _w, z _w) and bird's eye view coordinate system X _au Y _au coordinates (x _au , Y _au ) is expressed by the following equation (4). The height H of the virtual camera is set in advance. Furthermore, the following formula (5) is obtained by modifying the formula (4).

Substituting the obtained equation (5) into the above equation (3), the following equation (6) is obtained.

From the above equation (6), the coordinates (x _bu , y _bu ) of the coordinate system X _bu Y _{bu on} the imaging surface S are converted into the coordinates (x _au , y _au ) of the bird's eye view coordinate system X _au Y _au Equation (7) is obtained.

Since the coordinates (x _bu , y _bu ) of the coordinate system X _bu Y _{bu on} the imaging surface S represent the coordinates in the captured image of the camera 1, the captured image of the camera 1 is converted into a bird's eye view image by using the above equation (7). Converted. In practice, image processing such as lens distortion correction is appropriately performed on the captured image of the camera 1, and the captured image after the image processing is converted into a bird's eye view image using the above equation (7).

The embodiment of the present invention can be appropriately modified in various ways within the scope of the technical idea shown in the claims.

The present invention can be applied to the field of image processing.

Claims (6)

1. A viewpoint conversion unit that converts each of the images captured by the plurality of imaging devices into a bird's eye view image viewed from a virtual viewpoint;
   An image processing apparatus comprising: an image synthesis unit that synthesizes each bird's eye view image converted by the viewpoint conversion unit to generate a synthesized bird's eye view image;
   An obstacle information extraction unit that extracts obstacle information related to the obstacle based on the bird's eye view image;
   An obstacle original image extraction unit that extracts an obstacle original image that is an image including the obstacle from an image captured by the imaging device based on the obstacle information extracted by the obstacle information extraction unit; ,
   An obstacle bird's-eye image detection unit that detects an obstacle bird's-eye image corresponding to the obstacle in the synthetic bird's-eye view image from the synthetic bird's-eye view image based on the obstacle information extracted by the obstacle information extraction unit. When,
   A replacement processing unit that outputs an image obtained by replacing the obstacle bird's-eye image detected by the obstacle bird's-eye image detection unit with the obstacle original image extracted by the obstacle original image extraction unit as a replacement synthesized bird's-eye view image;
   An image processing apparatus comprising: a video data generation unit configured to generate video data for causing the display device to display the replacement composite bird's-eye view image output from the replacement processing unit.
2. The obstacle information extraction unit
   Difference processing is performed between the target bird's-eye view images in an image of a region where the target bird's-eye view image that is the bird's-eye view image captured by the target imaging device that is at least two of the plurality of imaging devices. The image processing apparatus according to claim 1, wherein the obstacle information is extracted.
3. A viewpoint conversion unit that converts each of the images captured by the plurality of imaging devices into a bird's eye view image viewed from a virtual viewpoint;
   An image processing apparatus comprising: an image synthesis unit that synthesizes each bird's eye view image converted by the viewpoint conversion unit to generate a synthesized bird's eye view image;
   An obstacle information extraction unit that extracts obstacle information related to the obstacle based on the bird's eye view image;
   An obstacle bird's-eye image detection unit that detects an obstacle bird's-eye image corresponding to the obstacle in the synthetic bird's-eye view image from the synthetic bird's-eye view image based on the obstacle information extracted by the obstacle information extraction unit. When,
   Based on the information on the obstacle bird's-eye image detected by the obstacle bird's-eye image detection unit, an obstacle original image that is an image including the obstacle is extracted from the image captured by the imaging device, and the obstacle A replacement processing unit that outputs, as a replacement composite bird's-eye view image, an image obtained by replacing the obstacle bird's-eye image detected by the object bird's-eye image detection unit with the obstacle original image;
   An image processing apparatus comprising: a video data generation unit configured to generate video data for causing the display device to display the replacement composite bird's-eye view image output from the replacement processing unit.
4. A viewpoint conversion step of converting each of the images captured by the plurality of imaging devices into a bird's eye view image viewed from a virtual viewpoint;
   An image processing program that causes a computer to operate as an image processing apparatus, including an image synthesis step of generating a combined bird's-eye view image by combining the bird's-eye view images converted in the viewpoint conversion step,
   Obstacle information extraction step for extracting obstacle information related to the obstacle based on the bird's eye view image;
   An obstacle original image extraction step for extracting an obstacle original image that is an image including the obstacle from an image captured by the imaging device based on the obstacle information extracted in the obstacle information extraction step; ,
   Obstacle bird's-eye image detection step of detecting an obstacle bird's-eye image corresponding to the obstacle in the synthetic bird's-eye view image from the synthetic bird's-eye view image based on the obstacle information extracted in the obstacle information extracting step When,
   A replacement processing step of outputting an image obtained by replacing the obstacle bird's eye image detected by the obstacle bird's eye image detection unit with the obstacle original image extracted by the obstacle original image extraction unit as a replacement synthetic bird's eye view image;
   An image processing program comprising: a video data generation step for generating video data for causing the display device to display the replacement composite bird's-eye view image output from the replacement processing step.
5. An image processing apparatus according to any one of claims 1 to 3,
   An image processing system comprising the plurality of imaging devices and the display device.
6. An image processing method for converting each of images captured by a plurality of imaging devices into a bird's-eye view image viewed from a virtual viewpoint, and combining the converted bird's-eye view images to generate a combined bird's-eye view image,
   Extracting obstacle information related to the obstacle based on the bird's eye view image,
   Based on the extracted obstacle information, an obstacle original image that is an image including the obstacle is extracted from an image captured by the imaging device,
   Based on the extracted obstacle information, an obstacle bird's-eye image corresponding to the obstacle in the synthetic bird's-eye view image is detected from the synthetic bird's-eye view image,
   An image obtained by replacing the obstacle bird's-eye image detected by the obstacle bird's-eye image detection unit with the obstacle original image extracted by the obstacle original image extraction unit is output as a replacement synthesized bird's-eye view image,
   An image processing method for generating video data for displaying the output replacement synthetic bird's-eye view image on a display device.

Images