JP5829851B2 - Image processing system, vehicle apparatus, image processing apparatus, program, and image display method - Google Patents

Description

  The present invention relates to a technique for processing an image obtained by a camera mounted on a vehicle.

  2. Description of the Related Art Conventionally, an in-vehicle device that displays an image showing a region around a vehicle on a display using a captured image obtained by a camera mounted on the vehicle such as an automobile is known. By using such a vehicle-mounted device, a user (typically a driver) can grasp the situation around the vehicle.

  For example, if a back monitor device that is an in-vehicle device that displays an image showing the rear of the vehicle is used, the user can easily grasp an object that exists behind the vehicle when the vehicle is moved backward. Therefore, the user can safely move the vehicle backward while avoiding contact between the vehicle and the object.

  In recent years, an in-vehicle device that generates a composite image viewed from a virtual viewpoint using a plurality of captured images respectively obtained by a plurality of cameras that capture different directions around the vehicle and displays the composite image has also been proposed. (For example, refer to Patent Document 1). The user can intuitively grasp the position of an object existing in the vicinity of the vehicle by checking such a composite image.

Japanese Patent No. 3286306

  However, in order to generate the composite image as described above, complicated and enormous arithmetic processing is required, and thus the in-vehicle device needs to include a high-performance integrated circuit with high arithmetic capability. For this reason, when a user desires such a composite image display function, it is necessary to introduce a relatively high-cost in-vehicle device, and thus a reduction in cost has been demanded.

  The present invention has been made in view of the above problems, and an object of the present invention is to provide a technique capable of displaying a composite image using a vehicular apparatus with relatively low cost.

In order to solve the above-mentioned problem, an invention of claim 1 is an image processing system, comprising: a vehicle device used in a vehicle; and an image processing device configured separately from the vehicle device, The vehicle apparatus includes an acquisition unit that acquires a plurality of captured images respectively obtained by a plurality of cameras mounted on the vehicle, a first transmission unit that transmits the plurality of captured images to the image processing device, First receiving means for receiving a composite image generated by the image processing device based on a plurality of photographed images and showing a region around the vehicle as viewed from a virtual viewpoint; and outputting the composite image to a display device for display Output means for causing the image processing apparatus to generate a composite image based on the second reception means for receiving the plurality of photographed images transmitted from the vehicle device and the plurality of photographed images. Generator When, and a second transmission means for transmitting the composite image device for the vehicle, said output means, said first transmission means based on the plurality of captured images from the transmission of the plurality of captured images During the period until the first receiving means receives the composite image, at least one of the plurality of captured images is output and displayed on the display device .

The invention of claim 2 is an image processing system comprising a vehicle device used in a vehicle and an image processing device configured separately from the vehicle device, wherein the vehicle device includes: An acquisition unit that acquires a plurality of captured images respectively obtained by a plurality of cameras mounted on the vehicle, a first transmission unit that transmits the plurality of captured images to the image processing device, and a plurality of captured images. First receiving means for receiving a composite image generated by the image processing device based on a virtual viewpoint and showing a region around the vehicle, and output means for outputting and displaying the composite image on a display device; The image processing apparatus includes: a second receiving unit configured to receive the plurality of captured images transmitted from the vehicle device; a generating unit configured to generate the composite image based on the plurality of captured images; Composite image Second transmitting means for transmitting to the vehicular apparatus, wherein the acquiring means periodically acquires each of the plurality of captured images, and the vehicular apparatus includes the plurality of captured images. Among them, the information processing apparatus further includes a specifying unit that specifies a change image in which an included subject has changed with time, and when there is the change image, the first transmission unit only includes the change image of the plurality of captured images. And the generation means replaces only the captured image corresponding to the change image among the plurality of captured images used for generation of the composite image in the past with the change image, thereby generating a new composite image. The output means outputs the change image to the display device for display before outputting the composite image based on the change image.

Further, the invention of claim 3, in the image processing system according to claim 2, wherein the vehicle device, when there is the change in image, a warning means for warning the user, the more.

According to a fourth aspect of the present invention, there is provided an image processing system comprising: a vehicle device used in a vehicle; and an image processing device configured separately from the vehicle device, wherein the vehicle device includes: An acquisition unit that acquires a plurality of captured images respectively obtained by a plurality of cameras mounted on the vehicle, a first transmission unit that transmits the plurality of captured images to the image processing device, and a plurality of captured images. First receiving means for receiving a composite image generated by the image processing device based on a virtual viewpoint and showing a region around the vehicle, and output means for outputting and displaying the composite image on a display device; The image processing apparatus includes: a second receiving unit configured to receive the plurality of captured images transmitted from the vehicle device; a generating unit configured to generate the composite image based on the plurality of captured images; Composite image And a second transmission means for transmitting to the device for the vehicle, the image processing apparatus, based on a storage means for storing the ground image showing the surface of a state, the position information indicating the position of said vehicle, said vehicle Combining means for generating a combined image including the ground image indicating the periphery of the image and the composite image, wherein the first transmission means transmits the position information to the image processing device, and the second transmission The means transmits the combined image to the vehicle device, and the output means outputs the combined image to the display device for display.

According to a fifth aspect of the present invention, in the image processing system according to the fourth aspect , the first transmission unit transmits scale information related to the scale of the ground image to the image processing device, and the combination unit includes: The combined image including the ground image having a scale corresponding to the scale information is generated.

The invention of claim 6 is an image processing system, comprising: a vehicle device used in a vehicle; and an image processing device configured separately from the vehicle device, wherein the vehicle device includes: An acquisition unit that acquires a plurality of captured images respectively obtained by a plurality of cameras mounted on the vehicle, a first transmission unit that transmits the plurality of captured images to the image processing device, and a plurality of captured images. First receiving means for receiving a composite image generated by the image processing device based on a virtual viewpoint and showing a region around the vehicle, and output means for outputting and displaying the composite image on a display device; The image processing apparatus includes: a second receiving unit configured to receive the plurality of captured images transmitted from the vehicle device; a generating unit configured to generate the composite image based on the plurality of captured images; Composite image And a second transmission means for transmitting to the device for the vehicle, the image processing apparatus, a plurality of buttons for designating each of the plurality of cameras, further wherein the output means during the composite image The composite image in which the identification information of the button corresponding to the camera is superimposed in the vicinity of the position of each of the plurality of cameras is output and displayed on the display device.

The invention according to claim 7 is the image processing system according to claim 6 , wherein, when any of the plurality of buttons is operated by a user, the generation unit corresponds to the operated button. The composite image in which the vicinity of the camera position is enlarged is generated.

The invention according to claim 8 is the image processing system according to claim 6 , wherein when the user operates any of the plurality of buttons, the second transmission unit specifies the operated button. Button information to be transmitted to the vehicular device, and the output means outputs, based on the button information, the captured image obtained by the camera corresponding to the operated button to the display device for display. Let

The invention according to claim 9 is the image processing system according to any one of claims 6 to 8 , wherein the composite image includes a vehicle image, and the identification information of the button is the vehicle image in the composite image. Is superimposed so as to overlap.

The invention according to claim 10 is the image processing system according to any one of claims 6 to 9 , wherein the image processing apparatus corresponds to the position of each of the plurality of cameras in the composite image. Display means for displaying the composite image on which the identification information of the button to be superimposed is displayed.

The invention according to claim 11 is the image processing system according to claim 10 , wherein the display means is configured so that the longitudinal direction of the screen matches the longitudinal direction of the vehicle image in the composite image. Display an image.

The invention of claim 12 is an apparatus for a vehicle used in a vehicle, the acquisition means for acquiring a plurality of captured images respectively obtained by a plurality of cameras mounted on the vehicle, and the plurality of captured images. Transmitting means for transmitting the image to an image processing device configured separately from the vehicle device, and a region around the vehicle as viewed from a virtual viewpoint, generated by the image processing device based on the plurality of captured images Receiving means for receiving a composite image indicating the output, and output means for outputting and displaying the composite image on a display device , wherein the output means transmits the plurality of captured images after the transmission means transmits the plurality of captured images. During the period until the receiving unit receives the composite image based on a plurality of photographed images, at least one of the plurality of photographed images is output and displayed on the display device .
The invention of claim 13 is an apparatus for a vehicle used in a vehicle, the acquisition means for acquiring a plurality of captured images respectively obtained by a plurality of cameras mounted on the vehicle, and the plurality of captured images. Transmitting means for transmitting the image to an image processing device configured separately from the vehicle device, and a region around the vehicle as viewed from a virtual viewpoint, generated by the image processing device based on the plurality of captured images Receiving means for receiving a composite image indicating the output and output means for outputting and displaying the composite image on a display device, wherein the acquisition means periodically acquires each of the plurality of captured images. And the vehicle apparatus further includes a specifying unit that specifies a change image in which a subject included in the plurality of captured images changes with time, and when there is the change image, the transmission unit includes: Previous Only the change image of the plurality of photographed images is transmitted, and the receiving unit transmits only the photographed image corresponding to the change image among the plurality of photographed images used for generation of the composite image in the past. The new composite image generated by the image processing device is received instead of the change image, and the output means outputs the change image to the display device before outputting the composite image based on the change image. To display.
The invention of claim 14 is an apparatus for a vehicle used in a vehicle, the acquisition means for acquiring a plurality of captured images respectively obtained by a plurality of cameras mounted on the vehicle, and the plurality of captured images. Transmitting means for transmitting the image to an image processing device configured separately from the vehicle device, and a region around the vehicle as viewed from a virtual viewpoint, generated by the image processing device based on the plurality of captured images Receiving means for receiving a composite image indicating the output and output means for outputting and displaying the composite image on a display device, wherein the transmission means transmits position information indicating the position of the vehicle to the image processing device. The receiving means receives a combined image generated by the image processing device based on the position information and including a ground image indicating the state of the ground surface around the vehicle and the composite image, and the output means Is The serial combination image to be displayed is outputted to the display device.
The invention of claim 15 is an apparatus for a vehicle used in a vehicle, the acquisition means for acquiring a plurality of captured images respectively obtained by a plurality of cameras mounted on the vehicle, and the plurality of captured images. Transmitting means for transmitting the image to an image processing device configured separately from the vehicle device, and a region around the vehicle as viewed from a virtual viewpoint, generated by the image processing device based on the plurality of captured images Receiving means for receiving a composite image showing, and output means for outputting and displaying the composite image on a display device, the image processing device, a plurality of buttons for designating each of the plurality of cameras, And the output means superimposes the composite image in which the identification information of the button corresponding to the camera is superimposed on the display device in the vicinity of the position of each of the plurality of cameras in the composite image. To display the force.

The invention of claim 16 is an image display method for displaying an image using a vehicle apparatus used in a vehicle and an image processing apparatus configured separately from the vehicle apparatus, wherein: (a) The vehicle device acquiring a plurality of captured images respectively obtained by a plurality of cameras mounted on the vehicle; and (b) transmitting the plurality of captured images from the vehicle device to the image processing device. And (c) the image processing device generates a composite image showing a region around the vehicle viewed from a virtual viewpoint based on the plurality of captured images, and (d) the composite image is converted into the composite image. Transmitting from the image processing device to the vehicle device; and (e) the vehicle device outputting and displaying the composite image on a display device, wherein the step (e) includes the step In (b), the vehicle apparatus is During the period from when the captured image is transmitted until the vehicle device receives the composite image based on the plurality of captured images in step (d), at least one of the plurality of captured images is sent to the display device. Output and display.
The invention according to claim 17 is an image display method for displaying an image using a vehicle apparatus used in a vehicle and an image processing apparatus configured separately from the vehicle apparatus, wherein (a) The vehicle device acquiring a plurality of captured images respectively obtained by a plurality of cameras mounted on the vehicle; and (b) transmitting the plurality of captured images from the vehicle device to the image processing device. And (c) the image processing device generates a composite image showing a region around the vehicle viewed from a virtual viewpoint based on the plurality of captured images, and (d) the composite image is converted into the composite image. A step of transmitting from the image processing device to the vehicle device; and (e) the vehicle device outputting and displaying the composite image on a display device, wherein the step (a) includes the plurality of steps. Each of the captured images of The image display method further includes: (f) the vehicle device further including a step of identifying a change image in which an included subject has changed in time among the plurality of captured images. When there is an image, the step (b) transmits only the changed image among the plurality of captured images, and the step (c) includes the plurality of captured images used for generating the composite image in the past. Of the images, only the captured image corresponding to the change image is replaced with the change image to generate a new composite image, and the step (e) is performed before outputting the composite image based on the change image. In addition, the change image is output and displayed on the display device.
The invention of claim 18 is an image display method for displaying an image using a vehicle device used in a vehicle and an image processing device configured separately from the vehicle device, wherein: (a) The vehicle device acquiring a plurality of captured images respectively obtained by a plurality of cameras mounted on the vehicle; and (b) transmitting the plurality of captured images from the vehicle device to the image processing device. And (c) the image processing device generates a composite image showing a region around the vehicle viewed from a virtual viewpoint based on the plurality of captured images, and (d) the composite image is converted into the composite image. A step of transmitting from the image processing device to the vehicle device; and (e) the vehicle device outputting and displaying the composite image on a display device, and the image display method comprises (f) The image processing device is provided on the vehicle. Generating a combined image including a ground image indicating the state of the ground surface around the vehicle and the composite image based on position information indicating a position, wherein the step (b) The vehicle apparatus transmits the combined image to the image processing apparatus, the step (d) transmits the combined image from the image processing apparatus to the vehicle apparatus, and the step (e) displays the combined image on the display. Output to device and display.
The invention according to claim 19 is an image display method for displaying an image using a vehicle apparatus used in a vehicle and an image processing apparatus configured separately from the vehicle apparatus. The vehicle device acquiring a plurality of captured images respectively obtained by a plurality of cameras mounted on the vehicle; and (b) transmitting the plurality of captured images from the vehicle device to the image processing device. And (c) the image processing device generates a composite image showing a region around the vehicle viewed from a virtual viewpoint based on the plurality of captured images, and (d) the composite image is converted into the composite image. A step of transmitting from the image processing device to the vehicle device; and (e) the vehicle device outputting and displaying the composite image on a display device, wherein the image processing device includes the plurality of the image processing devices. To specify each camera And the step (e) includes displaying the composite image in which the identification information of the button corresponding to the camera is superimposed in the vicinity of the position of each of the plurality of cameras in the composite image. Output to and display.

  According to the first to nineteenth aspects of the present invention, the composite image can be displayed on the display device even if the vehicle device used in the vehicle does not have a function of generating the composite image viewed from the virtual viewpoint. For this reason, a composite image can be displayed on a display apparatus using the apparatus for vehicles with comparatively low cost.

In particular, according to the first, twelfth, and sixteenth aspects of the invention, at least one of the plurality of photographed images is displayed until the composite image is received. You can see the surrounding area.

Further, according to the inventions of claims 2, 13 and 17 in particular, since the change image is displayed on the display device before the composite image is output, the user can display the image in the vicinity of the vehicle even before the composite image is displayed. The subject that has changed can be confirmed.

In particular, according to the invention of claim 3 , since the user is warned when there is a change image, the user can recognize that there is a subject that has changed around the vehicle.

In particular, according to the inventions of claims 4, 14 and 18, since the combined image including the ground image indicating the periphery of the vehicle and the composite image is displayed, the user can view the vehicle information together with the information on the object in the vicinity of the vehicle. Information on the surrounding surface can also be obtained.

In particular, according to the fifth aspect of the present invention, since the ground image of a scale corresponding to the scale information transmitted from the vehicle device is generated, the ground image of a scale necessary for the vehicle device can be displayed. .

According to the sixth, fifteenth and nineteenth aspects of the present invention, the combined image in which the identification information of the buttons corresponding to the vicinity of the positions of the plurality of cameras in the combined image is superimposed is displayed on the display device. Therefore, the user can clearly grasp which button corresponds to which camera.

In particular, according to the invention of claim 7 , since the composite image in which the vicinity of the position of the camera corresponding to the operated button is enlarged is displayed, the user can enlarge and confirm the desired position of the composite image. Can do.

In particular, according to the eighth aspect of the present invention, since the captured image obtained by the camera corresponding to the operated button is displayed, the user can confirm the desired captured image.

In particular, according to the ninth aspect of the present invention, it is possible to prevent subjects around the vehicle from being hidden by the button identification information.

In particular, according to the invention of claim 10 , the composite image in which the identification information of the button corresponding to the vicinity of the position of each of the plurality of cameras in the composite image is superimposed in the composite image is displayed. For this reason, the user can also grasp which button corresponds to which camera on the display means of the image processing apparatus.

In particular, according to the invention of claim 11 , the composite image is displayed so that the longitudinal direction of the screen and the longitudinal direction of the vehicle image in the composite image coincide with each other. Can be displayed.

FIG. 1 is a diagram illustrating an overview of an image processing system according to the first embodiment. FIG. 2 is a diagram illustrating a configuration of the in-vehicle device according to the first embodiment. FIG. 3 is a diagram illustrating a position where the in-vehicle camera is arranged. FIG. 4 is a diagram illustrating a configuration of the portable device according to the first embodiment. FIG. 5 is a diagram illustrating a method for generating a composite image. FIG. 6 is a diagram illustrating a processing flow of the image processing system according to the first embodiment. FIG. 7 is a diagram illustrating an example of a display image. FIG. 8 is a diagram illustrating a processing flow of the image processing system according to the second embodiment. FIG. 9 is a diagram illustrating a transition example of the display image. FIG. 10 is a diagram illustrating transition of the display image. FIG. 11 is a diagram illustrating a configuration of the in-vehicle device according to the third embodiment. FIG. 12 is a diagram illustrating a transition example of the display image. FIG. 13 is a diagram illustrating a processing flow of the image processing system according to the third embodiment. FIG. 14 is a diagram illustrating the relationship between the operation unit of the portable device and the display content of the in-vehicle device. FIG. 15 is a diagram illustrating a transition example of the display image. FIG. 16 is a diagram illustrating a processing flow of the image processing system according to the fourth embodiment. FIG. 17 is a diagram illustrating an example of a display image. FIG. 18 is a diagram illustrating a transition example of the display image. FIG. 19 is a diagram illustrating a processing flow of the image processing system according to the fifth embodiment. FIG. 20 is a diagram illustrating an overview of an image processing system according to the sixth embodiment. FIG. 21 is a diagram illustrating a configuration of the in-vehicle device according to the sixth embodiment. FIG. 22 is a diagram illustrating a configuration of a server device according to the sixth embodiment. FIG. 23 is a diagram illustrating a processing flow of the image processing system according to the sixth embodiment. FIG. 24 is a diagram illustrating a configuration of the in-vehicle device according to the seventh embodiment. FIG. 25 is a diagram illustrating a configuration of a server device according to the seventh embodiment. FIG. 26 is a diagram illustrating a processing flow of the image processing system according to the sixth embodiment. FIG. 27 is a diagram illustrating an example of a combined image. FIG. 28 is a diagram illustrating transition of a display image. FIG. 29 is a diagram illustrating a processing flow of the image processing system for changing the scale.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings.

<1. First Embodiment>
<1-1. System overview>
FIG. 1 is a diagram illustrating an overview of an image processing system 10 according to the first embodiment. The image processing system 10 includes an in-vehicle device 2 mounted on a vehicle 9 such as an automobile, and a portable device 3 configured separately from the in-vehicle device 2.

  Both the in-vehicle device 2 and the portable device 3 have a wireless communication function based on a predetermined communication method such as Bluetooth (registered trademark). Therefore, the in-vehicle device 2 and the portable device 3 can transmit / receive data to / from each other by wireless communication. The in-vehicle device 2 and the portable device 3 may be physically connected by a communication cable so that the in-vehicle device 2 and the portable device 3 can transmit and receive data to each other through wired communication.

  The in-vehicle device 2 is a vehicle device used in the vehicle 9, and includes a plurality of in-vehicle cameras 5 that capture the periphery of the vehicle 9 and a display 29. The in-vehicle device 2 has a function of displaying images based on captured images obtained by the plurality of in-vehicle cameras 5 on the display 29 in the vehicle interior of the vehicle 9.

  Moreover, the portable apparatus 3 is a portable apparatus possessed by a user who gets on the vehicle 9, such as a mobile phone or a smartphone. The portable device 3 has an image processing function as well as a call function, and functions as an image processing device. One of the image processing functions provided in the portable device 3 is a function of generating a composite image viewed from a virtual viewpoint.

  In the image processing system 10, since the portable device 3 has a function of generating a composite image, the composite image is displayed on the display 29 of the in-vehicle device 2 even if the in-vehicle device 2 does not have a function of generating a composite image. It can be displayed. The in-vehicle device 2 transmits a plurality of captured images respectively obtained by the plurality of in-vehicle cameras 5 to the portable device 3. The portable device 3 generates a composite image viewed from a virtual viewpoint based on the plurality of captured images. The portable device 3 returns the generated composite image to the in-vehicle device 2, and the in-vehicle device 2 displays the composite image on the display 29. Hereinafter, the configuration and processing of such an image processing system 10 will be described in detail.

<1-2. Configuration of in-vehicle device>
First, the configuration of the in-vehicle device 2 will be described. FIG. 2 is a diagram illustrating a configuration of the in-vehicle device 2. The in-vehicle device 2 includes a main body 21 and an operation unit 22 together with the plurality of in-vehicle cameras 5 and the display 29 described above.

  The display 29 is a thin display device including a liquid crystal panel, for example. The display 29 is integrated with the main body 21 and displays a display image output from the main body 21. The display 29 is arranged on an instrument panel or the like in the passenger compartment so that the display screen can be visually recognized by the user.

  Each of the plurality of in-vehicle cameras 5 includes a lens and an imaging device, and electronically acquires a captured image showing the periphery of the vehicle 9. Each of the in-vehicle cameras 5F, 5B, 5L, and 5R periodically shoots at a predetermined cycle (for example, 1/30 second).

  Each of the plurality of in-vehicle cameras 5 is disposed at an appropriate position of the vehicle 9 separately from the main body 21 and inputs the acquired captured image to the main body 21. The plurality of in-vehicle cameras 5 include a front camera 5F, a back camera 5B, a left side camera 5L, and a right side camera 5R. These four in-vehicle cameras 5F, 5B, 5L, and 5R are arranged at different positions of the vehicle 9 and photograph different directions around the vehicle 9.

  FIG. 3 is a diagram illustrating positions where the in-vehicle cameras 5F, 5B, 5L, and 5R are arranged on the vehicle 9. The front camera 5F is provided in the vicinity of the left and right center of the front end of the vehicle 9, and its optical axis 5Fa is directed to the front (straight direction) of the vehicle 9. The back camera 5B is provided in the vicinity of the left and right center of the rear end of the vehicle 9, and its optical axis 5Ba is directed to the rear of the vehicle 9 (the direction opposite to the straight traveling direction). The left side camera 5L is provided on the left side mirror 93L of the vehicle 9, and its optical axis 5La is directed to the left side of the vehicle 9 (a direction orthogonal to the straight traveling direction). The right side camera 5R is provided on the right side mirror 93R of the vehicle 9, and its optical axis 5Ra is directed to the right side of the vehicle 9 (a direction orthogonal to the straight traveling direction).

  A fish-eye lens or the like is adopted as the lens of these in-vehicle cameras 5F, 5B, 5L, and 5R, and each of the in-vehicle cameras 5F, 5B, 5L, and 5R has an angle of view θ of 180 degrees or more. For this reason, it is possible to photograph the entire periphery of the vehicle 9 by using the four in-vehicle cameras 5F, 5B, 5L, and 5R.

  Returning to FIG. 1, the operation unit 22 is a member that can be directly operated by the user, and includes a touch panel and operation buttons. The touch panel is provided so as to overlap the display screen of the display 29, and the operation buttons are provided around the display screen of the display 29. When the user operates the operation unit 22, a signal indicating the operation content is input to the main body unit 21.

  The main body unit 21 includes a control unit 23, an image acquisition unit 24, a storage unit 25, an image output unit 26, a speaker 27, and a data communication unit 28.

  The image acquisition unit 24 acquires captured images respectively obtained by the four in-vehicle cameras 5F, 5B, 5L, and 5R. Therefore, the image acquisition unit 24 acquires four captured images that respectively indicate the front, rear, left side, and right side of the vehicle 9. The image acquisition unit 24 periodically acquires each of these four captured images at a predetermined cycle (for example, 1/30 seconds).

  The storage unit 25 is a non-volatile memory such as a flash memory, for example, and stores various types of information. The storage unit 25 stores vehicle information 25a and a program 25b. The vehicle information 25a is information that identifies the type of the vehicle 9 on which the in-vehicle device 2 is mounted. The type of the vehicle 9 includes a model (vehicle type), a body color, and the like.

  The image output unit 26 outputs and displays a display image to be displayed on the display 29. Such display images include photographed images and composite images as necessary.

  The speaker 27 outputs various sounds such as an operation sound and a warning sound based on the electrical signal from the control unit 23.

  The data communication unit 28 communicates with an external communication device by wireless communication. The data communication unit 28 can exchange data with the portable device 3. The data communication unit 28 transmits a plurality of captured images to the portable device 3 and receives a composite image from the portable device 3.

  The control unit 23 is a microcomputer that comprehensively controls the entire in-vehicle device 2. The control unit 23 includes a CPU, a RAM, a ROM, and the like, and various control functions are realized by the CPU performing arithmetic processing according to a program 25b stored in the storage unit 25.

  The display control unit 23a shown in the drawing is one of the functions of the control unit 23 realized by the CPU performing arithmetic processing according to the program 25b.

  The display control unit 23 a performs various controls related to the display on the display 29. The display control unit 23a generates, for example, a display image to be displayed on the display 29 using a captured image or a composite image according to the operation mode at that time.

  The in-vehicle device 2 includes, as operation modes, a normal mode that mainly displays captured images obtained by the in-vehicle camera 5 and an all-around mode that mainly displays a composite image viewed from a virtual viewpoint. The in-vehicle device 2 is in the normal mode when it is activated, but shifts to the all-around mode when communication with the portable device 3 is established and there is an instruction from the user.

  In the normal mode, the display control unit 23a generates a display image including a captured image obtained by any of the in-vehicle cameras 5 selected by the user. Then, the display control unit 23 a controls the image output unit 26 to cause the display 29 to output this display image. Thereby, a display image including the photographed image is displayed on the display 29.

  On the other hand, in the all-around mode, the display control unit 23 a controls the data communication unit 28 to transmit four captured images to the portable device 3. Then, the display control unit 23 a controls the data communication unit 28 to receive the composite image generated by the portable device 3. Then, the display control unit 23a generates a display image including the composite image, and controls the image output unit 26 to output the display image to the display 29. As a result, a display image including the composite image viewed from the virtual viewpoint is displayed on the display 29.

<1-3. Configuration of portable device>
Next, the configuration of the portable device 3 will be described. FIG. 4 is a diagram illustrating a configuration of the portable device 3. The portable device 3 includes a control unit 31, a microphone 32, a speaker 33, a voice communication unit 34, an operation unit 35, a display 36, a data communication unit 37, and a storage unit 38.

  The microphone 32 converts surrounding sounds into electrical signals. The microphone 32 converts, for example, a voice uttered by a user during a call into an electric signal. The speaker 33 outputs various sounds based on the electrical signal. The speaker 33 outputs, for example, a voice uttered by the other party. The voice communication unit 34 transmits / receives a voice electrical signal for a call to / from the base station. The microphone 32, the speaker 33, and the voice communication unit 34 realize the call function of the portable device 3.

  The operation unit 35 is a member that can be directly operated by the user, and includes a plurality of operation buttons. When the user operates the operation unit 35, a signal indicating the operation content is input to the control unit 31.

  The display 36 includes, for example, a liquid crystal panel, and displays various images based on signals from the control unit 31.

  The data communication unit 37 communicates with an external communication device by wireless communication. The data communication unit 37 can exchange data with the in-vehicle device 2. The data communication unit 37 receives a plurality of captured images from the in-vehicle device 2 and transmits a composite image to the in-vehicle device 2.

  The storage unit 38 is a non-volatile memory such as a flash memory, for example, and stores various types of information. The storage unit 38 stores a plurality of vehicle images 90 and a program 39. Each of the plurality of vehicle images 90 is image data indicating the appearance of the vehicle, and is used when generating a composite image. The storage unit 38 stores in advance vehicle images 90 corresponding to various vehicle types (models and body colors).

  The control unit 31 is a microcomputer that comprehensively controls the entire portable device 3. The control unit 31 includes a CPU, a RAM, a ROM, and the like, and various control functions are realized by the CPU performing arithmetic processing according to a program 39 stored in the storage unit 38.

  A composite image generation unit 31 a shown in the drawing is one of the functions of the control unit 31 realized by the CPU performing arithmetic processing according to the program 39. Based on the four captured images acquired by the in-vehicle device 2, the composite image generation unit 31 a generates a composite image that indicates an area around the vehicle 9 as viewed from the virtual viewpoint. The composite image generation unit 31a is validated when communication is established with the in-vehicle device 2 and there is a user instruction.

  FIG. 5 is a diagram for explaining a method in which the composite image generation unit 31a generates a composite image.

  When shooting is performed with each of the front camera 5F, the back camera 5B, the left side camera 5L, and the right side camera 5R of the in-vehicle device 2, four shootings that respectively indicate the front, rear, left side, and right side of the vehicle 9 are performed. Images PF, PB, PL, and PR are acquired. These four captured images PF, PB, PL, and PR include data around the entire vehicle 9.

  These four captured images PF, PB, PL, and PR are transmitted from the in-vehicle device 2 to the portable device 3. The composite image generation unit 31a of the portable device 3 generates a composite image CP using these four captured images PF, PB, PL, and PR.

  First, data (values of each pixel) of the four captured images PF, PB, PL, and PR are projected onto a projection surface TS that is a three-dimensional curved surface in a virtual three-dimensional space. The projection surface TS has, for example, a substantially hemispherical shape (a bowl shape). The center part (bottom part of the bowl) of the projection surface TS is determined as a position where the vehicle 9 exists. On the other hand, the other part of the projection surface TS is associated with the data of the captured images PF, PB, PL, PR, and the corresponding data is projected. Information indicating the correspondence between the data of the captured images PF, PB, PL, PR and the position of the projection plane TS is stored in the storage unit 25 in advance.

  A region corresponding to the front side of the vehicle 9 on the projection surface TS is projected with data of a captured image (hereinafter referred to as “front image”) PF obtained by the front camera 5F. On the projection surface TS, an area corresponding to the rear of the vehicle 9 is projected with data of a captured image (hereinafter referred to as “back image”) PB obtained by the back camera 5B. In addition, a region corresponding to the left side of the vehicle 9 on the projection surface TS is projected with data of a captured image PL (hereinafter referred to as “left side image”) PL obtained by the left side camera 5L. The area corresponding to the right side of the image is projected data of a captured image (hereinafter referred to as “right side image”) PR obtained by the right side camera 5R.

  Thus, when the data of each part of the projection surface TS is determined, the virtual viewpoint VP is set for the three-dimensional space. The virtual viewpoint VP can be set at an arbitrary viewpoint position in the three-dimensional space in an arbitrary viewing direction. Then, a region included in a predetermined viewing angle as viewed from the set virtual viewpoint VP is cut out as an image in the projection plane TS. The image obtained by cutting out in this way becomes a composite image CP showing the area around the vehicle 9 as viewed from the virtual viewpoint VP.

  Furthermore, the vehicle image 90 is superimposed on the position where the vehicle 9 exists in the composite image CP generated in this way. As the vehicle image 90 superimposed on the composite image CP, an image corresponding to the type of the vehicle 9 is read from the storage unit 25 and used. As a result, the vehicle image 90 is included in the composite image CP. The user can grasp the position of the subject around the vehicle 9 by the relative position with respect to the vehicle 9 by visually recognizing the composite image CP.

  As a result of such processing, for example, as shown in the figure, when a virtual viewpoint VPa is set with the viewpoint position directly above the vehicle 9 and the visual field direction directly below, the composition for overlooking the vehicle 9 and the area around the vehicle 9 An image CPa is generated. Further, when the virtual viewpoint VPb is set with the viewpoint position at the left rear of the vehicle 9 and the visual field direction at the front of the vehicle 9, the vehicle 9 and the vehicle 9 A composite image CPb showing the peripheral region is generated.

<1-4. Process flow>
Next, a processing flow of the image processing system 10 will be described. FIG. 6 is a diagram showing a flow of processing of the image processing system 10. The left side in the figure shows the processing flow of the in-vehicle device 2, and the right side in the figure shows the processing flow of the portable device 3. These processes are executed when communication is established between the in-vehicle device 2 and the portable device 3 and the user performs a predetermined operation on each device (in the case of the all-around mode). . In the in-vehicle device 2, the process shown in FIG. 6 is repeatedly executed at a predetermined cycle (for example, 1/30 second) under the control of the display control unit 23 a. Moreover, in the portable apparatus 3, the process shown in FIG. 6 is performed in response to the signal from the vehicle-mounted apparatus 2 by control of the control part 31. FIG.

  First, each of the four vehicle-mounted cameras 5F, 5B, 5L, and 5R of the vehicle-mounted device 2 images the periphery of the vehicle 9. And the image acquisition part 24 of the main-body part 21 acquires four picked-up images PF, PB, PL, PR obtained by four vehicle-mounted cameras 5F, 5B, 5L, 5R, respectively (step SA11).

  Next, the data communication unit 28 of the in-vehicle device 2 transmits the four captured images PF, PB, PL, PR to the portable device 3 by wireless communication (step SA12). In addition, the data communication unit 28 transmits the vehicle information 25a and the size information to the portable device 3 together with the captured images PF, PB, PL, and PR. As described above, the vehicle information 25 a is information that identifies the type of the vehicle 9. On the other hand, the size information is information for designating the size of the composite image CP suitable for display on the display 29 of the in-vehicle device 2 by the number of vertical and horizontal pixels.

  The data communication unit 37 of the portable device 3 receives these four captured images PF, PB, PL, PR, vehicle information 25a, and size information transmitted from the in-vehicle device 2 (step SB11).

  The portable device 3 generates a composite image CP in response to receiving these pieces of information. That is, the composite image generation unit 31a generates a composite image CP that indicates a region around the vehicle 9 viewed from the virtual viewpoint based on the four captured images by the above-described method (step SB12). As described above, the virtual viewpoint VP can be set at an arbitrary viewpoint position and an arbitrary visual field direction, but in this embodiment, a virtual viewpoint VPa is set with the viewpoint position directly above the vehicle 9 and the visual field direction immediately below. . As a result, a composite image CP is generated that overlooks the vehicle 9 and the area around the vehicle 9.

  Further, when generating the composite image CP, the composite image generation unit 31a reads the vehicle image 90 corresponding to the type of the vehicle 9 from the storage unit 38 based on the vehicle information 25a transmitted from the in-vehicle device 2, and this The vehicle image 90 is included in the composite image CP. As a result, a vehicle image 90 having a shape that conforms to the model of the actual vehicle 9 and a color that conforms to the body color of the actual vehicle 9 is included in the composite image CP.

  Further, the composite image generation unit 31a generates a composite image CP having a size specified by the size information based on the size information transmitted from the in-vehicle device 2. As a result, a composite image CP having a size suitable for display on the in-vehicle device 2 is generated.

  Next, the data communication unit 37 of the portable device 3 transmits the generated composite image CP to the in-vehicle device 2 by wireless communication (step SB13). The data communication unit 28 of the in-vehicle device 2 receives this composite image CP transmitted from the portable device 3 (step SA13).

  After transmitting the four captured images PF, PB, PL, and PR (after step SA12), the in-vehicle device 2 waits for processing until the composite image CP is received, and responds to reception of the composite image CP (step SA13). To resume processing.

  Specifically, first, the display control unit 23a generates a display image including the received composite image CP. Since the composite image CP has already been adjusted to a size suitable for display, the display control unit 23a can easily generate a display image. Then, the image output unit 26 outputs the generated display image to the display 29. As a result, a display image including the composite image CP is displayed on the display 29 (step SA14).

  FIG. 7 is a diagram showing an example of the display image DP displayed on the display 29 in this way. As shown in the figure, the display image DP includes a composite image CP showing a region around the vehicle 9 as viewed from the virtual viewpoint. The composite image CP includes a vehicle image 90 corresponding to the model of the vehicle 9 and the body color. As described above, since the process shown in FIG. 6 is repeated at a predetermined cycle, the content of the composite image CP is also updated at a predetermined cycle. Therefore, the user can grasp an object existing around the vehicle 9 by visually recognizing such display image DP.

  As described above, the image processing system 10 includes the in-vehicle device 2 used in the vehicle 9 and the portable device 3 configured separately from the in-vehicle device 2. The in-vehicle device 2 transmits a plurality of captured images respectively obtained by the plurality of in-vehicle cameras 5 to the portable device 3. The portable device 3 generates a composite image CP viewed from a virtual viewpoint based on the plurality of captured images. Then, the portable device 3 returns the generated composite image CP to the in-vehicle device 2, and the in-vehicle device 2 displays the composite image CP on the display 29. Therefore, the composite image CP can be displayed on the display 29 even if the in-vehicle device 2 does not have a function of generating the composite image CP. Thereby, even if it is a case where the vehicle-mounted apparatus with comparatively low cost is used, synthetic | combination image CP can be displayed on the display 29. FIG.

  Even if the user introduces the in-vehicle device 2 that does not have the function of generating the composite image CP, the user can display the composite image CP by introducing the portable device 3 that has the function of generating the composite image CP. Functions can be added later. Moreover, if the portable apparatus 3 acquires the said program from the predetermined | prescribed server apparatus and recording medium in which the program was memorize | stored, it can be provided with the production | generation function of such a composite image CP. For this reason, such a display function of the composite image CP can be easily added.

  The data communication unit 28 of the in-vehicle device 2 transmits size information specifying the size of the composite image CP to the portable device 3. On the other hand, the composite image generation unit 31a of the portable device 3 generates a composite image CP having a size specified by the size information. Therefore, the portable device 3 can generate a composite image CP having a size according to a request on the in-vehicle device 2 side. For this reason, the portable device 3 can generate a composite image CP of an appropriate size according to the hardware of the in-vehicle device that is the communication partner even if the in-vehicle device that is the communication partner is changed.

  In addition, the data communication unit 28 of the in-vehicle device 2 transmits vehicle information 25 a indicating the type (model and body color) of the vehicle 9 to the portable device 3. On the other hand, the composite image generation unit 31a of the portable device 3 includes the vehicle image 90 corresponding to the type of the vehicle 9 in the composite image CP based on the vehicle information 25a. Therefore, the vehicle image 90 corresponding to the type of the actual vehicle 9 can be included in the composite image CP, and a realistic composite image CP can be generated.

<2. Second Embodiment>
Next, a second embodiment will be described. In the first embodiment, the in-vehicle device 2 waits for processing during a period from when the captured image is transmitted to when the composite image is received. The time from when the in-vehicle device 2 transmits the captured image to when the composite image is received depends on the communication speed between the in-vehicle device 2 and the portable device 3 and the processing speed of the portable device 3. Therefore, depending on these speeds, the composite image CP may not be displayed on the display 29 for a while immediately after the vehicle-mounted device 2 is shifted to the all-around mode.

  For this reason, the in-vehicle device 2 according to the second embodiment displays the photographed image on the display 29 during a period from when the photographed image is transmitted to when the composite image is received. Since the configuration and processing of the image processing system 10 of the second embodiment are substantially the same as those of the first embodiment, the following description will be focused on differences from the first embodiment.

  FIG. 8 is a diagram illustrating a processing flow of the image processing system 10 according to the second embodiment. The processing of the in-vehicle device 2 in steps SA21, SA22, SA25, and SA26 in FIG. 8 is the same as the processing in steps SA11, SA12, SA13, and SA14 in FIG. Moreover, the process of the portable apparatus 3 of step SB21-SB23 in FIG. 8 is the same as that of step SB11-SB13 of FIG. Accordingly, also in the present embodiment, the in-vehicle device 2 transmits a plurality of captured images to the portable device 3, and the portable device 3 generates a composite image CP and transmits it to the in-vehicle device 2, and the in-vehicle device 2 The composite image CP is displayed.

  However, in the second embodiment, steps SA23 and SA24 are executed in a period (a period from step SA22 to step SA25) from when the in-vehicle device 2 transmits a plurality of captured images to when a composite image is received. Is done.

  First, it is determined whether or not the composite image CP is already displayed on the display 29 (step SA23). When the process in FIG. 8 is performed for the second time or later, the composite image CP is already displayed. Therefore, in this case (Yes in step SA23), priority is given to the display of the composite image CP, and the process directly proceeds to step SA25.

  On the other hand, when the process in FIG. 8 is the first time, that is, immediately after the transition to the all-around mode, the composite image CP is not displayed on the display 29. In this case (No in step SA23), the display control unit 23a generates a display image DP including the same four captured images PF, PB, PL, PR as those transmitted to the portable device 3. Then, the display control unit 23 a controls the image output unit 26 to cause the display 29 to output this display image DP. As a result, the display image DP including the four captured images PF, PB, PL, and PR is displayed on the display 29 (step SA24). Thereafter, the processing proceeds to step SA25.

  The upper part of FIG. 9 shows an example of the display image DP displayed in step SA24. This display image DP includes four captured images PF, PB, PL, and PR. Thereafter, when the in-vehicle device 2 receives the composite image CP from the portable device 3, a display image DP including the composite image CP as shown in the lower part of FIG. Even before the composite image CP is displayed, the user visually recognizes the display image DP including the captured images PF, PB, PL, and PR as shown in the upper part of FIG. Can be grasped.

  As described above, in the image processing system 10 according to the second embodiment, the data communication unit 28 of the in-vehicle device 2 transmits a plurality of captured images and then receives a composite image CP based on the plurality of captured images. Until this period, the image output unit 26 outputs the plurality of captured images to the display 29 for display. For this reason, even before the composite image CP is displayed, the user can grasp the situation around the vehicle 9.

  In the above description, all four captured images PF, PB, PL, and PR are displayed during the period from when the in-vehicle device 2 transmits the captured image to when the composite image is received. It is only necessary to display at least one of the images PF, PB, PL, and PR. For example, you may make it display only the back image PB which shows the back which becomes the blind spot most easily. Further, as shown in FIG. 10, the four captured images PF, PB, PL, and PR may be sequentially displayed in time series. Further, the in-vehicle device 2 only needs to display the captured image during a period from when the captured image is transmitted to when the composite image is received, and displays the captured image before transmission of the captured image or after reception of the composite image. May be.

<3. Third Embodiment>
Next, a third embodiment will be described. In the first embodiment, the in-vehicle device 2 always transmits all four captured images to the portable device 3. However, when all of the four captured images are always transmitted in this way, the amount of data to be transmitted increases, so that the overall processing speed may be reduced.

  For this reason, the in-vehicle device 2 of the third embodiment makes the cycle for executing all image transmissions for transmitting all four captured images to the portable device 3 longer than that of the first embodiment. . On the other hand, the in-vehicle device 2 is configured to transmit only a captured image in which an included subject is temporally changed to the portable device 3 between all image transmissions. Since the configuration and processing of the image processing system 10 of the third embodiment are substantially the same as those of the first embodiment, the following description will be focused on differences from the first embodiment.

  FIG. 11 is a diagram illustrating a configuration of the in-vehicle device 2 according to the third embodiment. The in-vehicle device 2 according to the third embodiment has a change detection unit 23b and a warning control unit 23c in addition to the display control unit 23a as functions of the control unit 23 realized by the CPU performing arithmetic processing according to the program 25b. It has. Other configurations of the in-vehicle device 2 of the third embodiment are the same as those of the first embodiment.

  The change detection unit 23b specifies a change image in which an included subject has changed in time among captured images periodically obtained by the in-vehicle cameras 5F, 5B, 5L, and 5R. That is, the change detection unit 23b takes a difference between corresponding pixel values in captured images (previous frame and current frame) obtained continuously in time with the same in-vehicle camera, and the sum of the differences is obtained. A captured image that exceeds a predetermined threshold is specified as a change image.

  More specifically, the change detection unit 23b determines the difference from the front image PF of the previous frame for the front image PF, the difference from the back image PB of the previous frame for the back image PB, and the previous frame for the left side image PL. For the right side image PF, the difference from the right side image PR of the previous frame is derived respectively. Then, the change detection unit 23b determines that the subject included in the captured image in which the sum of the derived differences exceeds the predetermined threshold among the four captured images PF, PB, PL, and PR. The change detection unit 23b specifies a captured image including the subject that has changed in this way as a change image.

  The warning control unit 23c warns the user when the change detection unit 23b determines that there is a change image. When there is a change image, the warning control unit 23c controls the speaker 27 to output a predetermined warning sound from the speaker 27.

  In addition, when the change detection unit 23b determines that there is a change image, the display control unit 23a of the present embodiment displays the change image on the display 29. Even when there is a change image, that is, when a change occurs in a subject around the vehicle 9, there is a time lag before the change in the subject is reflected in the content of the composite image CP. For this reason, if there is a change image, the display control unit 23a of the present embodiment displays the change image on the display 29 prior to the display of the composite image CP generated based on the change image. That is, the image output unit 26 outputs and displays the change image on the display 29 before outputting the composite image CP based on the change image.

  FIG. 12 is a diagram showing a transition of the display image DP displayed on the display 29 in the all-around mode of the present embodiment. When there is no change image, that is, when there is no change in the subject, only the composite image CP is included in the display image DP (state ST1).

  In such a state, when there is a change image, that is, when a change occurs in the subject, the change image is included in the display image DP, and the display image DP including the change image is displayed on the display 29 (state) ST2). In the example of FIG. 12, a change has occurred in the subject Sb in the back image PB, and this back image PB is displayed on the display 29 as a change image. Note that the composite image CP is also included in the display image DP, but since the time immediately after the subject Sb changes, the state of the subject Sb is not reflected in the content of the composite image CP.

  Thereafter, when the composite image CP reflecting the state of the subject Sb generated based on the change image can be displayed, the composite image CP is displayed on the display 29. On the other hand, the change image (back image PB) is not displayed on the display 29 (state ST3).

  By visually recognizing such display image DP, the user can recognize that the subject has changed (that is, there is a moving object around the vehicle 9). Further, when a change occurs in the subject, the change image including the subject is displayed before the composite image CP including the subject is displayed, so that the user can quickly confirm the subject.

  FIG. 13 is a diagram illustrating a processing flow of the image processing system 10 according to the third embodiment. Also in the present embodiment, in the in-vehicle device 2, the process shown in FIG. 13 is repeatedly executed at a predetermined first period (for example, 1/30 second) under the control of the display control unit 23 a. However, in the third embodiment, a specific process different from the other cases is executed every second period (for example, 1 second) longer than the first period.

  In the specific process for each second period (Yes in step SA30), substantially the same process as in the first embodiment is executed. That is, first, the image acquisition unit 24 of the in-vehicle device 2 acquires four captured images PF, PB, PL, PR (step SA31), and the data communication unit 28 of the in-vehicle device 2 acquires four captured images PF, PB, PL. , PR all images are transmitted to the portable device 3 (step SA32).

  When the data communication unit 37 of the portable device 3 receives these four captured images PF, PB, PL, PR (step SB31), all of the four captured images received by the storage unit 38 of the portable device 3 are stored. To do. Thereby, when the memory | storage part 38 memorize | stored the picked-up image in the past, all four picked-up images PF, PB, PL, PR are updated (step SB32). Then, the composite image generation unit 31a generates a composite image CP using the four captured images PF, PB, PL, and PR stored in the storage unit 38 (step SB33), and the data communication unit 37 is generated. The composite image CP is transmitted to the in-vehicle device 2 (step SB34).

  When the data communication unit 28 of the in-vehicle device 2 receives this composite image CP (step SA38), the image output unit 26 of the in-vehicle device 2 outputs and displays the display image DP including the composite image CP on the display 29 (step SA39). ).

  Since this specifying process is executed every second period, a new composite image CP reflecting all the contents of the four captured images PF, PB, PL, PR is displayed on the display 29 every second period. become.

  On the other hand, in cases other than the specific processing (between all image transmission and all image transmission) (No in step SA30), first, the image acquisition unit 24 of the in-vehicle device 2 has four captured images PF, PB, and PL. , PR are acquired (step SA33). Next, the change detection unit 23b specifies a change image in which the included subject has temporally changed from these four captured images PF, PB, PL, and PR by the above-described method. If there is no change image (No in step SA34), the process ends.

  If a change image exists (Yes in step SA34), the data communication unit 28 of the in-vehicle device 2 sends only the change image of the four captured images PF, PB, PL, PR to the portable device 3. Transmit (step SA35).

  When the data communication unit 37 of the portable device 3 receives the change image (step SB31), the change image received by the storage unit 38 of the portable device 3 is stored. As a result, of the four captured images PF, PB, PL, PR obtained in the past stored in the storage unit 38, only the captured image corresponding to the changed image is updated (step SB32). For example, when the change image is the front image PF, only the front image PF is updated, and when the change image is the back image PB, only the back image PB is updated. Therefore, the past photographed image that does not correspond to the change image is maintained in the state stored in the storage unit 38.

  Then, the composite image generation unit 31a generates a composite image CP using the four captured images PF, PB, PL, PR stored in the storage unit 38 (step SB33). As a result, the composite image generation unit 31a replaces only the captured image corresponding to the change image among the four captured images PF, PB, PL, and PR used to generate the composite image CP in the past, and changes the new image. A composite image CP is generated. As a result, a new composite image CP including the changed subject is generated. When a new composite image CP is generated, the data communication unit 37 transmits the generated composite image CP to the in-vehicle device 2 (step SB34).

  Moreover, after transmitting the change image (after step SA35), the in-vehicle device 2 displays the change image until the composite image CP based on the change image is received. Specifically, the display control unit 23 a generates a display image DP including the same change image as that transmitted to the portable device 3. Then, the display control unit 23 a controls the image output unit 26 to cause the display 29 to output this display image DP. Thereby, the display image DP including the change image as in the state ST2 of FIG. 12 is displayed on the display 29 (step SA36).

  Next, the warning control unit 23c controls the speaker 27 to output a predetermined warning sound such as a beep sound from the speaker 27 (step SA37). Thereby, when there is a change image, that is, when there is a moving object around the vehicle 9, the user is notified of that.

  Thereafter, when the data communication unit 28 of the in-vehicle device 2 receives the composite image CP based on the change image (step SA38), the display control unit 23a generates a display image DP including the composite image CP based on the change image. Then, the image output unit 26 outputs the display image DP to the display 29 for display. As a result, the composite image CP including the changed subject Sb as in the state ST3 in FIG. 12 is displayed on the display 29 (step SA39).

  As described above, in the image processing system 10 according to the third embodiment, the change detection unit 23b of the in-vehicle device 2 identifies a change image in which a subject included in the four captured images has changed over time. When there is such a change image, the data communication unit 28 of the in-vehicle device 2 transmits only the change image to the portable device 3. Then, the composite image generation unit 31a of the portable device 3 generates a new composite image by replacing only the captured image corresponding to the change image among the four captured images used for generating the composite image CP in the past with the change image. To do. Therefore, it is possible to generate a composite image CP including a subject that has changed around the vehicle 9 while effectively reducing the amount of data transmitted from the in-vehicle device 2 to the portable device 3 and improving the processing speed. .

  In addition, when there is a change image, the image output unit 26 of the in-vehicle device 2 outputs the change image to the display 29 and displays it before outputting the composite image CP based on the change image. For this reason, even before the composite image CP based on the change image is displayed, the user can check the subject that has changed around the vehicle 9.

  Further, when there is a change image, the warning control unit 23c outputs a predetermined warning sound from the speaker 27 to warn the user. For this reason, the user can recognize that there is a subject (that is, a moving object) that has changed around the vehicle 9.

<4. Fourth Embodiment>
Next, a fourth embodiment will be described. Since the configuration and processing of the image processing system 10 of the fourth embodiment are substantially the same as those of the first embodiment, the following description will be focused on differences from the first embodiment. In the fourth embodiment, the content of the display image DP displayed on the in-vehicle device 2 can be changed by the user operating the portable device 3.

  FIG. 14 is a diagram illustrating the relationship between the operation unit 35 of the portable device 3 and the display content of the in-vehicle device 2.

  The portable device 3 includes a plurality of operation buttons that function as numeric keys on a part of the operation unit 35. Among these operation buttons, the “2” button 35F, the “8” button 35B, the “4” button 35L, and the “6” button 35R are associated with the four in-vehicle cameras 5 mounted on the vehicle 9, respectively. ing. Specifically, the “2” button 35F is associated with the front camera 5F, the “8” button 35B is associated with the back camera 5B, the “4” button 35L is associated with the left side camera 5L, and the “6” button 35R is associated with the right side camera 5R. It has been. These four operation buttons 35F, 36B, 36L, and 36R function as operation members for designating the in-vehicle camera 5 associated therewith.

  On the other hand, in the composite image CP displayed on the display 29 of the in-vehicle device 2, the position of each of the four in-vehicle cameras 5 (more precisely, the position corresponding to the actual arrangement position of each of the four in-vehicle cameras 5). In the vicinity, labels CF, CL, CB, and CR are superimposed. These labels CF, CL, CB, CR serve as identification information for identifying the operation buttons corresponding to the in-vehicle cameras 5.

  Specifically, a label CF indicating “2” is superimposed in the vicinity of the position of the front camera 5F in the composite image CP, and a label CB indicating “8” is superimposed in the vicinity of the position of the back camera 5B. A label CL indicating “4” is superimposed near the position of the left side camera 5L, and a label CR indicating “6” is superimposed near the position of the right side camera 5R. Therefore, the user visually recognizes the composite image CP on which the labels CF, CL, CB, and CR are superimposed, so that the “2” button 35F is the front camera 5F, the “8” button 35B is the back camera 5B, “ It can be clearly understood that the “4” button 35L is associated with the left side camera 5L, and the “6” button 35R is associated with the right side camera 5R.

  Further, the composite image CP on which such labels CF, CL, CB, CR are superimposed is also displayed on the display 36 of the portable device 3. For this reason, the user can grasp which operation button corresponds to which in-vehicle camera 5 by checking the display 36 of the portable device 3. The display 36 displays the composite image CP so that the longitudinal direction of the screen matches the longitudinal direction of the vehicle image 90 in the composite image CP. For this reason, the vehicle image 90 can be displayed in a relatively large size even on the relatively small display 36 provided in the portable device 3.

  In the state shown in FIG. 14, when the user operates any of the four operation buttons 35F, 36B, 36L, and 36R, the vicinity of the position of the in-vehicle camera 5 corresponding to the operated operation button is enlarged. A composite image CP is displayed. For example, as illustrated in FIG. 15, when the user operates the “4” button 35L, the composite image CP in which the vicinity of the position of the left side camera 5L corresponding to the “4” button 35L is enlarged is displayed. Become. Therefore, the user can enlarge and confirm a desired position of the composite image CP by operating the operation button of the portable device 3.

  FIG. 16 is a diagram illustrating a processing flow of the image processing system 10 according to the fourth embodiment. First, the image acquisition unit 24 of the in-vehicle device 2 acquires four captured images PF, PB, PL, PR (step SA41), and the data communication unit 28 of the in-vehicle device 2 acquires four captured images PF, PB, PL, PR. Is transmitted to the portable device 3 (step SA42).

  The data communication unit 37 of the portable device 3 receives these four captured images PF, PB, PL, PR (step SB41). Next, it is determined whether or not any of the four operation buttons 35F, 36B, 36L, and 36R of the portable device 3 has been operated by the user.

  When none of the four operation buttons is operated (No in step SB42), the composite image generation unit 31a generates a composite image CP in which labels CF, CL, CB, and CR are superimposed at appropriate positions (step). SB43). On the other hand, when any of the four operation buttons is operated (Yes in Step SB42), the composite image generation unit 31a enlarges the vicinity of the position of the in-vehicle camera 5 corresponding to the operated operation button. A CP is generated (step SB44).

  Next, the display 36 of the portable device 3 displays the generated composite image CP (step SB45). At the same time, the data communication unit 37 transmits the generated composite image CP to the in-vehicle device 2 (step SB46).

  When the data communication unit 28 of the in-vehicle device 2 receives this composite image CP (step SA43), the image output unit 26 of the in-vehicle device 2 outputs and displays the display image DP including the composite image CP on the display 29 (step SA44). ).

  Thereby, when none of the four operation buttons of the portable device 3 is operated, the composite image CP on which the labels CF, CL, CB, and CR are superimposed is displayed on the display 29 of the in-vehicle device 2. On the other hand, when any of the four operation buttons of the portable device 3 is operated, the composite image CP in which the vicinity of the position of the vehicle-mounted camera 5 corresponding to the operated button is enlarged is displayed on the display 29 of the vehicle-mounted device 2. Is displayed.

  As described above, in the image processing system 10 according to the fourth embodiment, labels serving as identification information of corresponding operation buttons are superimposed in the vicinity of the positions of the plurality of in-vehicle cameras 5 in the composite image CP. The composite image CP is displayed on the display 29 of the in-vehicle device 2. For this reason, the user can clearly grasp which operation button corresponds to which in-vehicle camera.

  When any of the four operation buttons is operated by the user, the composite image generation unit 31a generates a composite image CP in which the vicinity of the position of the in-vehicle camera 5 corresponding to the operated operation button is enlarged, This composite image CP is displayed on the display 29 of the in-vehicle device 2. For this reason, the user can enlarge and confirm a desired position of the composite image CP.

  14 and 15, the labels CF, CL, CB, and CR that are identification information of the operation buttons are superimposed on the periphery of the vehicle image 90. On the other hand, as shown in FIG. 17, the labels CF, CL, CB, and CR may be superimposed so as to overlap the vehicle image 90 in the composite image CP. In this way, it is possible to prevent the subject around the vehicle 9 from being hidden by the labels CF, CL, CB, and CR in the composite image CP.

<5. Fifth embodiment>
Next, a fifth embodiment will be described. Since the configuration and processing of the image processing system 10 of the fifth embodiment are substantially the same as those of the fourth embodiment, the following description will be focused on differences from the fourth embodiment. In the fourth embodiment, when any of the four operation buttons of the portable device 3 is operated, the composite image CP in which the vicinity of the position of the in-vehicle camera 5 corresponding to the operated operation button is enlarged is displayed. It was displayed on the display 29 of the in-vehicle device 2. On the other hand, in the fifth embodiment, when any of the four operation buttons of the portable device 3 is operated, a captured image obtained by the in-vehicle camera 5 corresponding to the operated operation button. Is displayed on the display 29 of the in-vehicle device 2.

  For example, as shown in FIG. 18, when the user operates the “4” button 35L, the display image DP including the left side image PL obtained by the left side camera 5L corresponding to the “4” button 35L is displayed on the display 29. Will be displayed. Therefore, the user can confirm the captured image obtained by the desired in-vehicle camera 5 by operating the operation button of the portable device 3.

  FIG. 19 is a diagram illustrating a processing flow of the image processing system 10 according to the fifth embodiment. First, the image acquisition unit 24 of the in-vehicle device 2 acquires four captured images PF, PB, PL, PR (step SA51), and the data communication unit 28 of the in-vehicle device 2 acquires four captured images PF, PB, PL, PR. Is transmitted to the portable device 3 (step SA52).

  The data communication unit 37 of the portable device 3 receives these four captured images PF, PB, PL, PR (step SB51).

  Next, it is determined whether or not any of the four operation buttons 35F, 36B, 36L, and 36R of the portable device 3 has been operated by the user. If none of the four operation buttons has been operated (No in step SB52), the process proceeds directly to step SB54. On the other hand, when any one of the four operation buttons is operated (Yes in Step SB42), the button information for specifying the operation button operated by the data communication unit 28 is transmitted to the in-vehicle device 2 (Step SB53). ), The process proceeds to step SB54.

  When the data communication unit 28 of the in-vehicle device 2 receives this button information (Yes in step SA53), the display control unit 23a specifies the operation button operated based on the button information. And the display control part 23a produces | generates display image DP containing the picked-up image of the vehicle-mounted camera 5 corresponding to the operated operation button. At this time, when the composite image CP is already displayed, a display image DP including the composite image and a captured image of the in-vehicle camera 5 corresponding to the operated operation button is generated.

  Then, the display control unit 23 a controls the image output unit 26 to cause the display 29 to output this display image DP. Thereby, the display image DP including the captured image of the in-vehicle camera 5 corresponding to the operated operation button is displayed on the display 29 (step SA54).

  In step SB54, the composite image generation unit 31a of the portable device 3 generates a composite image CP in which labels CF, CL, CB, and CR are superimposed at appropriate positions. Next, the display 36 of the portable device 3 displays the generated composite image CP (step SB55). At the same time, the data communication unit 37 of the portable device 3 transmits the generated composite image CP to the in-vehicle device 2 (step SB56).

  When the data communication unit 28 of the in-vehicle device 2 receives this composite image CP (step SA55), the image output unit 26 of the in-vehicle device 2 outputs and displays the display image DP including the composite image CP on the display 29 (step SA56). ).

  As described above, in the image processing system 10 according to the fifth embodiment, when one of the four operation buttons is operated by the user, the button information for specifying the operated operation button is the portable device 3. To the in-vehicle device 2. In the in-vehicle device 2, a captured image obtained by the in-vehicle camera 5 corresponding to the operated operation button is displayed on the display 29 based on the button information. Therefore, the user can confirm a desired captured image.

  Also in the present embodiment, as shown in FIG. 17, the labels CF, CL, CB, and CR may be superimposed so as to overlap the vehicle image 90 in the composite image CP.

<6. Sixth Embodiment>
Next, a sixth embodiment will be described. In the image processing system 10 of the first embodiment, the composite image is generated in the portable device 3 possessed by the user. In contrast, in the image processing system according to the sixth embodiment, a composite image is generated in a server device on the network instead of the portable device 3. Hereinafter, the difference from the first embodiment will be mainly described.

  FIG. 20 is a diagram illustrating an overview of the image processing system 11 according to the sixth embodiment. The image processing system 11 includes an in-vehicle device 2 mounted on the vehicle 9 and a server device 4 configured separately from the in-vehicle device 2. The vehicle 9 and the server device 4 can transmit and receive data to and from each other via a network 8 such as the Internet. The server device 4 has an image processing function and functions as an image processing device. One of the image processing functions provided in the server device 4 is a function of generating a composite image viewed from a virtual viewpoint.

  FIG. 21 is a diagram illustrating a configuration of the in-vehicle device 2 according to the sixth embodiment. The in-vehicle device 2 of the sixth embodiment has substantially the same configuration as that shown in FIG. 2, but includes a data communication unit 28 a instead of the data communication unit 28. The data communication unit 28a has a function of connecting to the network 8 using wireless communication. Thereby, the data communication unit 28 a can transmit and receive data to and from the server device 4 connected to the network 8. The data communication unit 28a can transmit / receive data to / from the server device 4 connected to the network 8 by using another communication device such as a mobile phone possessed by the user as a relay. It may be.

  FIG. 22 is a diagram illustrating a configuration of the server device 4. The server device 4 includes a control unit 41, a data communication unit 42, and a storage unit 43.

  The data communication unit 42 is connected to the network 8 and performs data communication with other communication devices having a function of connecting to the network 8. Thereby, the data communication part 42 can transmit / receive data between the vehicle-mounted apparatuses 2.

  The storage unit 43 is a non-volatile storage device such as a hard disk, and stores various types of information. The storage unit 38 stores a plurality of vehicle images 90 and a program 44.

  The control unit 41 is a computer that includes a CPU, a RAM, a ROM, and the like. Various functions of the control unit 41 are realized by the CPU performing arithmetic processing according to the program 44 stored in the storage unit 43.

  A composite image generation unit 41 a shown in the figure is one of the functions of the control unit 41 realized by the CPU performing arithmetic processing according to the program 44. The composite image generation unit 41a generates a composite image indicating a region around the vehicle 9 viewed from the virtual viewpoint based on the four captured images, using the same method as the composite image generation unit 31a of the first embodiment. .

  FIG. 23 is a diagram illustrating a processing flow of the image processing system 11 according to the sixth embodiment. The left side in the drawing shows the processing flow of the in-vehicle device 2, and the right side in the drawing shows the processing flow of the server device 4. The processing flow of the in-vehicle device 2 is almost the same as that of the first embodiment (FIG. 6). In the server device 4, the processing shown in FIG. 23 is executed in response to a signal from the in-vehicle device 2 under the control of the control unit 41.

  First, the image acquisition unit 24 of the in-vehicle device 2 acquires four captured images PF, PB, PL, PR (step SA61). Next, the data communication unit 28a of the in-vehicle device 2 transmits the four captured images PF, PB, PL, PR, vehicle information 25a, and size information to the server device 4 (step SA62).

  The data communication unit 42 of the server device 4 receives these four captured images PF, PB, PL, PR, vehicle information 25a, and size information (step SC11). In response to this reception, the composite image generation unit 41a of the server device 4 generates a composite image CP that indicates an area around the vehicle 9 as viewed from the virtual viewpoint, using the four captured images (step SC12).

  The composite image generation unit 41a reads the vehicle image 90 corresponding to the type of the vehicle 9 from the storage unit 43 based on the vehicle information 25a transmitted from the in-vehicle device 2 when generating the composite image CP. 90 is included in the composite image CP. The composite image generation unit 41a generates a composite image CP having a size specified by the size information based on the size information transmitted from the in-vehicle device 2.

  Next, the data communication unit 42 of the server device 4 transmits the generated composite image CP to the in-vehicle device 2 (step SC13).

  When the data communication unit 28a of the in-vehicle device 2 receives this composite image CP (step SA63), the image output unit 26 of the in-vehicle device 2 outputs and displays the display image DP including the composite image CP on the display 29 (step SA64). ).

  As described above, in the image processing system 11 according to the sixth embodiment, a composite image is generated in the server device 4 on the network 8. For this reason, also in the image processing system 11 of the present embodiment, the composite image CP can be displayed on the display 29 even if the in-vehicle device 2 does not have a function of generating the composite image CP. Therefore, the composite image CP can be displayed on the display 29 even when an in-vehicle device having a relatively low cost is used. Further, since the user does not need to introduce a portable device or the like having a function for generating the composite image CP, a display function for the composite image CP can be easily added.

  In the present embodiment, the case where the server device 4 is used instead of the portable device 3 of the first embodiment has been described. However, the portable device 3 of the second or third embodiment is used instead. The server device 4 may be used.

<7. Seventh Embodiment>
Next, a seventh embodiment will be described. In general, a server device has a large-capacity storage device compared to a portable device or the like. For this reason, in the case where the server device having a relatively large capacity storage device generates a composite image, it is also possible to combine the composite image with another image.

  In the seventh embodiment, an aerial photograph image (hereinafter referred to as “aerial image”) is stored in advance in the server device 4, and a combined image obtained by combining the aerial image and the composite image is generated. It is like that. Since the configuration and processing of the image processing system 11 of the seventh embodiment are substantially the same as those of the sixth embodiment, the following description will be focused on differences from the sixth embodiment.

  FIG. 24 is a diagram illustrating a configuration of the in-vehicle device 2 according to the seventh embodiment. The in-vehicle device 2 of the seventh embodiment includes a position acquisition unit 20 in addition to the configuration of the in-vehicle device 2 of the sixth embodiment shown in FIG. The position acquisition unit 20 receives signals from a plurality of GPS satellites and acquires position information (latitude and longitude) indicating the position of the vehicle 9.

  FIG. 25 is a diagram illustrating a configuration of the server device 4 according to the seventh embodiment. In the server device 4 of the seventh embodiment, the storage unit 43 stores an aerial image database 45 in addition to the vehicle image 90 and the program 44. In the aerial image database 45, a large number of aerial images obtained by photographing the state of the ground surface from above are stored in association with position information (latitude and longitude) indicating the respective positions.

  In addition to the composite image generation unit 41a, the server device 4 includes an image combination unit 41b as a function of the control unit 41 realized by the CPU performing arithmetic processing according to the program 44. The image combining unit 41b combines the aerial image and the combined image generated by the combined image generating unit 41a to generate a combined image including the aerial image and the combined image.

  FIG. 26 is a diagram illustrating a processing flow of the image processing system 11 according to the sixth embodiment. First, the image acquisition unit 24 of the in-vehicle device 2 acquires four captured images PF, PB, PL, PR (step SA71). Next, the position acquisition unit 20 of the in-vehicle device 2 acquires position information indicating the position of the vehicle 9 at that time (step SA72).

  Next, the data communication unit 28a of the in-vehicle device 2 transmits four captured images PF, PB, PL, and PR to the server device 4 (step SA73). At this time, the data communication unit 28a transmits the position information to the server device 4 together with the four captured images PF, PB, PL, and PR.

  The data communication unit 42 of the server device 4 receives the position information together with the four captured images PF, PB, PL, PR (step SC21). In response to this reception, the composite image generation unit 41a of the server device 4 generates a composite image CP that indicates the area around the vehicle 9 as viewed from the virtual viewpoint, using the four captured images (step SC22).

  Next, based on the position information received from the in-vehicle device 2, the image combining unit 41 b acquires an aerial image centered on the position indicated by the position information (that is, the position of the vehicle 9) from the aerial image database 45. (Step SC23). Thereby, an aerial image showing the periphery of the vehicle 9 is acquired.

  Then, the image combination unit 41b generates a combined image including the aerial image and the composite image CP by superimposing the composite image CP on the central portion of the aerial image (step SC24). At this time, the image combining unit 41b adjusts the size of the composite image CP so as to match the scale of the aerial image.

  FIG. 27 is a diagram illustrating an example of the combined image OP generated in this way. The combined image OP has a composite image CP at the center thereof, and an aerial image AP at the outer portion of the composite image CP. By visually recognizing such a combined image OP, the user can obtain information on the surface of a relatively wide area around the vehicle 9 as well as information on objects near the vehicle 9. For example, when the vehicle 9 exists in the parking lot, the user can grasp the exit of the parking lot, the traveling direction determined in the parking lot, the overall shape of the parking lot, and the like based on the combined image OP.

  Next, the data communication unit 42 of the server device 4 transmits the combined image OP including the composite image CP and the aerial image AP to the in-vehicle device 2 (step SC25 in FIG. 26).

  When the data communication unit 28a of the in-vehicle device 2 receives this combined image OP (Step SA74), the display control unit 23a generates a display image DP including the received combined image OP. Then, the image output unit 26 outputs the generated display image DP to the display 29. Thereby, the display image DP including the combined image OP is displayed on the display 29 (step SA75).

  The upper part of FIG. 28 is a diagram showing an example of the display image DP including the combined image OP displayed in this way. As shown in the figure, the display image DP includes two command buttons B1 and B2 for changing the scale. The user can change the scale of the aerial image AP in the combined image OP by touching these two command buttons B1 and B2 displayed on the display screen of the display 29.

  For example, when the detail button B1 is touched, as shown in the lower part of FIG. 28, the state of the ground surface indicated by the aerial image AP is enlarged, and the range of the ground surface included in the aerial image AP is narrowed. On the other hand, when the wide area button B2 is touched, the state of the ground surface indicated by the aerial image AP is reduced, and the range of the ground surface included in the aerial image AP is widened. Further, when the scale of the aerial image AP is changed in this way, the size (scale) of the composite image CP is also adjusted in accordance with the scale of the aerial image AP. Thus, since the user can change the scale of the aerial image AP as desired, the aerial image AP including the necessary ground surface range can be displayed on the display 29.

  FIG. 29 is a diagram showing a processing flow of the image processing system 11 corresponding to such a scale change instruction. In the in-vehicle device 2, the process shown in FIG. 29 is repeatedly executed at a predetermined cycle independently of the process shown in FIG. 26 under the control of the control unit 23. Moreover, in the server apparatus 4, the process shown in FIG. 29 is performed in response to the signal from the vehicle-mounted apparatus 2 by control of the control part 41. FIG.

  First, the control unit 23 of the in-vehicle device 2 determines whether or not an instruction to change the scale of the aerial image AP has been given by the user (step SA81). That is, the control unit 23 determines whether or not the user has touched one of the two command buttons B1 and B2 based on a signal from the touch panel included in the operation unit 22. If there is no instruction to change the scale of the aerial image AP (No in step SA81), the process ends.

  When there is an instruction to change the scale of the aerial image AP (Yes in step SA81), the data communication unit 28a of the in-vehicle device 2 transmits to the server device 4 scale information that instructs to change the scale of the aerial image AP. (Step SA82). When the touched command button is the detail button B1, scale information for instructing expansion of the appearance of the ground surface is transmitted. When the touched command button is the wide area button B2, scale information for instructing reduction of the appearance of the ground surface is transmitted. Is done.

  The data communication unit 42 of the server device 4 receives this scale information (step SC31). In response to the reception of the scale information, the image combining unit 41b of the server device 4 changes the scale of the aerial image AP according to the scale information (step SC32). Thereafter, the image combining unit 41b generates a combined image OP including the aerial image AP with a scale corresponding to the scale information.

  As described above, in the image processing system 11 according to the sixth embodiment, the storage unit 25 of the server device 4 stores the aerial image indicating the state of the ground surface, and the image combining unit 41b determines the position of the vehicle 9. Based on the position information shown, a combined image including an aerial image showing the periphery of the vehicle 9 and a composite image is generated. The in-vehicle device 2 displays the combined image OP on the display 29. Since such a combined image OP is displayed on the display 29, the user can obtain information on the surface of a relatively wide area around the vehicle 9 as well as information on objects near the vehicle 9.

  Further, since the aerial image AP having a reduced scale corresponding to the reduced scale information transmitted from the in-vehicle device 2 is generated, the aerial image AP having a reduced scale required by the user can be displayed on the display 29.

  In the present embodiment, the server device 4 has been described as storing aerial images. However, the portable device 3 stores the aerial image, and the portable device 3 includes the combined image and the aerial image. An image may be generated.

<8. Modification>
Although the embodiments of the present invention have been described above, the present invention is not limited to the above-described embodiments, and various modifications are possible. Below, such a modification is demonstrated. All the forms including the above-described embodiment and the form described below can be appropriately combined.

  In the said embodiment, although the vehicle-mounted apparatus 2 was demonstrated to the example as a vehicle apparatus used with a vehicle, the apparatus for vehicles may be comprised as portable communication apparatuses, such as a mobile telephone and a smart phone.

  Moreover, in the said embodiment, although the synthesized image which looks down the area | region of the vehicle 9 and the periphery of the vehicle 9 was produced | generated, you may make it produce | generate the synthesized image seen from arbitrary virtual viewpoints.

  Moreover, in the said embodiment, although the display 29 was integrated with the main-body part 21, the display may be provided as a different body from the main-body part.

  Moreover, in the said embodiment, whenever it transmitted the some picked-up image from the vehicle-mounted apparatus 2, the vehicle information 25a and size information were transmitted to the image processing apparatus (the portable apparatus 3 or the server apparatus 4). On the other hand, the vehicle information 25a and the size information are transmitted only once from the in-vehicle device 2 to the image processing device, and the vehicle information 25a and the size information are associated with the identification information of the in-vehicle device 2 to the image processing device. It may be stored.

  In the seventh embodiment, an aerial image is used as a ground image indicating the state of the ground surface. However, an image of a satellite photograph or a general map image represented by an illustration may be used.

  In the seventh embodiment, the scale information is transmitted to the server device 4 to change the range of the ground surface included in the ground surface image (aerial image). On the other hand, the scale information may be transmitted to the server device 4 so that the scale of the ground image and the scale of the composite image can be adjusted.

  Further, in the above-described embodiment, it has been described that various functions including a function of generating a synthesis process by software are realized by a CPU arithmetic process according to a program. However, some of these functions are electrically It may be realized by a simple hardware circuit.

2 In-vehicle device 3 Portable device 4 Server device 5 In-vehicle camera 9 Vehicle 35 Operation unit 10 Image processing system 11 Image processing system 90 Vehicle image AP Aerial image CP Composite image DP Display image OP Combined image

Claims (19)

An image processing system,
A vehicle device used in a vehicle;
An image processing device configured separately from the vehicle device;
With
The vehicle device includes:
Obtaining means for obtaining a plurality of captured images respectively obtained by a plurality of cameras mounted on the vehicle;
First transmission means for transmitting the plurality of captured images to the image processing apparatus;
First receiving means for receiving a composite image generated by the image processing device based on the plurality of captured images and indicating a region around the vehicle as viewed from a virtual viewpoint;
Output means for outputting and displaying the composite image on a display device;
With
The image processing apparatus includes:
Second receiving means for receiving the plurality of captured images transmitted from the vehicle device;
Generating means for generating the composite image based on the plurality of captured images;
Second transmission means for transmitting the composite image to the vehicle device;
Equipped with a,
The output means is a period of time from when the first transmission means transmits the plurality of captured images to when the first reception means receives the composite image based on the plurality of captured images. An image processing system characterized in that at least one is output and displayed on the display device . An image processing system,
A vehicle device used in a vehicle;
An image processing device configured separately from the vehicle device;
With
The vehicle device is:
Obtaining means for obtaining a plurality of captured images respectively obtained by a plurality of cameras mounted on the vehicle;
First transmission means for transmitting the plurality of captured images to the image processing apparatus;
First receiving means for receiving a composite image generated by the image processing device based on the plurality of captured images and indicating a region around the vehicle as viewed from a virtual viewpoint;
Output means for outputting and displaying the composite image on a display device;
With
The image processing apparatus includes:
Second receiving means for receiving the plurality of captured images transmitted from the vehicle device;
Generating means for generating the composite image based on the plurality of captured images;
Second transmission means for transmitting the composite image to the vehicle device;
With
The acquisition means periodically acquires each of the plurality of captured images,
The vehicle device is:
A specifying means for specifying a change image in which an included subject has temporally changed among the plurality of captured images;
Further comprising
If there is a change image,
The first transmission means transmits only the change image among the plurality of captured images,
The generation unit generates a new composite image by replacing only the captured image corresponding to the change image among the plurality of captured images used for generating the composite image in the past, and the change image.
The output means outputs the change image to the display device for display before outputting the composite image based on the change image . The image processing system according to claim 2 ,
The vehicle device includes:
Warning means for warning the user when there is the change image,
An image processing system further comprising: An image processing system,
A vehicle device used in a vehicle;
An image processing device configured separately from the vehicle device;
With
The vehicle device is:
Obtaining means for obtaining a plurality of captured images respectively obtained by a plurality of cameras mounted on the vehicle;
First transmission means for transmitting the plurality of captured images to the image processing apparatus;
First receiving means for receiving a composite image generated by the image processing device based on the plurality of captured images and indicating a region around the vehicle as viewed from a virtual viewpoint;
Output means for outputting and displaying the composite image on a display device;
With
The image processing apparatus includes:
Second receiving means for receiving the plurality of captured images transmitted from the vehicle device;
Generating means for generating the composite image based on the plurality of captured images;
Second transmission means for transmitting the composite image to the vehicle device;
With
The image processing apparatus includes:
Storage means for storing a ground image showing a state of the ground surface;
A combining unit configured to generate a combined image including the ground image indicating the periphery of the vehicle and the composite image based on position information indicating the position of the vehicle;
Further comprising
The first transmission means transmits the position information to the image processing apparatus,
The second transmission means transmits the combined image to the vehicle device,
The image processing system , wherein the output means outputs the combined image to the display device for display . The image processing system according to claim 4,
The first transmission means transmits scale information related to a scale of the ground image to the image processing apparatus;
The image processing system , wherein the combining unit generates the combined image including the ground image having a scale corresponding to the scale information . An image processing system,
A vehicle device used in a vehicle;
An image processing device configured separately from the vehicle device;
With
The vehicle device is:
Obtaining means for obtaining a plurality of captured images respectively obtained by a plurality of cameras mounted on the vehicle;
First transmission means for transmitting the plurality of captured images to the image processing apparatus;
First receiving means for receiving a composite image generated by the image processing device based on the plurality of captured images and indicating a region around the vehicle as viewed from a virtual viewpoint;
Output means for outputting and displaying the composite image on a display device;
With
The image processing apparatus includes:
Second receiving means for receiving the plurality of captured images transmitted from the vehicle device;
Generating means for generating the composite image based on the plurality of captured images;
Second transmission means for transmitting the composite image to the vehicle device;
With
The image processing apparatus includes:
A plurality of buttons for designating each of the plurality of cameras;
Further comprising
The output means outputs the composite image in which the identification information of the button corresponding to the camera is superimposed in the vicinity of the position of each of the plurality of cameras in the composite image to be displayed on the display device. Image processing system. The image processing system according to claim 6 .
The generation unit generates the composite image in which the vicinity of the position of the camera corresponding to the operated button is enlarged when any of the plurality of buttons is operated by a user. Processing system. The image processing system according to claim 6 .
The second transmission means transmits button information for identifying the operated button to the vehicle device when any of the plurality of buttons is operated by a user,
The output means is configured to output and display the captured image obtained by the camera corresponding to the operated button on the display device based on the button information . The image processing system according to any one of claims 6 to 8 ,
The composite image includes a vehicle image,
The identification information of the button is superimposed so as to overlap the vehicle image in the composite image . The image processing system according to any one of claims 6 to 9 ,
The image processing apparatus includes:
Display means for displaying the composite image in which the identification information of the button corresponding to the camera is superimposed in the vicinity of the position of each of the plurality of cameras in the composite image;
An image processing system further comprising: The image processing system according to claim 10.
The image processing system , wherein the display unit displays the composite image so that a longitudinal direction of a screen matches a longitudinal direction of a vehicle image in the composite image .   A vehicle device used in a vehicle,
  Obtaining means for obtaining a plurality of captured images respectively obtained by a plurality of cameras mounted on the vehicle;
  Transmitting means for transmitting the plurality of captured images to an image processing device configured separately from the vehicle device;
  Receiving means for receiving a composite image generated by the image processing device based on the plurality of photographed images and indicating a region around the vehicle as viewed from a virtual viewpoint;
  Output means for outputting and displaying the composite image on a display device;
With
  The output means outputs at least one of the plurality of photographed images during a period from when the transmission means transmits the plurality of photographed images to when the reception means receives the composite image based on the plurality of photographed images. An apparatus for a vehicle, wherein the display device outputs and displays the display device.   A vehicle device used in a vehicle,
  Obtaining means for obtaining a plurality of captured images respectively obtained by a plurality of cameras mounted on the vehicle;
  Transmitting means for transmitting the plurality of captured images to an image processing device configured separately from the vehicle device;
  Receiving means for receiving a composite image generated by the image processing device based on the plurality of photographed images and indicating a region around the vehicle as viewed from a virtual viewpoint;
  Output means for outputting and displaying the composite image on a display device;
With
  The acquisition means periodically acquires each of the plurality of captured images,
  The vehicle device includes:
    A specifying means for specifying a change image in which an included subject has temporally changed among the plurality of captured images;
Further comprising
  If there is a change image,
    The transmission means transmits only the change image among the plurality of captured images,
    The receiving unit replaces only the photographed image corresponding to the change image among the plurality of photograph images used in the generation of the composite image in the past with the change image, and generates a new image generated by the image processing apparatus. Receiving the composite image;
    The output device outputs the change image to the display device and displays the change image before outputting the composite image based on the change image.   A vehicle device used in a vehicle,
  Obtaining means for obtaining a plurality of captured images respectively obtained by a plurality of cameras mounted on the vehicle;
  Transmitting means for transmitting the plurality of captured images to an image processing device configured separately from the vehicle device;
  Receiving means for receiving a composite image generated by the image processing device based on the plurality of photographed images and indicating a region around the vehicle as viewed from a virtual viewpoint;
  Output means for outputting and displaying the composite image on a display device;
With
  The transmission means transmits position information indicating the position of the vehicle to the image processing device,
  The receiving means receives a combined image including the ground image indicating the state of the ground surface around the vehicle and the composite image generated by the image processing device based on the position information,
  The vehicle device, wherein the output means outputs the combined image to the display device for display.   A vehicle device used in a vehicle,
  Obtaining means for obtaining a plurality of captured images respectively obtained by a plurality of cameras mounted on the vehicle;
  Transmitting means for transmitting the plurality of captured images to an image processing device configured separately from the vehicle device;
  Receiving means for receiving a composite image generated by the image processing device based on the plurality of photographed images and indicating a region around the vehicle as viewed from a virtual viewpoint;
  Output means for outputting and displaying the composite image on a display device;
With
  The image processing apparatus includes a plurality of buttons for designating the plurality of cameras,
  The output means outputs the composite image in which the identification information of the button corresponding to the camera is superimposed in the vicinity of the position of each of the plurality of cameras in the composite image to be displayed on the display device. A vehicle device.   An image display method for displaying an image using a vehicular apparatus used in a vehicle and an image processing apparatus configured separately from the vehicular apparatus,
  (A) the vehicle device acquiring a plurality of captured images respectively obtained by a plurality of cameras mounted on the vehicle;
  (B) transmitting the plurality of captured images from the vehicle device to the image processing device;
  (C) The image processing device generates a composite image indicating a region around the vehicle viewed from a virtual viewpoint based on the plurality of captured images;
  (D) transmitting the composite image from the image processing device to the vehicle device;
  (E) the vehicle device outputting and displaying the composite image on a display device;
With
  In the step (e), the vehicular apparatus transmits the plurality of captured images in the step (b), and then the combined image based on the plurality of captured images is displayed in the step (d). An image display method, comprising: outputting and displaying at least one of the plurality of photographed images on the display device during a period until reception.   An image display method for displaying an image using a vehicular apparatus used in a vehicle and an image processing apparatus configured separately from the vehicular apparatus,
  (A) the vehicle device acquiring a plurality of captured images respectively obtained by a plurality of cameras mounted on the vehicle;
  (B) transmitting the plurality of captured images from the vehicle device to the image processing device;
  (C) The image processing device generates a composite image indicating a region around the vehicle viewed from a virtual viewpoint based on the plurality of captured images;
  (D) transmitting the composite image from the image processing device to the vehicle device;
  (E) the vehicle device outputting and displaying the composite image on a display device;
With
  The step (a) periodically acquires each of the plurality of captured images.
  The image display method includes:
  (F) a step in which the vehicular device specifies a change image in which an included subject has changed in time among the plurality of captured images;
Further comprising
  If there is a change image,
    The step (b) transmits only the change image among the plurality of captured images,
    The step (c) generates a new composite image by replacing only the photographed image corresponding to the change image among the plurality of photograph images used for generating the composite image in the past with the change image. And
    The step (e) outputs the change image to the display device for display before outputting the composite image based on the change image.   An image display method for displaying an image using a vehicular apparatus used in a vehicle and an image processing apparatus configured separately from the vehicular apparatus,
  (A) the vehicle device acquiring a plurality of captured images respectively obtained by a plurality of cameras mounted on the vehicle;
  (B) transmitting the plurality of captured images from the vehicle device to the image processing device;
  (C) The image processing device generates a composite image indicating a region around the vehicle viewed from a virtual viewpoint based on the plurality of captured images;
  (D) transmitting the composite image from the image processing device to the vehicle device;
  (E) the vehicle device outputting and displaying the composite image on a display device;
With
  The image display method includes:
  (F) The image processing device generates a combined image including a ground image indicating the state of the ground surface around the vehicle and the composite image based on position information indicating the position of the vehicle.
Further comprising
  The step (b) transmits the position information from the vehicle device to the image processing device,
  The step (d) transmits the combined image from the image processing device to the vehicle device,
  In the step (e), the combined image is output and displayed on the display device. An image display method for displaying an image using a vehicular apparatus used in a vehicle and an image processing apparatus configured separately from the vehicular apparatus,
(A) the vehicle device acquiring a plurality of captured images respectively obtained by a plurality of cameras mounted on the vehicle;
(B) transmitting the plurality of captured images from the vehicle device to the image processing device;
(C) The image processing device generates a composite image indicating a region around the vehicle viewed from a virtual viewpoint based on the plurality of captured images;
(D) transmitting the composite image from the image processing device to the vehicle device;
(E) the vehicle device outputting and displaying the composite image on a display device;
Equipped with a,
The image processing apparatus includes a plurality of buttons for designating the plurality of cameras,
In the step (e), the composite image in which the identification information of the button corresponding to the camera is superimposed in the vicinity of the position of each of the plurality of cameras in the composite image is output and displayed on the display device. Characteristic image display method.

Images