JP2024112036A - Electronic mirror device and control program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To simultaneously confirm the situation behind a vehicle and the situation in the back part of the vehicle cabin by the minimum vision move.

SOLUTION: A first imaging unit (30) takes an image of a region behind a vehicle. A second imaging unit (40) takes an image of the back part of the vehicle cabin. An electronic mirror display unit (20) displays the back image of the vehicle taken by the first imaging unit (30). A display control unit (12) cuts off a region including a predetermined object from the image of the back part of the vehicle cabin taken by the second imaging unit (40), and overlaps an image including the cut off object on the back image, and causes the electronic mirror display unit (20) to display the image.

SELECTED DRAWING: Figure 3

COPYRIGHT: (C)2024,JPO&INPIT

Description

The present invention relates to an electronic mirror device used as a digital rearview mirror and a control program.

In recent years, there has been a growing trend to replace conventional optical room mirrors (also called rearview mirrors) with electronic digital room mirrors. Generally, a rear camera that takes pictures of the area behind the vehicle is installed on the rear window, so when a digital room mirror is used, the view to the rear of the vehicle is not blocked by luggage or other items inside the vehicle. On the other hand, the image displayed on the digital room mirror does not show the rear seats, so even if there are passengers in the back seats, they cannot be seen. In order to see the passengers in the back seats, it was necessary to switch the digital room mirror from image display mode to optical mirror mode.

As such, with conventional digital rearview mirrors, it was difficult to simultaneously check the image from the rear camera and the situation at the rear of the vehicle. If you wanted to check the situation of the following vehicle while also checking the status of rear seat passengers and luggage in the rear luggage compartment, you basically had to switch modes each time. For example, some commercial vans have rearmost glass that is not transparent but wall-like, making it almost impossible to see behind the vehicle. In such vehicles, if you want to check the situation of the following vehicle and the status of luggage in the rear luggage compartment at the same time, it is difficult to check the situation of the following vehicle in optical mirror mode, so you have to frequently switch modes.

Patent Document 1 discloses a driving assistance device that detects the driver's line of sight and switches the display object to be displayed on a monitor mounted on an instrument panel based on the detected line of sight. Patent Document 2 discloses an electronic mirror camera system that can switch between displaying image information from an electronic mirror camera and image information from an electronic rearview camera on an electronic mirror display for a vehicle mirror. However, neither of these prior art technologies allows the situation of the following vehicle and the situation at the rear of the vehicle cabin to be checked simultaneously.

JP 2009-23565 A JP 2020-164115 A

It is also possible to use an in-car camera to capture the rear of the vehicle cabin and install a separate monitor to display the image of the rear of the vehicle cabin. However, installing a separate monitor increases costs and reduces the design quality of the vehicle cabin. In addition, the driver would need to move his or her line of sight between the monitor that displays the image behind the vehicle and the monitor that displays the image of the rear of the vehicle cabin.

This embodiment was made in consideration of these circumstances, and its purpose is to provide technology that allows the driver to simultaneously check the situation behind the vehicle and the situation at the rear of the passenger compartment with minimal eye movement.

To solve the above problem, the electronic mirror device of one aspect of this embodiment includes a first imaging unit that captures an image of the rear of the vehicle, a second imaging unit that captures an image of the rear of the vehicle's cabin, an electronic mirror display unit that displays the rear image of the vehicle captured by the first imaging unit, and a display control unit that cuts out an area including a specified object from the image of the rear of the vehicle's cabin captured by the second imaging unit and superimposes the image including the cut-out object on the rear image and displays it on the electronic mirror display unit.

In addition, any combination of the above components, and conversions of the expressions of this embodiment between methods, devices, systems, recording media, computer programs, etc., are also valid aspects of this embodiment.

According to this embodiment, the situation behind the vehicle and the situation at the rear of the passenger compartment can be checked simultaneously with minimal eye movement.

1 is a diagram for explaining an example of installation of a digital mirror apparatus according to an embodiment. FIG. 2 is a diagram showing an example of the external configuration of a main body of a digital mirror apparatus. 1 is a diagram showing a system configuration of a digital mirror apparatus according to an embodiment. 4A and 4B are diagrams showing a specific example of a rear image captured by a rear imaging section; 5A to 5C are diagrams showing specific examples of images of the rear part of the vehicle interior captured by the vehicle interior imaging unit. This figure shows a composite image displayed on an electronic mirror display unit, in which an image cut out from the cut-out range of the rear image shown in Figure 4 is superimposed on an image cut out from the cut-out range of the rear image of the vehicle compartment shown in Figure 5 (c). 7(a) and 7(b) are diagrams showing specific examples in which the driver's head posture or line of sight height moves in the vertical direction in the display state shown in FIG. 13 is a diagram showing a first modified example in which a cut-out image of an image of the rear part of the vehicle compartment is superimposed on a cut-out image of a rear image. FIG. 9A to 9C are diagrams showing a second modified example in which an extracted image of the rear passenger compartment image is superimposed on an extracted image of the rear image. FIG. 13 is a diagram showing an example display of a state in which a following vehicle is changing lanes to the right lane. 11A to 11C are diagrams showing specific examples of shifting of the cropping range of a rear image in response to the movement of the driver's line of sight. 12A and 12B are diagrams showing a specific example in which the transparency of a cut-out image of the rear passenger compartment image is changed according to the distance to the following vehicle. 5 is a flowchart showing an example of the operation of the digital mirror apparatus according to the embodiment.

1 is a diagram for explaining an example of installation of an electronic mirror device 1 according to an embodiment. The electronic mirror device 1 according to the embodiment includes an electronic mirror device main body 1a, a rear imaging unit 30, and a vehicle interior imaging unit 40. In the example shown in FIG. 1, the electronic mirror device main body 1a is placed on the mirror surface of an existing optical rearview mirror and is fixed to the existing rearview mirror via a plurality of bands or a predetermined fixing holder (not shown). The rear imaging unit 30 is fixed, for example, with double-sided tape to the inside of the rear window of the vehicle V1. The rear imaging unit 30 may be installed at any position as long as it is capable of capturing an image of the rear of the vehicle V1, and may be stored in a waterproof case and installed outside the vehicle V1. The optical rearview mirror may be omitted, and only the electronic mirror device main body 1a may be installed.

The vehicle interior imaging unit 40 is installed, for example, on the ceiling of the vehicle V1. An omnidirectional camera capable of 360° shooting may be used for the vehicle interior imaging unit 40. The vehicle interior imaging unit 40 may be installed anywhere in the vehicle V1 as long as the angle of view is such that the faces of the passengers in the rear seats are not obstructed by the headrests of the front seats. If the faces of the passengers in the rear seats can be included in the angle of view, the unit may be installed in the electronic mirror device main body 1a. If an in-cabin monitoring system that monitors the entire vehicle interior, including the passenger seat and rear seats, is already installed in the vehicle V1, the camera of the in-cabin monitoring system may be diverted to the vehicle interior imaging unit 40 according to this embodiment.

Figure 2 is a diagram showing an example of the external configuration of the electronic mirror device main body 1a. The electronic mirror display unit 20 and the driver imaging unit 50 are arranged on the front of the electronic mirror device main body 1a. The driver imaging unit 50 is arranged at a predetermined position in the frame area surrounding the electronic mirror display unit 20. In the example shown in Figure 2, a right-hand drive vehicle is assumed, and the driver imaging unit 50 is arranged at the top right of the electronic mirror display unit 20, but in the case of a left-hand drive vehicle, it is desirable to arrange it at the top left of the electronic mirror display unit 20.

The driver imaging unit 50 may be configured with an external imaging unit. The external imaging unit may be fixed and installed, for example, at the right end of the electronic mirror device main body 1a. The external imaging unit may also be installed at the top of the steering column so as to face the driver's face through the gap in the steering wheel. If a driver monitoring system (DMS) for detecting the driver's inattentiveness or drowsiness is already installed in the vehicle V1, the camera of the driver monitoring system may be repurposed as the driver imaging unit 50 according to this embodiment.

Figure 3 is a diagram showing the system configuration of the electronic mirror device 1 according to the embodiment. The electronic mirror device 1 includes a processing unit 10, an electronic mirror display unit 20, a rear imaging unit 30, a vehicle interior imaging unit 40, and a driver imaging unit 50.

The rear imaging unit 30, the vehicle interior imaging unit 40, and the driver imaging unit 50 each include a lens and a solid-state imaging element. For example, a CMOS (Complementary Metal Oxide Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor can be used as the solid-state imaging element. The solid-state imaging element converts the light incident through the lens into an electrical video signal and outputs it to the processing unit 10.

The electronic mirror display unit 20 includes a display 20a and a half mirror 20b. The display 20a is a display such as a liquid crystal display, an organic EL display, or a mini LED display, and is a display for displaying a rear image of the vehicle V1 captured by the rear imaging unit 30 and an image of the rear of the vehicle interior captured by the vehicle interior imaging unit 40.

When the electronic mirror device 1 is configured as an electronic mirror device (also called a mirror recorder) integrated with a drive recorder, the display 20a can also display an image of the front of the vehicle V1. In this case, an imaging unit for capturing an image of the front of the vehicle V1 is further installed on the back surface of the electronic mirror device main body 1a (the surface facing the traveling direction of the vehicle V1).

The half mirror 20b is disposed on the exit side of the display 20a, and reflects light from the front and transmits light from the rear. The half mirror 20b is a mirror with reduced reflectance by adjusting the amount of metal used in the reflective film, and the reflectance of a typical half mirror 20b is adjusted to about 50%. In the electronic mirror display unit 20, the transmitted light can be erased by turning off the display 20a, so it is desirable for the half mirror 20b to have a high transmittance. For example, the transmittance is adjusted to about 50%. Note that instead of the half mirror 20b, a mirror liquid crystal display whose transmittance can be controlled by electronic control may be used.

The processing unit 10 includes an image recognition unit 11 and a display control unit 12. The processing unit 10 is realized by a combination of hardware resources and software resources, or by hardware resources alone. As hardware resources, a CPU, ROM, RAM, DSP (Digital Signal Processors), GPU (Graphics Processing Unit), ASIC (Application Specific Integrated Circuit), FPGA (Field Programmable Gate Array), and other LSIs can be used. As software resources, programs such as firmware can be used.

The image recognition unit 11 recognizes passengers sitting in the back seats as specified objects from the images captured by the vehicle interior imaging unit 40. The image recognition unit 11 is equipped with a detector for detecting a person's face or face and body (mainly the upper body). In this embodiment, it is not necessary to identify individual people. The detector is generated by learning feature amounts contained in a large number of images showing a person's face or face and body. For example, known technologies such as HOG (Histograms of Oriented Gradients), Haar-like, and LBP (Local Binary Patterns) can be used as feature amounts.

The image recognition unit 11 uses a detector to search within each frame image of the video input from the vehicle interior imaging unit 40. When the image recognition unit 11 detects a person's face or a face and body in a frame image, it tracks the detected person's face or face and body in the subsequent frame image. For example, a particle filter or a mean shift method can be used to track the person's face or face and body.

As a detector, a detector for detecting the face or the face and body of an animal (e.g., a pet such as a dog or cat) may be provided, and the image recognition unit 11 may recognize the animal in the back seat as a specified object.

The image recognition unit 11 can also classify the frame images of the video input from the vehicle interior imaging unit 40 into areas with motion and areas without motion. When the vehicle V1 is stopped, the image recognition unit 11 sets a group of areas including multiple pixels with motion between chronologically adjacent frame images as moving areas, and sets the other areas as still areas. When the vehicle V1 is moving, the entire frame image changes slightly due to vibration. The image recognition unit 11 estimates the motion vector of the entire frame image based on the motion vectors of the pixels in the entire area between chronologically adjacent frame images. The image recognition unit 11 sets a group of areas including multiple pixels having a motion vector in a different direction from the estimated overall motion vector as moving areas, and sets the other areas as still areas.

Even if object detection using a detector is not performed, if there is movement in the object, the image recognition unit 11 can execute a process to detect moving areas in the frame image, and as a result detect the object. For example, if a person or animal in the back seat moves, or if luggage in the back seat or in the rear luggage compartment collapses, it will be detected as a moving area.

The image recognition unit 11 also recognizes the behavior of the driver Pd from the image captured by the driver imaging unit 50. The image recognition unit 11 can use a known detection algorithm for detecting the line of sight or head posture of the driver Pd. For example, the image recognition unit 11 detects the face area of the driver Pd from the image captured by the driver imaging unit 50 using a face detector. It is desirable to prepare a face detector for the specific driver Pd who actually sits in the driver's seat in advance. The face detector for the specific driver Pd can be generated by capturing an image of the driver Pd from the driver imaging unit 50 multiple times while the target driver Pd is sitting in the driver's seat and looking at the electronic mirror display unit 20.

When a visible light camera is used for the driver imaging unit 50, the image recognition unit 11 identifies the positions of the inner corner of the eye and the iris included in the eye area based on feature points in the face area of the visible light image. The image recognition unit 11 uses the position of the inner corner of the eye as a reference point and the position of the iris as a moving point, and detects the gaze direction or gaze movement based on the relative position of the iris to the inner corner of the eye. For example, if the iris of the left eye is moving away from the inner corner of the left eye and the iris of the right eye is approaching the inner corner of the right eye, the image recognition unit 11 estimates that the driver Pd's gaze is moving to the left.

When an infrared camera is used for the driver imaging unit 50, an infrared light emitting unit that irradiates infrared rays toward the driver Pd is provided near the infrared camera. The image recognition unit 11 detects the face area of the driver Pd from the infrared image captured by the infrared camera using a face detector. The image recognition unit 11 identifies the corneal reflection point included in the eye area and the position of the pupil based on feature points in the face area. The corneal reflection point is a reflection point on the cornea that reflects the infrared light irradiated from the infrared light emitting unit. The image recognition unit 11 detects the gaze direction or gaze movement based on the relative position of the pupil to the corneal reflection point, with the position of the corneal reflection point as the reference point and the position of the pupil as the moving point. The infrared method is basically able to estimate the gaze position of the driver Pd with a higher degree of accuracy.

The image recognition unit 11 can estimate the distance to an object that the driver Pd is gazing at based on the positional relationship (distance, angle) between the installation position of the driver image capture unit 50 and the position of the driver Pd's eyes, which was acquired in advance, the interpupillary distance between the driver Pd's eyes in normal times, which was acquired in advance, and the interpupillary distance between the driver Pd's eyes in the captured image. Gazing at an object is preferably a time during which the driver Pd's gaze is fixed on the object, which is about 2 seconds. The image recognition unit 11 can also estimate whether the driver Pd is moving his/her gaze to a distant or near direction based on the amount of change in the convergence angle of the eyes while the driver Pd is moving his/her gaze. The image recognition unit 11 can also estimate the roll angle, pitch angle, and yaw angle of the head from the arrangement of each feature point in the face area, and estimate the head posture of the driver Pd.

In this embodiment, it is desirable to be able to determine which area of the electronic mirror display unit 20 the driver Pd is looking at when he or she is directing their gaze toward the electronic mirror device 1. For example, the display area of the electronic mirror display unit 20 is divided into four or nine parts, and multiple facial images of people looking at each display area are prepared. Machine learning is performed on the multiple facial images for each display area to generate a detector for detecting the state of looking at each display area. The image recognition unit 11 can refer to these detectors and estimate, from the image captured by the driver imaging unit 50, which display area the driver Pd is looking at when he or she is directing their gaze toward the electronic mirror device 1.

As a basic process, the display control unit 12 displays the rear image of the vehicle V1 captured by the rear imaging unit 30 on the display 20a of the electronic mirror display unit 20. The display control unit 12 can cut out an area including a specific object from the image of the rear of the vehicle cabin captured by the vehicle cabin imaging unit 40, and display the image including the cut-out object (hereinafter referred to as an object image as appropriate) on the electronic mirror display unit 20 by superimposing it on the rear image of the vehicle V1. For example, the display control unit 12 can use an OSD (On-Screen Display) function to superimpose the object image of the rear of the vehicle cabin on the rear image of the vehicle V1.

When searching for a person within a frame image using a detector as described above, the cut-out range may be, for example, the smallest rectangular area that includes the person's face and body, or a region that follows the contours of the person's face and body. In this case, objects that are in contact with the person's face or body may also be included in the cut-out range. Furthermore, if the detector is not used to search for an object, the cut-out range may be an area that has movement within a specified time. Furthermore, as described below, an object image may be generated by deleting areas other than the area where the object is detected as background areas.

The display control unit 12 can move the object image at the rear of the vehicle cabin on the rear image of the vehicle V1 according to the movement of the driver Pd's line of sight or head posture detected by the image recognition unit 11 analyzing the image captured by the driver imaging unit 50. For example, the display control unit 12 moves the object image at the rear of the vehicle cabin vertically on the rear image in accordance with the vertical movement of the driver Pd's line of sight or head posture.

The display control unit 12 can also move the cut-out position of the rear image displayed on the display 20a in accordance with the detected movement of the driver Pd's line of sight or head posture. For example, the display control unit 12 shifts the cut-out position of the rear image displayed on the display 20a to the left or right, following the left or right movement of the driver Pd's line of sight or head posture.

The image recognition unit 11 determines whether the driver Pd is in a line of sight state (hereinafter referred to as a mirror viewing state) looking to the upper left where the electronic mirror device 1 is located, based on the position of the driver Pd's iris or pupil.

For example, when the driver Pd is looking at the mirror and thrusts his head forward, causing the head area in the image to expand by a predetermined magnification or more, the image recognition unit 11 estimates that this behavior has occurred in an attempt to look at the rear of the vehicle cabin. The display control unit 12 moves the object image of the rear of the vehicle cabin upward in accordance with the magnification rate of the head area, thereby enlarging the display area of the image of the rear of the vehicle cabin. Conversely, when the driver Pd is looking at the mirror and thrusts his head back, causing the head area in the image to shrink by a predetermined magnification or more, the image recognition unit 11 estimates that this behavior has occurred in an attempt to look at the rear of the vehicle cabin. The display control unit 12 moves the object image of the rear of the vehicle cabin downward in accordance with the reduction rate of the head area, thereby reducing the display area of the image of the rear of the vehicle cabin.

For example, when the driver Pd is looking through the mirror and raises his head, the image recognition unit 11 estimates that a behavior of attempting to look at the rear of the vehicle cabin has occurred when the pitch angle of the head region in the image increases by a predetermined value or more. The display control unit 12 moves the object image of the rear of the vehicle cabin upward in accordance with the increase in the pitch angle of the head region, thereby enlarging the display area of the image of the rear of the vehicle cabin. Conversely, when the driver Pd is looking through the mirror and lowers his head, the image recognition unit 11 estimates that a behavior of attempting to look at the rear of the vehicle has occurred when the driver Pd is looking through the mirror and lowers his head, thereby decreasing the pitch angle of the head region in the image by a predetermined value or more. The display control unit 12 moves the object image of the rear of the vehicle cabin downward in accordance with the decrease in the pitch angle of the head region, thereby reducing the display area of the image of the rear of the vehicle cabin.

For example, when the driver Pd turns his head to the left while looking at the mirror, the image recognition unit 11 estimates that a behavior of attempting to look at the right rear of the vehicle V1 has occurred when the yaw angle of the head region in the image increases by a predetermined value or more as a result of the driver Pd turning his head to the left. The display control unit 12 shifts the cut-out position of the rear image of the vehicle V1 to the right in accordance with the increase in the yaw angle of the head region. Conversely, when the driver Pd turns his head to the right while looking at the mirror, the image recognition unit 11 estimates that a behavior of attempting to look at the left rear of the vehicle V1 has occurred when the driver Pd turns his head to the right and the yaw angle of the head region in the image decreases by a predetermined value or more. The display control unit 12 shifts the cut-out position of the rear image of the vehicle V1 to the left in accordance with the decrease in the yaw angle of the head region.

For example, when the driver Pd looks in the mirror and raises his/her gaze, the image recognition unit 11 estimates that a behavior of attempting to look at the rear of the vehicle cabin has occurred when the driver Pd raises his/her gaze and the iris or pupil in the image moves upward by a predetermined value or more. The display control unit 12 moves the object image of the rear of the vehicle cabin upward according to the amount of upward movement of the iris or pupil, thereby enlarging the display area of the image of the rear of the vehicle cabin. Conversely, when the driver Pd looks in the mirror and lowers his/her gaze and the iris or pupil in the image moves downward by a predetermined value or more, the image recognition unit 11 estimates that a behavior of attempting to look at the rear of the vehicle has occurred. The display control unit 12 moves the object image of the rear of the vehicle cabin downward according to the amount of downward movement of the iris or pupil, thereby reducing the display area of the image of the rear of the vehicle cabin.

For example, when the driver Pd moves his/her gaze to the left while looking at the mirror, the image recognition unit 11 estimates that a behavior of attempting to look at the left rear of the vehicle V1 has occurred when the driver Pd moves his/her gaze to the left and the iris or pupil in the image moves to the left by a predetermined value or more. The display control unit 12 shifts the cut-out position of the rear image of the vehicle V1 to the left in accordance with the amount of leftward movement of the iris or pupil. Conversely, when the driver Pd moves his/her gaze to the right while looking at the mirror, the image recognition unit 11 estimates that a behavior of attempting to look at the right rear of the vehicle V1 has occurred when the driver Pd moves his/her gaze to the right and the iris or pupil in the image moves to the right by a predetermined value or more. The display control unit 12 shifts the cut-out position of the rear image of the vehicle V1 to the right in accordance with the amount of rightward movement of the iris or pupil.

If the image recognition unit 11 does not detect a moving object for a predetermined period of time or longer in the image of the rear of the vehicle cabin captured by the vehicle cabin imaging unit 40, the display control unit 12 does not cause the image of the rear of the vehicle cabin to be displayed on the electronic mirror display unit 20. The display control unit 12 causes the electronic mirror display unit 20 to display an object image of the rear of the vehicle cabin including a moving object only if a moving object is detected within the predetermined period of time in the image of the rear of the vehicle cabin.

The motion determination threshold for determining whether an object detected in an image is moving is set by the designer. The motion determination threshold may be adjustable afterwards by the driver Pd. If the motion determination threshold is set to a strict value, the image of the object at the rear of the vehicle cabin will continue to be displayed on the electronic mirror display unit 20 if the object at the rear of the vehicle cabin moves even slightly. Conversely, if the motion determination threshold is set to a lenient value, the image of the object at the rear of the vehicle cabin will disappear from the electronic mirror display unit 20 even if the object at the rear of the vehicle cabin moves slightly. The driver Pd can adjust the motion determination threshold according to his or her preference.

The image recognition unit 11 may further include a detector for detecting a following vehicle from the rear image captured by the rear image capturing unit 21. In this case, the image recognition unit 11 can detect the following vehicle from the rear image using the following vehicle detector. The following vehicle to be detected is basically a following vehicle traveling in the same lane. The image recognition unit 11 can estimate the inter-vehicle distance to the following vehicle captured in the rear image based on a known distance estimation algorithm.

If vehicle V1 is equipped with a proximity sensor such as a millimeter wave radar or sonar, the display control unit 12 can determine whether or not there is a following vehicle based on the detection value of the proximity sensor. If there is a following vehicle, the display control unit 12 can estimate the distance between the following vehicle. The presence or absence of a following vehicle and the distance between the following vehicle may be determined using publicly known technology based on the image captured by the rear imaging unit 21 of vehicle V1.

When a following vehicle is detected behind vehicle V1, the display control unit 12 may superimpose an object image of the rear of the vehicle cabin as a semi-transparent image onto the rear image of vehicle V1. In this case, the display control unit 12 may change the transparency of the semi-transparent image depending on the distance between the following vehicle and the following vehicle. Specifically, the display control unit 12 increases the transparency of the semi-transparent image as the distance between the following vehicle and the following vehicle becomes shorter.

The display control unit 12 may also control the electronic mirror unit 20 not to display an image of the rear of the vehicle compartment when the vehicle distance to the following vehicle becomes less than a predetermined distance, even if a moving object is present in the rear of the vehicle compartment.

In addition, when the object at the rear of the vehicle compartment has stopped moving, the display control unit 12 may change the time until the image of the rear of the vehicle compartment is switched from a displayed state to a hidden state depending on the distance between the vehicle and the following vehicle. Specifically, the shorter the distance between the vehicle and the following vehicle, the shorter the time until the image is switched to the hidden state will be.

Figure 4 is a diagram showing a specific example of a rear image captured by the rear imaging unit 30. The rear image shown in Figure 4 shows a following vehicle V2. As shown in Figure 4, in the default state, a cut-out range C1 is set in the center of the imaging range 30i of the solid-state imaging element of the rear imaging unit 30. An image cut out from the cut-out range C1 is displayed on the display 20a.

Figures 5(a)-(c) are diagrams showing specific examples of images of the rear of the vehicle interior captured by the vehicle interior imaging unit 40. In the example shown in Figures 5(a)-(c), a first occupant P1 is sitting in the right rear seat of the vehicle V1, and a second occupant P2 is sitting in the left rear seat. In the default state as shown in Figures 5(a)-(b), a cut-out range C2 is set in the center of the imaging range 40i of the solid-state imaging element of the vehicle interior imaging unit 400. The image cut out from the cut-out range C2 is superimposed on the rear image captured by the rear imaging unit 30.

From the image of the rear of the vehicle cabin shown in FIG. 5(a), the image recognition unit 11 detects the first occupant P1 and the second occupant P2 as objects. As shown in FIG. 5(b), the display control unit 12 deletes the area other than the area in which the detected objects, the first occupant P1 and the second occupant P2, exist as background areas.

In the example shown in FIG. 5(c), the display control unit 12 shifts the cut-out range C2 upward so that the first occupant P1 and the second occupant P2 are positioned at the bottom of the screen when superimposed on the rear image, and cuts out the image from the cut-out range C2 after the shift. The display control unit 12 may also cut out the image from the default cut-out range C2 shown in FIG. 5(b).

Figure 6 shows a composite image displayed on the electronic mirror display unit 20, in which an image cut out from the cut-out range C2 of the image of the rear of the vehicle compartment shown in Figure 5(c) is superimposed on an image cut out from the cut-out range C1 of the rear image shown in Figure 4.

Figures 7(a)-(b) are diagrams showing specific examples of when the head posture or line of sight of the driver Pd moves up or down in the display state shown in Figure 6. Figure 7(a) shows a display example of when the head posture or line of sight of the driver Pd moves downward, and Figure 7(b) shows a display example of when the head posture or line of sight of the driver Pd moves upward.

When the driver Pd's head posture or line of sight height moves downward as shown in FIG. 7(a), it can be assumed that the driver Pd is trying to look at the rear of the vehicle. The display control unit 12 expands the display area of the rear image of the vehicle V1 by shifting the cut-out range C2 of the image of the rear of the vehicle compartment upward or shifting the composite position of the cut-out image downward. Conversely, when the driver Pd's head posture or line of sight height moves upward as shown in FIG. 7(b), it can be assumed that the driver Pd is trying to look at the rear of the vehicle compartment. The display control unit 12 expands the display area of the image of the rear of the vehicle compartment by shifting the cut-out range C2 of the image of the rear of the vehicle compartment downward or shifting the composite position of the cut-out image upward.

8 is a diagram showing a first modified example when a cut-out image of the rear image of the vehicle cabin is superimposed on a cut-out image of the rear image. In the first modified example, the display control unit 12 shifts an object included in the cut-out image of the rear image of the vehicle cabin to the left or right end of the cut-out image. In the example shown in FIG. 8, the display control unit 12 shifts the area of the first occupant P1 to the left end and the area of the second occupant P2 to the right end in the cut-out image of the rear image of the vehicle cabin. Specifically, the display control unit 12 superimposes an image after object shifting of an image cut out from the cut-out range C2 of the rear image of the vehicle cabin shown in FIG. 5(c) on an image cut out from the cut-out range C1 of the rear image shown in FIG. 4, and displays the composite image after superposition on the electronic mirror display unit 20. This makes it easier to check the situation behind the vehicle V1.

The display control unit 12 may expand the area in which the rear image of the vehicle V1 is displayed by shrinking an object included in the cropped image of the image of the rear of the vehicle interior. Furthermore, the display control unit 12 may combine shifting an object to the left or right edge in the cropped image of the image of the rear of the vehicle interior with shrinking the object.

Figures 9(a)-(c) show variant example 2 when a cut-out image of the rear image of the vehicle cabin is superimposed on a cut-out image of the rear image. Figure 9(a) shows an image cut out from the cut-out range C1 of the rear image shown in Figure 4. As shown in Figure 9(b), the display control unit 12 reduces the cut-out image of the rear image at a predetermined reduction rate in the vertical direction. The display control unit 12 reduces the image cut out from the cut-out range C2 of the rear image of the vehicle cabin shown in Figure 5(c) at a predetermined reduction rate in the vertical direction.

As shown in FIG. 9(c), the display control unit 12 superimposes a cut-out image of the rear image of the vehicle interior, which has been reduced in the vertical direction, on a cut-out image of the rear image, which has been reduced in the vertical direction, and displays the composite image after superposition on the electronic mirror display unit 20. In the display example shown in FIG. 9(c), the aspect ratios of the cut-out image of the rear image and the cut-out image of the rear image of the vehicle interior are distorted, but the area in which they overlap is reduced. This reduces the possibility that the following vehicle and objects at the rear of the vehicle interior will overlap in the composite image, making it easier to check the situation behind vehicle V1.

Figure 10 shows an example display of the following vehicle V2 changing lanes to the right lane. Specifically, it shows the driver Pd following the following vehicle V2 reflected in the electronic mirror display unit 20 with his eyes. The driver Pd is moving his gaze to the left.

Figures 11(a)-(c) are diagrams showing a specific example of shifting the clipping range C1 of the rear image in response to the line of sight movement of the driver Pd. Figure 11(a) shows the clipping range C1 of the rear image before the line of sight movement, and Figure 11(b) shows the clipping range C1 of the rear image after the line of sight movement. The display control unit 12 shifts the clipping range C1 of the rear image to the left in response to the line of sight movement of the driver Pd to the left.

Figure 11(c) shows a state in which an image after object shift of an image cut out from cut-out range C2 of the image of the rear of the vehicle compartment shown in Figure 5(c) is superimposed on an image cut out from cut-out range C1 of the rear image shown in Figure 11(b), and the composite image after superposition is displayed on the electronic mirror display unit 20. In the display example shown in Figure 11(c), even though the following vehicle V2 has moved to the right, the following vehicle V2 is positioned in a more central position on the electronic mirror display unit 20, improving the visibility of the following vehicle V2.

Figures 12(a)-(b) are diagrams showing a specific example in which the transparency of a cropped image of the image of the rear of the vehicle interior is changed depending on the distance from the following vehicle V2. As shown in Figure 12(a), when the distance from the following vehicle V2 is equal to or greater than a threshold, the display control unit 12 superimposes the cropped image of the image of the rear of the vehicle interior on the rear image with a transparency of 0%. As shown in Figure 12(b), when the distance from the following vehicle V2 falls below the threshold, the display control unit 12 increases the transparency of the cropped image of the image of the rear of the vehicle interior as the distance from the following vehicle V2 becomes shorter.

As another display mode, the display control unit 12 may change the time for which the cropped image of the rear cabin image is superimposed on the rear image depending on the distance between the following vehicle V2. The display ratio may be changed within a unit time. Specifically, for example, when the distance between the following vehicle V2 is long, a combination of displaying the rear image for 0.3 seconds and displaying the image with the rear cabin image superimposed thereon for 0.7 seconds is repeatedly displayed. When the distance between the following vehicle V2 becomes short, a combination of displaying the rear image for 0.7 seconds and displaying the image with the rear cabin image superimposed thereon for 0.3 seconds is repeatedly displayed.

Figure 13 is a flowchart showing an example of the operation of the digital mirror device 1 according to the embodiment. When the driver Pd turns on the ignition of the vehicle V1 (step S10: Yes), the display control unit 12 acquires a cropped image of the rear image from the rear imaging unit 30, and acquires a cropped image of the rear image of the vehicle interior from the vehicle interior imaging unit 40 (step S11).

The display control unit 12 deletes the background area from the cut-out image of the rear image of the vehicle interior (step S12). The display control unit 12 superimposes the cut-out image of the rear image of the vehicle interior after the background has been deleted onto the cut-out image of the rear image (step S13). The display control unit 12 displays the composite image obtained by superimposing on the display 20a (step S14). The display control unit 12 changes the layout when combining the cut-out image of the rear image and the cut-out image of the rear image according to changes in the head posture or line of sight height of the driver Pd (step S15). The process proceeds to step S10. When the driver Pd turns off the ignition of the vehicle V1 (step S10: Yes), the operation of the electronic mirror device 1 stops.

The control program of the digital mirror device 1 may be installed in the processing unit 10 in advance, or may be installed later. In the latter case, the control program may be downloaded from an application program store via a network to the digital mirror device 1 and installed.

As described above, according to this embodiment, a composite image in which a cropped image of the rear of the vehicle compartment image is superimposed on a cropped image of the rear image is displayed on the electronic mirror display unit 20, allowing the driver Pd to simultaneously check the situation behind the vehicle and the situation at the rear of the vehicle compartment with minimal eye movement. Because the situation at the rear of the vehicle compartment can be checked through video, it is easy to grasp the situation of rear seat passengers, who were difficult to see because they were hidden by the headrests of the front seats with conventional optical mirrors.

In addition, there is no need to install a separate monitor for displaying the rear image and a monitor for displaying the image of the rear of the vehicle cabin, which helps to prevent increases in costs. The aesthetic appearance of the vehicle interior is also maintained. In addition, the need to shift one's gaze between looking at the rear of the vehicle and looking at the rear of the vehicle cabin can be minimized, which contributes to improved safety. In particular, it can improve safety for elderly drivers whose eyes have a reduced ability to adjust their focus.

In addition, by hiding areas in the image of the rear of the vehicle cabin that remain unchanged, the image of the rear of the vehicle cabin can be displayed on the electronic mirror display unit 20 only when there is a change in the condition of the rear of the vehicle cabin. This makes it possible to accurately indicate to the driver Pd the timing when he or she should pay attention to the condition of the rear of the vehicle cabin.

The display control unit 12 also adaptively changes the display range of the rear image in response to changes in the driver Pd's head posture or line of sight height based on the image captured by the driver imaging unit 50, thereby giving the driver Pd a sensation similar to the change in visibility caused by line of sight movement in a conventional optical mirror. For example, if the driver Pd's head posture moves to the right or his line of sight moves to the left, the display control unit 12 determines that the driver Pd is trying to look at the left rear, and increases the image display area of the left rear so that the left edge area that was not visible until now can be seen. This allows the driver Pd to smoothly see the place he wants to see.

The present invention has been described above based on an embodiment. This embodiment is merely an example, and it will be understood by those skilled in the art that various modifications are possible in the combination of each component and each processing process, and that such modifications are also within the scope of the present invention.

In the above embodiment, the electronic mirror device 1 has been described as being capable of being used as a conventional optical mirror by installing the half mirror 20b. In this regard, the half mirror 20b may be omitted. In this case, the electronic mirror display unit 20 becomes a pure display. The display brightness of the display can be made higher than when the half mirror 20b is installed.

In the above embodiment, the driver image capture unit 50 may be omitted. In this case, the layout of the composite image of the cropped image of the rear image and the cropped image of the rear of the vehicle compartment is fixed, but costs can be reduced.

1 Digital mirror device, 1a Digital mirror device main body, 10 Processing unit, 11 Image recognition unit, 12 Display control unit, 20 Digital mirror display unit, 20a Display, 20b Half mirror, 30 Rear image capture unit, 40 Vehicle interior image capture unit, 50 Driver image capture unit, V1 Vehicle, V2 Following vehicle, Pd Driver, P1 First passenger, P2 Second passenger.

Claims (5)

A first imaging unit that images a rear area of the vehicle;
A second imaging unit that images a rear portion of a passenger compartment of the vehicle;
an electronic mirror display unit that displays a rear image of the vehicle captured by the first imaging unit;
a display control unit that cuts out an area including a predetermined object from the image of the rear of the vehicle compartment captured by the second imaging unit, and displays the image including the cut-out object on the electronic mirror display unit by superimposing the image on the rear image; and
An electronic mirror device comprising: The display control unit moves an image including the cut-out object on the rear image, or moves a cut-out position as a display range of the rear image captured by the first imaging unit, in response to a movement of the driver's line of sight or head posture detected by analyzing an image captured by a third imaging unit that captures an image of the driver in the vehicle.
The digital mirror apparatus according to claim 1 . When a moving object is not detected for a predetermined time or longer in the image of the rear of the vehicle interior captured by the second imaging unit, the display control unit controls the electronic mirror display unit not to display an image including the clipped object.
The digital mirror apparatus according to claim 1 . when a following vehicle is detected behind the vehicle, the display control unit superimposes an image including the clipped object on the rear image as a semi-transparent image;
The display control unit changes the transparency of the semi-transparent image in accordance with a distance between the vehicle and the following vehicle.
The digital mirror apparatus according to claim 1 . A control program for an electronic mirror device including an electronic mirror display unit that displays a rear image of a vehicle captured by a first imaging unit that captures an image of the rear of the vehicle,
a function of cutting out an area including a predetermined object from an image of the rear of the vehicle cabin captured by a second imaging unit that captures an image of the rear of the vehicle cabin, and displaying the image including the cut-out object on the electronic mirror display unit by superimposing the image on the rear image;
A control program for an electronic mirror device that causes a computer to execute the above.

Images