JP6729958B1 - Display control system, display control method and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a display control system, a display control method, and a program capable of allowing a crew member to accurately grasp the position of a monitoring target that cannot be recognized by a camera that captures a situation around a vehicle. SOLUTION: An outside-vehicle image position specifying unit 42 specifies an outside-vehicle image position, which is a position in the outside-vehicle image of a monitoring target appearing in an outside-vehicle image captured by an outside-vehicle photographing device provided outside the vehicle. .. The display internal position specifying unit 56 determines the display internal position, which is the position of the monitoring target in the display unit, based on the position outside the vehicle image and the position and orientation of the display unit arranged in front of the crew of the vehicle. Identify. The display control unit 58 displays the mark image at the specified position in the display unit. [Selection diagram] Figure 8

Description

The present invention relates to a display control system, a display control method and a program.

In Patent Document 1, when the presence of a pedestrian is recognized based on camera image information obtained from a camera that captures a situation around a vehicle, a mark is displayed on a head-up display (HUD) by a virtual image by superimposing it on the pedestrian. Describes the technology for displaying alerts.

Further, Patent Document 1 describes that a road surface sign such as a front pedestrian crossing that cannot be recognized by video information is displayed as a virtual image at a position where it is grasped from the map information, a travel log, and the like as they exist. ..

JP, 2019-59247, A

The inventor displays a mark image such as a mark at the appearance position of the monitored object in the display unit such as the HUD arranged in front of the crew of the vehicle when the monitored object such as a fallen person from the platform appears. Therefore, we are considering making the crew accurately grasp the position of the monitored object.

Here, even if the technique described in Patent Document 1 is used, it is not possible to display a landmark image at the appearance position of a monitoring target object that cannot be recognized by a camera that captures the situation around the vehicle. Further, since such a monitoring target suddenly appears, its position cannot be grasped in advance from the map information, the travel log, etc. described in Patent Document 1.

The present invention has been made in view of the above problems, and one of the objects thereof is a display that allows a crew member to accurately grasp the position of a monitoring target that cannot be recognized by a camera that captures a situation around a vehicle. It is to provide a control system, a display control method, and a program.

The display control system according to the present invention is an in-vehicle image position for identifying an in-vehicle image position, which is a position in the outside image of a monitored object, which is shown in an outside image captured by an outside image capturing device provided outside the vehicle. Based on the specifying means, the position outside the vehicle image, and the position and orientation of the display unit arranged in front of the crew of the vehicle, the position inside the display unit that is the position of the monitored object in the display unit is specified. And a display control unit for displaying a mark image at the specified position in the display unit.

In one aspect of the present invention, based on the travel route data indicating the linear travel route of the vehicle and the position inside the vehicle image, the position of the monitored object on the travel route indicated by the travel route data is determined. It further includes a travel route position specifying unit that specifies a certain travel route position, and the display unit internal position specifying unit specifies the display unit internal position based on the travel route position.

Further, according to an aspect of the present invention, a determination unit that determines whether or not the monitoring target object has been captured by an in-vehicle image capturing device provided in the vehicle, and the monitoring that appears in an in-vehicle image captured by the in-vehicle image capturing device. When it is determined that the monitoring target has been captured by the vehicle-mounted image capturing device, the vehicle-mounted image capturing device further includes a vehicle-mounted image internal position identifying unit that identifies an in-vehicle image internal position that is a position in the vehicle-mounted image of the object. The display section internal position specifying means specifies the display section internal position based on the vehicle-mounted image internal position.

In this aspect, when the in-vehicle imaging device changes from a situation in which it is not determined that the monitoring target has been photographed to a situation in which it is determined, the display unit internal position specifying unit determines the display unit internal position by the vehicle exterior image. It may be changed from the one based on the internal position to the one based on the in-vehicle image internal position.

Further, in one aspect of the present invention, the display control means, when the position of the monitored object is outside the display range of the display unit, on the edge of the display unit according to the position of the monitored object. Display the landmark image.

Moreover, in one aspect of the present invention, the display unit is a head-up display provided in the vehicle.

Alternatively, the display unit is a head mounted display worn by the crew member.

Further, the display control method according to the present invention includes a step of identifying an in-vehicle image position which is a position of the monitoring target in the out-vehicle image captured in an out-vehicle image captured by an out-vehicle image capturing device provided outside the vehicle. A position inside the display unit that is the position of the monitored object in the display unit based on the position inside the image outside the vehicle and the position and orientation of the display unit arranged in front of the crew member of the vehicle; And displaying a landmark image at the specified position in the display unit.

Further, the program according to the present invention is a procedure for identifying an in-vehicle image position, which is a position of the monitoring target in the outside image captured in an outside image taken by an outside image pickup device provided outside the vehicle, the outside of the vehicle. Based on the position in the image and the position and orientation of the display unit arranged in front of the crew member of the vehicle, the procedure for specifying the position in the display unit, which is the position of the monitored object in the display unit, is specified. A computer is made to perform the procedure of displaying a landmark image at the position in the display unit.

It is a figure showing an example of composition of a display control system concerning one embodiment of the present invention. It is a figure which shows an example of an image outside a vehicle. It is a figure which shows an example of an image outside a vehicle. It is a figure which shows an example of an image outside a vehicle. It is a figure which shows an example of the image displayed on the display of a surveillance system. It is a figure which shows an example of a mode that the crew member of the vehicle looked ahead. It is a figure which shows an example of a mode that the crew member of the vehicle looked ahead. It is a functional block diagram which shows an example of the function mounted in the vehicle and monitoring system which concern on one Embodiment of this invention. It is a figure which shows an example of virtual space. It is a flow figure showing an example of a flow of processing performed with a vehicle concerning one embodiment of the present invention. It is a figure which shows an example of a mode that the crew member of the vehicle looked ahead. It is a figure which shows an example of a mode that the crew member of the vehicle looked ahead.

Hereinafter, an embodiment of the present invention will be described in detail with reference to the drawings.

FIG. 1 is a diagram showing an example of a configuration of a display control system 1 according to an embodiment of the present invention. As shown in FIG. 1, the display control system 1 according to the present embodiment includes a vehicle 10 that operates on a route, a plurality of monitoring systems 12, and an operation management system 14 that manages the operation of the vehicle 10.

Each of the plurality of monitoring systems 12 is provided, for example, at different places on the route on which the vehicle 10 operates. Here, for example, each station on the route may be provided with a monitoring system 12 that monitors the platform of the station. Further, for example, the monitoring system 12 may be provided in each of the plurality of platforms provided in one station. Further, a plurality of monitoring systems 12 may be provided on one platform.

The vehicle 10, the monitoring system 12, and the operation management system 14 are connected to a computer network 16 such as the Internet. Therefore, the vehicle 10, the monitoring system 12, and the operation management system 14 can communicate with each other via the computer network 16.

As shown in FIG. 1, the vehicle 10 includes a processor 10a, a storage unit 10b, a communication unit 10c, a vehicle-mounted image capturing device 10d, and a head-up display (HUD) 10e.

The processor 10a is a program control device such as a CPU that operates according to a program stored in the storage unit 10b, for example.

The storage unit 10b is a storage element such as a ROM or RAM, a hard disk drive, or the like. The storage unit 10b stores a program executed by the processor 10a and the like.

The communication unit 10c is a communication interface such as a network board.

The vehicle-mounted imaging device 10d is an imaging device such as a digital camera that images the front of the vehicle 10, for example. The in-vehicle image capturing device 10d may be a camera (for example, a stereo camera) capable of capturing a three-dimensional image or a depth image. Hereinafter, the image captured by the in-vehicle image capturing device 10d will be referred to as an in-vehicle image. The vehicle-mounted photographing device 10d according to the present embodiment photographs the front of the vehicle 10 at a predetermined frame rate.

The HUD 10e is a display device that displays an image in front of the eyes of a crew member such as a driver. The mounting method of the HUD 10e according to this embodiment is not particularly limited. For example, the HUD 10e according to this embodiment may be of a type that projects an image on a windshield provided on the front surface of the vehicle 10. Further, the HUD 10e according to the present embodiment may be of a type that projects an image on a combiner arranged in front of the crew member. In the present embodiment, the image is displayed by the HUD 10e, so that the crew can visually recognize both the image displayed by the HUD 10e and the state of the real space in front of the vehicle 10. The HUD 10e may be configured to show a crew member a virtual image floating in a space in front of a windshield provided on the front surface of the vehicle 10.

As shown in FIG. 1, the monitoring system 12 includes a processor 12a, a storage unit 12b, a communication unit 12c, a vehicle exterior imaging device 12d, and a display 12e.

The processor 12a is a program control device such as a CPU that operates according to a program stored in the storage unit 12b, for example.

The storage unit 12b is a storage element such as a ROM or RAM, a hard disk drive, or the like. The storage unit 12b stores programs and the like executed by the processor 12a.

The communication unit 12c is a communication interface such as a network board.

The vehicle exterior imaging device 12d is, for example, an imaging device such as a digital camera that captures an image of a station platform or the like. The vehicle exterior imaging device 12d according to the present embodiment photographs a fixed point at a predetermined frame rate. The vehicle exterior image capturing device 12d may be a camera (for example, a stereo camera) capable of capturing a three-dimensional image or a depth image. Hereinafter, the image captured by the vehicle exterior imaging device 12d will be referred to as the vehicle exterior image.

The display 12e is, for example, a display device such as a liquid crystal display or an organic EL display that displays an image outside the vehicle captured by the vehicle exterior image capturing device 12d.

An image 20 outside the vehicle is shown in FIG. In the present embodiment, as shown in FIG. 2, four areas of a safety area 22a, an approach area 22b, a falling area 22c, and a track keeping escape area 22d are set in the vehicle exterior image 20.

The above-mentioned four areas set in the image 20 outside the vehicle may be set in advance by using the vehicle limit frame model 24 as shown in FIG. 3, for example. In the setting, for example, first, the position of the track in the image 20 outside the vehicle is identified. Then, a plurality of vehicle limit frame models 24 are arranged along the identified track. The size of each vehicle limit frame model 24 is, for example, proportional to the width of the track on which the vehicle limit frame model 24 is arranged.

Then, a line connecting the positions corresponding to the diagonal lower ends on the side opposite to the home of the plurality of vehicle limit frame models 24, a line extending from the upper end of the lines to the side opposite to the home (left in the example of FIG. 4), and An area surrounded by a side (left side in the example of FIG. 4) on the side opposite to the home of the image 20 outside the vehicle may be set as the track maintenance escape area 22d.

A line connecting the positions corresponding to the diagonal lower ends of the plurality of vehicle limit frame models 24 on the side opposite to the home, a line extending from the upper end of the line to the home side (right in the example of FIG. 4), a plurality of vehicle limit frames A region that is surrounded by a line connecting a position corresponding to the lower end of the platform side of the model 24, the lower side of the outside image 20 and the side of the outside image 20 opposite to the home (the left side in the example of FIG. 4 ). , May be set in the falling area 22c.

Then, a line connecting the positions corresponding to the lower ends of the side positions on the home side of the plurality of vehicle limit frame models 24, a line extending from the upper end of the lines to the home side (right in the example of FIG. 4), a braille block or a white line home An area surrounded by the end on the opposite side (the left end in the example of FIG. 4) and the lower side of the vehicle exterior image 20 may be set as the approach area 22b.

Then, a line extending from the upper end of the line connecting the positions corresponding to the lower ends of the side positions on the home side of the plurality of vehicle limit frame models 24 to the home side (right in the example of FIG. 4), the side on the home side of the outside image 20 ( Even if the area surrounded by the right side in the example of FIG. 4, the lower side of the image 20 outside the vehicle, and the end of the braille block or the white line opposite to the home (the left end in the example of FIG. 4) is set as the safety area 22a. Good.

As described above, the safety area 22a, the approaching area 22b, the falling area 22c, and the track keeping escape area 22d are set in the vehicle exterior image 20. Note that the settings of the safety area 22a, the approach area 22b, the fall area 22c, and the track keeping escape area 22d are not limited to the above-mentioned settings, and these four areas can be set by another predetermined method. May be set in the image 20 outside the vehicle. Further, the falling area 22c may be set with a margin with respect to the vehicle limit frame indicated by the vehicle limit frame model 24.

As described above, since the vehicle exterior imaging device 12d captures a fixed point, the above-described area is common in the series of vehicle exterior images 20 captured by the vehicle exterior imaging device 12d at a predetermined frame rate.

Hereinafter, the area including the approaching area 22b, the falling area 22c, and the track keeping escape area 22d will be referred to as a monitoring target area. Then, the monitoring system 12 according to the present embodiment detects a predetermined monitoring target object appearing in the monitoring target area in the captured outside image 20 by using a known image recognition technique or the like.

FIG. 4 shows an outside-vehicle image 26 which is an outside-vehicle image different from the outside-vehicle image 20 shown in FIGS. 2 and 3. The outside image 26 shown in FIG. 4 shows a fallen person 28 who has fallen off the platform, which is an example of a predetermined monitoring target.

A safety area 22a, an approach area 22b, and a fall area 22c are set in the vehicle exterior image 26 shown in FIG. Unlike the image 26 outside the vehicle shown in FIG. 4, the track keeping escape area 22d may not be set in the image outside the vehicle 26. Then, the fallen person 28 is shown in the falling area 22c which is a part of the monitoring target area of the vehicle exterior image 26 shown in FIG. Further, FIG. 4 shows the center position P1 of the fallen person 28.

In this way, when it is detected that a predetermined monitoring target object appears in the monitoring target area, for example, a mark image 30 corresponding to the detected monitoring target object is displayed on the display 12e outside the vehicle shown in FIG. An image 32 illustrated in FIG. 5 superimposed on the image 26 is displayed.

The landmark image 30 is, for example, an image showing the position of the fallen person 28 in the image 26 outside the vehicle. In the example of FIG. 5, the mark image 30 is a frame-shaped image centered on the position P1.

Further, the mark image 30 may be an image in a mode (for example, color) corresponding to the type of the area where the detected monitoring target object exists. For example, when the detected monitored object is present in the track maintenance area 22d, the mark image 30 may be a purple frame image. When the detected monitoring target is present in the falling area 22c, the mark image 30 may be an image with a red frame. When the detected monitoring target object exists in the approaching area 22b, the mark image 30 may be an image of a yellow frame. In the example of FIG. 4, the falling person 28 exists in the falling area 22c, so the mark image 30 shown in FIG. 5 may be, for example, a red frame image.

FIG. 6 is a diagram illustrating an example of a state in which a crew member of the vehicle 10 looks forward when a situation in which a predetermined monitoring target is reflected in the monitoring target area continues for a predetermined time. In the present embodiment, when the situation in which the predetermined monitoring target appears in the monitoring target area continues for a predetermined time, as shown in FIG. 6, the position P2 of the monitoring target in the HUD 10e of the vehicle 10 is marked. The image 34 is displayed.

The position P2 is the position of the monitoring target that is associated with the position P1 shown in FIG. Then, in the example of FIG. 6, the mark image 34 is a frame-shaped image centered on the position P2. Here, the aspect ratio of the mark image 34 may be the same as the aspect ratio of the mark image 30. The size of the mark image 34 may be smaller as the distance between the vehicle 10 and the monitored object is longer. Further, similarly to the mark image 30, the mark image 34 may be an image of a mode (for example, a color) corresponding to the type of the area where the detected monitoring target object exists. In the example of FIG. 6, the mark image 34 may be, for example, a red frame image.

As described above, in the present embodiment, even if the predetermined monitoring target such as the fallen person 28 is not shown in the vehicle-mounted image captured by the vehicle-mounted image capturing device 10d, the landmark image 34 is displayed at the position of the detected monitoring target. Is displayed. Therefore, according to the present embodiment, it becomes possible for the crew to accurately grasp the position of the monitoring target that cannot be recognized by the in-vehicle image capturing device 10d. In the present embodiment, the mark image 34 may be displayed at the position P2 of the monitored object in the HUD 10e as soon as the predetermined monitored object is captured in the monitored area.

Further, the vehicle 10 according to the present embodiment detects a predetermined monitoring target object shown in the vehicle-mounted image captured by the vehicle-mounted image capturing device 10d by using a known image recognition technique or the like.

FIG. 7 is a diagram showing an example of a state in which a crew member of the vehicle 10 looks forward when the vehicle 10 advances from the situation shown in FIG. Here, in the situation shown in FIG. 7, it is assumed that the fallen person 28, which is a monitoring target, is shown in the vehicle-mounted image captured by the vehicle-mounted image capturing device 10d.

In this situation, the mark image 34 is displayed not at the position associated with the position P1 shown in FIG. 4 but at the position P3 of the monitored object detected based on the in-vehicle image. In the example of FIG. 7, the mark image 34 is a frame-shaped image centered on the position P3.

Since both the vehicle-mounted image capturing device 10d and the HUD 10e are fixed to the vehicle 10, the position and orientation of the HUD 10e are always constant with reference to the position and orientation of the vehicle-mounted image capturing device 10d. In the present embodiment, the position and orientation of the vehicle-mounted imaging device 10d and the position and orientation of the HUD 10e are associated in advance. Therefore, the position P3 of the monitored object in the HUD 10e can be uniquely specified based on the position of the monitored object in the vehicle-mounted image.

By doing as described above, when it is possible to detect that a predetermined monitoring target object appears in the vehicle-mounted image captured by the vehicle-mounted image capturing device 10d, the crew member can accurately grasp the position of the monitoring target object. Is possible.

Hereinafter, the functions of the vehicle 10 and the monitoring system 12 according to the present embodiment, and the processing executed by the vehicle 10 according to the present embodiment will be further described.

FIG. 8 is a functional block diagram showing an example of functions implemented in the vehicle 10 and the monitoring system 12 according to this embodiment. It should be noted that the vehicle 10 and the monitoring system 12 according to the present embodiment do not need to have all the functions shown in FIG. 8 installed, and may have functions other than those shown in FIG.

As shown in FIG. 8, the monitoring system 12 according to the present embodiment functionally includes, for example, a detection unit 40, a vehicle outside image position specifying unit 42, and a detection notification unit 44. The detection unit 40 is mainly mounted on the processor 12a and the vehicle exterior imaging device 12d. The vehicle outside image position specifying unit 42 is mainly mounted on the processor 12a. The detection notification unit 44 is mainly mounted on the communication unit 12c.

The above functions may be implemented by causing the processor 12a to execute a program that is installed in the monitoring system 12 that is a computer and that includes a command corresponding to the above functions. This program is supplied to a computer installed in the monitoring system 12 via a computer-readable information storage medium such as an optical disc, a magnetic disc, a magnetic tape, a magneto-optical disc, a flash memory, or the like, or via the Internet or the like. May be done.

Further, as shown in FIG. 8, in the vehicle 10 according to the present embodiment, functionally, for example, the notification receiving unit 50, the detection data storage unit 52, the traveling route position specifying unit 54, and the display position inside specifying unit 56. The display control unit 58, the determination unit 60, and the in-vehicle image position specifying unit 62 are included. The notification receiving unit 50 is implemented mainly by the communication unit 10c. The detection data storage unit 52 is mainly mounted on the storage unit 10b. The travel route position specifying unit 54 is implemented mainly by the processor 10a and the storage unit 10b. The in-display position specifying unit 56 is mainly mounted on the processor 10a, the storage unit 10b, and the communication unit 10c. The display control unit 58 is mainly mounted on the processor 10a and the HUD 10e. The determination unit 60 is mainly mounted on the processor 10a and the vehicle-mounted imaging device 10d. The in-vehicle image position specifying unit 62 is mainly mounted on the processor 10a.

The above functions may be implemented by executing a program, which is installed in a computer mounted on the vehicle 10 and includes a command corresponding to the above functions, in the processor 10a. This program is supplied to a computer mounted on the vehicle 10 via a computer-readable information storage medium such as an optical disc, a magnetic disc, a magnetic tape, a magneto-optical disc, a flash memory, or the like, or via the Internet or the like. May be.

In the present embodiment, for example, the detection unit 40 uses a known image recognition technique or the like to detect the monitoring target object in the vehicle exterior image captured by the vehicle exterior imaging device 12d provided outside the vehicle 10. The detection unit 40 may detect, for example, a monitoring target object appearing in the monitoring target area.

The outside-vehicle image position specifying unit 42 specifies the inside-vehicle image position, which is the position of the monitoring target in the outside-vehicle image in which the monitoring target is detected. Here, for example, based on the vehicle exterior image 26 shown in FIG. 4, the two-dimensional coordinate value of the position P1 of the center of the fallen person 28 who is the monitoring target in the image plane of the vehicle exterior image 26 is specified.

In the present embodiment, for example, the detection notification unit 44 transmits a detection notification indicating that the monitoring target has been detected to the vehicle 10 when the detection unit 40 detects the monitoring target. Here, the detection notification unit 44 may transmit the detection notification to the vehicle 10 when the situation in which the monitoring target in the monitoring target area is reflected continues for a predetermined time.

The detection notification unit 44 may also send the detection notification to the vehicle 10 that satisfies a predetermined condition. For example, the detection notification unit 44 may transmit the detection notification to the vehicle 10 heading for the station where the monitoring system 12 is provided and having a distance from the station that is equal to or less than a predetermined distance. Further, for example, the detection notification unit 44 may transmit the detection notification to the vehicle 10 heading for the station where the monitoring system 12 is provided and between the station and the station in front of the station.

The detection notification includes, for example, the identification information of the monitoring system 12 that transmits the detection notification or the station where the monitoring system 12 is provided, the detection target position data indicating the position of the detected monitoring target, and the like. The data is associated. The monitored object position data may be, for example, data indicating the two-dimensional coordinate value of the position P1 described above.

In the present embodiment, for example, the notification receiving unit 50 receives the above-described detection notification from the monitoring system 12 that has detected the monitoring target object. Then, the notification receiving unit 50 causes the detection data storage unit 52 to store the detection data associated with the received detection notification.

The detection data storage unit 52 stores the detection data in the present embodiment, for example.

In the present embodiment, for example, the travel route position specifying unit 54 specifies the position of the monitored object on the linear travel route of the vehicle 10 (hereinafter, referred to as the travel route position).

In the present embodiment, for example, the travel route position specifying unit 54 stores travel route data indicating a linear travel route of the vehicle 10 in advance. In the travel route data, for example, three-dimensional coordinate values in a real space at a plurality of positions along the linear travel route of the vehicle 10 are shown. Here, the three-dimensional coordinate value may be represented by, for example, latitude, longitude, and altitude.

Further, in the present embodiment, for example, the outside-vehicle image/traveling route correspondence relationship data for each of the plurality of monitoring systems 12 is stored in the traveling route position specifying unit 54 in advance. The outside-vehicle image/running route correspondence data is, for example, data indicating a given correspondence between the position in the outside-vehicle image captured by the monitoring system 12 and the position on the running route indicated by the running route data.

Then, in the present embodiment, it is associated with the in-vehicle image position based on the in-vehicle image position, the traveling route data, and the outside-vehicle image/driving route correspondence relationship data indicated by the monitored object position data included in the detection data. The position on the travel route (the above-mentioned position on the travel route) is specified.

In the present embodiment, the position identifying unit 56 in the display unit, for example, in the image outside the vehicle and the position and orientation of the display unit (for example, the HUD 10e) arranged in front of the crew member of the vehicle 10 in the display unit. The position of the monitored object (hereinafter referred to as the position in the display unit) is specified.

The specification of the position in the display unit by the position-in-display unit 56 will be described below with reference to FIG.

FIG. 9 is a diagram showing an example of a virtual space associated with a real space. The position in the real space is in one-to-one correspondence with the position in the virtual space shown in FIG. The data indicating the virtual space is stored in, for example, the display internal position specifying unit 56.

In this embodiment, for example, the operation management system 14 manages the position and traveling direction of the vehicle 10 on the travel route R in real time. Then, operation data indicating the current position and traveling direction of the vehicle 10 on the travel route R is transmitted to the vehicle 10 from the operation management system 14 at predetermined time intervals, for example.

Then, in the present embodiment, for example, the vehicle-HUD correspondence relationship data indicating a given correspondence relationship between the position and orientation of the vehicle 10 indicated by the operation data and the position and orientation of the HUD 10e is previously stored in the display position identifying unit 56. Remembered Then, the display intra-position specifying unit 56 specifies the current position and orientation of the HUD 10e based on the vehicle-HUD correspondence relationship data and the operation data indicating the current position and traveling direction of the vehicle 10.

In addition, in the present embodiment, for example, the display-internal position specifying unit 56 preliminarily provides a vehicle/viewpoint correspondence indicating a given correspondence relationship between the position and direction of the vehicle 10 indicated by the operation data and the viewpoint position and line-of-sight direction of the crew. Related data is stored. Then, the position-in-display part specifying unit 56 specifies the current position of the crew member's viewpoint and the line-of-sight direction based on the vehicle/viewpoint correspondence data and the operation data indicating the current position and traveling direction of the vehicle 10. ..

FIG. 9 shows a plate-shaped virtual HUD object 70 associated with the HUD 10e in the real space. The virtual HUD object 70 shown in FIG. 9 is arranged at the position in the virtual space associated with the position of the HUD 10e in the real space, in the orientation in the virtual space associated with the orientation of the HUD 10e in the real space.

Further, FIG. 9 shows a position P4 in the virtual space that is associated with the current position of the viewpoint of the crew member in the real space specified as described above. Further, FIG. 9 shows the line-of-sight direction D in the virtual space that is associated with the current line-of-sight direction of the crew member in the real space identified as described above.

Further, FIG. 9 shows a position P5 in the virtual space that is associated with the position on the travel route in the real space that is specified based on the position in the image outside the vehicle (for example, the position P1 shown in FIG. 4).

Then, the in-display portion position specifying unit 56 specifies the position P6 of the intersection of the line connecting the position P4 and the position P5 and the virtual HUD object 70. Here, for example, the two-dimensional coordinate value of the position P6 in the plane represented by the plate-shaped virtual HUD object 70 is specified.

Then, the in-display unit position specifying unit 56 specifies the position in the display unit (for example, the position P2 shown in FIG. 6) which is the position of the monitored object in the HUD 10e associated with the position P6 shown in FIG. In the present embodiment, the position in the plane represented by the virtual HUD object 70 and the position in the display unit are preliminarily associated with each other on a one-to-one basis. Therefore, the position in the display unit can be specified based on the two-dimensional coordinate value of the position P6. As described above, the display internal position specifying unit 56 may specify the display internal position based on the position on the travel route.

In the present embodiment, for example, the display control unit 58 causes the mark image 34 to be displayed at the position inside the display unit specified by the position inside display unit specifying unit 56. Here, the display control unit 58 may display the mark image 34 at a position in the display unit of the monitored object that satisfies a predetermined condition. For example, the mark image 34 may be displayed at a position in the display unit of the monitored object that is present in the traveling direction of the vehicle 10 and is located at a distance from the vehicle 10 that is equal to or less than a predetermined distance. Alternatively, the mark image 34 may be displayed at a position in the display unit of the monitored object detected by the monitoring system 12 provided in a station up to a predetermined number of stations in the traveling direction of the vehicle 10. Alternatively, the mark image 34 may be displayed at the position in the display unit of the monitored object detected by the monitoring system 12 provided at the station where the vehicle 10 will next stop or pass.

In the present embodiment, for example, the determination unit 60 determines whether or not the monitoring target has been imaged by the vehicle-mounted imaging device 10d provided in the vehicle 10. Here, for example, it is determined whether or not the monitored object appears in the in-vehicle image.

In the present embodiment, for example, the in-vehicle image position specifying unit 62 specifies the in-vehicle image position, which is the position of the monitoring target in the in-vehicle image captured in the in-vehicle image captured by the in-vehicle image capturing device 10d.

When the determination unit 60 determines that the monitoring target object has been captured by the in-vehicle image capturing device 10d, the display unit internal position identification unit 56 may identify the display unit internal position based on the in-vehicle image internal position. Then, in this case, the display control unit 58 may display the mark image 34 at the position inside the display unit that is specified based on the position inside the vehicle-mounted image.

In the present embodiment, the in-vehicle image position and the display unit position are associated with each other in advance. Therefore, the in-display unit position specifying unit 56 can uniquely specify the in-display unit position based on the in-vehicle image position.

In addition, in the present embodiment, there may be a change from a situation in which it is not determined that the monitoring target object has been photographed by the vehicle-mounted photographing device 10d to a situation in which it is determined. In this case, the display internal position specifying unit 56 may change the display internal position from one based on the vehicle outside image position to one based on the vehicle image inside position.

Hereinafter, an example of the flow of processing that is repeatedly performed at the predetermined frame rate in the vehicle 10 according to the present embodiment will be described with reference to the flowchart illustrated in FIG. 10.

First, the determination unit 60 acquires the vehicle-mounted image captured in the frame (S101).

Then, the travel route position identifying unit 54 identifies one of the detection data stored in the detection data storage unit 52 that satisfies a predetermined condition (S102). Here, for example, based on the identification information included in the detection data, the detection data transmitted from the monitoring system 12 existing in the traveling direction of the vehicle 10 and having a distance from the vehicle 10 equal to or less than a predetermined distance may be specified. .. Alternatively, based on the identification information included in the detection data, the detection data transmitted from the monitoring system 12 provided in the station up to a predetermined number of stations existing in the traveling direction of the vehicle 10 may be specified. Alternatively, the detection data transmitted from the monitoring system 12 provided at the station where the vehicle 10 next stops or passes may be specified based on the identification information included in the detection data.

Then, the travel route on-position specifying unit 54 specifies the travel route on-position corresponding to the in-vehicle image position indicated by the monitored object position data included in the detection data specified in the process of S102 (S103).

Then, the position identifying unit 56 in the display unit identifies the current position and direction of the HUD 10e and the position and line-of-sight direction of the crew member's viewpoint based on the latest operation data (S104).

Then, the in-display position specifying unit 56 determines the position on the travel route specified in the process shown in S103, the current position and orientation of the HUD 10e specified in the process shown in S104, and the position and line-of-sight direction of the crew member's viewpoint. Based on, the position in the display unit is specified (S105).

Then, the determination unit 60 determines whether or not the monitoring target object appears in the range corresponding to the position in the display unit specified in the process shown in S105 in the vehicle-mounted image acquired in the process shown in S101 (S106). .. Here, for example, it is determined whether or not the monitored object appears in a range in which the distance from the position in the vehicle-mounted image corresponding to the position in the display unit specified in the process of S105 is equal to or less than a predetermined distance.

When it is determined that the image is not captured in the process shown in S106 (S106: N), the display control unit 58 displays the mark image 34 at the position in the display unit specified in the process shown in S105 (S107), It returns to the process shown in S101.

If it is determined that the image is captured in the process shown in S106 (S106: Y), the in-vehicle image position specifying unit 62 is in the in-vehicle image of the monitoring target determined to be captured in the process shown in S106. The position is specified (S108).

Then, the display unit internal position specifying unit 56 specifies the display unit internal position corresponding to the vehicle-mounted image internal position specified in the process shown in S108 (S109).

Then, the display control unit 58 causes the mark image 34 to be displayed at the position in the display unit identified in the process of S109 (S110), and returns to the process of S101.

By executing the process shown in FIG. 10, the position inside the display unit is based on the position inside the image outside the vehicle when the in-vehicle image capturing device 10d changes from the state in which it is not determined that the monitored object is captured to the determined state. It changes from the one based on the position in the vehicle-mounted image.

In the process shown in FIG. 10, once it is determined that the monitoring target object appears in the vehicle-mounted image, the processes shown in S102 to S107 are not executed for the monitoring target object, and the vehicle-mounted image is displayed. The internal position of the display unit may be specified based on the internal position.

Further, in the present embodiment, when the position of the monitored object is outside the display range of the HUD 10e, the display control unit 58 marks the edge of the HUD 10e corresponding to the position of the monitored object as shown in FIG. The image 72 may be displayed.

For example, it is assumed that there is no intersection between the line connecting the position P4 and the position P5 shown in FIG. 9 and the virtual HUD object 70. In this case, the position on the side of the virtual HUD object 70 that is closest to the line connecting the position P4 and the position P5 may be specified. Then, the mark image 72 may be displayed on the edge of the HUD 10e associated with the specified position.

Further, in the present embodiment, as shown in FIG. 12, a warning image 74 may be displayed together with the mark image 34 at a place where the visual field of the crew member is not obstructed. Alternatively, the warning image 74 may be displayed instead of the mark image 34.

Further, in the present embodiment, when the vehicle exterior imaging device 12d is a camera capable of capturing a three-dimensional image or a depth image, the monitoring system 12 specifies the three-dimensional coordinate value of the monitoring target in the camera coordinate system. Good. Then, based on the three-dimensional coordinate value, the position of the monitored object in the real space or the position in the display unit may be specified. In this case, the detection notification associated with the detection data including the monitoring target position data indicating the three-dimensional coordinate value may be transmitted.

Further, when the vehicle-mounted photographing device 10d is a camera capable of photographing a three-dimensional image or a depth image, the vehicle 10 may specify the three-dimensional coordinate value of the monitoring target in the camera coordinate system. Then, the position in the display unit may be specified based on the three-dimensional coordinate value.

The division of roles of the vehicle 10 and the monitoring system 12 is not limited to the above. For example, the vehicle exterior image may be transmitted from the monitoring system 12 to the vehicle 10. Then, the vehicle 10 may specify the position inside the vehicle outside image based on the vehicle outside image. Further, the monitoring system 12 may specify the position of the monitored object in the real space, such as the position on the travel route.

In addition, in the present embodiment, when it is detected that a person with a white cane, an auxiliary dog, a person using a wheelchair, a drunken person, etc. are included in the image outside the vehicle, an image representing that is detected by the vehicle. You may make it displayed on 10 HUD10e.

Further, in the present embodiment, it may be determined whether or not there is an obstacle inside the railroad crossing based on an image captured by a camera that monitors the railroad crossing. The determination may be performed based on, for example, the output of the operation sensor of the special light emission signal or the recognition result of the image of the special light emission signal. Then, when it is determined whether or not there is an obstacle inside the railroad crossing, an image indicating this may be displayed on the HUD 10e of the vehicle 10. Here, for example, the mark image may be displayed at the position in the display unit associated with the obstacle by the same method as described above.

The present invention is not limited to the above embodiment.

For example, the method of identifying the position in the display unit is not limited to the above method. For example, image recognition may be performed on the vehicle-mounted image, and the position of the railroad that is not shown in the vehicle-mounted image may be estimated by extrapolation. Then, the position in the display unit on the estimated position of the track may be determined. Alternatively, the position of the monitoring target in the real space or the position in the display unit may be estimated by using a machine learning model in which an image captured by the vehicle-mounted image capturing device 10d in the past traveling of the vehicle 10 is learned. Good.

Further, for example, the display unit arranged in front of the crew of the vehicle 10 may be a head mounted display (HMD) worn by the crew instead of the HUD 10e. Then, the landmark image may be displayed on the HMD. In this case, for example, the position and orientation of the HMD may be specified based on the outputs of various sensors such as an acceleration sensor provided on the HMD. Then, the position in the display unit may be specified based on the specified position and orientation of the HMD.

Further, the specific character strings and numerical values described above and the specific character strings in the drawings are examples, and the present invention is not limited to these character strings and numerical values.

1 display control system, 10 vehicle, 10a processor, 10b storage unit, 10c communication unit, 10d vehicle-mounted image capturing device, 10e head-up display (HUD), 12 monitoring system, 12a processor, 12b storage unit, 12c communication unit, 12d exterior image capturing Device, 12e display, 14 operation management system, 16 computer network, 20 exterior image, 22a safety area, 22b approach area, 22c fall area, 22d track maintenance area, 24 vehicle limit frame model, 26 exterior image, 28 fallen person, 30 Mark image, 32 image, 34 mark image, 40 detection unit, 42 vehicle outside image position specifying unit, 44 detection notification unit, 50 notification receiving unit, 52 detection data storage unit, 54 travel route position specifying unit, 56 display position Specific part, 58 Display control part, 60 Judgment part, 62 In-vehicle image position specific part, 70 Virtual HUD object, 72 Mark image, 74 Warning image.

Claims (8)

An outside-vehicle image position specifying means for specifying an outside-vehicle image position, which is a position in the outside-vehicle image of the monitoring target shown in the outside-vehicle image taken by the outside-vehicle imaging device provided outside the vehicle,
Based on the traveling route data indicating the linear traveling route of the vehicle and the position inside the vehicle image, the position on the traveling route which is the position of the monitoring target on the traveling route indicated by the traveling route data is specified. Position specifying means on the traveling route,
Based on the position on the travel route and the position and orientation of the display unit arranged in front of the crew of the vehicle, the position in the display unit that identifies the position in the display unit that is the position of the monitored object in the display unit. Identification means,
Display control means for displaying a mark image at the specified position in the display section;
A display control system comprising: Determination means for determining whether or not the monitored object is photographed by the vehicle-mounted photographing device provided in the vehicle,
The vehicle-mounted image capturing device further includes a vehicle-mounted image internal position identifying unit that identifies a vehicle-mounted image internal position that is a position in the vehicle-mounted image of the monitoring target captured in the vehicle-mounted image captured by the vehicle-mounted image capturing device,
When it is determined that the monitoring target has been photographed by the vehicle-mounted image capturing device, the display unit internal position identifying means identifies the display unit internal position based on the vehicle-mounted image internal position,
The display control system according to claim 1 , wherein: When the situation in which it is not determined that the monitoring target has been photographed by the vehicle-mounted image capturing device is changed to the determined situation, the display unit internal position specifying unit determines the display unit internal position based on the vehicle outside image internal position. Change from the one based on the position in the in-vehicle image,
The display control system according to claim 2 , wherein: The display control means, when the position of the monitoring target is outside the display range of the display unit, displays a mark image on the edge of the display unit according to the position of the monitoring target,
The display control system according to any one of claims 1 to 3, characterized in that. The display unit is a head-up display provided in the vehicle,
The display control system according to claim 1, any one of 4, characterized in that. The display unit is a head mounted display worn by the crew member,
The display control system according to claim 1, any one of 4, characterized in that. A step of identifying an in-vehicle image position that is a position in the out-of-vehicle image of the monitoring target shown in an out-of-vehicle image captured by an out-of-vehicle image capturing device provided outside the vehicle,
A position on the travel route that is the position of the monitored object on the travel route indicated by the travel route data is specified based on the travel route data indicating the linear travel route of the vehicle and the position outside the vehicle image. Steps to
Based on the position on the traveling route and the position and orientation of the display unit arranged in front of the crew of the vehicle, a step of specifying a position in the display unit that is the position of the monitored object in the display unit,
A step of displaying a mark image at the specified position in the display section;
A display control method comprising: A procedure for identifying the in-vehicle image position, which is the position of the monitored object in the in-vehicle image shown in the in-vehicle image captured by the in-vehicle image capturing device provided outside the vehicle,
A position on the travel route that is the position of the monitored object on the travel route indicated by the travel route data is specified based on the travel route data indicating the linear travel route of the vehicle and the position outside the vehicle image. Steps to
Based on the position on the travel route and the position and orientation of the display unit arranged in front of the crew member of the vehicle, a procedure for specifying a position in the display unit that is the position of the monitored object in the display unit,
A procedure for displaying a landmark image at the specified position in the display unit,
A program characterized by causing a computer to execute.

Images