JP2019073190A - Program and parking support device - Google Patents

Abstract

To provide a technology for executing processing suitable to automatic parking through remote control.SOLUTION: An acquisition section 112 acquires first information on an obstacle detected by a vehicle 200 and second information on an estimated operation of the vehicle 200 relative to the obstacle of the vehicle 200. Relative to an image taken by an imaging section 126 of a terminal device 100, a superposition section 114 superposes mixed reality UI indicating operation in the second information on a position of the obstacle indicated in the first information. A display I/F 130 causes a display device 132 of the terminal device 100 to display an image on which the mixed reality UI is superposed. An operation I/F 136 receives an instruction to designate the operation of the vehicle 200 relative to the mixed reality UI superposed on the image displayed by the display device 132. A communication section 110 transmits the designation of the received operation to a parking support device 300.SELECTED DRAWING: Figure 1

Description

  The present disclosure relates to parking assistance technology, in particular, a program for assisting parking and a parking assistance apparatus.

  A system for remotely operating a vehicle has been proposed as a driving support system that assists the driver's operation when parking the vehicle in a narrow space or the like. In the driving support system up to now, the vehicle operation by the remote control transmission device can be realized, but the button operation is complicated and the movement of the operator itself is necessary. In order to improve the operability in such a driving support system, the vehicle transmits the generated overhead image to the portable remote control device, and the portable remote control device displays the overhead image and onto the overhead image. Accept the input of the route.

JP, 2009-292254, A

  The low obstacles of low height include a loop to be touched, a planter which should not be stepped on, a flap of a parking lot to be overcome, etc., and a route according to the type should be generated. On the other hand, when the type of the low obstacle is automatically detected, the result of the automatic detection may be erroneous, so a function of correcting the result is desired.

  This indication is made in view of such a situation, and the object is to provide art which performs processing suitable for automatic parking by remote control.

  In order to solve the above problems, a program according to an aspect of the present disclosure is a program in a terminal device capable of communicating with a communication device mounted on a vehicle, and first information related to an obstacle detected by the vehicle; In the first information, a mixed reality UI in which an operation in the second information is shown in the step of acquiring the second information on the operation of the vehicle with respect to the obstacle and the image captured by the imaging unit of the terminal device The step of superimposing on the position of the obstacle, the step of displaying the image on which the mixed reality UI is superimposed on the display unit of the terminal device, and the vehicle on the mixed reality UI superimposed on the image displayed on the display unit And causing the computer to execute the steps of: receiving an instruction for specifying the operation of and transmitting the instruction for specifying the received operation to the communication device.

  Another aspect of the present disclosure is a parking assistance device. This device is a parking assistance device mounted on a vehicle to be automatically parked, and for an image captured by an imaging unit of a terminal device, the obstacle is detected at the position of an obstacle detected for the vehicle. The mixed reality UI showing the motion of the vehicle estimated is superimposed, and a video on which the mixed reality UI is superimposed is displayed on the display unit of the terminal device, and is superimposed on the video displayed on the display unit. When the terminal device receives an instruction to change the operation of the vehicle to the mixed reality UI, the communication unit receives information on the operation of the vehicle after the change from the terminal device, and the information received by the communication unit And a parking path generation unit that generates a parking path by reflecting information related to the operation of the vehicle after the change.

  Yet another aspect of the present disclosure is also a program. This program is a program of a communication device mounted on a vehicle to be automatically parked and a server program capable of communicating with a terminal device, receiving the search results searched for in the vehicle from the communication device, and the received search The step of detecting an obstacle from the result, the step of estimating the operation of the vehicle with respect to the detected obstacle, the first information about the detected obstacle, and the second information about the estimated operation of the vehicle are transmitted to the terminal device Have the computer execute the steps.

  In the present disclosure, any combination of the above components, the expression of the present disclosure converted between an apparatus, a system, a method, a program, a recording medium recording the program, a vehicle equipped with the present apparatus, etc. It is effective as an aspect.

  According to the present disclosure, processing suitable for automatic parking by remote control can be performed.

FIG. 1 is a diagram showing a configuration of a parking assistance system according to a first embodiment. It is a figure which shows the outline | summary of the process in the projection conversion part of FIG. It is a figure which shows the image | video displayed on the display apparatus of FIG. 4a-4b are diagrams showing another image displayed on the display device of FIG. 5a to 5m are diagrams showing a mixed reality UI displayed on the display device of FIG. 1 and operated by the operation unit. It is a sequence diagram which shows the parking assistance procedure in the parking assistance system of FIG. It is a flowchart which shows the transmission procedure in the terminal device of FIG. It is a flowchart which shows another transmission procedure in the terminal device of FIG. It is a flowchart which shows the obstruction condition superimposed display procedure in the terminal device of FIG. It is a flowchart which shows another obstruction condition superimposed display procedure in the terminal device of FIG. It is a flowchart which shows another obstruction condition superimposed display procedure in the terminal device of FIG. It is a figure which shows the structure of the parking assistance system which concerns on this Embodiment 2. FIG. It is a sequence diagram which shows the parking assistance procedure in the parking assistance system of FIG. It is a figure which shows the structure of the parking assistance system which concerns on this Embodiment 3. As shown in FIG. It is a sequence diagram which shows the parking assistance procedure in the parking assistance system of FIG.

Embodiment 1
Before specifically describing the present embodiment, an outline will be described. The present embodiment relates to a parking assistance system for assisting parking of a vehicle. Here, it is assumed that the vehicle equipped with the parking assist device executes automatic parking under a situation where a driver who possesses a terminal device such as a mobile telephone device or a smartphone exists outside the vehicle. The vehicle is provided with an imaging unit such as a camera and a sonar, and an obstacle is searched for by these. The parking assistance apparatus generates a parking path while reflecting a search result of an obstacle, and moves the vehicle along the parking path. Under such a parking situation, a loop stop, a planter, a flap of a parking lot, etc. can be detected as a low obstacle about 10 cm high (hereinafter referred to as "low obstacle"). If a clasp is detected, the parking assistance device should generate a parking path so that the clasp is in contact with the vehicle's tire. Also, if a planter is detected, the parking assistance device should generate a parking path that avoids stepping on the planter. In addition, if a flap is detected, the parking assistance device should generate a parking path over the flap. That is, the parking assistance device must change the operation of the vehicle according to the type of low obstacle, and the identification of the type of low obstacle becomes important. On the other hand, false detection may occur in the imaging unit and the sonar.

  In order to accurately identify the type of low obstacle under such circumstances, the parking assistance system according to the present embodiment performs the following process. Low obstacles are searched for by the imaging unit of the vehicle (hereinafter referred to as “in-vehicle camera”) and sonar. The terminal device receives the search result, detects a low obstacle from the search result, and specifies the operation of the vehicle according to the type of the obstacle. The terminal device superimposes a UI (User Interface) in which the operation of the specified vehicle is shown on the captured image on the position of the low obstacle and displays the superimposed image on the display device. By this display, the driver can recognize the operation of the vehicle with respect to the low obstacle.

  When the operation of the vehicle is erroneous, the driver touches the displayed UI to display the operation options of the vehicle, and selects the correct operation of the vehicle by flick operation. On the other hand, when the UI is not displayed despite the presence of the low obstacle, the low obstacle is not detected, so that the driver designates the displayed low obstacle to detect the reference information. To improve detection performance. The vehicle generates a parking path while also reflecting the modified vehicle behavior. Hereinafter, the present embodiment will be described in detail with reference to the drawings. In addition, each embodiment described below is an example, and this indication is not limited by these embodiments.

  FIG. 1 shows the configuration of a parking assistance system 1000 according to the first embodiment. The parking assistance system 1000 includes a terminal device 100 and a vehicle 200. The terminal device 100 includes a positioning device 116, a digital compass 118, an imaging unit 126, a display device 132, an operation unit 134, and a processing unit 160. The processing unit 160 includes a communication unit 110, an acquisition unit 112, an overlapping unit 114, a position / orientation detection unit 120, an imaging I / F 128, a display I / F 130, an operation I / F 136, a setting unit 138, a control unit 150, an obstacle detection. And a motion estimation unit 502 and a projection conversion unit 504. The position / orientation detection unit 120 includes a positioning I / F 122 and a compass I / F 124.

  The vehicle 200 includes a movement control device 210, a parking assistance device 300, a positioning device 310, a digital compass 312, an imaging unit 320, and a sonar 322. The parking assistance device 300 includes a position / orientation detection unit 314, a search unit 324, a communication unit 330, a parking path generation unit 332, and a control unit 350. The position / orientation detection unit 314 includes a positioning I / F 316 and a compass I / F 318, and the search unit 324 includes an imaging I / F 326 and a sonar I / F 328. Here, the communication unit 110 of the terminal device 100 and the communication unit 330 of the parking assistance device 300 are connected by a wireless communication system. Note that these may be directly connected or may be connected via a base station apparatus.

  The driver is present outside the vehicle 200 while holding the terminal device 100. In such a state, when the vehicle 200 is to be parked automatically, the driver inputs an instruction to execute the automatic parking to the operation unit 134 of the terminal device 100. The operation unit 134 can receive an operation from the driver, and, for example, configures a touch panel in combination with the display device 132. The operation unit 134 receives an instruction for performing automatic parking as an operation from the driver. The operation unit 134 outputs the received operation to the operation I / F 136. When the operation received from the operation unit 134 is an instruction to execute automatic parking, the operation I / F 136 notifies the control unit 150 of an instruction to execute automatic parking and outputs the instruction to the communication unit 110 via the setting unit 138. When the control unit 150 receives an instruction to execute automatic parking from the operation I / F 136, the control unit 150 starts execution of automatic parking processing. When communication unit 110 receives an instruction to execute automatic parking from operation I / F 136, communication unit 110 transmits a signal including an instruction to execute automatic parking to parking assist device 300.

  The communication unit 330 of the parking assistance device 300 receives a signal from the terminal device 100. The communication unit 330 outputs an instruction for performing automatic parking included in the received signal to the control unit 350. When the control unit 350 receives an instruction for executing automatic parking from the communication unit 330, the control unit 350 starts execution of automatic parking processing. The imaging unit 320 is a camera mounted on the vehicle 200. A plurality of imaging units 320 may be mounted on the vehicle 200. For example, four imaging units 320 may be mounted on the front, rear, left, and right of the vehicle 200. The imaging unit 320 captures an image of the outside of the vehicle 200. Here, the video is configured by arranging a plurality of images in time series. The imaging unit 320 outputs the captured video to the imaging I / F 326.

  The sonar 322 is a detection device mounted on the vehicle 200. A plurality of sonars 322 may be mounted on the vehicle 200. For example, four sonars 322 are mounted on the front and the rear of the vehicle 200, and one sonar 322 is mounted on the front left, front right, rear left, and rear right of the vehicle 200. Twelve sonars 322 may be mounted. The sonar 322 is a device for searching for an object such as a low obstacle by sound waves, and can measure the distance to the object. The sonar 322 outputs the search result to the sonar I / F 328.

  The imaging I / F 326 receives an image from the imaging unit 320, and the sonar I / F 328 receives a search result from the sonar 322. Since it is possible to search for a low obstacle also by the image captured by the imaging unit 320, receiving the image is also equivalent to receiving the search result. Therefore, it can be said that the search unit 324 including the imaging I / F 326 and the sonar I / F 328 searches the outside of the vehicle 200. The search unit 324 outputs the video and the search results (hereinafter, these are collectively referred to as “search results”) to the communication unit 330.

  The positioning device 310 measures the position of the vehicle 200 based on a signal received from a satellite of Global Navigation Satellite System (GNSS). The position of the vehicle 200 is indicated by, for example, latitude, longitude, and altitude. The positioning device 310 outputs information on the measured position (hereinafter referred to as “position information”) to the positioning I / F 316. The digital compass 118 detects geomagnetism by an electronic sensor to determine the direction. Here, the direction in which the front or rear of the vehicle 200 is directed is determined. This direction is called the direction of the vehicle 200. The digital compass 118 outputs information on the determined azimuth (hereinafter referred to as “azimuth information”) to the compass I / F 318.

  Positioning I / F 316 receives position information from positioning device 310, and compass I / F 318 receives azimuth information from digital compass 312. Therefore, it can be said that the position / orientation detection unit 314 including the positioning I / F 316 and the compass I / F 318 detects the position information of the vehicle 200 and the azimuth information of the vehicle 200. The position / orientation detection unit 314 outputs the position information of the vehicle 200 and the direction information of the vehicle 200 to the communication unit 330.

  When communication unit 330 receives the search result from search unit 324, communication unit 330 transmits a signal including the search result to terminal apparatus 100. When communication unit 330 receives the position information of vehicle 200 and the direction information of vehicle 200 from position / direction detection unit 314, the terminal device is a signal including the position information of vehicle 200 and the direction information of vehicle 200. Send to 100 These processes in the parking assistance device 300, the positioning device 310, the digital compass 312, the imaging unit 320, and the sonar 322 are repeatedly executed.

  The communication unit 110 of the terminal device 100 receives a signal from the parking assistance device 300. The signal includes the position information of the vehicle 200, the azimuth information of the vehicle 200, and the search result. The communication unit 110 outputs the search result to the obstacle detection unit 500, and also outputs the image of the search result, the position information of the vehicle 200, and the direction information of the vehicle 200 to the projection conversion unit 504. The positioning device 116 is similar to the positioning device 310, and the digital compass 118 is similar to the digital compass 312. Here, the positioning device 116 measures the position of the terminal device 100, and the digital compass 118 determines the direction in which the imaging unit 126 faces.

  Positioning I / F 122 is similar to positioning I / F 316, and compass I / F 124 is similar to compass I / F 318. The positioning I / F 122 receives position information from the positioning device 116, and the compass I / F 124 receives direction information from the digital compass 118. Therefore, it can be said that the position / orientation detection unit 120 including the positioning I / F 122 and the compass I / F 124 detects the position information of the terminal device 100 and the azimuth information of the imaging unit 126. The position / orientation detection unit 120 outputs the position information of the terminal device 100 and the azimuth information of the imaging unit 126 to the projection conversion unit 504 and the superposition unit 114.

  The projection conversion unit 504 receives the position information of the vehicle 200, the direction information of the vehicle 200, the position information of the terminal device 100, and the direction information of the imaging unit 126, and an image captured by the imaging unit 320 of the vehicle 200. (Hereinafter referred to as "the image of the vehicle 200"). Here, FIG. 2 is used to explain the processing of the projection conversion unit 504. FIG. 2 shows an outline of processing in the projection conversion unit 504. This is a top view of the vehicle 200 as viewed from above. Behind the vehicle 200, a first other vehicle 220a and a second other vehicle 220b collectively referred to as another vehicle 220 are parked in parallel, and a parking area 230 is disposed between them. Here, it is assumed that the vehicle 200 is parked in the parking area 230. Further, a driver 50 exists outside the vehicle 200, and the driver 50 carries the terminal device 100. Here, the terminal position 180 indicates the position information of the terminal device 100, the terminal direction 182 indicates the direction information of the imaging unit 126, the vehicle position 280 indicates the position information of the vehicle 200, and the vehicle direction 282 indicates the vehicle Indicates 200 azimuth information.

  The projection conversion unit 504 generates the space model 600 in a coordinate system based on the vehicle 200. The space model 600 is a model composed of a curved surface that maps the image of the vehicle 200 on a three-dimensional space coordinate system. As illustrated, the space model 600 is a curved surface having a cylindrical shape, and the image of the vehicle 200 is mapped on the inner surface thereof. Here, the position of each pixel constituting the image included in the image of the vehicle 200 is indicated as a coordinate on the UV plane including the imaging unit 320. The projection transformation unit 504 transforms the pixel indicated as the coordinates of the U-V coordinate system into the coordinates in the world coordinate system. Further, the projection conversion unit 504 calculates a projection point of the pixel onto the space model 600 from the vehicle position 280, the vehicle direction 282, and the world coordinate of the pixel. Furthermore, the projection conversion unit 504 maps the color data of the pixel to the projection point of the space model 600.

  The projection conversion unit 504 converts the video mapped to the space model 600 into a video projected from the terminal position 180 and the terminal orientation 182. In the following, the image thus project-transformed may be referred to as a “projected-transformed image”. In other words, based on the position information and orientation information of the vehicle 200, the position information of the terminal device 100, and the orientation information of the imaging unit 126, the projection conversion unit 504 displays the image of the vehicle 200 at the viewpoint of the imaging unit 126. Projective transformation. Return to FIG. The projection conversion unit 504 outputs, to the superimposing unit 114, the correspondence between the projection conversion video and the coordinates of the video of the vehicle 200 and the projection conversion video.

  The obstacle detection unit 500 receives the search result from the communication unit 110. The obstacle detection unit 500 detects an obstacle from the search result. Here, among the obstacles, a low obstacle will be described as a processing target. The detection of the low obstacle in the obstacle detection unit 500 includes detecting the position of the low obstacle and the shape of the low obstacle. In addition, the position of the low obstacle is indicated as coordinates on the image of the vehicle 200, and also indicated by latitude, longitude, and altitude. A known technique may be used for the processing in the obstacle detection unit 500, so the description is omitted here. The obstacle detection unit 500 outputs information on the detected position of the low obstacle (hereinafter referred to as "first information") to the acquisition unit 112, and outputs information on the shape of the detected low obstacle to the motion estimation unit 502. Do.

  The motion estimation unit 502 receives from the obstacle detection unit 500 information on the shape of the low obstacle. The motion estimation unit 502 stores in advance a table indicating the correspondence between the shape of the low obstacle and the motion of the vehicle 200. In the table, for example, the shape of the “ring fixed” is associated with the “contact with a low obstacle” operation, the shape of the “planter” is associated with the “avoid with a low obstacle” operation, “flap” The shape of "" and the action of "getting over low obstacles" are associated. The contents of the table are not limited to these. The motion estimation unit 502 selects a shape closest to the shape of the received low obstacle from the plurality of types of shapes included in the table. Known techniques may be used for the selection. In addition, the motion estimation unit 502 specifies a motion corresponding to the selected shape from the table. This corresponds to estimating the motion of the vehicle 200 with respect to the detected low obstacle. The motion estimation unit 502 outputs information related to the motion of the vehicle 200 (hereinafter, referred to as “second information”) to the acquisition unit 112.

  The acquisition unit 112 receives the first information from the obstacle detection unit 500 and receives the second information from the motion estimation unit 502. The second information indicates one of the following as the operation of the vehicle 200: contact with a low obstacle, avoiding a low obstacle, or overcoming a low obstacle. The acquisition unit 112 outputs a combination of the first information and the second information to the superimposing unit 114 and the setting unit 138 as obstacle information. The imaging unit 126 is similar to the imaging unit 320, and the imaging I / F 128 is similar to the imaging I / F 326. The imaging I / F 128 outputs an image captured by the imaging unit 126 (hereinafter referred to as “image of the terminal device 100”) to the superimposing unit 114.

  The superposing unit 114 receives obstacle information from the acquiring unit 112, receives an image of the terminal device 100 from the imaging I / F 128, and receives a projection conversion image from the projection conversion unit 504. Also, the superimposing unit 114 receives, from the projection conversion unit 504, the correspondence relationship between the image of the vehicle 200 and the coordinates of the projection conversion image. The superimposing unit 114 performs coordinate conversion of the position indicated by the first information in the obstacle information using the correspondence relationship. The coordinate-transformed position indicates the position of the low obstacle in the image of the terminal device 100. Also, the superimposing unit 114 extracts an area near the position of the low obstacle from the projection conversion video. The superimposing unit 114 superimposes the extracted projection conversion video (hereinafter, referred to as “extracted projection conversion video”) on the video from the terminal device 100 while aligning the positions. The superimposing unit 114 superimposes an icon indicating the operation of the vehicle 200 (hereinafter, referred to as “mixed reality UI”) on the position of the low obstacle in the video on which the extraction projection conversion video is superimposed. The superimposing unit 114 outputs a video (hereinafter, referred to as “superimposed video”) in which the extraction projection conversion video and the mixed reality UI are superimposed to the display I / F 130. The display I / F 130 causes the display device 132 to display the superimposed video. The display device 132 displays the superimposed video.

  FIG. 3 shows an image displayed on the display device 132. This shows the situation shown in FIG. A superimposed image is shown on the display device 132, and the superimposed image includes the first other vehicle 220a and the second other vehicle 220b. In addition, the extraction projection conversion image 622 is disposed between the first other vehicle 220a and the second other vehicle 220b. The extracted projection conversion image 622 includes a looped hook 630, and the mixed reality UI 620 is arranged to point to the looped hook 630. In the mixed reality UI 620, a text message of "Appr." Is shown, which means "approach with low obstacle".

  4a-4b illustrate another image displayed on display 132. FIG. In FIG. 3, since the extraction projection conversion image 622 is superimposed on the first other vehicle 220a, a part of the first other vehicle 220a is missing. In order to prevent this chipping, when the obstacle detection unit 500 detects an obstacle such as the other vehicle 220, the superimposing unit 114 performs mask processing on the extraction projection conversion image 622 to extract on the obstacle. The projection conversion image 622 may not be superimposed. FIG. 4 shows a display that has been subjected to such processing. As illustrated, part of the first other vehicle 220a is not missing.

  The obstacle detection unit 500 of FIG. 1 may measure the distance from the vehicle 200 to the low obstacle. A known technique may be used to measure the distance, so the description is omitted here. The obstacle detection unit 500 also includes information on the measured distance (hereinafter, referred to as “distance information”) in the first information and outputs the information to the acquisition unit 112. Therefore, the first information acquired by the acquisition unit 112 also includes distance information from the vehicle 200 to the low obstacle. The superposition unit 114 also superimposes the distance information included in the first information. FIG. 4b shows the display in which such processing has been made. Distance information 624 is shown above the mixed reality UI 620. Return to FIG.

  When the driver sees the mixed reality UI 620 of the display device 132 as shown in FIGS. 3 and 4a and 4b and determines that the operation of the vehicle 200 is correct, the OK button shown in FIGS. 3 and 4a and 4b Touch 626. The operation unit 134 of the terminal device 100 receives an OK instruction when the OK button 626 is touched. The operation unit 134 outputs the received OK instruction to the operation I / F 136. The operation I / F 136 outputs an OK instruction to the setting unit 138. When the setting unit 138 receives the OK instruction, the setting unit 138 outputs the obstacle information received from the acquisition unit 112 to the communication unit 110.

  On the other hand, when the driver sees the mixed reality UI 620 of the display device 132 and determines that the operation of the vehicle 200 is not correct, the driver performs the following operation. 5a-5m show the mixed reality UI 620 displayed on the display device 132 and operated by the operation unit 134. FIG. FIG. 5 a shows the case where the text message “Appr.” Is shown in the mixed reality UI 620 for the planter 632. Since the operation of the vehicle 200 with respect to the planter 632 is "avoid low obstacles", "Guard" must be indicated as a text message.

  At this time, the driver touches the planter 632 or the mixed reality UI 620. The operation unit 134 of the terminal device 100 receives an instruction to display a plurality of actions when the planter 632 or the mixed reality UI 620 is touched. The operation I / F 136 outputs an instruction to display a plurality of actions to the setting unit 138. When the setting unit 138 receives an instruction to display a plurality of actions, the setting unit 138 instructs the superimposing unit 114 via the acquiring unit 112 to display the plurality of actions in the mixed reality UI 620. In response to this, the superimposing unit 114, the display I / F 130, and the display device 132 cause the mixed reality UI 620 to display the text messages of "Appr.", "Guard", and "Over" as illustrated in FIG. 5B. In addition, the text message of "Over" means "get over a low obstacle".

  The driver selects "Guard" by flicking operation as shown in FIG. 5c. As a result, as shown in FIG. 5 d, in mixed reality UI 620 for planter 632, a text message of “Guard” is shown. The operation unit 134 of the terminal device 100 receives an instruction for changing the operation of the vehicle 200 (hereinafter, referred to as a “change instruction”) by such an operation. The change instruction indicates a change from "Appr." To "Guard". The operation unit 134 outputs the received change instruction to the operation I / F 136. The operation I / F 136 outputs the change instruction to the setting unit 138. When the setting unit 138 receives the change instruction, the setting unit 138 changes the obstacle information received from the acquisition unit 112 according to the change instruction.

  FIG. 5 e shows the case where the text message “Appr.” Is shown in the mixed reality UI 620 for the flap 634. Essentially, the operation of the vehicle 200 with respect to the flap 634 is “over low obstacle”, so “Over” must be indicated as a text message. Therefore, when the driver performs the same operation as before, the state transitions to FIGS. 5 f, 5 g, and 5 h. In this case, the change instruction indicates a change from "Appr." To "Over".

  The above description assumes that the obstacle detection unit 500 detects a low obstacle. On the other hand, there may be a case where the obstacle detection unit 500 does not detect a low obstacle. As shown in FIG. 5i, the display device 132 displays a looping 630, but the mixed reality UI 620 for the looping 630 is not displayed. At the same time, the driver touches the snap ring 630. The operation unit 134 of the terminal device 100 receives the redetection instruction by touching the loop fastener 630. The instruction for redetection can be said to be an instruction for the mixed reality UI 620 to select a non-overlapping portion in the image. The operation I / F 136 outputs a redetection instruction to the setting unit 138. When the setting unit 138 receives the redetection instruction, the setting unit 138 instructs the obstacle detection unit 500 to perform redetection.

  When the obstacle detection unit 500 receives the instruction, the obstacle detection unit 500 executes the low obstacle detection process targeting only a certain range around the looping 630. At this time, the obstacle detection unit 500 changes detection parameters so that detection is facilitated. When the clasp 630 is detected in the obstacle detection unit 500, the motion estimation unit 502, the acquisition unit 112, the superposition unit 114, the display I / F 130, and the display device 132 execute the same processing as described above. That is, the acquisition unit 112 acquires new first information and new second information for the portion indicated in the received instruction. As a result, as shown in FIG. 5j, the mixed reality UI 620 for the loop clip 630 is displayed on the display device 132.

  5k to 5m are also processing for the same situation as FIG. 5i to FIG. 5j, but specifying the area to be re-detected by the obstacle detection unit 500 by swipe operation as shown in FIG. Is different. At this time, a rectangular area whose line segment designated by the swipe operation is a diagonal is taken as an area to be re-detected. Return to FIG.

  The communication unit 110 of the terminal device 100 receives obstacle information from the setting unit 138. The obstacle information also includes information on the operation of the vehicle 200 after the change. The communication unit 110 transmits a signal including obstacle information to the parking assistance device 300. When the communication unit 330 of the parking assistance device 300 receives a signal from the terminal device 100, the communication unit 330 outputs the obstacle information included in the signal to the parking path generation unit 332. The parking path generation unit 332 receives obstacle information from the communication unit 110, receives position information of the vehicle 200 from the position / orientation detection unit 314, and direction information of the vehicle 200, and receives a search result from the search unit 324. The parking path generation unit 332 generates a parking path by reflecting these pieces of information. A known technique may be used to generate the parking path, so the description is omitted here. The parking path generation unit 332 outputs the parking path to the movement control device 210. The movement control device 210 moves the vehicle 200 along the parking path.

  The operation of the parking assistance system 1000 having the above configuration will be described. FIG. 6 is a sequence diagram showing a parking assistance procedure in the parking assistance system 1000. The parking assistance device 300 captures an image (S10), and the terminal device 100 also captures an image (S12). The parking assistance device 300 transmits the search result to the terminal device 100 (S14). The parking assistance device 300 detects the position and orientation (S16), and the terminal device 100 also detects the position and orientation (S18). The parking assistance device 300 transmits the position information and the direction information to the terminal device 100 (S20). The terminal device 100 detects the low obstacle and estimates the operation of the vehicle 200 (S22). The terminal device 100 performs projection conversion on the image of the vehicle 200 (S24), superimposes the projection conversion image and the mixed reality UI 620 on the image of the terminal device 100, and displays the superimposed image (S26). The terminal device 100 receives an instruction to change the operation of the vehicle 200 (S28). The terminal device 100 outputs the obstacle information to the parking assistance device 300 (S30). The parking assistance device 300 generates a parking path (S32).

  FIG. 7 is a flowchart showing a transmission procedure in the terminal device 100. The communication unit 110 receives the position information of the vehicle 200 and the direction information of the vehicle 200 (S50). The position / orientation detection unit 120 detects the position information of the terminal device 100 and the azimuth information of the terminal device 100 (S52). The obstacle detection unit 500 detects a low obstacle, and the motion estimation unit 502 estimates the motion of the vehicle 200 (S54). The superposing unit 114 superimposes and displays the state of the low obstacle (S56). If there is no operation by the driver (N at S58), the process returns to step 50. If there is an operation by the driver (Y at S58), if the operation is not determined (N at S60), the setting unit 138 corrects the low obstacle determination (S62), and returns to step 50. If the operation is determined (Y at S60). The communication unit 110 transmits obstacle information (S64).

  FIG. 8 is a flowchart showing another transmission procedure in the terminal device 100. The communication unit 110 receives the position information of the vehicle 200 and the direction information of the vehicle 200 (S100). The position / orientation detection unit 120 detects the position information of the terminal device 100 and the azimuth information of the terminal device 100 (S102). The obstacle detection unit 500 detects a low obstacle, and the motion estimation unit 502 estimates the motion of the vehicle 200 (S104). The superposing unit 114 superimposes and displays the condition of the low obstacle (S106). If there is no operation by the driver (N in S108), the process returns to step 100. If there is an operation by the driver (Y in S108), the operation is not determined (N in S110), and if the operation is a mixed reality UI 620 (Y in S112), the setting unit 138 corrects the low obstacle determination Then (S114), the process returns to step 100. If the operation is not the operation of the mixed reality UI 620 (N in S112), the obstacle detection unit 500 executes low obstacle detection manually (S116), and returns to step S106. If the operation is determined (Y in S110). The communication unit 110 transmits obstacle information (S118).

  FIG. 9 is a flowchart showing the low obstacle condition superimposed display procedure in the terminal device 100. This corresponds to the processing of step 56 of FIG. 7 and step 106 of FIG. The projection conversion unit 504 projects and converts the image of the vehicle 200 into an image from the viewpoint of the terminal device 100 based on the position and orientation of the vehicle 200 and the position and orientation of the terminal device 100 (S150). The superimposing unit 114 superimposes the extracted projection conversion video on a part of the video of the terminal device 100 (S152). The superposing unit 114 determines the coordinates on the image of the low obstacle shown in the image of the terminal device 100 based on the determined position of the low obstacle, the position and orientation of the vehicle 200, and the position and orientation of the terminal device 100. Calculate (S154). The superimposing unit 114 generates a mixed reality UI 620 based on the determination (S156). The superimposing unit 114 superimposes the mixed reality UI 620 on the coordinates (S158).

  FIG. 10 is a flowchart showing another low obstacle situation superimposed display procedure in the terminal device 100. This also corresponds to the processing of step 56 of FIG. 7 and step 106 of FIG. The projection conversion unit 504 projects and converts the image of the vehicle 200 into an image from the viewpoint of the terminal device 100 based on the position and orientation of the vehicle 200 and the position and orientation of the terminal device 100 (S200). The superimposing unit 114 performs mask processing on the extraction projection conversion image on the portion hidden by the obstacle based on the arrangement of the obstacle in the three-dimensional space around the vehicle 200 (S202). The superposing unit 114 superimposes the extracted projection conversion video on a part of the video of the terminal device 100 (S204). The superposing unit 114 determines the coordinates on the image of the low obstacle shown in the image of the terminal device 100 based on the determined position of the low obstacle, the position and orientation of the vehicle 200, and the position and orientation of the terminal device 100. Calculate (S206). The superimposing unit 114 generates a mixed reality UI 620 based on the determination (S208). The superimposing unit 114 superimposes the mixed reality UI 620 on the coordinates (S210).

  FIG. 11 is a flowchart showing yet another low obstacle situation superimposed display procedure in the terminal device 100. This also corresponds to the processing of step 56 of FIG. 7 and step 106 of FIG. Based on the determined position of the low obstacle, the position and orientation of the vehicle 200, and the position and orientation of the terminal device 100, the projection conversion unit 504 is on the image of the low obstacle identified in the image of the terminal device 100. Coordinates at are calculated (S250). The superimposing unit 114 generates a mixed reality UI 620 based on the determination (S 252). The superimposing unit 114 superimposes the mixed reality UI 620 on the coordinates (S254). The obstacle detection unit 500 calculates the distance between the determined low obstacle and the rear end of the vehicle 200 based on the determined position of the low obstacle and the position and direction of the vehicle 200 (S256). The obstacle detection unit 500 generates distance information based on the distance (S258). The superimposing unit 114 superimposes the distance information on the coordinates of the image (S260).

  According to the present embodiment, the undetected / false detection of object detection in automatic parking by remote control is corrected by displaying the mixed reality on the display device of the terminal device, and operability can be improved. In addition, despite the fact that detection is difficult due to detection of low height looped / low obstacle / step differences, the parking path is different depending on the judgment, and a predetermined option is selected by flick operation, so the correct selection is performed. it can. In addition, since the false detection of the low-level looped / low obstacle / step difference which is difficult to detect by detection is corrected by the flick operation, it is possible to quickly correct it by a simple operation.

  In addition, the mixed reality UI superimposed on the displayed image receives an instruction to change the operation of the vehicle, and transmits the received change instruction to the parking assistance apparatus, so that the detection result of the type of low obstacle is You can correct In addition, the estimated operation is suitable for the low obstacle because one of the low obstacle is contacted, the low obstacle is avoided, and the low obstacle is overcome. Can be corrected to Further, since the distance information is also superimposed on the image, the driver can be notified of the distance from the vehicle to the low obstacle.

  Further, since the motion of the vehicle with respect to the detected low obstacle is estimated after the low obstacle is detected, the motion with respect to the low obstacle can be estimated automatically. In addition, when the mixed reality UI receives an instruction to select a non-overlapping portion, the detection process and the estimation process are re-executed on the portion, so that the low obstacle which has not been detected can be detected. In addition, when re-executing the detection process and the estimation process, parameters that facilitate detection are set, so it is possible to easily detect an undetected low obstacle. In addition, since the detection process and the estimation process are re-executed to acquire new first information and new second information, it is possible to correct the detection result of the undetected low obstacle. In addition, since the image of the vehicle is projected and converted, and then superimposed on the image of the terminal device, it is possible to provide an easy-to-view image for the driver.

  Further, when the information on the operation of the vehicle after the change is received from the terminal device, the information on the operation of the vehicle after the change is reflected to generate the parking path, so that it is possible to appropriately cope with the low obstacle. In addition, any one of the low obstacle touching, avoiding the low obstacle, and overcoming the low obstacle is indicated, so that it is possible to generate a parking path reflecting the operation suitable for the low obstacle. In addition, the position information of the vehicle and the direction information of the vehicle are transmitted, and based on these, the image captured by the imaging unit of the vehicle is projected and converted, so that the processing amount of the parking assistance device can be reduced. Further, the search result is transmitted, and the low obstacle is detected from the search result, and the operation of the vehicle with respect to the low obstacle is estimated, so that the processing amount of the parking assistance device can be reduced.

Second Embodiment
Next, the second embodiment will be described. Similar to Embodiment 1, Embodiment 2 relates to a parking assistance system for assisting parking of a vehicle, and includes a parking assistance device and a terminal device. In the first embodiment, the obstacle detection unit, the motion estimation unit, and the projection conversion unit are included in the terminal device, but in the second embodiment, these are included in the parking assistance device. Here, we will focus on the differences from the previous ones.

  FIG. 12 shows the configuration of a parking assistance system 1000 according to the second embodiment. The parking assistance system 1000 includes a terminal device 100 and a vehicle 200. The terminal device 100 includes a positioning device 116, a digital compass 118, an imaging unit 126, a display device 132, an operation unit 134, and a processing unit 160. The processing unit 160 includes a communication unit 110, an acquisition unit 112, a superposition unit 114, a position / orientation detection unit 120, an imaging I / F 128, a display I / F 130, an operation I / F 136, a setting unit 138, and a control unit 150. The position / orientation detection unit 120 includes a positioning I / F 122 and a compass I / F 124.

  The vehicle 200 includes a movement control device 210, a parking assistance device 300, a positioning device 310, a digital compass 312, an imaging unit 320, and a sonar 322. The parking assistance device 300 includes a position / orientation detection unit 314, a search unit 324, a communication unit 330, a parking path generation unit 332, a control unit 350, an obstacle detection unit 500, an operation estimation unit 502, and a projection conversion unit 504. The position / orientation detection unit 314 includes a positioning I / F 316 and a compass I / F 318, and the search unit 324 includes an imaging I / F 326 and a sonar I / F 328.

  The position / orientation detection unit 120 including the positioning I / F 122 and the compass I / F 124 detects the position information of the terminal device 100 and the azimuth information of the imaging unit 126 as in the above. The position / orientation detection unit 120 outputs the position information of the terminal device 100 and the azimuth information of the imaging unit 126 to the communication unit 110 and the superposition unit 114. The communication unit 110 receives from the position / orientation detection unit 120 the position information of the terminal device 100 and the azimuth information of the imaging unit 126. The communication unit 110 transmits a signal including the position information of the terminal device 100 and the azimuth information of the imaging unit 126 to the parking assistance device 300.

  The communication unit 330 of the parking assistance device 300 receives a signal from the terminal device 100. The communication unit 330 outputs the position information of the terminal apparatus 100 included in the signal and the azimuth information of the imaging unit 126 to the projection conversion unit 504. The search unit 324 including the imaging I / F 326 and the sonar I / F 328 searches the outside of the vehicle 200 as before. The search unit 324 outputs the search result to the obstacle detection unit 500 and the projection conversion unit 504. The position / orientation detection unit 314 including the positioning I / F 316 and the compass I / F 318 detects the position information of the vehicle 200 and the azimuth information of the vehicle 200 as described above. The position / orientation detection unit 314 outputs the position information of the vehicle 200 and the direction information of the vehicle 200 to the projection conversion unit 504, the communication unit 330, and the parking path generation unit 332.

  The projection conversion unit 504 receives the position information of the vehicle 200 and the orientation information of the vehicle 200 from the position / orientation detection unit 314, and the communication unit 330 transmits the position information of the terminal device 100 and the orientation information of the imaging unit 126. Accept and accept the search result from the search unit 324. Based on the position information of the vehicle 200, the direction information of the vehicle 200, the position information of the terminal device 100, and the direction information of the imaging unit 126, the projection conversion unit 504 sets the vehicle 200 to the viewpoint of the imaging unit 126. Project and convert the image. The projection conversion unit 504 outputs, to the communication unit 330, the correspondence relationship between the projection conversion video and the coordinates of the video of the vehicle 200 and the projection conversion video. Communication unit 330 transmits a signal including these to terminal device 100.

  The obstacle detection unit 500 receives the search result from the search unit 324. The obstacle detection unit 500 detects a low obstacle from the search result. The obstacle detection unit 500 outputs the first information on the detected position of the low obstacle to the communication unit 330, and outputs the information on the detected shape of the low obstacle to the motion estimation unit 502. The motion estimation unit 502 receives from the obstacle detection unit 500 information on the shape of the low obstacle. The motion estimation unit 502 estimates the motion of the vehicle 200 with respect to the detected low obstacle. The motion estimation unit 502 outputs the second information on the motion of the vehicle 200 to the communication unit 330. The communication unit 330 sets the combination of the first information and the second information as obstacle information, and transmits a signal including the obstacle information to the terminal device 100.

  The communication unit 110 of the terminal device 100 receives a signal from the parking assistance device 300. The acquisition unit 112 acquires obstacle information from the communication unit 110. The superposing unit 114 receives obstacle information from the acquisition unit 112, receives an image of the terminal device 100 from the imaging I / F 128, and receives a projection conversion image from the communication unit 110. Also, the superimposing unit 114 receives, from the communication unit 110, the correspondence between the coordinates of the image of the vehicle 200 and the projection conversion image. The superimposing unit 114 uses these to generate a superimposed image in which the extraction projection conversion video and the mixed reality UI 620 are superimposed on the video of the terminal device 100. The superimposing unit 114 outputs the superimposed image to the display I / F 130. The display I / F 130 causes the display device 132 to display the superimposed video. The display device 132 displays the superimposed video.

  When the obstacle detection unit 500 does not detect a low obstacle, the driver executes the same processing as before. The operation unit 134 of the terminal device 100 receives the redetection instruction, and outputs the redetection instruction to the communication unit 110 via the setting unit 138. Communication unit 110 transmits a signal including a redetection instruction to parking assistance device 300. The communication unit 330 of the parking assistance device 300 receives a signal from the terminal device 100, and outputs an instruction for redetection included in the signal to the obstacle detection unit 500. The obstacle detection unit 500 and the motion estimation unit 502 execute a new detection process and a new estimation process according to the re-detection instruction. The communication unit 330 uses, as obstacle information, new first information and new second information based on the new detection process result and the new estimation process result as a signal including obstacle information. Are transmitted to the terminal device 100. The communication unit 110 of the terminal device 100 receives a signal from the parking assistance device 300, and outputs obstacle information included in the signal to the acquisition unit 112. That is, the acquisition unit 112 acquires new first information and new second information for the portion indicated in the received instruction. The subsequent processing is the same as before.

  The operation of the parking assistance system 1000 having the above configuration will be described. FIG. 13 is a sequence diagram showing a parking assistance procedure in the parking assistance system 1000. The parking assistance device 300 captures a video (S300), and the terminal device 100 also captures a video (S302). The parking assistance device 300 detects the position and orientation (S304), and the terminal device 100 also detects the position and orientation (S306). The terminal device 100 transmits the position information and the direction information to the parking assistance device 300 (S308). The parking assistance device 300 detects a low obstacle and estimates the operation of the vehicle 200 (S310). The parking assistance device 300 performs projection conversion of the image of the vehicle 200 (S312), and transmits obstacle information and the projection conversion image to the terminal device 100 (S314). The terminal device 100 superimposes the projection conversion image and the mixed reality UI 620 on the image of the terminal device 100 and displays the image (S316). The terminal device 100 receives an instruction to change the operation of the vehicle 200 (S318). The terminal device 100 outputs the obstacle information to the parking assistance device 300 (S320). The parking assistance device 300 generates a parking path (S322).

  According to the present embodiment, since the first information and the second information are received from the parking assistance device, the terminal device can dispense with the detection processing of the low obstacle and the estimation processing of the operation. Further, since the projection conversion video generated in the parking assistance device is received, the processing amount of the terminal device can be reduced. Also, when the mixed reality UI receives an instruction to select a non-overlapping portion, it transmits new first information and new second information for the portion, so detection of a low obstacle which has not been detected is performed. You can correct the result. Further, since the image of the vehicle is subjected to projection conversion and then transmitted, it is possible to provide an easy-to-see image to the driver.

Third Embodiment
Next, the third embodiment will be described. The third embodiment relates to a parking assistance system for assisting parking of a vehicle, as described above. On the other hand, the parking assistance system according to the third embodiment includes a server in addition to the parking assistance device and the terminal device. The server includes an obstacle detection unit, a motion estimation unit, and a projection conversion unit. Here, we will focus on the differences from the previous ones.

  FIG. 14 shows the configuration of a parking assistance system 1000 according to the third embodiment. The parking assistance system 1000 includes a terminal device 100, a vehicle 200, and a server 400. Vehicle 200 includes a parking assistance device 300. Server 400 includes a communication unit 410, a data exchange unit 412, an obstacle detection unit 500, an operation estimation unit 502, and a projection conversion unit 504. The server 400 can communicate with the terminal device 100 and the parking assistance device 300 via a network.

  The communication unit 410 of the server 400 receives a signal from the terminal device 100 and receives a signal from the vehicle 200. Thus, the communication unit 410 receives the position information of the terminal device 100 and the direction information of the imaging unit 126 of the terminal device 100 from the terminal device 100, and receives the search result searched for in the vehicle 200 from the parking assistance device 300. Do. Furthermore, the communication unit 410 receives the position information of the vehicle 200 and the direction information of the vehicle 200 from the parking assistance device 300. The data exchange unit 412 is connected to the communication unit 410, and is also connected to the obstacle detection unit 500, the motion estimation unit 502, and the projection conversion unit 504, and exchanges data among them.

  The projection conversion unit 504 receives, from the data exchange unit 412, the position information of the vehicle 200, the direction information of the vehicle 200, the position information of the terminal device 100, the direction information of the imaging unit 126, and the search result. Based on the position information of the vehicle 200, the direction information of the vehicle 200, the position information of the terminal device 100, and the direction information of the imaging unit 126, the projection conversion unit 504 sets the vehicle 200 to the viewpoint of the imaging unit 126. Project and convert the image. The projection conversion unit 504 outputs the correspondence relationship between the projection conversion video and the coordinates of the video of the vehicle 200 and the projection conversion video to the communication unit 410 via the data exchange unit 412. Communication unit 410 transmits a signal including these to terminal device 100.

  The obstacle detection unit 500 receives the search result from the data exchange unit 412. The obstacle detection unit 500 detects a low obstacle from the search result. The obstacle detection unit 500 outputs the first information on the detected position of the low obstacle to the communication unit 410 via the data exchange unit 412, and outputs the information on the shape of the detected low obstacle to the motion estimation unit 502. The motion estimation unit 502 receives from the obstacle detection unit 500 information on the shape of the low obstacle. The motion estimation unit 502 estimates the motion of the vehicle 200 with respect to the detected low obstacle. The motion estimation unit 502 outputs the second information on the motion of the vehicle 200 to the communication unit 410 via the data exchange unit 412. Communication unit 410 sets the combination of the first information and the second information as obstacle information, and transmits a signal including the obstacle information to terminal device 100.

  The operation of the parking assistance system 1000 having the above configuration will be described. FIG. 15 is a sequence diagram showing a parking assistance procedure in the parking assistance system 1000. The parking assistance device 300 captures an image (S350), and the terminal device 100 also captures an image (S352). The parking assistance device 300 transmits the search result to the server 400 (S354). The parking assistance device 300 detects the position and orientation (S356), and the terminal device 100 also detects the position and orientation (S358). The parking assistance device 300 transmits the position information and the direction information to the server 400 (S360), and the terminal device 100 transmits the position information and the direction information to the server 400 (S362). The server 400 detects a low obstacle and estimates the operation of the vehicle 200 (S364). The server 400 performs projection conversion of the image of the vehicle 200 (S366), and transmits obstacle information and the projection conversion image to the terminal device 100 (S368). The terminal device 100 superimposes the projection conversion image and the mixed reality UI 620 on the image of the terminal device 100 and displays the image (S370). The terminal device 100 receives an instruction to change the operation of the vehicle 200 (S372). The terminal device 100 outputs the obstacle information to the parking assistance device 300 (S374). The parking assistance device 300 generates a parking path (S376).

  According to the present embodiment, since the image of the vehicle is subjected to projection conversion and then transmitted, an image that can be easily viewed by the driver can be provided. In addition, since the generation of the projection conversion video is executed in the server, the processing amount of the terminal device and the parking assistance device can be reduced. Further, since the motion of the vehicle with respect to the detected low obstacle is estimated, the motion with respect to the low obstacle can be automatically estimated. In addition, since the detection of the low obstacle and the estimation of the operation of the vehicle are performed in the server, the throughput of the terminal device and the parking assistance device can be reduced.

  Although the embodiments according to the present disclosure have been described in detail with reference to the drawings, the functions of the above-described apparatus and each processing unit may be realized by a computer program. The computer that realizes the functions described above by a program includes input devices such as a keyboard, a mouse, and a touch pad, output devices such as a display and a speaker, a central processing unit (CPU), a ROM, a RAM, a hard disk drive, and a solid state drive (SSD). Etc., a reading device that reads information from a recording medium such as a DVD-ROM (Digital Versatile Disk Read Only Memory) or a USB memory, a network card that communicates via a network, etc. . Further, the reading device reads the program from the recording medium having the program recorded thereon and stores the program in the storage device. In addition, the function of each device is realized by the CPU copying the program stored in the storage device to the RAM, sequentially reading out the instructions included in the program from the RAM, and executing the instruction.

  The outline of one aspect of the present disclosure is as follows. A program according to an aspect of the present disclosure is a program in a terminal device capable of communicating with a communication device mounted on a vehicle, and relates to first information on an obstacle detected by the vehicle and an operation of the vehicle with respect to the obstacle The step of acquiring the second information, and the mixed reality UI in which the operation in the second information is indicated on the image captured by the imaging unit of the terminal device is superimposed on the position of the obstacle indicated in the first information. And a step of causing the display unit of the terminal device to display an image on which the mixed reality UI is superimposed, and an instruction for causing the mixed reality UI superimposed on the image displayed on the display unit to specify the operation of the vehicle And causing the computer to execute the steps of accepting and transmitting an instruction for specifying the accepted operation to the communication device.

  According to this aspect, the mixed reality UI superimposed on the displayed image receives an instruction for changing the operation of the vehicle, and transmits the received change instruction to the communication device, so that the type of obstacle is detected. You can correct the result.

  The second information acquired in the acquiring step indicates that the operation of the vehicle is to perform any one of contact with an obstacle, avoiding an obstacle, and overcoming an obstacle, and accepting the second step Receive an instruction to specify one operation different from the one indicated in the information. In this case, since any one of touching an obstacle, avoiding an obstacle, and overcoming an obstacle is changed to another one, it is possible to correct the action suitable for the obstacle.

  The first information acquired in the acquiring step may further include distance information from the vehicle to the obstacle. The step of superimposing further superimposes the distance information included in the first information. In this case, since the distance information is also superimposed, it is possible to notify the driver of the distance from the vehicle to the obstacle.

  The method may further comprise the steps of receiving a search result searched for in the vehicle, detecting an obstacle from the received search result, and estimating an action of the vehicle with respect to the detected obstacle. The acquiring step acquires first information based on the detection result and acquires second information based on the estimation result. In this case, since the obstacle is detected and then the motion of the vehicle with respect to the detected obstacle is estimated, the motion with respect to the obstacle can be estimated automatically.

  Receives the first information generated based on the detection result of the obstacle for the search result searched for in the vehicle and the second information generated based on the estimation result of the operation of the vehicle for the detected obstacle The method may further comprise the step of The acquiring step acquires the received first information and second information. In this case, since the first information and the second information are received, the obstacle detection process and the operation estimation process can be omitted.

  In the image displayed on the display unit, the method may further include a step of receiving an instruction for selecting a non-overlapping portion by the mixed reality UI. The acquiring step acquires new first information and new second information for the portion indicated in the received instruction. In this case, when the mixed reality UI receives an instruction to select a non-overlapping portion, new first information and new second information for the portion are acquired, so detection of an undetected obstacle is performed. You can correct the result.

  The steps of receiving the position information of the vehicle, the direction information of the vehicle, the steps of detecting the position information of the terminal device, the direction information of the imaging unit of the terminal device, the position information of the vehicle, the direction information of the vehicle And projecting the image captured by the imaging unit of the vehicle to the viewpoint of the imaging unit of the terminal based on the position information of the terminal and the orientation information of the imaging unit of the terminal. It is also good. The superimposing step superimposes the projection-converted video and the mixed reality UI on the video captured by the imaging unit of the terminal device. In this case, the image captured by the imaging unit of the vehicle is projected and converted, and then superimposed on the image captured by the imaging unit of the terminal device, so that an easy-to-view video can be provided to the driver.

  Detecting the position information of the terminal device and the direction information of the imaging unit of the terminal device; transmitting the detected position information of the terminal device; and the direction information of the imaging unit of the terminal device; Based on the direction information of the vehicle, the position information of the terminal device, and the direction information of the imaging unit of the terminal device, the image captured by the imaging unit of the vehicle is projected and converted to the viewpoint of the imaging unit of the terminal device The method may further comprise receiving the projected and transformed image. The superimposing step superimposes the projection-converted video and the mixed reality UI on the video captured by the imaging unit of the terminal device. In this case, since the image captured by the imaging unit of the vehicle is subjected to projection conversion and then received, the processing amount can be reduced.

  Another aspect of the present disclosure is a parking assistance device. This device is a parking assistance device mounted on a vehicle to be automatically parked, and for an image captured by an imaging unit of a terminal device, the obstacle is detected at the position of an obstacle detected for the vehicle. The mixed reality UI showing the motion of the vehicle estimated is superimposed, and a video on which the mixed reality UI is superimposed is displayed on the display unit of the terminal device, and is superimposed on the video displayed on the display unit. When the terminal device receives an instruction to change the operation of the vehicle to the mixed reality UI, the communication unit receives information on the operation of the vehicle after the change from the terminal device, and the information received by the communication unit And a parking path generation unit that generates a parking path by reflecting information related to the operation of the vehicle after the change.

  According to this aspect, when the information on the operation of the vehicle after the change is received from the terminal device, the information on the operation of the vehicle after the change is reflected to generate the parking path, so that the detection result of the obstacle type is corrected. A suitable parking path can be generated.

  The operation of the vehicle indicates any one of contact with an obstacle, avoiding an obstacle and overcoming an obstacle. In this case, any one of touching an obstacle, avoiding an obstacle, and overcoming an obstacle is indicated, so that it is possible to generate a parking path reflecting an action suitable for the obstacle.

  The search unit for searching the outside of the vehicle, an obstacle detection unit for detecting an obstacle from the search result in the search unit, and a motion estimation unit for estimating the motion of the vehicle with respect to the obstacle detected by the obstacle detection unit It is also good. The communication unit transmits first information on an obstacle detected by the obstacle detection unit and second information on an operation of the vehicle estimated by the movement estimation unit. In this case, since the obstacle is detected and then the motion of the vehicle with respect to the detected obstacle is estimated, the motion with respect to the obstacle can be estimated automatically.

  The communication unit instructs the mixed reality UI to select the non-overlapping part when the terminal device receives an instruction to select the non-overlapping part in the mixed reality UI in the image displayed on the display unit. Is received from the terminal device, and the obstacle detection unit and the motion estimation unit execute new detection processing and new estimation processing according to the instruction received by the communication unit, and the communication unit operates the obstacle detection unit and the motion detection unit. New first information and new second information are transmitted based on the result of the new detection process executed by the estimation unit and the result of the new estimation process. In this case, when the mixed reality UI receives an instruction to select a non-overlapping portion, it transmits new first information and new second information to the portion, so detection of an undetected obstacle is performed. You can correct the result.

  You may further provide the search part which searches the exterior of a vehicle. The communication unit transmits the search result in the search unit, detects an obstacle from the search result transmitted by the communication unit, and estimates the operation of the vehicle with respect to the obstacle. In this case, the search result is transmitted, the obstacle is detected from the search result, and the movement of the vehicle with respect to the obstacle is estimated, so that the amount of processing can be reduced.

  The communication unit receives the position information of the terminal device and the direction information of the imaging unit of the terminal device, and the parking assistance device detects the position information of the vehicle and the direction information of the vehicle. Based on the position information of the vehicle, the direction information of the vehicle, the position information of the terminal device, and the direction information of the image pickup unit of the terminal device, the image pickup unit of the vehicle The image processing apparatus may further include a projection conversion unit that performs projection conversion on the selected image. The communication unit transmits the image projected and converted by the projection conversion unit. In this case, since the image captured by the imaging unit of the vehicle is subjected to projection conversion and then transmitted, an image that can be easily viewed by the driver can be provided.

  You may further provide the position and direction detection part which detects the position information of a vehicle, and the direction information of a vehicle. The communication unit transmits the position information of the vehicle detected by the position / direction detection unit and the direction information of the vehicle, and the position information of the vehicle and the direction information of the vehicle transmitted by the communication unit, and the position information of the terminal device Based on the direction information of the imaging unit of the terminal device, the image captured by the imaging unit of the vehicle is projected and converted to the viewpoint of the imaging unit of the terminal device. In this case, the position information of the vehicle and the direction information of the vehicle are transmitted, and based on these, the image captured by the imaging unit of the vehicle is projection-transformed, so that the amount of processing can be reduced.

  Yet another aspect of the present disclosure is also a program. This program is a program of a communication device mounted on a vehicle to be automatically parked and a server program capable of communicating with a terminal device, receiving the search results searched for in the vehicle from the communication device, and the received search The step of detecting an obstacle from the result, the step of estimating the operation of the vehicle with respect to the detected obstacle, the first information about the detected obstacle, and the second information about the estimated operation of the vehicle are transmitted to the terminal device Have the computer execute the steps.

  According to this aspect, since the motion of the vehicle with respect to the detected obstacle is estimated, the motion with respect to the obstacle can be estimated automatically.

  The present disclosure has been described above based on the embodiments. It is understood by those skilled in the art that these embodiments are exemplifications, and that various modifications can be made to the combinations of the respective components and processing processes, and such modifications are also within the scope of the present disclosure. It is a place.

  In the first to third embodiments, the obstacle detection unit 500, the motion estimation unit 502, and the projection conversion unit 504 are collectively included in any of the terminal device 100, the parking assistance device 300, and the server 400. However, the present invention is not limited to this. For example, the obstacle detection unit 500 and the motion estimation unit 502 are included in any of the terminal device 100, the parking assistance device 300, and the server 400, and the projection conversion unit 504 is an obstacle detection unit 500 and motion estimation. The unit other than the unit including the unit 502 may be included. According to this modification, the degree of freedom of the configuration can be improved.

  In the first to third embodiments, the extraction projection conversion video is superimposed on the superimposed video. However, the present invention is not limited to this. For example, the extraction projection conversion video may not be superimposed on the superimposed video. According to this modification, processing can be simplified.

  According to the present disclosure, processing suitable for automatic parking by remote control can be performed.

  Reference Signs List 100 terminal device, 110 communication unit, 112 acquisition unit, 114 superposition unit, 116 positioning device, 118 digital compass, 120 position / direction detection unit, 122 positioning I / F, 124 compass I / F, 126 imaging unit, 128 imaging I / F, 130 display I / F, 132 display device, 134 operation unit, 136 operation I / F, 138 setting unit, 150 control unit, 160 processing unit, 180 terminal position, 182 terminal direction, 200 vehicle, 210 movement control device , 220 other vehicle, 230 parking area, 280 vehicle position, 282 vehicle direction, 300 parking assist device, 310 positioning device, 312 digital compass, 314 position / direction detection unit, 316 positioning I / F, 318 compass I / F, 320 Imaging unit, 322 sona , 324 search unit, 326 imaging I / F, 328 sonar I / F, 330 communication unit, 332 parking path generation unit, 350 control unit, 500 obstacle detection unit, 502 motion estimation unit, 504 projection conversion unit, 620 mixed reality UI, 1000 parking assistance system.

Claims (16)

A program in a terminal device capable of communicating with a communication device mounted on a vehicle,
Acquiring first information on an obstacle detected by the vehicle and second information on an operation of the vehicle with respect to the obstacle;
Superimposing a mixed reality UI in which an action in the second information is indicated on an image captured by an imaging unit of the terminal device on a position of an obstacle indicated in the first information;
Displaying a video on which a mixed reality UI is superimposed on the display unit of the terminal device;
Accepting an instruction for causing the mixed reality UI superimposed on the image displayed on the display unit to specify the operation of the vehicle;
Transmitting to the communication device an instruction for specifying the received operation;
A program to make a computer run. The second information acquired in the acquiring step indicates, as the operation of the vehicle, performing one of contact with an obstacle, avoiding an obstacle, and overcoming an obstacle.
The accepting step accepts an instruction for specifying one operation different from the one indicated in the second information.
The program according to claim 1. The first information acquired in the acquiring step further includes distance information from the vehicle to an obstacle,
The superimposing step further superimposes the distance information included in the first information.
The program according to claim 1 or 2. Receiving a search result searched for in the vehicle;
Detecting an obstacle from the received search result;
Estimating the motion of the vehicle with respect to the detected obstacle;
And further
The acquiring step acquires first information based on the detection result and acquires second information based on the estimation result.
The program according to any one of claims 1 to 3. First information generated based on a detection result of an obstacle with respect to a search result searched for in the vehicle, and second information generated based on an estimation result of the operation of the vehicle with respect to the detected obstacle Further comprising the step of receiving
The acquiring step acquires the received first information and second information.
The program according to any one of claims 1 to 3. In the image displayed on the display unit, the method further comprises the step of accepting an instruction for selecting a non-overlapping portion by the mixed reality UI.
The acquiring step acquires new first information and new second information for the portion indicated in the received instruction.
The program according to any one of claims 1 to 5. Receiving position information of the vehicle and heading information of the vehicle;
Detecting position information of the terminal device and orientation information of an imaging unit of the terminal device;
Based on the position information of the vehicle, the direction information of the vehicle, the position information of the terminal device, and the direction information of the imaging unit of the terminal device, in the viewpoint of the imaging unit of the terminal device And projecting the image captured by the imaging unit.
The superimposing step superimposes the projection-converted video and the mixed reality UI on the video captured by the imaging unit of the terminal device.
The program according to any one of claims 1 to 6. Detecting position information of the terminal device and orientation information of an imaging unit of the terminal device;
Transmitting the detected position information of the terminal device and the azimuth information of the imaging unit of the terminal device;
Based on the position information of the vehicle, the direction information of the vehicle, the position information of the terminal device, and the direction information of the imaging unit of the terminal device, in the viewpoint of the imaging unit of the terminal device The image captured by the imaging unit is project-transformed, and the method further comprises the step of receiving the project-transformed video,
The superimposing step superimposes the projection-converted video and the mixed reality UI on the video captured by the imaging unit of the terminal device.
The program according to any one of claims 1 to 6. A parking assistance device mounted on a vehicle to be automatically parked,
The mixed reality UI showing the operation of the vehicle estimated for the obstacle is superimposed on the image captured by the imaging unit of the terminal device at the position of the obstacle detected for the vehicle And a video on which the mixed reality UI is superimposed is displayed on the display unit of the terminal device, and the operation of the vehicle is changed with respect to the mixed reality UI superimposed on the video displayed on the display unit. A communication unit that receives, from the terminal device, information related to the operation of the vehicle after the change when the terminal device receives an instruction;
A parking path generation unit that generates a parking path by reflecting information received by the communication unit and information related to the operation of the vehicle after the change;
Parking assistance device provided with. The operation of the vehicle indicates any one of contact with an obstacle, avoiding an obstacle and overcoming an obstacle.
The parking assistance apparatus of Claim 9. A search unit for searching the outside of the vehicle;
An obstacle detection unit that detects an obstacle from the search result in the search unit;
It further comprises a motion estimation unit for estimating the motion of the vehicle with respect to the obstacle detected by the obstacle detection unit,
The communication unit transmits first information on an obstacle detected by the obstacle detection unit and second information on an operation of the vehicle estimated by the movement estimation unit.
A parking assistance device according to claim 9 or 10. The communication unit selects the non-overlapping portion of the mixed reality UI when the terminal device receives an instruction for selecting the non-overlapping portion of the mixed reality UI in the image displayed on the display unit. Receive instructions for the communication from the terminal device,
The obstacle detection unit and the motion estimation unit execute a new detection process and a new estimation process according to the instruction received by the communication unit,
The communication unit performs new first information and new second information based on the result of the new detection process performed by the obstacle detection unit and the motion estimation unit and the result of the new estimation process. Send,
The parking assistance device according to claim 11. It further comprises a search unit for searching the outside of the vehicle,
The communication unit transmits a search result in the search unit;
As well as detecting an obstacle from the search result transmitted by the communication unit, the operation of the vehicle with respect to the obstacle is estimated.
A parking assistance device according to claim 9 or 10. The communication unit receives position information of the terminal device and orientation information of an imaging unit of the terminal device,
The parking assistance device
A position / orientation detection unit that detects the position information of the vehicle and the azimuth information of the vehicle;
Based on the position information of the vehicle, the direction information of the vehicle, the position information of the terminal device, and the direction information of the imaging unit of the terminal device, in the viewpoint of the imaging unit of the terminal device And a projection conversion unit for projecting and converting an image captured by the imaging unit.
The communication unit transmits an image projected and converted by the projection conversion unit.
The parking assistance apparatus of Claim 9. It further comprises a position / orientation detection unit for detecting the position information of the vehicle and the azimuth information of the vehicle,
The communication unit transmits position information of the vehicle detected by the position / orientation detection unit and direction information of the vehicle.
The viewpoint of the imaging unit of the terminal device based on the position information of the vehicle, the orientation information of the vehicle, the position information of the terminal device, and the orientation information of the imaging unit of the terminal device transmitted by the communication unit And the image captured by the imaging unit of the vehicle is projection-transformed,
The parking assistance apparatus of Claim 9. A communication device mounted on a vehicle to be automatically parked, and a program of a server capable of communicating with a terminal device,
Receiving from the communication device a search result searched for in the vehicle;
Detecting an obstacle from the received search result;
Estimating the motion of the vehicle with respect to the detected obstacle;
Transmitting to the terminal device first information related to the detected obstacle and second information related to the estimated operation of the vehicle;
A program to make a computer run.

Images