JP6473048B2 - Mobile device operation terminal, mobile device operation method, and mobile device operation program - Google Patents

Description

  The present invention relates to a technique for remotely operating a mobile device with an operation terminal.

  Currently, telepresence robots (hereinafter referred to as robots) for communication purposes are being researched and developed. The user can remotely control the moving direction and the rotating direction of the robot using the operation terminal. When the user wants to talk with a person around the robot, the user operates the robot so that the person enters the camera angle of view. For example, if the robot is operated so that the front of the robot faces the other party and it is desired to talk about an object (person / thing) near the other party, the operation is performed so that the angle of view is within the camera angle.

Hashimoto, 3 others, “TouchMe: Direct Robot Operation Method Using CG Overlay Display”, Information Processing Society of Japan, Interaction, 2011

  However, it is troublesome for the user to manually operate the robot so that the target is within the video every time a conversation is performed. For example, since the robot movement direction and rotation direction are repeatedly adjusted while searching for surrounding people while watching the camera image, the operation procedure becomes complicated. Non-Patent Document 1 proposes a method of intuitively operating a robot using computer graphics superimposed display, but even in this case, it is difficult to instantaneously place an object in an image.

  This invention is made | formed in view of the said situation, and aims at improving the operation efficiency of a moving apparatus.

  In order to solve the above-described problem, the mobile device operation terminal according to claim 1 displays a captured image captured by an imaging unit of the mobile device on a screen, and relates to an object existing outside the range of the captured image. The gist of the invention is to include display means for superimposing the object information on the captured image in association with a position in the real space.

  According to the present invention, the captured image captured by the imaging unit of the mobile device is displayed on the screen, and the object information relating to the object existing outside the range of the captured image is associated with the position in the real space, and Since it is superimposed on the captured image, the operation efficiency of the mobile device can be improved.

  The mobile device operation terminal according to claim 2 is the mobile device operation terminal according to claim 1, wherein the object of the target object information selected on the screen is within an imaging range of the imaging means. The gist of the present invention is to further include control means for moving the moving device.

  The mobile device operation terminal according to claim 3 is the mobile device operation terminal according to claim 1, wherein a plurality of objects selected from the target object in the captured image and / or the target object related to the target object information are provided. The gist further includes control means for moving the moving device so that the body falls within the imaging range of the imaging means.

  The mobile device operation terminal according to claim 4 is the mobile device operation terminal according to claim 2 or 3, wherein the selected object is at least a living object.

  The mobile device operation terminal according to claim 5 is the mobile device operation terminal according to any one of claims 1 to 4, wherein the display unit is configured to change the object according to a distance from the mobile device to the object. The gist is to change the display size of information.

  The mobile device operation method according to claim 6, wherein the mobile device operation terminal displays a captured image captured by an imaging unit of the mobile device on a screen, and the imaging The gist of the invention is to include a display step of superimposing the object information relating to the object existing outside the range of the image on the captured image in association with the position in the real space.

  A gist of a mobile device operation program according to a seventh aspect is to cause a computer to function as the mobile device operation terminal according to any one of the first to fifth aspects.

  According to the present invention, the operation efficiency of the mobile device can be improved.

It is a figure which shows the functional block structure of an operating terminal. It is a figure which shows the processing flow (Example 1) of the robot operation method. It is a figure which shows the positional relationship (Example 1) with the periphery with respect to a robot. It is a figure which shows the example of a display of a shortcut button (Example 1). It is a figure which shows the example of selection of a shortcut button (Example 1). It is a figure which shows the example of control (Example 1) of a robot. It is a figure which shows the example of a screen (Example 1) after robot control. It is a figure which shows the processing flow (Example 2, operation example 1) of the robot operation method. It is a figure which shows the positional relationship (Example 2) with the periphery with respect to a robot. It is a figure which shows the example of a display of a shortcut button (Example 2, operation example 1). It is a figure which shows the example of selection of a shortcut button (Example 2, operation example 1). It is a figure which shows the example of control (Example 2, operation | movement example 1) of a robot. It is a figure which shows the example of a screen (Example 2, operation example 1) after robot control. It is a figure which shows the processing flow (Example 2, operation example 2) of the robot operation method. It is a figure which shows the selection example (Example 2, operation example 2) of a shortcut button. It is a figure which shows the example of control (Example 2, operation | movement example 2) of a robot. It is a figure which shows the example of a screen (Example 2, operation example 2) after robot control. It is a figure which shows the processing flow (Example 2, operation example 3) of the robot operation method. It is a figure which shows the selection example (Example 2, operation example 3) of a shortcut button. It is a figure which shows the example of control (Example 2, operation example 3) of a robot. It is a figure which shows the example of a screen (Example 2, operation example 3) after robot control.

  Hereinafter, an embodiment for carrying out the present invention will be described with reference to the drawings.

[First Embodiment]
FIG. 1 is a diagram illustrating a functional block configuration of the operation terminal 1 according to the first embodiment. The operation terminal 1 is a terminal for remotely operating a telepresence robot (robot) 2 and includes a camera image display unit 11 and a robot control unit 12.

  The camera image display unit 11 includes a touch panel, and is configured to display a camera image captured by the camera of the robot 2 on the touch panel. In addition, a shortcut button (object information) that intuitively represents the position and direction of a target (subject / object) existing outside the range of the camera video and the distance to the target is displayed superimposed on the camera video. Has been.

  The robot controller 12 is configured to move the robot 2 so as to be in front and at a certain distance with respect to the selected shortcut target.

  The operation terminal 1 including the plurality of functional units described above can be realized by a computer including a touch panel, a CPU, a memory, and the like. For example, it is realized using a smartphone, a tablet terminal, or a personal computer. It is also possible to create a program for causing a computer to function as the operation terminal 1 and a storage medium for the program.

  The robot 2 is a moving device that includes a speaker, a microphone, and a 360-degree digital camera having a front direction of photographing, and can control the moving direction and the rotating direction of the own device based on a control signal from the operation terminal 1. In addition, a function for recognizing a person's face or object is provided, and the position direction and distance of the person / object can be detected from the position and size of the face / object on the camera image. Furthermore, the captured camera image and all detected data can be transmitted to the operation terminal 1. The telepresence robot is an example of a robot, and an unmanned air vehicle (drone) such as a small airplane or a helicopter may be used. Any mobile device having the functions and mechanisms described above may be used. The communication path between the operation terminal 1 and the robot 2 may be wireless or wired.

  Next, a robot operation method performed on the operation terminal 1 will be described with reference to FIG. It is assumed that the robot 2 detects the position and direction of a person existing around the own machine and the distance from the own machine to the person.

First, in step S <b> 101, the camera video display unit 11 acquires the position direction and distance of the person (target person) detected by the robot 2. For example, when there are three target persons HA to HC at the position shown in FIG. 3 with respect to the robot 2, the rotation angle θ _HA from the front direction of the camera (upward in the figure) to the target person HA in the clockwise direction, A distance L _HA from the robot 2 to the subject _HA is acquired. Similarly, the rotation angle θ _HB and distance L _HB related to the subject _HB , and the rotation angle θ _HC and distance L _HC related to the subject _HC are also acquired.

  Next, in step S102, the camera image display unit 11 uses the acquired data to generate a shortcut button that intuitively indicates the position direction of the subject and the distance to the subject, and displays the shortcut button on the camera image on the touch panel. Superimposed display. Specifically, a button image (icon) representing the face is generated, the display size is changed according to the distance to the subject, and the button is displayed at a location corresponding to the position direction of the subject.

For example, as shown in FIG. 4, the screen center of the touch panel is regarded as the current position of the robot 2, and the shortcut button SB _HA of the subject _HA is displayed on the screen の in the axial direction of θ _HA ° clockwise relative to the upward direction. Similarly, the shortcut button SB _HB of the subject _HB is displayed on the screen 淵 in the clockwise direction of θ _HB ° in the clockwise direction. Here, since the distance from the robot 2 to the subject person HA is closer to the subject person HB (L _HA <L _HB ), the shortcut button SB _HA is displayed larger and the shortcut button SB _HB is displayed smaller. As the button image, for example, a photographed face image of the subject may be used by reducing or enlarging it, or a prescribed face image may be used. In addition, since the target person (target person HC) that falls within the camera angle of view θ _cam is already displayed on the touch panel, the shortcut button is not displayed.

Next, in step S <b> 103, when a pressing touch operation by the user is detected for the shortcut button being displayed, the camera image display unit 11 transmits the ID of the pressed shortcut button to the robot control unit 12. Here, it is assumed that the shortcut button SB _HB is selected as shown in FIG. 5 in order to have a conversation with the target person HB.

Next, in step S104, the robot control unit 12 generates a control signal for moving the shortcut button so that the shortcut button is in front of the subject and located at a certain distance from the subject. Transmit to the robot 2. When the shortcut button SB _HB is pressed down as described above, the robot 2 is rotated clockwise by θ _HB ° from the upward direction (old front direction), and the distance obtained by subtracting the predetermined distance from the distance L _HB as shown in FIG. Move forward.

  In step S <b> 105, the camera image display unit 11 displays the camera image of the moving robot 2 on the touch panel, and uses shortcut buttons based on the position direction and distance of surrounding people that change every moment as the robot 2 moves. Re-generation / re-display is performed sequentially.

Then, after the movement is completed, the robot 2 stops its own movement operation. After the robot stops, as shown in FIG. 7, a camera image showing the selected target person HB is displayed on the touch panel. Shortcut buttons SB _HA are displayed for the target persons HA and HC outside the camera angle of view θ _cam. , SB _HC is displayed.

  In the present embodiment, the case where a person is selected for the purpose of communication has been described as an example. However, for example, a tree, a personal computer, or an animal may be selected for the purpose of investigating a failure. In addition, the other party may be a robot having a call function or the like.

[Second Embodiment]
In the first embodiment, the method for displaying shortcut buttons on the touch panel and the method for selecting one target person have been described. On the other hand, there may be a case where one wants to talk with a plurality of people or another person who wants to talk about a nearby object. Therefore, in the second embodiment, a method of operating the robot 2 so that a plurality of objects (subjects / objects) are included in the camera angle of view will be described. As with the first embodiment, the operation terminal 1 according to the present embodiment includes a camera video display unit 11 and a robot control unit 12.

(Operation example 1)
First, the robot operation method of the operation example 1 performed on the operation terminal 1 will be described with reference to FIG.

First, in step S <b> 201, the camera image display unit 11 acquires the position direction and distance of a person (subject) / tree (subject) detected by the robot 2. For example, when there is one target person HA and two objects OA and OB at the position shown in FIG. 9 with respect to the robot 2, from the front direction of the camera (upward in the figure) to the target person HA in the clockwise direction. The rotation angle θ _HA and the distance L _HA from the robot 2 to the subject _HA are acquired. Similarly, the rotation angle theta _OA and the distance _{L OA} about the object OA, the rotation angle theta _OB and the distance _{L OB} regarding the object OB to retrieve.

Next, in step S202, the camera image display unit 11 generates a shortcut button that intuitively represents the position direction of the target (target person / target object) and the distance to the target using the acquired data, and the touch panel. Display superimposed on the upper camera image. The shortcut button generation / display method is the same as that of the first embodiment. As a result, as shown in FIG. 10, for the object OA within the camera angle of view θ _cam , the image itself is displayed so as to be selectable. Shortcut buttons SB _HA and SB _OB are displayed for the subject HA and the subject OB outside the camera angle of view θ _cam .

Next, in step S <b> 203, when a push-down touch operation is detected with respect to the displayed object / shortcut button, the camera image display unit 11 transmits the IDs of the plurality of pushed down objects to the robot control unit 12. To do. Here, since the conversation about the target object OA is performed with the target person HA, it is assumed that the shortcut button SB _HA and the target object OA are selected as shown in FIG. For example, the user drags the object OA while pressing the shortcut button SB _HA , and then releases the finger.

Next, in step S204 to step S206, the robot control unit 12 operates the robot 2 so that a plurality of depressed objects are within the camera angle of view θ _cam .

Specifically, first, in step S204, as shown in FIG. 12, the rotation angle θ ′ _OA from the camera front direction with respect to the selected object OA and the camera front direction with respect to the subject _HA of the shortcut button SB _HA. The robot 2 is rotated so that the added value with the rotation angle θ ′ _HA is 0 (= θ ′ _OA + θ ′ _HA ).

Subsequently, in step S205, it is determined whether or not the selected object OA and the subject HA are within the camera angle of view θ _cam after the rotation is completed. Specifically, it is confirmed whether | θ ′ _OA | + | θ ′ _HA | <θ _cam is satisfied. If it does not fit, the process ends. If it does not fit, the process proceeds to step S206.

In step S206, the robot 2 moves backward until | θ ′ _OA | + | θ ′ _HA | <θ _cam is established so that the selected object OA and the subject HA are within the camera angle of view θ _cam . Let After the robot stops, a camera image showing the selected target object OA and the target person HA is displayed on the touch panel as shown in FIG. 13, and a shortcut button is displayed for the target object OB outside the camera angle of view θ _cam. SB _OB is displayed.

(Operation example 2)
Subsequently, an operation example 2 will be described with reference to FIG. Step S301 and step S302 are the same as step S201 and step S202. In step S303, as shown in FIG. 15, it is assumed that two shortcut buttons SB _HA and SB _OA are selected in order to have a conversation about the object OB with the subject _HA .

Next, in step S304, as shown in FIG. 16, the rotation angle θ ′ _HA from the camera front direction with respect to the subject _HA of the selected shortcut button SB _{HA and} the camera front direction of the shortcut button SB _OB with respect to the object OB. The robot 2 is rotated so that the added value of the rotation angle θ ′ _OB from the rotation angle becomes 0 (= θ ′ _HA + θ ′ _OB ).

Subsequently, in step S305, it is determined whether or not the selected subject HA and the subject OB are within the camera angle of view θ _cam after the rotation is completed. Specifically, it is confirmed whether | θ ′ _HA | + | θ ′ _OB | <θ _cam is satisfied. If it does not fit, the process ends. If it does not fit, the process proceeds to step S306.

In step S306, the robot 2 moves backward until | θ ′ _HA | + | θ ′ _OB | <θ _cam is established so that the selected subject HA and the subject OB are within the camera angle of view θ _cam . Let After the robot is stopped, a camera image showing the selected target person HA and the target object OB is displayed on the touch panel as shown in FIG. 17, and a shortcut button is displayed for the target object OA outside the camera angle of view θ _{cam. SBOA} is displayed.

(Operation example 3)
Subsequently, an operation example 3 will be described with reference to FIG. Step S401 and step S402 are the same as step S201 and step S202. In step S403, as shown in FIG. 19, the object OA and the two shortcut buttons SB _HA and SB _OA are selected in order to have a conversation about the two objects OA and OB with the subject _HA . Shall be. For example, a plurality of objects selected by the user are specified by displaying a button for selecting a plurality of objects, depressing the button, selecting these three objects, and then pressing the button again.

Next, in step S404 to step S407, the robot control unit 12 operates the robot 2 so that the plurality of pushed objects are within the camera angle of view θ _cam .

Specifically, first, in step S404, the rotation angles θ _N [= {θ _HA , θ _OA , θ _OB }] before movement in all the selected target groups N [= {HA, OA, OB}]]. from determining the minimum angle Minshita _N and the maximum angle maxθ _N. Here, minθ _N = θ _OA and maxθ _N = θ _OB from the positional relationship before rotation shown in FIG.

Subsequently, in step S405, the angle before movement of the object of minθ _N is θ _min , the angle after movement is θ ′ _min , the angle before movement of the object of maxθ _N is θ _max , and the angle after movement is θ ′. _If it is set to _max , the robot 2 is rotated so that the added value of θ ′ _min and θ ′ _max becomes 0 (= θ ′ _min + θ ′ _max ). Specifically, as shown in FIG. 20, the robot 2 is rotated so that | θ ′ _OA | and | θ ′ _OB | are equal.

In step S406, it is determined whether or not the selected target person HA and the target objects OA and OB are within the camera angle of view θ _cam after the rotation is completed. Specifically, it is confirmed whether or not | θ ′ _min | + | θ ′ _max | <θ _cam holds. If it does not fit, the process ends. If it does not fit, the process proceeds to step S407.

Thereafter, in step S407, the selected subjects HA and the object OA, as OB fit the camera angle theta in _{cam, | θ 'min | +} | θ' max | < robot until theta _cam holds 2 Retreat. After the robot stops, a camera image showing all the selected objects is displayed on the touch panel as shown in FIG.

  As described above, according to each embodiment, in the operation terminal 1, the camera image display unit 11 displays the camera image captured by the camera of the robot 2 on the touch panel, and the target (outside the range of the camera image) Since the shortcut button relating to the subject / object is associated with the position and distance in the real space and superimposed on the screen of the camera image, the operation efficiency of the robot 2 can be improved. That is, it is possible to intuitively confirm on the touch panel how much the robot 2 should be rotated and how much the robot 2 should be moved based on the display position and display size of the shortcut button. It is possible to easily operate the robot 2 so that a nearby object is within the camera angle of view.

  In addition, the shortcut button has a function of displaying the shape of the object (human face / tree) and a function of moving the robot 2 so that it is in front and at a certain distance from the object. Effective use of the limited display area on the touch panel enables efficient display.

  Finally, in the present embodiment, the case where a touch panel is used has been described, but the camera image display means and the touch operation accepting means may be realized by physically different means. For example, it is possible to use a general-purpose monitor and a pen tablet. An effect similar to the above can be obtained by indirectly touching a camera image or shortcut button on the monitor on the pen tablet using the electronic pen of the pen tablet.

1 ... Operation terminal (mobile device operation terminal)
11 ... Camera image display section (display means)
12 ... Robot controller (control means)
2. Telepresence robot (mobile device)
S101-S105, S201-S206, S301-S306, S401-S407 ... step

Claims (6)

A display in which a captured image captured by an imaging unit of a mobile device is displayed on a screen, and object information relating to an object existing outside the range of the captured image is associated with a position in real space and superimposed on the captured image. Means,
Control means for moving the moving device so that a target object of the target object information selected by a touch operation on the screen falls within an imaging range of the imaging means ,
The display means includes
A mobile device operating terminal, wherein a display size of the target object information is changed according to a distance from the mobile device to the target object . A display in which a captured image captured by an imaging unit of a mobile device is displayed on a screen, and object information relating to an object existing outside the range of the captured image is associated with a position in real space and superimposed on the captured image. Means,
Control means for moving the moving device so that a plurality of objects selected by a touch operation among objects in the captured image and / or objects related to the object information are within the imaging range of the imaging means; Have
The display means includes
A mobile device operating terminal, wherein a display size of the target object information is changed according to a distance from the mobile device to the target object . The selected object is
The mobile device operation terminal according to claim 1, wherein the mobile device operation terminal is at least a living body. In the mobile device operation method performed at the mobile device operation terminal,
The mobile device operation terminal is
A display in which a captured image captured by an imaging unit of a mobile device is displayed on a screen, and object information relating to an object existing outside the range of the captured image is associated with a position in real space and superimposed on the captured image. Steps,
Performing a control step of moving the moving device so that a target object of the target object information selected by a touch operation on the screen falls within an imaging range of the imaging means,
In the display step,
A moving device operating method, wherein a display size of the object information is changed according to a distance from the moving device to the object. In the mobile device operation method performed at the mobile device operation terminal,
The mobile device operation terminal is
A display in which a captured image captured by an imaging unit of a mobile device is displayed on a screen, and object information relating to an object existing outside the range of the captured image is associated with a position in real space and superimposed on the captured image. Steps,
A control step of moving the moving device so that a plurality of objects selected by a touch operation among objects in the captured image and / or objects related to the object information are within the imaging range of the imaging means; , And
In the display step,
A moving device operating method, wherein a display size of the object information is changed according to a distance from the moving device to the object. A mobile device operating program for causing a computer to function as the mobile device operating terminal according to any one of claims 1 to 3.

Images