JP6470112B2 - Mobile device operation terminal, mobile device operation method, and mobile device operation program - Google Patents

Description

  The present invention relates to a technique for remotely operating a mobile device.

  Currently, techniques for remotely operating mobile devices are being researched and developed. According to Non-Patent Document 1, an operation terminal having an input screen for adjusting parameters such as a flight altitude, a joystick and a body model that can be rotated in any direction are proposed for operating a small airplane. ing.

M. Quigley, “Semi-Autonomous Human-UAV Interfaces for Fixed-Wing Mini-UAVs”, Proc. Of IEEE’04, Intelligent Robots and Systems, Vol.3, 2005, p.2457-p.2462

  However, in the case of a conventional operation terminal, it is difficult to intuitively control the flight altitude and the like because the mobile device is operated based on the input numerical values. On the other hand, in the case of a joystick or an aircraft model, it can be controlled by the movement of the wrist and can realize intuitive operability compared to the operation terminal. However, it is difficult to adjust the flight altitude or the like appropriately because the operation is performed with a roughly sensory wrist rotation. That is, with the conventional technology, it is difficult to intuitively perform an operation of how much the moving device is moved.

  The present invention has been made in view of the above circumstances, and an object thereof is to improve the remote operability of a mobile device.

  In order to solve the above problem, the mobile device operation terminal according to claim 1 designates a display unit that displays a captured image captured by an imaging unit of the mobile device on a screen, and an area included in the captured image. The gist of the invention is to include an accepting unit that accepts an operation, and a control unit that moves the moving device so that the captured image approaches the designated region.

  According to the present invention, the captured image captured by the imaging unit of the moving device is displayed on the screen, an operation for designating an area included in the captured image is received, and the movement is performed so that the captured image approaches the designated area. Since the device is moved, the remote operability of the moving device can be improved.

  The mobile device operation terminal according to claim 2 includes a display unit that displays a captured image captured by an imaging unit of the mobile device on a screen, a reception unit that receives an operation for designating an area included in the captured image, and an imaging unit. And a control means for moving the moving device so that the image moves away from the designated area.

  The mobile device operation terminal according to claim 3 is the mobile device operation terminal according to claim 1 or 2, wherein the display unit and the reception unit are touch panels, and the operation for designating the region is performed by the touch panel. The gist of the operation is to surround the region by touching the finger.

  The mobile device operation terminal according to claim 4 is the mobile device operation terminal according to claim 1 or 3, wherein the display unit and the reception unit are touch panels, and the control unit pinches out the touch panel. The gist is to move the moving device forward based on the operation.

  The mobile device operation terminal according to claim 5 is the mobile device operation terminal according to claim 2 or 3, wherein the display means and the reception means are touch panels, and the control means is a pinch-in to the touch panel. The gist is to retract the moving device based on the operation.

  The mobile device operation terminal according to claim 6, a display unit that displays a captured image captured by an imaging unit of the mobile device on a screen, a reception unit that receives an operation for designating a moving direction of the captured image, and the movement And a control means for moving the moving device so that the direction is imaged.

The mobile device operation terminal according to claim 7 is the mobile device operation terminal according to claim 6, wherein the display unit and the reception unit are touch panels.
The operation for designating the moving direction is a drag operation in which a finger touches and slides on the touch panel, and the control means performs at least one of rotation, ascending or descending based on the drag operation on the moving device. It is made to perform.

  The mobile device operation method according to claim 8, wherein the mobile device operation terminal is a mobile device operation method performed by a mobile device operation terminal, wherein the mobile device operation terminal displays a captured image captured by an imaging unit of the mobile device on a screen. The gist of the invention is to include an accepting step for accepting an operation for designating an area included in the captured image, and a control step for moving the moving device so that the captured image approaches the designated area.

  A mobile device operation program according to a ninth aspect causes a computer to function as the mobile device operation terminal according to any one of the first to seventh aspects.

  According to the present invention, the remote operability of the mobile device can be improved.

It is a figure which shows the functional block structure of an operating terminal. It is a figure which shows the processing flow of a camera image display part. It is a figure which shows the processing flow of a moving apparatus control part. It is a figure which shows the operation example (at the time of a pinch out) of a moving apparatus. It is a figure which shows the operation example (at the time of pinch in) of a moving apparatus. It is a figure which shows the operation example (at the time of a right direction drag) of a moving apparatus. It is a figure which shows the example of operation of a moving apparatus (at the time of a downward drag).

  Hereinafter, an embodiment for carrying out the present invention will be described with reference to the drawings.

  FIG. 1 is a diagram showing a functional block configuration of operation terminal 1 according to the present embodiment. The operation terminal 1 is a terminal for remotely operating the mobile device 2 including a camera, and includes a camera video display unit 11 and a mobile device control unit 12.

  The camera video display unit 11 includes a touch panel, and is configured to display a camera image captured by the camera of the mobile device 2 on the touch panel. Further, it is configured to accept a user's touch operation and discriminate the type of gesture from the touch operation.

  The moving device control unit 12 is configured to move the moving device 2 based on a gesture on the touch panel. Further, the moving direction of the moving device 2 is changed according to the type of gesture, and the moving amount of the moving device 2 is changed according to the magnitude of the gesture operation. Further, when the moving device 2 is moved forward or backward, the moving angle of the moving device 2 with respect to the traveling direction is changed according to the operation position on the touch panel where the gesture is performed.

  The operation terminal 1 including the plurality of functional units described above can be realized by a computer including a touch panel, a CPU, a memory, and the like. For example, it is realized using a smartphone, a tablet terminal, or a personal computer. It is also possible to create a program for causing a computer to function as the operation terminal 1 and a storage medium for the program.

  The moving device 2 is, for example, a drone (unmanned aerial vehicle such as a small airplane or a helicopter), an automobile, an underwater ship, or a working robot. Any mobile device may be used as long as it is equipped with a digital camera, can transmit a camera image taken by the own device, and moves the own device based on the received control signal. The communication path between the operation terminal 1 and the mobile device 2 may be wireless or wired.

  Next, a remote operation method of the mobile device 2 performed at the operation terminal 1 will be described. The operation terminal 1 receives a camera image from the mobile device 2 and displays it on the touch panel.

  First, the operation of the camera video display unit 11 will be described with reference to FIG.

  First, in step S101, when the user touches the touch panel displaying a camera image with a finger, the touch of the user's finger is detected. The presence or absence of contact with the touch panel can be detected using an existing interface provided by the touch panel.

  Next, in step S102, the touch position when the user's finger touches is stored. For example, the touch panel is converted into two-dimensional coordinates, and the position coordinates touched by the user's finger are stored in the memory as the touch position at the time of contact.

  Next, in step S <b> 103, the type of gesture performed by the user is determined from the operation of the finger on the touch panel and the number of touches. For example, when the number of touches is two and the distance between the two touches gradually increases, the user's gesture is determined to be a pinch out. For example, when a thumb and an index finger are placed on the touch panel and an operation of enlarging a rectangular area surrounded by two fingers is performed, it is determined that the image is pinched out.

  Further, when the number of touches is two and the distance between the two touches gradually decreases, it is determined that the pinch-in has occurred. For example, it is determined to be pinch-in when an operation of reducing a rectangular area surrounded by two fingers is performed.

  On the other hand, when the number of touches is one, it is determined as a drag. For example, a gesture is determined when an operation of sliding an index finger placed on the touch panel in a certain direction is performed.

  If the gesture type is any one of pinch-out, pinch-in, and drag, the process proceeds to step S104, and if not, the process ends. Although operations such as flicking and swiping can be considered, they are synonymous with dragging.

  Finally, in step S104, every time the user's finger moves by a certain distance, the locus of the finger is repeatedly transmitted to the moving device control unit 12 until it leaves the touch panel. Specifically, the coordinates of the touch position moving while touching the touch panel are transmitted as a finger trajectory at regular intervals or at regular intervals. The type of gesture determined in step S103 is also transmitted to the mobile device control unit 12.

  Next, the operation of the mobile device control unit 12 will be described with reference to FIG.

  First, in step S <b> 201, a finger trajectory and a gesture type are received from the camera image display unit 11. As described above, the finger trajectory includes virtual two-dimensional coordinates on the touch panel corresponding to the finger contact position.

  Next, in step S202, the movement direction (forward, backward, rotation, up, down) of the moving device 2 is determined based on the type of gesture determined by the camera image display unit 11, and based on the movement amount of the finger. The movement amount (movement distance, rotation angle, altitude) of the moving device 2 is calculated. In particular, when the moving direction of the moving device 2 is forward or backward, the moving angle of the moving device 2 with respect to the forward or backward moving direction is also determined.

  For example, when the type of gesture is pinch out, the camera image on the touch panel is enlarged and displayed, so as shown in FIG. 4A, the moving device 2 is advanced so that the actual image of the camera matches the enlarged display. Let Further, as shown in FIG. 4B, the distance d between the two touch positions when the touch panel is touched and the distance d ′ after the pinch out are obtained using the information of the finger trajectory, and the difference between the two distances is obtained. A value {= α (d′−d)} obtained by multiplying the value by a constant α (positive real number) is set as a moving distance of the moving device 2. Further, the average coordinates of all touch positions included in the finger trajectory information are obtained, and the moving angle of the moving device 2 with respect to the forward direction is determined based on how far the touch panel is deviated in the horizontal direction and / or the vertical direction. To do. For example, when the center coordinate of the touch panel is (0, 0) and the average coordinate is (100, 100), the touch panel is moved in the diagonally forward right direction of 45 °.

  In the case of pinch-in, since the camera image on the touch panel is reduced and displayed, the moving device 2 is moved backward so that the actual image of the camera matches the reduced display as shown in FIG. Further, as shown in FIG. 5B, a value {= β (positive real number) obtained by multiplying the difference between the distance d between the two touch positions when the touch panel is touched and the distance d ′ after the pinch-in is obtained by a constant β (positive real number). d′−d)} is the moving distance of the moving device 2. Furthermore, the movement angle with respect to the backward movement direction is also determined by the same method as in the case of pinch out.

  Further, in the case of dragging, the camera image on the touch panel moves in the dragged direction. Therefore, as shown in FIG. 6A or FIG. 7A, the moving device 2 so that the actual image of the camera matches the drag direction. To work. Specifically, when the drag direction is the right direction (left direction), the moving device 2 is rotated to the right (left rotation) as viewed from the user. When the drag direction is downward (upward), the moving device 2 is raised (lowered).

  When the drag direction is right or left, as shown in FIG. 6B, a movement vector ν = (x, y) is obtained from two touch positions at the start and end of the drag, as shown in FIG. And the value obtained by multiplying the circumference ratio π (= θπν.y) as the rotation angle of the moving device 2. When the drag direction is downward or upward, as shown in FIG. 7B, a value (= γν.y) obtained by multiplying the movement vector ν = (x, y) by a constant γ is used as the movement device 2. Altitude.

  Whether the drag direction is the horizontal direction or the vertical direction is determined using the movement vector ν. For example, when the value of arctan (y / x) is within a certain range, that is, when the movement vector ν is horizontal to a certain degree, it is determined that the drag is in the left-right direction, and the moving device 2 is rotated. On the other hand, when it does not fall within the predetermined range, it is determined that the drag is in the vertical direction, and the moving device 2 is raised or lowered.

  Finally, in step S203, a control signal for moving in the moving direction and moving amount determined in step S202 is generated and transmitted to the moving device 2. The moving device 2 receives the control signal from the operation terminal 1 and moves the own device in the moving direction and the moving amount included in the control signal. Specifically, if the user's gesture is a pinch out, the user moves forward. At this time, the greater the movement amount of the pinch out, the greater the forward distance. Moreover, if it is pinch-in, it will move backward. The greater the amount of pinch-in movement, the greater the reverse distance. In the case of a drag, it is rotated, raised or lowered depending on the drag direction. The greater the amount of drag movement, the greater the amount of rotation, the amount of increase, or the amount of decrease.

  As described above, according to the present embodiment, in the operation terminal 1, the camera video display unit 11 displays the camera image captured by the camera of the mobile device 2 on the touch panel, and the user's gesture (pinch-in, Since the moving device control unit 12 moves the moving device 2 based on the user's gesture, the remote operability of the moving device 2 can be improved. That is, it is possible to remotely control with a finger gesture on the touch panel while viewing the camera image of the mobile device 2 at hand, and it is possible to intuitively operate how much the mobile device 2 is moved.

  Finally, in the present embodiment, the case where a touch panel is used has been described. However, the camera image display unit and the touch operation reception unit may be realized by physically different units. For example, it is possible to use a general-purpose monitor and a pen tablet. An effect similar to the above can be obtained by indirectly touching the camera image on the monitor on the pen tablet using the electronic pen of the pen tablet. Specifically, for example, a part of the camera image is surrounded by a line using an electronic pen and a part of the surrounded line is pulled outward, so that substantially the same operation as that of pinch out can be realized. Further, tools for controlling the movement of the moving apparatus 2 such as a screen enlargement button, a reduction button, a slide bar for changing the screen size ratio, and a slide direction instruction button may be displayed and used.

1. Operation terminal (remote device operation terminal)
11 ... Camera image display section (display means, reception means)
12 ... Moving device control section (control means)
2 ... Mobile device S101-S104, S201-S203 ... Step

Claims (8)

Display means for displaying a captured image captured by the imaging means of the mobile device on a screen;
Accepting means for accepting an operation for designating an area included in the captured image;
Control means for moving the moving device so that the captured image approaches the designated area ,
The display means and the reception means are touch panels,
The control means includes
Based on a pinch-out operation on the touch panel, the moving device is advanced, average coordinates of a plurality of touch positions included in the locus of the operation are obtained, and a deviation width in the vertical direction and the horizontal direction with respect to the center coordinates of the touch panel is obtained. A moving device operation terminal, wherein a moving angle of the moving device with respect to a forward direction is determined . Display means for displaying a captured image captured by the imaging means of the mobile device on a screen;
Accepting means for accepting an operation for designating an area included in the captured image;
Control means for moving the moving device so that the captured image moves away from the designated area ,
The display means and the reception means are touch panels,
The control means includes
Based on a pinch-in operation on the touch panel, the moving device is retracted, an average coordinate of a plurality of touch positions included in the locus of the operation is obtained, and based on a deviation width in a vertical direction and a horizontal direction with respect to the center coordinate of the touch panel. A moving device operation terminal, wherein a moving angle of the moving device with respect to a backward direction is determined . The operation to specify the area is as follows:
The mobile device operation terminal according to claim 1, wherein the operation is performed by touching the touch panel with a finger to surround the area. Display means for displaying a captured image captured by the imaging means of the mobile device on a screen;
Accepting means for accepting an operation for designating a moving direction of the captured image;
Control means for moving the moving device so that the moving direction is imaged,
The display means and the reception means are touch panels,
The control means includes
Based on a drag operation in which the finger touches and slides on the touch panel, when the angle of the finger slide direction with respect to the horizontal direction of the touch panel is included in a certain angle, it is determined as a horizontal drag operation and the moving device is rotated, If it is not included in the certain angle, it is determined as a vertical drag operation and the moving device is raised or lowered.
A mobile device operation terminal characterized by that. In the mobile device operation method performed at the mobile device operation terminal,
The mobile device operation terminal is
A display step for displaying a captured image captured by the imaging means of the mobile device on the touch panel;
An accepting step of accepting an operation for designating an area included in the captured image on a touch panel;
Performing a control step of moving the moving device so that the captured image approaches the designated area,
In the control step,
Based on a pinch-out operation on the touch panel, the moving device is advanced, average coordinates of a plurality of touch positions included in the locus of the operation are obtained, and a deviation width in the vertical direction and the horizontal direction with respect to the center coordinates of the touch panel is obtained. A moving device operating method, comprising: determining a moving angle of the moving device with respect to a forward direction. In the mobile device operation method performed at the mobile device operation terminal,
The mobile device operation terminal is
A display step for displaying a captured image captured by the imaging means of the mobile device on the touch panel;
An accepting step of accepting an operation for designating an area included in the captured image on a touch panel;
Performing a control step of moving the moving device so that the captured image moves away from the designated region,
In the control step,
Based on a pinch-in operation on the touch panel, the moving device is retracted, an average coordinate of a plurality of touch positions included in the locus of the operation is obtained, and based on a deviation width in a vertical direction and a horizontal direction with respect to the center coordinate of the touch panel. A moving device operating method, wherein a moving angle of the moving device with respect to a backward direction is determined. In the mobile device operation method performed at the mobile device operation terminal,
The mobile device operation terminal is
A display step for displaying a captured image captured by the imaging means of the mobile device on the touch panel;
An accepting step of accepting an operation for designating a moving direction of the captured image on a touch panel;
Performing a control step of moving the moving device so that the moving direction is imaged,
In the control step,
Based on a drag operation in which the finger touches and slides on the touch panel, when the angle of the finger slide direction with respect to the horizontal direction of the touch panel is included in a certain angle, it is determined as a horizontal drag operation and the moving device is rotated, When not included in the certain angle, it is determined as a drag operation in the vertical direction, and the moving device is raised or lowered. A mobile device operating program for causing a computer to function as the mobile device operating terminal according to any one of claims 1 to 4.

Images