WO2018214029A1

WO2018214029A1 - Method and apparatus for manipulating movable device

Info

Publication number: WO2018214029A1
Application number: PCT/CN2017/085555
Authority: WO
Inventors: 苏冠华; 陈嘉勇; 李小军; 郭灼
Original assignee: 深圳市大疆创新科技有限公司
Priority date: 2017-05-23
Filing date: 2017-05-23
Publication date: 2018-11-29
Also published as: CN108496151A

Abstract

A method for manipulating a movable device, comprising: displaying at least one touch base point on a touch screen of a control terminal; when a touch operation is detected on the touch screen, and the distance between a touch point of the touch operation and the touch base point is less than or equal to a preset threshold, using the touch point as a touch starting point, and generating a virtual operation area around the touch starting point; generating, according to a sliding track of the touch point in the virtual operation area starting from the touch starting point, a displacement vector of the touch point relative to the touch base point; detecting the displacement vector and generating an instruction for manipulating the movable device based on the displacement vector; and sending the instruction to the movable device. Also disclosed is an apparatus for manipulating a movable device.

Description

Method and apparatus for manipulating a mobile device

Copyright statement

The disclosure of this patent document contains material that is subject to copyright protection. This copyright is the property of the copyright holder. Copyright The owner has no objection to the reproduction of the patent document or patent disclosure contained in the official records and files of the Patent and Trademark Office. Technical field

The present disclosure relates to the field of interactive control of a mobile device using a user interface, and more particularly to a method and apparatus for manipulating a mobile device, and more particularly to a mobile device or navigation for use as a motion carrier An interface interaction method and apparatus for controlling the motion state of the carried object and the carried object. Background technique

In recent years, mobile devices (eg, aircraft, including fixed-wing aircraft, rotorcraft including helicopters; motor vehicles, submarines or ships, and satellites, space stations, or spacecraft, aviation models, or the like; and camera rockers, poles) ) has been widely used, such as in the areas of detection, search and rescue, and videography. The manipulation of these moveable devices is typically accomplished, for example, by the user via a user interface on the remote control. Similarly, the manipulation of the movable device can also be achieved using a user interface on a control unit equipped in the cockpit.

These mobile devices / can carry the carrier. In some cases, for example, the carrier is a camera, a light, a mounted pod (such as a camera pod, an electronic countermeasure pod, or a detection pod such as a radar pod, a weapon pod). In other cases, a carrier such as a camera or a nacelle is typically not directly attached to the fuselage, but is coupled to the aircraft via a bracket (carrier; For example, a carrier device (or stand) that carries a camera is commonly referred to in some literature as a "pTZ".

Currently, one common mode of operation for a mobile device is achieved, for example, by a joystick controller, or by a virtual joystick controller on the portable terminal. However, with conventional solid rocker controllers, it is necessary to manually manipulate the magnitude of the rocker to manipulate the movable device. And if the physical joystick controller is added to display the display of the mobile device camera screen in real time, a more complicated combination and configuration is required. In addition, in the prior art, the control of the aircraft is directly performed on the touch screen, that is, the virtual joystick manipulation interface is superimposed on the display interface of the real-time captured image of the screen to realize the interactive interface. However, this kind of interactive interface also faces a big challenge, both To minimize the occlusion of the display interface on the display interface, it is also necessary to have a strong sense of control on the screen with weak feedback. In the existing virtual joystick control interface and the real-time shooting display interface superimposed interactive interface scheme, the virtual joystick has a weak sense of control, and the feedback feeling is weak, which is easy to cause an erroneous operation; and the hidden logic is not set so that the image is displayed for display. The display interface is too occluded; at the same time, it is impossible to control the pitch of the pan/tilt during the two-hand control. Summary of the invention

To at least partially overcome the above-discussed deficiencies and/or disadvantages of the prior art, embodiments of the present disclosure provide a method and apparatus for manipulating a movable device. The technical solution is as follows:

According to an aspect of an embodiment of the present disclosure, a method for operating a mobile device is provided, including: displaying at least one touch base point on a touch screen of a control terminal; touching the touch screen to generate a touch point When the touch operation on the touch screen is detected, and the distance between the touch point of the touch operation and the touch base point is less than or equal to a preset threshold, the touch point acts as a touch start point, and Generating a virtual operation area around the touch start point; generating a displacement vector of the touch point relative to the touch start point by the sliding track of the touch point in the virtual operation area from the touch start point; detecting the Displacement vector, and generating an instruction to manipulate the movable device based on the displacement vector; and transmitting the instruction to the movable device.

According to another aspect of an embodiment of the present disclosure, an apparatus for manipulating a mobile device is provided, including: a display module configured to display at least one touch base point on a touch screen of the control terminal; Configuring, when the touch operation on the touch screen is detected, and when the distance between the touch point of the touch operation and the touch base point is less than or equal to a preset threshold, using the touch point as a touch start point And instructing the display module to display a virtual operation area generated by expanding outward around the touch start point; the data processing module configured to detect by starting from the touch start point due to the touch point a displacement vector of the touch point generated relative to the touch start point generated by the sliding track in the virtual operation area, and generating an instruction to manipulate the movable device based on the displacement vector; and an instruction transmitting module configured to be movable The device transmits the instruction. DRAWINGS

The present application will be further described in detail below with reference to the accompanying drawings, in which: FIG. Schematic diagram of an application environment for manipulating a mobile device; FIG. 2(a) shows a flowchart of a method for manipulating a mobile device in accordance with an embodiment of the present disclosure; 2(b) shows a block diagram of an apparatus for manipulating a movable device in accordance with an embodiment of the present disclosure; FIG. 3(a) illustrates a control terminal for manipulating a movable device in accordance with an embodiment of the present disclosure Schematic diagram of the working state of the touch screen before activation, wherein the image captured by the movable device and the interface for manipulating the movable device are illustrated, the interface including two touch base points that can be shifted by touch;

3(b) is a diagram showing an operation state of a touch screen of a control terminal for manipulating a movable device after activation, in which an initial position of two touch base points is activated, according to an embodiment of the present disclosure. Generate two virtual operating areas for the geometry center;

4(a) and 4(b) respectively show schematic diagrams of the left virtual operating area and the right virtual operating area of the virtual operating area shown in FIG. 3(b) in an exemplary working state, according to an embodiment of the present disclosure. ;

5 shows a schematic flow chart of instructions for operating a mobile device based on acquired sensor data, in accordance with an embodiment of the present disclosure;

6 illustrates an operation state of a touch screen of a control terminal as shown in FIGS. 3( a ) and 3 ( b ) when performing pitch adjustment on a load carried by a movable device, according to an embodiment of the present disclosure. Schematic diagram

Fig. 7 shows a flow chart according to Fig. 6. detailed description

The technical solutions of the embodiments of the present invention are further specifically described below by way of embodiments and with reference to the accompanying drawings. In the specification, the same or similar reference numerals indicate the same or similar parts. The description of the embodiments of the present invention with reference to the accompanying drawings is intended to illustrate the general inventive concept of the invention, and should not be construed as a limitation of the invention.

In the following detailed description, numerous specific details are set forth Obviously, however, one or more embodiments may be practiced without these specific details. In other instances, well-known structures and devices are shown in the drawings in the drawings.

In the present disclosure, the terms "comprising" and "including" and their derivatives are intended to include and not limiting. Further, the size and shape of the elements in the drawings do not reflect the true proportions between the elements on the interactive interface of the method for manipulating the movable device of the embodiments of the present disclosure, and the purpose is only to illustrate the present disclosure.

Before describing various embodiments of the present disclosure in detail, various terms that may be used herein will be first introduced. Movable device: In the present disclosure, a movable device refers to an object that can be used as a moving vehicle in a real environment or a virtual environment, and is divided into an unmanned remote control and an internal manned/operated type, for example, an aircraft, including an aircraft (including But not limited to aircraft, more specifically, for example, fixed-wing aircraft, rotorcraft (such as helicopters), spacecraft (package) Including but not limited to space shuttles, spacecraft, satellites, space stations; motor vehicles, such as unmanned vehicles; ships (including but not limited to surface vessels, submersibles, hovercrafts, ground-effect boats), such as unmanned vessels, and camera rockers , poles, handheld heads; or similar.

Control terminal: A control terminal is a device for manipulating a mobile device and its components/devices such as a pan/tilt or a pod, including but not limited to: a portable device, such as a dedicated remote control, or with a touch screen and installed Smartphones, tablets, wearable electronic devices, etc. that control the APP; embedded control devices embedded in the mobile device, such as touch screens in the cockpit.

Image acquisition device: The image acquisition device is a collection device for real-time acquisition of still images and dynamic photography, such as a motion camera equipped on a drone, a camera mounted on a camera rocker mounted on a live ground, and a water on a deep submersible. The camera, the camera gun equipped on the reconnaissance plane, etc.

PTZ: PTZ is a supporting device for installing and fixing image acquisition devices, and is stabilized. It is usually divided into fixed pan/tilt and electric pan/tilt. The fixed gimbal is suitable for the case where the monitoring range is not large. After installing the camera on the fixed gimbal, the horizontal and vertical angles of the camera can be adjusted. After the best working attitude is reached, it is only necessary to lock the adjustment mechanism. The electric pan/tilt is suitable for scanning and monitoring a wide range, which can expand the surveillance range of the camera.

Specific embodiments of the present disclosure are described below in conjunction with the accompanying drawings.

1 shows a schematic diagram of an application environment for manipulating a mobile device, such as with a control terminal, in accordance with an embodiment of the present disclosure.

As shown in FIG. 1, the control terminal 101 transmits a manipulation command to the mobile device 102 (for example, a drone) via the network, and the mobile device 102 passes, for example, a screen captured by the carried image acquisition device (for example, a motion camera). The network transmits to the control terminal 101, and the screen captured by the mobile device 102 is displayed on the control terminal 101 in real time. Specifically, the removable device 102 can convert the image into a signal transmission to the image transmission relay, and the control terminal 101 receives the real-time image captured by the movable device 102 through a network (for example, a WiFi signal connecting the image transmission relay). More preferably, the control terminal 101 and the mobile device 102 respectively set a wireless communication module, and establish a communication channel through the wireless communication module. For example, the control terminal 101 and the mobile device 102 establish a 2.4G wireless communication channel. Through the exemplary wireless communication channel, the control terminal 101 can immediately send a manipulation command to the mobile device 102, and the mobile device 102 can transmit the captured image to the control terminal 101 for display in real time, and control The terminal 101 feeds back information about its current location (such as spatial position coordinates) and speed, acceleration, and the like. The control terminal 101 can be, but not limited to, various smart phones with touch screens, tablet computers, wearable electronic devices, and the like. The control terminal 101 and the removable device 102 can also transmit image data streams through the high definition image transmission channel. 2(a) shows a flow chart of a method for manipulating a mobile device in accordance with an embodiment of the present disclosure. Figure

3(a) is a schematic diagram showing an operation state of a touch screen of a control terminal for manipulating a movable device before activation, according to an embodiment of the present disclosure, wherein an interface 2 for manipulating the movable device is illustrated, The interface 2 includes two touch base points 21 that can be shifted by touch; and the interface can be separately displayed on the touch screen or appear on the background displayed on the touch screen. And FIG. 3(b) is a schematic diagram showing the working state of the touch screen of the control terminal for manipulating the movable device after activation, in which the two touch base points 21 are activated, according to an embodiment of the present disclosure. The initial position is the two virtual operating areas 22 generated by the geometric center.

In accordance with an overall concept of the present disclosure, in an aspect of an embodiment of the present disclosure, as shown in FIGS. 2(a), 3(a), and 3(b), a method for manipulating a movable device is provided The method includes: displaying step S101, displaying at least one touch base point 21 on the touch screen of the control terminal 101; touching step S101': touching the touch screen to generate a touch point, for example, as shown in FIGS. 3(a) and 2( b) showing the left touch point 211 and the right touch point 212; the activation detecting step S102, detecting a touch operation on the touch screen, and detecting a touch operation on the touch screen When the distance between the touch point of the touch operation and the touch base point is less than or equal to the preset threshold, the touch point acts as a touch start point, and a virtual operation area (ie, a virtual operation area is generated around the touch start point) Step S103 is generated to generate a virtual operation area around the touch start point. Data processing step S104, generating a touch by the sliding track in the virtual operation area starting from the touch start point by the touch point. Point relative to touch start a displacement vector, and detecting the displacement vector, for example, continuously acquiring the displacement vector by a sensor (eg, a pressure sensor and a displacement sensor) at predetermined time intervals, and generating an instruction to manipulate the movable device based on the displacement vector; Step S105, transmitting the instruction to the mobile device.

Specifically, for example, as shown in FIG. 3(a), for example, two dots, that is, touch base points are displayed on the left and right sides near the middle of the screen on the control terminal such as the screen of the mobile phone. When a touch operation on the touch screen is detected, and a distance between the touch point of the touch operation and the touch base point is less than or equal to a preset threshold, generating a preset radius based on the outward touch of the touch base point Touchable area. And generating, by the touch base point, the touchable area of the preset radius by outward radiation includes generating the touchable area that is opaque or translucent. The user can trigger the dot, that is, the touch base activation, by clicking the two dots for a certain time to generate the touchable area; then the touch point touching the touchable area is used as the touch start point, and the touch start point is outward The virtual operating area is diffused from the heart, and at the same time, the control terminal, for example, the smart phone, sends a vibration command to the control terminal, prompting that the virtual joystick control has been activated at this time. In the virtual operation area, for example, a user shifting a position from an initial position such as a touch start point to a touch direction by a touch operation, for example, by a touch operation in a sliding manner, for example, is proportional to a movable device being defined as being The speed corresponding to the movement mode/mode corresponding to the touch direction. It can be understood that in addition to vibrating the virtual joystick control, the light, sound, etc. can also be prompted.

Optionally, two virtual operation areas are respectively generated for the respective geometric centers with touch start points, for example, each of the left virtual operation area 221 and the right virtual operation area 222 shown in FIG. 3(b) The operating area is for example shaped into a predetermined shape and size, such as a regular polygonal area, an elliptical area, or a circular area as shown. The specific shape of the virtual operation area can be determined, for example, based on the relationship between the movements of the movable devices in different touch directions. For example, the speed of the roll is specified to be a certain percentage of the speed of the forward/reverse speed; or the angular velocity of the movable device rotated clockwise/counterclockwise in the vertical direction and the speed of the ascending/descending speed a certain ratio.

With the virtual operating area 22, by way of the exemplary embodiment shown in Fig. 2(a), it is no longer necessary to separately use a dedicated controller such as a solid joystick. Since the interface 2 includes a certain number of touch base points occupying a limited area on the touch screen (for example, two touch base points 21 to correspond to a single operator's eyes); of course, for example, there may be a plurality of, for example, four The touch base point is used for two people to share one display at the same time, or the front and rear cabin controllers respectively use two screens after screen expansion and image division to simultaneously control), and when the touch base point is detected (because, for example, by the user's finger When a touchable area is activated for a certain period of time, a virtual operation area corresponding to the number of activated touch base points is generated from the position where the touch start point is the geometric center, thereby simultaneously displaying the background and the interface for interaction. The occlusion between them is limited, and it is convenient to make full use of a single touch screen with limited display area to simultaneously display the collected information and control information to reduce the visual interference between each other. Moreover, the touch base point on the interface may be displaced by a touch action, for example, so that the touch base point for interaction and the position of the touchable area thus produced can be dynamically adjusted when a specific background exists. To reduce their occlusion of the background in a dynamic manner as efficiently as possible.

4(a) and 4(b) respectively illustrate an example work of the left virtual operating area 221 and the right virtual operating area 222 of the virtual operating area as shown in FIG. 3(b), according to an embodiment of the present disclosure. Schematic diagram of the state.

In an embodiment of the present disclosure, as shown in FIGS. 4(a) and 4(b), for example, in each of the virtual operating areas 221, 222 in a radial direction centered on the touch start point A plurality of main directions are defined on the upper side (as shown, each of the virtual operation areas has four main directions indicated by broken lines respectively), and each of the virtual operation areas includes: a plurality of main direction areas and a plurality of composite direction areas Each of the plurality of composite regions is defined as a region between two adjacent main direction regions. As an example, as shown in FIG. 4(a) and FIG. 4(b), each of the plurality of main direction regions is defined as a sector region including a corresponding main direction, for example, as shown, the left side is virtual There are four main direction areas 2211, 2213, 2215 and 2217 in the operation area 221, and four main parties exist in the right virtual operation area 222. To areas 2221, 2223, 2225, and 2227. Each of the plurality of composite regions is defined as an area between two adjacent main direction regions. For example, as shown, there are four composite direction regions 2212, 2214 in the left virtual operation region 221. There are four composite direction regions 2222, 2224, 2226 and 2228 in 2216 and 2218, and in the right virtual operating area 222.

And, for example, when the touch point is located in the main direction area, generating an instruction to manipulate the movable device based on a component of the displacement vector corresponding to the corresponding main direction; and when the touch point is located in the When the direction regions are combined, instructions for manipulating the movable device are generated based on components of the displacement areola corresponding to respective corresponding ΐ directions of the two adjacent ΐ direction regions, respectively.

In other words, based on the above, a plurality of main direction regions and a composite direction region are divided in each virtual operation area, that is, specifically, the main direction is defined based on each main direction and is offset from the main direction to the both sides. A main direction area that is radially at a specific central angle is formed at an oblique angle, and a region between adjacent main direction areas is defined as a composite direction area. Based on this, the movement of the touch point after entering the single main direction area with respect to the touch start point, that is, the movement of the touch point in a single main direction area, is simplified as being along the single main direction area. Containing movement in the main direction, ignoring components of the displacement vector in other directions than the corresponding main direction contained in the main direction region; in other words, once a single touch point enters one of the corresponding virtual operation regions In the main direction region, a single movement of the single touch point along the corresponding main direction included in the main direction region radially at a specific central angle is considered, and the movement of the single touch point in other directions is ignored. . With this arrangement, and the main direction is set to correspond to the most common movement direction and movement mode of the movable device, the most common movement mode and movement direction of the movable device can be conveniently realized in a simplified manner, while avoiding Malfunctions due to undesired slippage of the finger to the side that cannot be avoided during touch operation.

Also, in an embodiment of the present disclosure, as shown in FIGS. 4(a) and 4(b), for example, in each of the virtual operation areas 22, the plurality of main directions include: relative to the Up, down, left, and right of the touch start point, and components corresponding to the plurality of main directions based on the displacement vector respectively generate a manipulation movable device to translate forward and backward relative to the current state An instruction to pan to the left, to the right, or to generate an instruction to manipulate the movable device to rise, fall, counterclockwise, and clockwise with respect to the current state.

It should be noted that the up, down, left, and right directions mentioned in the present application are relative, and refer to the four directions of up, down, left, and right generated by the control terminal in the process of being used in the most common manner. 4(a) and 4(b), the user can use the control terminal to view the real-time display of the interface on the screen in a conventional manner, and then the four directions of up, down, left, and right can be determined. Specifically, for example, in practice, a Cartesian coordinate system is established by the geometric center of each virtual operating area 22, and as shown by the dashed lines in Figures 4(a) and 4(b), respectively, in the y-axis of the coordinate system The positive direction, the negative y-axis direction, the negative x-axis direction, and the positive X-axis direction are the main directions of up, down, left, and right. And, for example, as shown in FIG. 4(a), in the left virtual operation area 221, components based on the displacement vector shown in the y-axis forward direction, the y-axis negative direction, the x-axis negative direction, and the X-axis positive direction, respectively. Correspondingly generating a speed vector command for manipulating the movable device to translate forward, backward, to the left, to the right; and/or, for example, as shown in FIG. 4(b), in the virtual operating area 222 on the right side, Based on the displacement yaw shown in the y-axis lH direction, the y-axis negative direction, the X-axis negative direction, and the X-axis lH upward, respectively, correspondingly generating the manipulation movable device translational rise, translational decline, and counterclockwise, smooth Speed vector command for hour hand rotation.

With such an arrangement, it is substantially equivalent to controlling the movement of each degree of freedom of the movable device by establishing a two-dimensional linear coordinate system in at least two virtual operating regions without simplifying the three-dimensional coordinate system, thereby simplifying The interface display also reduces the visual interference between the various degrees of freedom during operation.

In an exemplary embodiment of the present disclosure, for example, when the touch point is located in the main direction area, that is, when the displacement vector enters one of the plurality of main direction areas, that is, in the left virtual operation area 221. One of the plurality of main direction regions that are entered is highlighted when one of the four main direction regions 2211, 2213, 2215, and 2217 or one of the four main direction regions 2221, 2223, 2225, and 2227 in the right virtual operation region 222 Display, easy to visually recognize the upcoming operation in a single main direction. Specifically, for example, as shown in FIG. 4(a), when the user travels forward using the left virtual remote controller 221, even if the finger is not in front and is offset, it is determined that the command is directed to the single said positive Go ahead.

In an exemplary embodiment of the present disclosure, for example, as shown in Figs. 4(a) and 4(b), the central angle of the sector area is less than or equal to 11 °. For example, the angles of the two sides of the sector-shaped region being respectively deflected toward the two sides with respect to the main direction included are -5 ° and 10 5 °, respectively.

Further, for example, the two side edges of the sector area may be disposed to form different angles with respect to the corresponding main direction. More specifically, for example, a user can apply a force variation trend according to a sliding trajectory of the touch point, and both sides of the sector area are preset to have different angles with respect to the corresponding main direction to achieve force feedback. Compensation. For example, the sector area includes an operation side half sector actually touched by a touch point divided by a covered main direction as a boundary line, and a non-operation side half sector not touched by the touch point, and the sector is located The angle between the side of the operating side half sector and the corresponding main direction is greater than the angle between the other side of the sector located on the non-operating side half sector and the corresponding main direction. As an example, as shown in FIG. 4(a), in the main direction region along the forward and backward main directions, the angle from the main direction to the left side and the angle to the right side may be set to respectively -8 ° and 10 ° ° ; and as shown in Figure 4 (b), in the main direction along the forward and backward main directions, from the main The angle of the direction to the left and the angle of the deflection to the right can be set to -3 ° and + 8 °, respectively, so that the compensation of the force feedback input can be realized according to the habit of the conventional user's direction of force.

FIG. 5 illustrates a schematic flow diagram of instructions for operating a mobile device based on acquired sensor data in accordance with an embodiment of the present disclosure.

In an embodiment of the present disclosure, as shown in FIG. 5, for example, the generating, based on the displacement vector, an instruction to manipulate the movable device further includes: S1031, (eg, at an initial time of a single predetermined time interval) obtained (eg, Using the acquired sensor data to determine an initial state of the movable device (eg, an initial state of velocity sagittal); S1032, (eg, at a termination time of a single predetermined time interval) obtaining a velocity vector of the movable device a final stage state (for example, a final stage state of the velocity vector); and S1033, generating a compensation control command of the movable device according to a change of the final segment state with respect to the initial state (for example, a change in a velocity vector) to correct the The state of movement of the device can be moved to compensate for undesired shifts such as caused by lateral interference such as cross wind. Alternatively, it may be provided to generate a correction amount of the displacement vector of the movable device to be traveled based on the displacement vector shown to facilitate intuitive manipulation.

In addition, in the embodiment of the present disclosure, in order to ensure that the touch activates the reliability of the touch base point, as an example, in a case where the distance between the touch point of the touch operation and the touch base point is less than or equal to a preset threshold, Determining the touch start point in response to detecting that the pressure received by the touch screen at the touch point is higher than a preset first pressure threshold, thereby ensuring activation reliability from both pressure and duration Preventing misoperations from causing undesired activation of the touch base to create a touchable area.

In addition, in an embodiment of the present disclosure, for example, detecting a duration of a case where a distance between the touch point and the touch base point is less than or equal to a size of the touchable area, and when the duration is greater than the first predetermined time, Determining the touch start point. By setting the first predetermined time, and is intended to be distinguished from the touch operation to avoid erroneous operations.

And, in an embodiment of the present disclosure, for example, as shown in FIG. 3(b) to FIG. 3(a), when the touch point is released (for example, for a second predetermined time), the virtual operation area disappears and The touch base point is redisplayed such that the generation of instructions to manipulate the moveable device is terminated. Specifically, for example, as shown in the figure, when the user finishes the operation, the user releases the finger and continues for a second predetermined time, so that the outer contour of the virtual operation area rebounds rapidly inward or slowly retracts, thereby returning to The outer circumference contours of the dot-shaped touch base points coincide, thereby achieving resetting and terminating the manipulation of the movable device. At this time, the movable device is held, for example, to travel in a state (for example, speed, acceleration) before the termination of the manipulation of the movable device, or to maintain a position hovering at the timing of the termination of the manipulation, depending on the initial setting for the control terminal. .

Under normal circumstances, the adjustment to the action of the mobile device is limited in the instructions given to the mobile device. For example, a velocity vector command output to a movable device needs to have a threshold with respect to a specific direction, and thus it is necessary to set a boundary corresponding to the threshold for the virtual operation region. Specifically, as an example, as shown in FIGS. 4(a) and 3(b), in an embodiment of the present disclosure, for example, each of the virtual operation areas 22 is shaped to represent a maximum value of the displacement vector The transparent or translucent area defined by the boundary mark. For ease of setup and simplification of the interface, consider common handling requirements. Typically, for example, in virtual operating areas that represent plane degrees of freedom, the speed thresholds for forward, backward, left shift, and right shift are generally equal. In an embodiment of the present disclosure, for example, each of the virtual operation areas is shaped into a circular shape having a center of the touch start point and a radius of a predetermined maximum value of the displacement areola.

In a further embodiment of the present disclosure, for example, as shown in Figures 4(a) and 3(b), the boundary mark is a hollow triangle, for example, as shown in Figure 4(a), adjacently disposed on the left A triangular region 2219 at the outer side of the circumferential boundary of the side virtual operating region 221, or a triangular region 2229 disposed adjacently outside the circumferential boundary of the right virtual operating region 222, and the bottom edges of the triangles 2219, 2229 are configured to have An arc of curvature that coincides with the circumference of the circular shape. And, when the touch point moves in the virtual operation area, the boundary mark slides along the circumference with the arc edge to indicate an actual direction after the touch point moves. Thereby, the direction of the manipulation to be performed by the movable device can be displayed in an intuitive manner in real time.

Further, as an example, when the touch point approaches a boundary mark of the virtual operation area, the boundary mark becomes highlighted. For example, as shown in FIG. 4(a) or 3(b), when the user hits the edge of the virtual operating area 21 and substantially reaches the edge, that is, when the touched point enters the circumference close to the left virtual operating area 221. The boundary is located in each of the critical regions 22110, 22130, 22150, 22170 of the main direction regions 2211, 2213, 2215, 2217, or enters a circumferential boundary close to the right virtual operation region 222 and is located in the main direction regions 2221, 2223, 2225, When the respective critical regions 22210, 22230, 22250, and 22270 are 2227, it is determined that the touch point is close to the boundary mark of the virtual operation area, and the full-stick prompt is triggered, that is, the speed value in the control command reaches a preset maximum value. value. At this time, the triangular area 2219 or 2229 is discolored correspondingly, for example, from a regular green color to a red state in a full-bar prompt state. It is suggested that the user can respond to the speed of the mobile device at this time or has reached the maximum, and should pay attention to safety.

Alternatively, in other embodiments of the present disclosure, for example, when the touch point approaches the boundary mark of the virtual operation area, the full bar prompt is triggered, the boundary mark becomes highlighted, and the boundary mark is in the following manner One or more of them are highlighted: by becoming a fill in a particular color, by becoming a fill in a particular pattern, by a change in the stretch of the boundary mark, and by a change in the shape of the boundary mark.

And, for example, when the touch point moves to reach the boundary mark, the instruction to manipulate the movable device is maintained, thereby, for example, causing the movable device to maintain stable operation at the threshold response speed. Moreover, further, for example, the image 1 is also illustrated, and the image is a dynamic and/or static image that can be used as a background acquired by the image capturing device carried by the mobile device. The user obtains the image 1 by using an image capturing device such as an fpv camera or a motion camera on the movable device, for example, and transmits it back to the control terminal, such as a mobile phone screen, and additionally displays the image 1 simultaneously and superimposedly for the touch operation. The interactive interface 2, for example, the interface 2 emerges above the image 1. For example, the virtual operating area has a higher transparency than the image 1. Alternatively, the interface 2 is also formed, for example, as a region of a light colored background; and, further, the ground color of the interface 2 can adjust the color depth according to the overall color depth of the interface for manipulation. In this way, the images acquired in real time and the interface for interaction are simultaneously displayed, and the occlusion between each other is limited. In other words, it not only facilitates reducing the mutual influence of visual confusion or occlusion between the fpv return screen and the interface for manipulation; and can dynamically adjust the touch base point for interaction when shooting for a specific target and The positions of the virtual operating areas thus produced are used to reduce their occlusion of the captured image in a dynamic manner as efficiently as possible. In addition, by setting the boundary mark, the virtual operation area in a transparent or semi-transparent state is easily perceived by the user as its approximate location.

The above embodiments of the present disclosure enable manipulation of the movable device; however, while the manipulation of the movable device requires a load carried by the movable device (for example, a pan/tilt and a crane for mounting the image capturing device) The cabin, and optionally an image capture device such as a camera mounted thereon, is manipulated to, for example, effect adjustment of its pitching angle, such as implementing a pan/tilt Pitch to achieve composition control, thereby effecting manipulation of the movable device The combination of the manipulation with the load enables the shooting of the full trajectory and angle of view of the target.

6 shows an operation state of a touch screen of a control terminal shown in FIGS. 3( a ) and 3 ( b ) when performing tilt adjustment on a load carried by a movable device, according to an embodiment of the present disclosure. Schematic diagram. Fig. 7 shows a flow chart according to Fig. 6.

In a further embodiment of the present disclosure, as shown in FIG. 1, 6 and 7, the method further includes: manipulating the load carried by the mobile device, specifically: displaying and controlling the touch on the touch screen of the control terminal The base point 21 and the virtual operation area 22 are spaced apart from each other by the manipulation mode switching key 23; and by touching the switching key, the detection of the posture change of the control terminal is initiated; based on the detected posture change, the generated load is relatively a second command of the angle of rotation and speed of the pitch in the horizontal direction; and transmitting the second command to the movable device.

Specifically, as an example, detecting a change in posture of the control terminal is implemented by an attitude sensor built in the control terminal.

Specifically, as an example, the pitch angle of the load relative to the horizontal direction is between a negative threshold and a positive threshold Within the range of values. And, setting a pitch range of the specific angle range before the load reaches a forward threshold and a forward threshold, for example, approximately 5°, in which the load performs pitching at a preset rotational speed and angle, thereby The motion buffer before the threshold is reached and the termination of the pitch once the threshold is reached can be achieved gently. More specifically, the negative threshold is set to -90° and the forward threshold is set to +30°.

And, as an example, the attitude sensor includes an IMU (Inertial Measurement Unit), an acceleration sensor, an angular velocity sensor, a magnetometer or an attitude direction reference system, and the attitude sensor is, for example, embedded in the control terminal, such as Inside the smartphone.

In order to avoid the influence on the manipulation of the movable device when the manipulation of the load is activated, for example, the manipulation mode switching key 23 is displayed near the edge of the touch screen. Moreover, in order to ensure the reliability of the actuation of the load to be activated, for example, it is also arranged to determine that the activation is started when the pressure of the touch acting on the manipulation mode switching key 23 is detected to be higher than the preset second pressure threshold. The detecting the change of the attitude of the terminal.

Alternatively, for the sake of display, a hidden logic can be set, for example, if the touch point is manipulated into a specific main direction in each virtual operating area 22 or remains stationary, if The manipulation of the load is activated by manipulating the mode switching key 23, and the hidden logic is activated such that the interface displays information related to the manipulation of the load, and hides the touch base point and the virtual operation area associated with the movable device, To avoid visual interference. And, for example, by touching the manipulation mode switching key 23 again, the hidden logic can be turned off to stop the manipulation of the load.

As an example, the load includes a pan/tilt.

As an example, the movable device includes at least one of the following: an unmanned vehicle in a real environment or a virtual environment, a drone, an unmanned ship, a handheld head.

As a supplemental embodiment, for example, a plurality of dynamic icons or scales are additionally displayed on the interface to display parameters of the motion of the movable device and/or parameters of the load carried thereon such as pan/tilt.

Heretofore, a scheme for a method of manipulating a movable device according to an embodiment of the present disclosure has been described in detail with reference to the accompanying drawings.

The functional configuration of the apparatus for manipulating the movable device according to an embodiment of the present disclosure will be described in detail below in conjunction with FIG. 2(b).

2(b) shows a block diagram of an apparatus for manipulating a mobile device in accordance with an embodiment of the present disclosure. According to another aspect of the embodiments of the present disclosure, as shown in FIG. 2(b), FIG. 3(a), and FIG. 3(b), there is further provided an apparatus for manipulating a movable device, wherein the device The method includes: a display module configured to display at least one touch base point 21 on a touch screen of the control terminal; and an activation detection module configured to detect the touch a touch operation on the screen, and when the distance between the touch point of the touch operation and the touch base point is less than or equal to a preset threshold, the touch point acts as a touch start point, and the display module is instructed to be in the touch Expanding around the control starting point to generate a virtual operating area, such as at least two virtual operating areas 22 as shown; a data processing module configured to detect, for example, by the sensor from the touch starting point due to the touch point a displacement vector of the touch point relative to the touch start point generated by the sliding track in the virtual operation area, and generating an instruction to manipulate the movable device based on the displacement vector; and an instruction sending module configured The instructions are sent to the mobile device.

For example, when the activation detection module detects a touch operation on the touch screen, and when the distance between the touch point of the touch operation and the touch base point is less than or equal to a preset threshold, the display module is based on the touch base point. Generating a touchable area of a predetermined radius to the external radiation, and generating a touchable area of the predetermined radius based on the outwardly radiating the touch base point comprises: generating the touchable area that is opaque or translucent.

In the virtual operation area, for example, the magnitude of the shift of the user to the touch direction by the touch start point is, for example, proportional to the speed of the movable device in the corresponding motion mode/mode defined as corresponding to the touch direction. And optionally, two virtual operation areas, such as a left virtual operation area 221 and a right virtual operation area 222, as shown in FIG. 3(b), are respectively generated in the interface with the touch start point as a geometric center. Each of the illustrated virtual operational zones is shaped, for example, into a predetermined shape and size, such as a regular polygonal region, an elliptical region, or a circular region as shown. The specific shape of the virtual operating area can be determined, for example, based on the relationship between the movements of the movable device in the untouched touch direction.

In an embodiment of the present disclosure, as shown in FIGS. 4(a) and 4(b), for example, the display module is in each of the virtual operation areas 221 when manipulation of the movable device is activated. 222 defines a plurality of main directions along a radial direction centered on the touch start point (as shown, each of the virtual operation areas has four main directions indicated by broken lines), each of which The virtual operating area includes: a plurality of main direction areas and a plurality of composite direction areas, each of the plurality of composite areas being defined as an area between two adjacent main direction areas. As an example, as shown in FIG. 4(a) and FIG. 4(b), each of the plurality of main direction regions is defined as a sector region including a corresponding main direction, that is, a radial divergence, and the range is from The corresponding main direction is defined toward the two sides of the two sides, for example, as shown, there are four main direction areas 2211, 2213, 2215 and 2217 in the left virtual operation area 221, and the virtual operation area on the right side There are four main direction regions 2221, 2223, 2225, and 2227 in 222. Each of the plurality of composite regions is defined as an area between two adjacent main direction regions. For example, as shown, there are four composite direction regions 2212, 2214 in the left virtual operation region 221. There are four composite direction regions 2222, 2224, 2226 and P 2228 in 2216 and 2218, and in the right virtual operating region 222. As an example, when the data processing module detects that the touch point is located in the main direction area, the data processing module generates a manipulation of the movable device based on a component of the displacement vector corresponding to the corresponding main direction. And when the data processing module detects that the touch point is located in the composite direction area, the data processing module is respectively based on a corresponding main body of the displacement vector and the two adjacent main direction areas The component corresponding to the direction generates an instruction to manipulate the movable device.

In an embodiment of the present disclosure, as shown in FIG. 4(a) and FIG. 4(b), for example, the display module, in each of the virtual operation areas 22, sets the plurality of ΐ directions to include: Upward, downward, leftward, and rightward directions of the touch start point, and components corresponding to the plurality of main directions based on the displacement vector respectively generate a manipulation movable device to shift forward relative to a current state An instruction to pan backward, pan to the left, pan to the right, or generate an instruction to manipulate the movable device to rise, fall, counterclockwise, and clockwise with respect to the current state.

It should be noted that the up, down, left, and right directions mentioned in the present application are relative, and refer to the four directions of up, down, left, and right generated by the control terminal in the process of being used in the most common manner. As shown in Fig. 4(a) and Fig. b), the user can use the control terminal to view the real-time display of the interface on the screen in a conventional manner, and then determine the four directions of up, down, left, and right.

In an exemplary embodiment of the present disclosure, for example, when the data processing module detects that the touch point is located in the main direction area, that is, the displacement vector enters one of the plurality of main direction areas, that is, the left side virtual The plurality of main direction regions entered when one of the four main direction regions 2211, 2213, 2215, and 2217 in the operation area 221 or one of the four main direction regions 2221, 2223, 2225, and 2227 in the right virtual operation area 222 One is highlighted to facilitate visual recognition of upcoming operations in a single main direction. Specifically, for example, as shown in FIG. 4(a), when the user travels forward using the left virtual remote controller 221, even if the finger is not in front and is offset, it is determined that the command is directed to the single said positive Go ahead.

In an exemplary embodiment of the present disclosure, for example, as shown in FIG. 4(a) and FIG. 4(b), the central angle of the sector area is less than or equal to 11 °, for example, from the sector area The main directions are respectively deflected to the sides of the sector by angles of -5 ° and + 5 °, respectively. Still further, for example, the two side edges may be disposed to form different angles with respect to the corresponding main direction. More specifically, for example, the display module is capable of presetting the two sides of the sector area to be different according to the corresponding main direction according to a trend of a user's force application change with the relative sliding trajectory of the touch point. Angled to compensate for force feedback. Specifically, for example, the sector area includes an operation side half sector actually touched by a touch point divided by a covered main direction as a boundary line, and a non-operation side half sector not touched by the touch point, and the fan An angle between a side of the operation side half sector and the corresponding main direction is greater than another side of the sector on the non-operation side half sector and the corresponding main side The angle of the direction.

In an embodiment of the present disclosure, as shown in FIG. 5, for example, the data processing module is further configured to: (eg, at an initial time of a single predetermined time interval) obtain (eg, by a sensor) the movable device An initial state, such as an initial state of a velocity vector; (eg, at a termination time of a single predetermined time interval) obtaining a final segment state of the movable device, such as a terminal segment state of a velocity vector; and A change control of the initial state generates a compensation control command of the movable device to correct a movement state of the movable device.

In addition, in the embodiment of the present disclosure, in order to ensure that the touch activates the reliability of the touch base point, as an example, the activation detection module receives the touch screen at a pressure higher than the preset pressure. The first touch threshold determines the touch start point, thereby activating the touch base to generate a touchable area. In addition, in an embodiment of the present disclosure, for example, detecting a duration of a case where a distance between the touch point and the touch base point is less than or equal to a size of the touchable area, and when the duration is greater than the first predetermined time, Determining the touch start point. By setting the first predetermined time, and is intended to be distinguished from the touch operation to avoid erroneous operations. This ensures the reliability of the activation from both pressure and duration, preventing misoperations leading to undesired activation.

Also, in an embodiment of the present disclosure, for example, as shown in FIGS. 3(b) to 3(a), when the touch point is released for a second predetermined time, the activation detecting module instructs the display module to cause The virtual operation area disappears and the touch base point is redisplayed, and the operation of the detection module, the data processing module, and the instruction transmission module is stopped. At this time, the movable device is held, for example, to travel in a state (for example, speed, acceleration) before the termination of the manipulation of the movable device, or to maintain a position hovering at the timing of the manipulation termination, depending on the initial setting for the control terminal. .

Also, in an embodiment of the present disclosure, as shown in FIGS. 4(a) and 3(b), for example, the display module shapes each of the virtual operation areas 22 into the displacement vector representing a touch point. The boundary of the maximum value is defined by a transparent or translucent area. In an embodiment of the present disclosure, for example, the display module shapes each of the virtual operation regions 22 to have a radius centered on the touch start point and a radius of a predetermined maximum value of the displacement vector. Round shape.

In a further embodiment of the present disclosure, for example, as shown in Figures 4(a) and 3(b), the display module constructs the boundary mark as a hollow triangle, for example, as in Figure 4(a) a triangular area 2219 disposed adjacently outside the circumferential boundary of the left virtual operation area 221, or a triangular area 2229 disposed adjacently outside the circumferential boundary of the right virtual operation area 222, and the bottom side of the triangle is Constructed as an arc having a curvature that coincides with the circumference of the circular shape. And, when the touch point moves within the virtual operation area, the display The display module is instructed to cause the boundary mark to slip along the circumference with the arc edge to indicate the actual direction after the touch point is moved.

In an embodiment of the present disclosure, for example, as shown in FIG. 4(a) or 3(b), when the touch point approaches a boundary mark of the virtual operation area (for example, when the touch point enters close to the left The circumferential boundary of the side virtual operation area 221 is located in the respective critical sections 22110, 22130, 22150, 22170 of the main direction areas 2211, 2213, 2215, 2217, or enters the circumferential boundary close to the right virtual operation area 222 and is located in the main direction. When the touch points are close to the boundary mark of the virtual operation area, when the touch point is close to the boundary mark of the virtual operation area, the full-stick prompt is triggered, that is, in the control command at this time. The speed value reaches a preset maximum value; and the boundary mark is caused to be highlighted by the display module. For example, when the touch point approaches a boundary mark of the virtual operation area, the display module causes the boundary mark to become highlighted, and the boundary mark is highlighted by one or more of the following manners: It becomes filled with a specific color, by being changed to a specific pattern, by the expansion and contraction of the boundary mark, and by the shape of the boundary mark.

And, for example, when the display module displays on the touch screen that the movement of the touch point reaches the boundary mark, the data processing module maintains an instruction to manipulate the movable device, thereby, for example, causing the movable device to maintain a threshold response The speed operates steadily.

Further, for example, the display module further displays the image 1 on the touch screen, and the image is a dynamic and/or static image that can be used as a background acquired by the image capturing device carried by the movable device. . The user obtains the image 1 by using an image capturing device such as an fpv camera or a motion camera on the movable device, for example, and transmits it back to the control terminal, such as a mobile phone screen, and additionally displays the image 1 simultaneously and superimposedly for the touch operation. The interactive interface 2, for example, the interface 2 emerges above the image 1. For example, the interface 2 and the virtual operating area shown therein are more transparent than the image 1. Alternatively, the interface 2 is also formed, for example, as a region of a light colored background; and, further, the ground color of the interface 2 can adjust the color depth according to the overall color depth of the interface for manipulation. In this way, the images acquired in real time and the interface for interaction are simultaneously displayed, and the occlusion between each other is limited. In other words, it not only facilitates reducing the mutual influence of visual confusion or occlusion between the fpv return screen and the interface for manipulation; and can dynamically adjust the touch base point for interaction when shooting for a specific target and The positions of the virtual operating areas thus produced are used to reduce their occlusion of the captured image in a dynamic manner as efficiently as possible. In addition, by setting the boundary mark, the virtual operation area in a transparent or translucent state is easily perceived by the user as its approximate location.

The apparatus of an embodiment of the present disclosure is also utilized to effect manipulation of a load carried by a mobile device, such as a pan/tilt and a pod for mounting an image capture device, to adjust its pitch. Further embodiments of the present disclosure As shown in FIGS. 1, 6, and 7, the display module is configured to display a manipulation mode switching key 23 disposed apart from the touch base point and the virtual operation interval on a touch screen of the control terminal, And the activation detection module is arranged to activate manipulation of the load carried by the mobile device by initiating detection of a change in attitude of the control terminal by touching the switch key.

The data processing module is further configured to detect a change in posture of the control terminal 101 when the activation detection module detects activation of manipulation of the load, and generate a manipulation based on the detected posture change a second command that adjusts a rotational angle and speed of the load relative to the horizontal pitch; and the command transmitting module is further configured to transmit the second command to the mobile device.

For example, the data processing module detects a change in posture of the control terminal through an attitude sensor built in the control terminal.

Specifically, as an example, the pitch angle of the load with respect to the horizontal direction is within a range between a negative threshold and a forward threshold, and a pitch buffer of a specific angular range before the load reaches a forward threshold and a forward threshold is set. a zone (e.g., approximately 5 °) in which the load performs pitching at a predetermined rotational speed and angle, whereby motion buffering before reaching the threshold and pitching once the threshold is reached can be achieved smoothly termination. More specifically, the negative threshold is set to -90° and the forward threshold is set to +30°.

Also, as an example, the attitude sensor includes an IMU, an acceleration sensor, an angular velocity sensor, a magnetometer or an attitude direction reference system, and the illustrated attitude sensor is embedded, for example, in the control terminal such as a smartphone.

In order to avoid the influence on the manipulation of the movable device when the manipulation of the load is activated, for example, the display module displays the manipulation mode switching key 23 near the edge of the touch screen. Moreover, in order to ensure the reliability of the actuation of the load is activated, for example, the activation detection module determines to activate when detecting that the pressure of the touch acting on the manipulation mode switching key is higher than a preset second pressure threshold. Detection of a change in attitude of the control terminal.

The method and apparatus for manipulating a movable device and a load carried thereon by the embodiments of the present disclosure can enhance positive feedback and reduce erroneous operation by defining a virtual rocker mode. Specifically, the solution adopts a method of performing area division on the inside of the virtual operation area, and defines a main direction area in a narrow radiation range substantially along a plurality of specific main directions, and the touch in the main direction area is regarded as an indication The manipulation of the single direction of the movable device, thereby reducing the probability of misoperation of its control. And also by adding a somatosensory way to control such as the gimbal Such a load mounted on the movable device enables the Pitch of the load such as the pan/tilt to be adjusted to achieve composition control during the manipulation of the movable device by operating the virtual operation area with both hands. Thereby, the shooting of the full trajectory and the angle of view of the object can be achieved by a combination of both manipulation of the movable device and manipulation of the load.

As an example, the load includes a pan/tilt.

It should be noted that those skilled in the art can understand all or part of the processes in the foregoing embodiments, such as at least the foregoing activation detection step, data processing step, and instruction transmission step, which can be programmed by the relevant hardware. Upon completion, the program can be stored in a computer readable storage medium, which, when executed, can include the flow of an embodiment of the methods described above. The storage medium is, for example, a magnetic disk, an optical disk, a hard disk drive, a flash memory, a read-only memory (ROM), or a random access memory (RAM).

In addition, the functions described herein as being implemented by the steps of the method may also be implemented by means of dedicated hardware, a combination of general hardware and software, and the like. For example, functions described as being implemented by dedicated hardware (eg, Field Programmable Gate Array (FPGA), Application Specific Integrated Circuit (ASIC), etc.) may be implemented by general purpose hardware (eg, central processing unit (CPU), microprocessor (μ Ρ), digital signal processor (DSP) and software are combined to achieve, and vice versa. In addition, for example, it is described as a function implemented by a WiFi chip, a Bluetooth module, an FC chip/coil, etc., and a general-purpose processor (for example, a CPU, a DSP, etc.) may be combined with an analog-to-digital conversion circuit, an amplifying circuit, an antenna, and the like, and Bluetooth. NFC, WiFi related processing software to achieve, and vice versa. In the following detailed description, numerous specific details are set forth Obviously, however, one or more embodiments may be practiced without these specific details.

The specific embodiments of the present invention have been described in detail with reference to the specific embodiments of the present application. It is to be understood that the foregoing description Any modifications, equivalent substitutions, improvements, etc., made within the spirit and scope of the present application are intended to be included within the scope of the present application.

Claims

Claim

A method for operating a mobile device, wherein the method comprises the steps of: displaying at least one touch base point on a touch screen of the control terminal;

When the touch operation on the touch screen is detected, and the distance between the touch point of the touch operation and the touch base point is less than or equal to a preset threshold, the touch point acts as a touch start point, and A virtual operation area is generated around the touch start point;

Generating, by the touch point, a sliding track in the virtual operation area from the touch start point, generating a displacement vector of the touch point relative to the touch start point;

Detecting the displacement vector and generating an instruction to manipulate the movable device based on the displacement vector; and transmitting the instruction to the mobile device.

The method according to claim 1, wherein when a touch operation on the touch screen is detected, and a distance between a touch point of the touch operation and a touch base point is less than or equal to a preset threshold, based on The touch base point radiates outward to generate a touchable area of a predetermined radius.

The method according to claim 2, wherein the touchable area for generating the preset radius based on the outward touch of the touch base point comprises:

The touchable area that is opaque or translucent is generated.

4. The method according to claim 1, wherein generating a virtual operation area around the touch start point comprises: generating two virtual operation areas.

5. The method according to claim 1, wherein in each of the virtual operation areas, a plurality of main directions are defined in a radial direction centered on the touch start point, each of the virtual operations The area includes:

Multiple main direction areas; and

A plurality of composite direction regions, each of the plurality of composite regions being defined as a region between two adjacent main direction regions.

6. The method of claim 5, wherein each of the plurality of main direction regions is defined as a sector region radiating from the touch start point and covering a corresponding main direction.

7. The method according to claim 5, wherein when the touch point is located in the main direction area, an instruction to manipulate the movable device is generated based on a component of the displacement vector corresponding to the corresponding main direction; And when the touch point is located in the composite direction area, an instruction to manipulate the movable device is generated based on the separation of the displacement vectors corresponding to the respective pupil directions of the two adjacent pupil direction regions, respectively.

8. The method according to claim 5, wherein, in each of the virtual operation areas, the plurality of main directions comprise: upward, downward, leftward, rightward relative to the touch start point a direction, and a component corresponding to the plurality of main directions based on the displacement vector respectively generates a control instruction for manipulating the movable device to move forward, backward, left, and right relative to the current state, or generate A control command that manipulates the movable device to rise, fall, counterclockwise, and clockwise with respect to the current state.

9. The method according to claim 7, wherein the main direction area is highlighted when the touch point is located in the main direction area.

10. The method of claim 6 wherein the central angle of the sector is less than or equal to 1 Γ.

11. The method according to claim 10, wherein two sides of the sector area are preset to be different with respect to the corresponding main direction according to a trend of user application force with a sliding trajectory of the touch point Angle.

12. The method according to claim 11, wherein the sector area includes an operation side half sector actually touched by a touch point divided by a covered main direction as a boundary line, and a non-operation side touched by the touch point a half sector, and an angle of a side of the sector on the side of the operation side with the corresponding main direction is greater than an other side of the sector on a side of the non-operation side half sector An angle with the corresponding main direction.

The method according to any one of claims 1 to 12, wherein the generating, by the displacement vector, an instruction to manipulate the movable device comprises:

Obtaining an initial state of the movable device; Obtaining a final state of the movable device; and

A compensation control command of the movable device is generated according to the change of the last segment state with respect to the initial state to correct a moving state of the movable device.

The method according to claim 1, wherein, in a case where the distance between the touch point of the touch operation and the touch base point is less than or equal to a preset threshold, the touch screen is received in response to detecting the touch point The touch start point is determined when the pressure is higher than the preset first pressure threshold.

The method according to claim 1 or 14, wherein a duration of a case where a distance between a touch point of the touch operation and a touch base point is less than or equal to a preset threshold is detected, and when the duration is greater than the first At the predetermined time, the touch start point is determined.

16. The method according to claim 1, wherein when the touch point is released, the virtual operation area disappears and the touch base point is redisplayed, and the instruction to manipulate the movable device is terminated.

17. The method of claim 1, wherein each of the virtual operating regions is shaped as a transparent or translucent region defined by a boundary mark representing a maximum value of the displacement vector.

18. The method according to claim 17, wherein each of the virtual operation areas is shaped into a circular shape having a center of the touch start point and a radius of a predetermined maximum value of the displacement vector .

19. The method of claim 18, wherein the boundary mark is a hollow triangle, and a bottom edge of the triangle is configured to have an arc edge having a curvature that coincides with a circumference of the circular shape.

The method according to claim 19, wherein, when the touch point moves in the virtual operation area, the boundary mark slides along the circumference with the arc edge to indicate that the touch point is moved. Actual direction.

21. The method according to claim 20, wherein the speed value in the control command reaches a preset maximum value when the touch point approaches a boundary mark of the virtual operation area.

22. The method according to claim 21, wherein when the touch point approaches a boundary mark of the virtual operation area, the boundary mark becomes highlighted, and the boundary mark passes one or more of the following manners To highlight: by becoming a specific color fill, by becoming a specific pattern, by the expansion and contraction of the boundary mark, and by the shape of the boundary mark.

The method of claim 17, wherein the instruction to manipulate the movable device is maintained when the touch point moves to reach the boundary mark.

24. The method according to claim 17, wherein the image on the touch screen of the control terminal is also displayed as a background.

The method according to claim 24, wherein the image is a dynamic and/or static image acquired by an image capturing device carried by the mobile device.

26. The method of claim 24, wherein the virtual operating area has a higher transparency than the image.

27. The method of claim 1, further comprising manipulating a load carried by the mobile device, comprising: displaying on the touch screen of the control terminal spaced apart from the touch base point and the virtual operation interval Switching a key, and by detecting the switching key, initiating detection of a change in posture of the control terminal;

Generating a second command of the rotational angle and speed of the load relative to the horizontal pitch based on the detected change in attitude; and

Sending the second instruction to the mobile device.

28. The method according to claim 27, wherein detecting a change in posture of the control terminal is implemented by an attitude sensor built in the control terminal.

29. The method according to claim 27, wherein a pitch angle of the load with respect to a horizontal direction is within a range between a negative threshold and a forward threshold, and before the load is set to reach a forward threshold and a forward threshold of A pitch buffer of a particular range of angles within which the load performs pitch motion at a predetermined rotational speed and angle.

30. The method of claim 29, wherein the negative threshold is set to -90[deg.] and the forward threshold is set to +30[deg.].

31. The method according to claim 28, wherein the attitude sensor comprises an inertial halo unit, an acceleration sensor, an angular velocity sensor, a magnetometer or an attitude direction reference system.

32. The method of claim 27, wherein the switch key is displayed adjacent an edge of the touch screen.

33. The method according to claim 27, further comprising determining to initiate a posture change for the control terminal when detecting that a pressure of a touch applied to the switching key is higher than a preset second pressure threshold Check

The method according to any one of claims 27 to 33, wherein the load comprises a pan/tilt.

35. The method according to claim 1, wherein the movable device comprises at least an unmanned vehicle, a drone, an unmanned ship, a handheld head in a real environment or a virtual environment.

36. An apparatus for manipulating a mobile device, wherein the device comprises:

The display module is configured to display at least one touch base point on the touch screen of the control terminal;

An activation detection module configured to detect a touch operation on the touch screen, and when the distance of the touch point of the touch operation and the touch base point is less than or equal to a preset threshold, act as the touch point Touching a starting point, and instructing the display module to display a virtual operating area generated by expanding outward around the touch starting point; the data processing module is configured to detect from the touch starting point by the touch point Generating a displacement vector in the virtual operation area to generate a displacement vector of the touch point with respect to a touch start point, and generating an instruction to manipulate the movable device based on the displacement vector; and

An instruction transmitting module configured to send the instruction to a mobile device.

The device according to claim 36, wherein when a touch operation on the touch screen is detected, and when a distance between the touch point of the touch operation and the touch base point is less than or equal to a preset threshold, Generating a touchable area of a predetermined radius based on the outward touch of the touch base point.

38. The device according to claim 37, wherein the touchable area for generating a preset radius based on the outward touch of the touch base point comprises:

The touchable area that is opaque or translucent is generated.

39. The device according to claim 36, wherein two virtual operation areas are generated around the touch start point.

40. The device according to claim 36, wherein the display module defines a plurality of main directions along a radial direction centered on the touch start point in each of the virtual operation areas, each The virtual operating area includes - a plurality of main direction areas; and

41. The apparatus according to claim 40, wherein each of the plurality of main direction regions is defined as a sector region radiating from the touch start point and covering a corresponding main direction.

42. The method according to claim 40, wherein, when the data processing module detects that the touch point is located in the main direction area, the data processing module is based on the displacement vector and the corresponding main The component corresponding to the direction generates an instruction to manipulate the movable device;

And when the data processing module detects that the touch point is located in the composite direction area, the data processing module respectively corresponds to a corresponding main direction of each of the two adjacent main direction areas based on the displacement vector The component is used to generate an instruction to manipulate the mobile device.

43. The device according to claim 40, wherein the display module sets the plurality of main directions in each of the virtual operation areas to include: upward and downward with respect to the touch start point , turn left turn Right a direction, and the data processing module generates, respectively, based on the components of the displacement vector corresponding to the plurality of main directions, respectively, the manipulation movable device is translated forward, backward, left, and right relative to the current state. Controlling instructions, or generating control commands that manipulate the movable device to rise, fall, counterclockwise, and clockwise with respect to the current state.

44. The apparatus according to claim 42, wherein, when the data processing module detects that the touch point is located in the ΐ direction area, the ΐ direction area is highlighted.

45. Apparatus according to claim 41 wherein the central angle of the sector is less than or equal to 1 Γ.

The device according to claim 45, wherein the display module presets two sides of the sector-shaped area to be opposite to the corresponding one according to a trend of a user's applied force with a sliding trajectory of the touched point The main direction is at a different angle.

47. The method according to claim 46, wherein the sector area includes an operation side half sector actually touched by a touch point divided by a covered main direction as a boundary line, and a non-operation side touched by the touch point a half sector, and an angle of a side of the sector on the side of the operation side with the corresponding main direction is greater than an other side of the sector on a side of the non-operation side half sector An angle with the corresponding main direction.

The apparatus according to any one of claims 36 to 47, wherein the data processing module is further configured to:

Obtaining an initial state of the movable device;

Obtaining a final state of the movable device; and

The device according to claim 36, wherein, in a case where the distance between the touch point of the touch operation and the touch base point is less than or equal to the size of the touchable area, the activation detection module is responsive to detecting The touch start point is determined when the pressure at the touch point is higher than a preset first pressure threshold.

The device according to claim 36 or claim 49, wherein the activation detecting module detects a duration of a case where a distance between the touch point and the touch base point is less than or equal to a size of the touchable area, and when The touch start point is determined when the duration is greater than the first predetermined time.

51. The device according to claim 36, wherein, when the touch point is released, the activation detecting module instructs the display module to cause the virtual operation area to disappear and the touch base point to be redisplayed, and activated The operation of the detection module, the data processing module, and the instruction transmission module stops.

52. The apparatus of claim 36, wherein the display module shapes each of the virtual operating regions into a transparent or translucent region defined by a boundary mark representing a maximum value of the displacement vector.

The device according to claim 52, wherein the display module shapes each of the virtual operation regions to have a radius of a predetermined starting point of the displacement vector and a radius of the displacement vector Round shape.

54. The apparatus according to claim 53, wherein the display module configures the boundary mark into a hollow triangle, and a bottom edge of the triangle is configured to have a curvature consistent with a circumference of the circular shape The arc side.

55. The apparatus according to claim 54, wherein, when the touch point moves within the virtual operation area, the display module causes the boundary mark to slide along the circumference into an indication position with the arc edge The actual direction after the touch point is moved.

56. The apparatus according to claim 55, wherein, when the touch point approaches a boundary mark of the virtual operation area, a speed value in the control command reaches a preset maximum value.

57. The apparatus according to claim 56, wherein, when the touch point approaches a boundary mark of the virtual operation area, the display module causes a boundary mark to become highlighted, and the display module causes the boundary The mark is highlighted by one or more of the following ways: by becoming a fill in a particular color, by becoming a fill in a particular pattern, by a change in the stretch of the boundary mark, and by a change in the shape of the mark.

58. The device according to claim 52, wherein the data processing module maintains an instruction to manipulate the movable device when the display module displays on the touch screen that the touch point moves to reach a boundary mark.

59. The device according to claim 52, wherein the display module is further configured to display an image as a background on a touch screen of the control terminal.

60. The device of claim 59, wherein the image is a dynamic and/or static image acquired by an image capture device carried by the mobile device.

61. The apparatus according to claim 59, wherein the virtual operation area has a higher transparency than the image.

62. The device according to claim 36, wherein the device is further configured to manipulate a load carried by the mobile device, comprising: the display module is further configured to display on a touch screen of the control terminal a switching button disposed apart from the touch base point and the virtual operation interval,

The activation detection module is further configured to activate manipulation of a load carried by the mobile device by initiating detection of a change in attitude of the control terminal by touching the switch key,

The data processing module is further configured to detect a change in posture of the control terminal when the activation detection module activates manipulation of the load, and generate a manipulation based on the detected posture change relative to the load adjustment a second command for the angle of rotation and speed of the pitch in the horizontal direction; and

The instruction transmitting module is further configured to transmit the second instruction to the mobile device.

63. The apparatus according to claim 62, wherein the data processing module detects a posture change of the control terminal through an attitude sensor built in the control terminal.

64. The apparatus according to claim 62, wherein a pitch angle of the load with respect to a horizontal direction is within a range between a negative threshold and a forward threshold, and before the load is set to reach a forward threshold and a forward threshold a pitch buffer of a specific angular range in which the load is at a predetermined rotational speed and angle Perform a pitch motion.

65. Apparatus according to claim 64, wherein said negative threshold is set to -90[deg.] and the forward threshold is set to +30[deg.].

66. Apparatus according to claim 63, wherein said attitude sensor comprises an inertial measurement unit, an acceleration sensor, an angular velocity sensor, a magnetometer or an attitude direction reference system.

67. The device according to claim 62, wherein the display module displays the switching key near an edge of the touch screen.

The device according to claim 62, wherein the activation detecting module determines to activate the control terminal when detecting that the pressure of the touch applied to the switching key is higher than a preset second pressure threshold Detection of posture changes.

The apparatus according to any one of claims 62 to 68, wherein the load comprises a pan/tilt.

70. The device according to claim 36, wherein the movable device comprises at least one of the following: an unmanned vehicle, a drone, an unmanned ship, a handheld head in a real environment or a virtual environment.