JP5974668B2 - Manipulation system - Google Patents

Description

  The present invention relates to a master-slave type manipulation system.

  In the field of robotics and virtual reality, various studies have been conducted on master-slave manipulation systems that incorporate humans into the system. For example, there is research related to tele-existence in which the master side operator's technical operations are transmitted to the slave side robot arm and the slave unit is operated with a high degree of realism as if the operator is in a remote place (for example, Non-patent document 1).

  In a conventional master-slave type manipulation system, a master arm reads an operator's intention to move as an arm displacement. The operator's intention to move is the displacement of the arm after transmitting the nerve, contracting the muscles and generating joint torque. Therefore, a time lag occurs before the operator's intention to move is transmitted to the master arm, and it cannot be adapted to an operation that is not output as a displacement. As a result, the operator cannot move the slave arm in the same way as his / her arm, and there is a problem that the operation is uncomfortable.

  In addition, information transmission to an operator in a master-slave type manipulation system requires not only feedback to somatic sensations such as haptic presentation, but also accurate visual feedback to the vision that occupies most of space recognition.

S.Tachi, T.Sakai, “Impedance controlled master-slave manipulation system. Part 1. Basic concept and application to the system with a time delay”, Advanced Robotics, Vol.6, No.4, 1992, pp483-503

  The present invention has been made in consideration of such points, and an object of the present invention is to provide a manipulation system having improved operability by performing operation assistance based on visual information.

  A manipulation system according to an aspect of the present invention includes a master arm device worn by a wearer, a slave arm device, an imaging device that photographs the slave arm device, and a display device that displays an image captured by the imaging device. The master arm device comprises: a biopotential sensor that detects a biopotential signal associated with muscle activity corresponding to movements of the shoulder joint and elbow joint of the wearer; and the shoulder joint of the wearer And a first angle sensor for detecting the angle of the elbow joint, a first drive source for applying power to the wearer, the drive source based on the bioelectric potential signal, and the first angle sensor A first control unit that calculates position information of the master arm from the detection result; and an impedancer based on the position information of the master arm and the biopotential signal A first communication unit that transmits information on the slave arm, the slave arm device detecting a force applied to the slave arm, a second drive source for operating the slave arm, and the slave arm A second angle sensor for detecting a joint angle; and the second drive source is controlled based on the position information received from the first communication unit, the impedance information, and a detection result of the force sensor, and the second A second control unit that calculates the position information of the slave arm from the detection result of the angle sensor; a second communication unit that transmits the position information of the slave arm and the detection result of the force sensor to the first communication unit; The first control unit controls the first drive source based on the detection result of the force sensor received from the second communication unit, and the imaging device includes a plurality of lenses. Is Okamera.

  In the manipulation system according to an aspect of the present invention, it is preferable that the imaging device controls the orientations of the plurality of lenses based on position information of the slave arm.

  In the manipulation system according to an aspect of the present invention, the manipulation system further includes a head sensor that measures the movement of the head of the wearer, and the imaging device includes a pitch angle, a roll angle, It is preferable to control the posture by adjusting the yaw angle.

  In the manipulation system according to the aspect of the present invention, the manipulation system further includes an input interface that receives information on an entry prohibition area that prohibits the entry of the slave arm. It is preferable to control the first drive source so as not to enter the entry prohibition region.

  In the manipulation system according to one aspect of the present invention, it is preferable that the display displays the entry prohibition area so as to overlap the video imaged by the imaging device.

  According to the present invention, the operability of the manipulation system can be improved by dynamically changing the impedance of the robot arm using the biopotential information of the operator and performing the operation assist by the visual information.

It is an external appearance perspective view of the manipulation system concerning this embodiment. It is a schematic block diagram of the master arm apparatus concerning the embodiment. It is a schematic block diagram of the slave arm apparatus concerning the embodiment. It is a schematic block diagram of the imaging device concerning the embodiment. It is a figure which shows an example of the approach prohibition area | region displayed on a display. It is a block block diagram of the manipulation system which concerns on the same embodiment. It is a block diagram in the bilateral control method which introduced the bioelectric potential variable impedance control by the same embodiment.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings.

  FIG. 1 is an external perspective view of a manipulation system according to the present embodiment. This manipulation system is a master-slave type manipulation system, a master arm device 1 that an operator (wearer) 0 wears and operates, and a slave arm device that is provided at a location away from the wearer 0 and that actually performs work. 2 and the imaging device 3. The master arm device 1 and the slave arm device 2 communicate bidirectionally so that the master arm operation of the wearer 0 is reflected in the operation of the slave arm. In addition, the imaging device 3 is provided on the slave arm device 2 side, and an image taken by the imaging device 3 is presented to the wearer 0 by the display 4. The display 4 may be anything as long as it can present an image to the wearer 0. For example, a head-mounted display can be used as shown in FIG.

  FIG. 2 shows a state where the master arm device 1 is worn by the wearer 0. In FIG. 2, the upper right half of the master arm device 1 corresponding to the right arm portion of the wearer 0 is shown. The master arm device 1 has the same configuration in the upper left half as in the upper right half.

  As shown in FIG. 2, the master arm device 1 includes a trunk member 131, an upper arm member 132, and a forearm member 133. The trunk member 131 is attached to the body of the wearer 0. The upper arm member 132 extends along the upper arm portion of the wearer 0 and is attached to the upper arm portion of the wearer 0. The forearm member 133 extends along the forearm portion of the wearer 0 and is worn on the forearm portion of the wearer 0.

  A first shoulder joint mechanism 134 is provided between the trunk member 131 and the upper arm member 132, and the trunk member 131 and the upper arm member 132 are rotatably connected to each other via the first shoulder joint mechanism 134. ing. Further, an elbow joint mechanism 135 is provided between the upper arm member 132 and the forearm member 133, and the upper arm member 132 and the forearm member 133 are connected to each other via the elbow joint mechanism 135 so as to be rotatable. Although not shown in FIG. 2 because it is covered with the trunk member 131, a second shoulder joint mechanism is provided on the shoulder portion of the wearer 0. The first shoulder joint mechanism 134 rotates along the vertical direction, while the second shoulder joint mechanism rotates along the horizontal direction. Thus, the master arm device 1 has a shoulder joint 2 degrees of freedom and an elbow joint 1 degrees of freedom.

  As shown in FIG. 2, the master arm device 1 is provided with a bending side biopotential sensor 136 and an extension side biopotential sensor 137. The upper arm member 132 is disposed along the humerus, and is fixed to the upper arm portion with bands or the like provided at two locations near the elbow joint and on the far side. The forearm member 133 is disposed along the ulna or the rib, and is fixed to the forearm with bands or the like provided at two locations near the elbow joint and far from the elbow joint.

  The drive unit 138 that connects the upper arm member 132 and the forearm member 133 includes an angle sensor, an actuator (drive source), and a torque sensor. As this torque sensor, for example, a sensor that detects a drive torque by detecting a current value supplied to the actuator and multiplying this current value by a torque constant inherent to the actuator can be applied. A rotary encoder or the like can be applied to the angle sensor.

  As shown in FIG. 2, the bending-side bioelectric potential sensor 136 is affixed to the surface of the body corresponding to the biceps brachii and humerus muscles that mainly work as voluntary muscles during the elbow joint bending operation, and detects the myoelectric potential. Further, as shown in FIG. 2, the extension-side bioelectric potential sensor 137 is attached to the body surface corresponding to the triceps surae mainly acting as a voluntary muscle during the extension operation of the elbow joint, and detects the myoelectric potential.

  The drive unit 138 described above is also provided in the first shoulder joint mechanism 134 and the second joint mechanism. In addition, the master arm device 1 is a bioelectric potential sensor (see FIG. 5) that is attached to the body surface corresponding to the deltoid and greater pectoralis muscles that mainly act as voluntary muscles during shoulder joint flexion / extension motion, abduction / addition motion Not shown).

  In addition, the master arm device 1 has a control unit 119 that is worn on the back of the wearer 0, and stores devices such as a host controller, a motor driver, a measuring device, a power supply circuit, and a communication device. The host controller can be operated from an external computer using TCP / IP. Information measured by the biopotential sensor is digitized by the A / D converter of the measuring device and sent to the host controller. The host controller determines the output of the drive unit 138 based on the sent information, and drives the drive unit 138. The communication device can communicate with the slave arm device 2 using TCP / IP. The communication device may be able to communicate with the display 4.

  FIG. 3 shows a schematic configuration of the slave arm device 2. In this embodiment, a 7-axis robot arm is used as the slave arm device 2, but the robot arm applicable to the slave arm device 2 is not limited to this. The slave arm device 2 is provided with eight actuators, the first to third actuators perform a turning operation, and the fourth to seventh actuators perform a bending / extending operation. The eighth actuator opens and closes the gripper gripper 201 attached to the arm tip of the slave arm device 2. Each joint of the slave arm device 2 is provided with a joint angle sensor so that the joint angle, angular velocity, and angular acceleration of each joint can be acquired. A 6-axis force sensor 202 is provided at the tip of the arm so that translational and rotational force applied to the tip can be detected.

  The slave arm device 2 includes a control unit (not shown). The control unit acquires the detection result of the joint angle sensor and the detection result of the six-axis force sensor. The control unit controls the operation of the slave arm device 2 by controlling the first to eighth actuators. Further, the control unit has a communication device and can perform data communication with the master arm device 1.

  Next, the communication method and the content of communication data between the master arm device 1 and the slave arm device 2 will be described. In this embodiment, TCP / IP is used as a protocol. TCP / IP has two types of transport protocols, TCP and UDP. In communication using TCP, transmission and response confirmation are performed for each packet. Therefore, communication can be temporarily stopped due to loss of a transmission packet or a response confirmation packet. In particular, retransmission processing may be performed a plurality of times for data with a large size. During this time, data does not reach and control is stopped. The data transfer cycle is limited in the time required for the packet to reciprocate on the network.

  On the other hand, in UDP, the transmitting side simply transmits data, and the receiving side only receives the received data. That is, since response confirmation is not performed, packet loss and data order are not guaranteed, and the transfer time on the network is not limited. Since communication using UDP can shorten the transfer stop period compared to communication using TCP, in this embodiment, communication is performed using UDP.

  The communication data array in the master-slave type manipulation system according to the present embodiment is as follows.

{Synchronization data; X movement amount; Y movement amount; Z movement amount; X rotation amount; Y rotation amount; Z rotation amount; gripper open / close data; i impedance data; error check calculation data}
The X movement amount, the Y movement amount, and the Z movement amount indicate the movement amounts in the X axis direction, the Y axis direction, and the Z axis direction, respectively. Further, the X rotation amount, the Y rotation amount, and the Z rotation amount indicate rotation amounts in the X axis direction, the Y axis direction, and the Z axis direction. The gripper opening / closing data corresponds to the opening / closing operation of the gripper 201 for gripping operation.

  In this embodiment, the arm is controlled using hand position coordinate data instead of joint angle data. If the master arm device 1 and the slave arm device 2 are similar (same degrees of freedom), each joint has a one-to-one correspondence. However, when the types of the master arm device 1 and the slave arm device 2 are different (the degrees of freedom are different), it is difficult to associate each joint. Therefore, in this embodiment, the arm is controlled using the hand position coordinate data. Thus, the slave arm can be freely selected according to the work, and the arm can be controlled even when the degree of freedom of the slave arm changes.

  The hand position coordinate data is not data in absolute coordinates but data on the hand position movement amount. The data to be transmitted from the master arm device 1 to the slave arm device 2 is not absolute coordinate value data, but the error amount cannot be processed on the receiving side due to data loss or the like by using the movement amount of the hand position as transmission data. However, since the amount of movement is not large, abnormal operation of the slave arm device 2 can be prevented. Also, by changing the ratio of the movement amount correspondence relationship between the master arm device 1 and the slave arm device 2 to 1: 1 or 2: 1 instead of 1: 1, it is flexible even for work that involves fine work or large movements. It is possible to ensure operability.

  FIG. 4 shows a schematic configuration of the imaging device 3. The imaging device 3 is a stereo camera including a plurality (two in this example) of lenses 301 and 302 that are arranged at a predetermined interval. The imaging device 3 simultaneously captures the work space with the lenses 301 and 302, and recognizes the position and three-dimensional shape of the object from the difference (parallax) in the position of the object captured by each lens on the image. To get depth information. The left-side image of the double image shifted to the left and right is made visible only to the left eye of the wearer 0, and the right-side image is made visible only to the right eye of the wearer 0, so 3D images are presented.

  As shown in FIG. 4, the imaging device 3 adjusts the roll angle (angle of rotational movement around the x axis), pitch angle (up and down swing angle around the y axis), and yaw angle (left and right swing angle around the z axis). And control the posture. The posture of the imaging device 3 is controlled based on the detection result of the movement and posture of the head of the wearer 0. For example, by mounting a head sensor incorporating a gyro sensor on the head of the wearer 0, the movement and posture of the head of the wearer 0 are detected. The head sensor may be provided in the head mounted display 4 shown in FIG.

  Further, the imaging device 3 can control the convergence angle (position of the convergence point) by adjusting the optical axis direction of the lenses 301 and 302. The convergence angle is dynamically adjusted according to the distance from the imaging device 3 to the gripping operation gripper 201 of the slave arm device 2 so that the focal position is aligned with the gripping operation gripper 201.

  In the manipulation system according to the present embodiment, in order to prevent the slave arm device 2 from colliding with a work target or the like, the virtual arm acting on the real space is indicated using the OSD technology and the AR technology, and the slave arm The operating range of the device 2 is limited. For example, an entry prohibition area that the slave arm device 2 should not enter is input using an input interface (not shown). As shown in FIG. 5, the information on the entry prohibition area can be presented on the display in a manner superimposed on the image captured by the imaging device 3.

  FIG. 6 is a block configuration diagram of the manipulation system according to the present embodiment. The biopotential sensor 11 of the master arm device 1 corresponds to the biopotential sensors 136 and 137 in FIG. 2, and the angle sensor 12 and the actuator 13 are provided in the drive unit 138 in FIG. Further, the controller 14 and the communication device 15 are provided in the control unit 119 of FIG.

  The force sensor of the slave arm device 21 corresponds to the 6-axis force sensor 202 of FIG. The joint angle sensor 22 is provided at each joint of the arm of the slave arm device 2 shown in FIG. The actuator 23 performs the turning operation of the arm of the slave arm device 2 shown in FIG. 3, the bending / extending operation, and the opening / closing operation of the gripper 201 for gripping operation. The controller 24 and the communication device 25 are provided in the control unit of the slave arm device 2 described above.

  The arm of the wearer 0 dynamically changes impedance depending on physical condition, environment, and muscle tension. Therefore, in order to move the robot arm like its own arm, it is necessary to dynamically change the impedance of the robot arm. In this embodiment, paying attention to the bioelectric potential information of the wearer 0, biopotential variable impedance control is performed in which the intuitive action of the wearer 0 can be arbitrarily expressed as the impedance of the robot.

In the bioelectric potential variable impedance control, paying attention to the bioelectric potential information of the wearer 0, the virtual impedance of the arm stiffness is calculated so that the characteristics of the robot arm can be controlled arbitrarily. The equation at the hand position in the impedance control is expressed by the following Equation 1.

  Here, F is an external force, M is an inertia term, D is a viscosity term, and K is an impedance term such as a stiffness term. P represents a position, and Δt represents a sampling time.

Equation 2 below shows a biopotential variable impedance control equation in which the stiffness term of Equation 1 is changed to that based on biopotential information.

The bio-variable stiffness matrix K _BES is calculated by the following formula 3.

In Equation 3, F _{BES0 to} F _BES2 are biopotential information, k _{0 to} k ₂ are adjustment gains, and K _{0 to} K ₂ are offset adjustment terms. In the present embodiment, as shown in FIG. 2, the bioelectric potential information at the three joints (shoulder 2 and elbow 1) is used, and thus the biovariable stiffness matrix K _BES is a 3 × 3 matrix. Further, the biopotential information F _BES is obtained by the following mathematical formula 4.

In Equation 4, w is a balance gain, E _f is a biopotential signal at the time of bending, and E _e is a biopotential signal at the time of extension.

  The operation of the master-slave manipulation system incorporating such biopotential variable impedance control will be described with reference to FIG.

  Images of the slave arm device 2 and the object captured by the imaging device 3 are presented on the display 4. The posture of the imaging device 3 is controlled in accordance with the movement of the head of the wearer 0 measured by the head sensor 6. The measurement result of the head sensor 6 may be transmitted to the imaging device 3 by the communication device 15 of the master arm device 1 or may be transmitted to the imaging device 3 by another communication device. The imaging device 3 controls the posture by adjusting the roll angle, the pitch angle, and the yaw angle based on the notified measurement result of the head sensor 6. Information (for example, coordinate information) of the entry prohibition area of the slave arm device 2 is given to the controller 14 via the input interface 5. The controller 14 presents the entry prohibition area on the display 4 so as to overlap the captured image of the imaging device 3. The wearer 0 looks at the video presented on the display 4 and starts to move the arm or the like in order to move the slave arm device 2 in a desired direction.

The biopotential sensor 11 of the master arm device 1 detects the biopotential signal of the wearer 0 and passes it to the controller 14. The controller 14 acquires joint torque information from the received biopotential signal (estimates the joint torque), and controls the actuator 13 to operate the master arm device 1. The angle sensor 12 detects the displacement of each joint of the arm of the master arm device 1 due to this operation. The controller 14 calculates arm hand position information (movement amount of the hand position) from the displacement of each joint. Further, the controller 14 calculates a bio-variable stiffness matrix K _BES from the bio-potential signal based on Expressions 3 and 4. The communication device 15 transmits the calculated hand position information and impedance information (biologically variable stiffness matrix K _BES or the like) to the communication device 25 of the slave arm device 2.

  The communication device 25 passes the hand position information and impedance information received from the communication device 15 to the controller 24. The controller 24 applies the received impedance information to the slave arm device 2, controls the actuator 23 based on the movement amount of the hand position and the force information obtained by the force sensor 21, and follows the hand position of the slave arm. I do. The controller 24 calculates the hand position information (movement amount of the hand position) of the slave arm from the detection result of the joint angle sensor 22. Then, the communication device 25 transmits force information and hand position information (a movement amount of the hand position) to the communication device 15 of the master arm device 1.

  The controller 14 of the master arm device 1 converts the hand force into each joint torque using the Jacobian from the force information received via the communication device 15, and further controls the actuator 13 using the position deviation to Force feedback to zero. Moreover, the actuator 13 is controlled from the hand position information of the slave arm so that the slave arm device 2 does not enter the set entry prohibited area, and the arm position of the wearer 0 is adjusted. Further, in this manipulation system, feedback control of the absolute coordinates of the hand position of the arm is performed by visual feedback using the imaging device 3 and the display 4.

  The communication device 15 notifies the imaging device 3 of the hand position information of the slave arm. The imaging device 3 adjusts the convergence angle based on the notified hand position information of the slave arm.

  FIG. 7 is a block diagram in the bilateral control method in which the biopotential variable impedance control according to the present embodiment is introduced. The BES variable impedance control model in FIG. 7 is a physical model that realizes virtual impedance in biopotential variable impedance control. By controlling the impedance in this model, each arm characteristic of the master arm device 1 can be freely set.

  Thus, by detecting the bioelectric potential signal, estimating the joint torque and operating the master arm, the characteristics of the arm / muscle of the wearer 0 can be reflected in real time on the master arm. In addition, the data transmitted and received between the master arm device 1 and the slave arm device 2 is not the joint angle data but the movement amount of the hand position coordinates, so that the master arm device 1 and the slave arm device 2 can type ( Even when the degree of freedom is different, control can be easily performed, and versatility and operability can be improved.

  In addition, the biopotential variable impedance control based on the biopotential information makes it possible to control the operation such as increasing the joint stiffness without displacement output, and to perform an intuitive and supple operation. Moreover, the impedance adjustment of the robot can be performed intuitively by the wearer 0 without requiring consciousness or special operation, and the master arm device 1 and the wearer 0 can be more seamlessly connected. . The wearer 0 can move the robot arm like his / her arm.

  Furthermore, it is possible to control the slave arm device 2 so as not to enter a preset area.

  As described above, according to the present embodiment, the operability of the manipulation system can be improved by dynamically changing the impedance of the robot arm using the biopotential information of the operator and performing operation assistance based on visual information. .

  In the above embodiment, the controller 14 presents the entry prohibition area on the display 4, but an external computer may perform it.

  Note that the present invention is not limited to the above-described embodiment as it is, and can be embodied by modifying the constituent elements without departing from the scope of the invention in the implementation stage. In addition, various inventions can be formed by appropriately combining a plurality of components disclosed in the embodiment. For example, some components may be deleted from all the components shown in the embodiment. Furthermore, constituent elements over different embodiments may be appropriately combined.

DESCRIPTION OF SYMBOLS 1 Master arm apparatus 2 Slave arm apparatus 3 Imaging apparatus 4 Display 5 Input interface 6 Head sensor 11 Bioelectric potential sensor 12 Angle sensor 13 Actuator 14 Controller 15 Communication apparatus 21 Force sensor 22 Joint angle sensor 23 Actuator 24 Controller 25 Communication apparatus

Claims (5)

A manipulation system comprising a master arm device worn by a wearer, a slave arm device, an imaging device that images the slave arm device, and a display device that displays an image captured by the imaging device,
The master arm device is
A biopotential sensor that detects a biopotential signal associated with muscle activity corresponding to the movement of the shoulder joint and elbow joint of the wearer;
A first angle sensor for detecting angles of the shoulder joint and elbow joint of the wearer;
A first drive source for applying power to the wearer;
A first controller that controls the drive source based on the biopotential signal and calculates position information of the master arm from the detection result of the first angle sensor;
A first communication unit for transmitting impedance information based on the position information of the master arm and the biopotential signal;
Have
The slave arm device is
A force sensor that detects the force applied to the slave arm;
A second drive source for operating the slave arm;
A second angle sensor for detecting a joint angle of the slave arm;
The second drive source is controlled based on the position information received from the first communication unit, the impedance information, and the detection result of the force sensor, and the position of the slave arm is determined from the detection result of the second angle sensor. A second control unit for calculating information;
A second communication unit for transmitting the position information of the slave arm and the detection result of the force sensor to the first communication unit;
Have
The first control unit controls the first drive source based on the detection result of the force sensor received from the second communication unit,
The manipulation system is a stereo camera having a plurality of lenses.   The manipulation system according to claim 1, wherein the imaging device controls directions of the plurality of lenses based on position information of the slave arm. A head sensor for measuring the movement of the wearer's head;
The manipulation system according to claim 1, wherein the imaging device controls the posture by adjusting a pitch angle, a roll angle, and a yaw angle based on a measurement result of the head sensor. An input interface for receiving information on an entry prohibition area that prohibits the entry of the slave arm;
The said 1st control part controls the said 1st drive source from the positional information on the said slave arm so that the said slave arm may not enter into the said entry prohibition area | region. Manipulation system.   The manipulation system according to claim 4, wherein the display displays the entry prohibition area so as to be superimposed on an image captured by the imaging device.

Images