JP2020175453A - Remote control device - Google Patents

Abstract

To associate a coordinate system of a mobile robot with a coordinate system of a work site without using a distance measurement device.SOLUTION: A remote control device comprises: an imaging unit imaging an actual video image of a work site of a mobile robot; a virtual video image generation unit generating a virtual three-dimensional video image of the work site on the basis of the actual video image and a three-dimensional model; a display unit displaying the virtual three-dimensional video image; and an operation unit generating an operation signal for remotely operating the mobile robot. The virtual video image generation unit adjusts a position of a basic virtual three-dimensional video image in such a way that common objects common to the basic virtual three-dimensional video image generated based on the three-dimensional model and the actual video image overlap each other.SELECTED DRAWING: Figure 1

Description

The present invention relates to a remote control device.

The following Patent Document 1 discloses a manipulator remote control method in a mobile robot system. In this remote control method of the manipulator, the positional relationship between the mobile manipulator and the obstacle is specified based on the distance to the obstacle measured by the distance measuring device, and the camera image captured by the camera is displayed on the display device of the control system. The operator remotely controls the mobile robot system while viewing the camera image displayed on the display device.

Japanese Patent No. 4558682

By the way, the distance measuring device measures the distance between the mobile robot and an obstacle by using a laser beam, but it may not be adopted depending on the environment of the work site. In such a case, the coordinate system of the mobile manipulator (coordinate system of the mobile robot) and the coordinate system of the obstacle (coordinate system of the work site) cannot be associated with each other, so that the mobile manipulator can be operated remotely. It disappears.

The present invention has been made in view of the above circumstances, and an object of the present invention is to associate the coordinate system of a mobile robot with the coordinate system of a work site without using a distance measuring device.

In order to achieve the above object, in the present invention, as a first solution relating to the remote control device, an imaging unit that captures an actual image of a mobile robot's work site, and a three-dimensional model of the actual image and the work site. A virtual image generation unit that generates a virtual three-dimensional image of the work site based on the above, a display unit that displays the virtual three-dimensional image, and an operation unit that generates an operation signal for remotely operating the mobile robot are provided. The virtual image generation unit adjusts the position of the basic virtual three-dimensional image so that the common object common to the basic virtual three-dimensional image generated based on the three-dimensional model and the real image overlaps with each other. The means of generating a virtual three-dimensional image is adopted.

In the present invention, as the second solution means for the remote control device, in the first solution means, the common object is a mark member provided at the work site.

In the present invention, as the third solution means for the remote control device, in the first or second solution means, the mobile robot is a self-propelled type robot.

In the present invention, as a fourth solution relating to the remote control device, in any of the first to third solutions, the display unit is a head-mounted display.

In the present invention, as a fifth solution according to the remote control device, in any of the first to fourth solutions, the work site is a place where the radiation level is higher than the normal level. Adopt the means.

According to the present invention, it is possible to associate the coordinate system of the mobile robot with the coordinate system of the work site without using a distance measuring device.

It is a system block diagram which shows the whole structure of the robot system in one Embodiment of this invention. It is a block diagram which shows the structure of the remote control device which concerns on one Embodiment of this invention. It is a schematic diagram (a) which shows the real image and the schematic diagram (b) which shows a virtual three-dimensional image in one embodiment of the present invention. It is a schematic diagram which shows the synthetic image in one Embodiment of this invention.

Hereinafter, an embodiment of the present invention will be described with reference to the drawings.
First, the robot system in this embodiment is composed of a robot body 1 and a remote control device 2 as shown in FIG.

The robot body 1 is a mobile robot that performs a predetermined work on the work W while moving at a predetermined work site, and has a mobile trolley 1a, a manipulator 1b, a work tool 1c, a camera 1d, and a robot controller 1e as shown in the figure. At least I have. The work site is a place where the radiation level is higher than the normal level, and it is difficult to use the conventional distance measuring device in the robot body 1.

At this work site, the work W, which is the work target of the robot body 1, is placed on the support base T. Further, in the vicinity of the work W, a mark member M for associating the position of the work W with the position of the robot body 1 is provided. The mark member M is an object having a peculiar shape.

The robot body 1 is a self-propelled robot that runs on the work site. The robot body 1 is controlled by the robot controller 1e to perform a predetermined work on the work W placed on the support base T.

The mobile carriage 1a includes a plurality of wheels and a drive device (motor or the like) for driving the wheels, and travels on the floor F of the work site based on a travel control signal input from the robot controller 1e. The mobile carriage 1a sets the position of the manipulator 1b mounted on the mobile carriage 1b at the work site to a predetermined work position.

The manipulator 1b is fixedly installed on the moving carriage 1a, and includes a plurality of arms and a plurality of joints connecting the arms. The manipulator 1b moves the work tool 1c attached to the tip portion by driving a motor provided in the joint portion based on a joint control signal input from the robot controller 1e. That is, the manipulator 1b is a mechanical device that optimally sets the position and posture of the work tool 1c according to the work content with respect to the work W.

The work tool 1c is detachably attached to the tip of the manipulator 1b, and is a portion for directly performing work on the work W. For example, when mechanically machining a work, the work tool 1c is a tool that exerts a shearing force, a pressing force, or the like on the work.

The camera 1d is fixedly installed on the front side of the mobile carriage 1a, that is, in front of the manipulator 1b on the mobile carriage 1a, and captures an image in front of the mobile carriage 1a as an actual image G1 of the work site. As shown in FIG. 3A, the actual image G1 is a moving image showing the state of the work W at the work site and the work tool 1c that performs work on the work. Such a camera 1d outputs the real video G1 as a real video signal to the remote control device 2.

Here, in FIG. 1, the camera 1d and the remote control device 2 are drawn as separate components, but the camera 1d is functionally a component included in the remote control device 2. Further, the camera 1d corresponds to an imaging unit that captures an actual image G1 at a work site.

The robot controller 1e is a control device communicatively connected to the remote control device 2 in the operation room, and controls the mobile carriage 1a and the manipulator 1b based on the operation signal received from the remote control device 2. The robot controller 1e is a kind of computer, and controls the mobile carriage 1a and the manipulator 1b according to the operation signal by processing the operation signal according to the control program stored in advance.

The remote control device 2 is provided in an operation room separated from the work site, and outputs an operation signal to the robot body 1 based on an operation input from the operator. The remote control device 2 is a kind of computer that generates an operation signal by processing an operation input based on an operation program, and has a virtual image generation unit 2a, a display unit 2b, and an operation shown in FIG. 2 as functional components. It includes at least part 2c.

The virtual image generation unit 2a generates a virtual three-dimensional image G2, that is, a virtual reality image at the work site. That is, the virtual image generation unit 2a is a virtual tertiary based on the three-dimensional model of the work site stored in advance, the viewpoint designation signal input from the display unit 2b, and the actual image G1 of the work site input from the camera 1d. Generates the original video G2.

The three-dimensional model of the work site includes data showing the positional relationship between the three-dimensional shapes of the main objects existing in the work site, that is, the work W, the robot body 1, the mark member M, and the like, and the main objects. That is, the virtual three-dimensional image G2 includes the main objects existing in the work site as individual objects. The mark member M is a common object common to the real image G1 and the virtual three-dimensional image G2.

Further, the virtual three-dimensional image G2 is an image in which a viewpoint is set based on a viewpoint designation signal. That is, in this virtual three-dimensional image G2, as shown in FIG. 3B, main objects such as the support base T', the work W', the robot body 1', and the mark member M'are designated by the viewpoint designation signal. It is displayed as a three-dimensional image viewed from a different viewpoint. The virtual image generation unit 2a outputs such a virtual three-dimensional image G2 to the display unit 2b.

Here, although the details will be described later, the virtual image generation unit 2a generates the basic virtual three-dimensional image Gk at the initial stage of the work using the remote control device 2, and the basic virtual three-dimensional image Gk and the real image G1. Performs alignment processing with. The virtual image generation unit 2a generates a virtual three-dimensional image G2 as a result of the alignment process.

The display unit 2b is a display device that displays the virtual three-dimensional image G2. The display unit 2b provides the operator with the virtual three-dimensional image G2 as support information for remotely controlling the robot body 1. That is, the display unit 2b has a form that is easy for the operator to see in the operation room, and is, for example, a head-mounted display (HMD).

Further, the display unit 2b incorporates a motion sensor 2d that detects the direction of the head of the wearer, that is, the operator. The motion sensor 2d outputs a detection signal indicating the direction of the operator's head to the virtual image generation unit 2a as a viewpoint designation signal. Such a motion sensor 2d corresponds to a viewpoint designation unit that designates the viewpoint of the virtual three-dimensional image G2.

The virtual image generation unit 2a described above captures the detection signal of the motion sensor 2d as a viewpoint designation signal to generate a virtual three-dimensional image G2 whose viewpoint changes according to the direction of the operator's head. ..

The operation unit 2c is a device for the operator to input an operation instruction. That is, the operation unit 2c receives an operation instruction for remotely controlling the robot body 1 from the operator, generates an operation signal indicating the operation instruction, and outputs the operation signal to the robot controller 1e. Such an operation unit 2c is, for example, Joyce Tech.

Next, the operation of the remote control device according to the present embodiment, particularly the positioning operation of the basic virtual three-dimensional image Gk and the real image G1 will be described in detail.

When the robot main body 1 is remotely controlled by using this remote control device, the operator attaches the display unit 2b, which is an HMD, to the face and inputs an operation to the operation unit 2c. That is, the operator remotely controls the robot body 1 by performing an operation input to the operation unit 2c while visually recognizing the virtual three-dimensional image G2 shown in FIG. 2 (b).

Here, the operator and the remote control device perform the positioning process of the basic virtual three-dimensional image Gk and the real image G1 prior to the actual work. That is, in this alignment process, the virtual image generation unit 2a first creates a basic virtual three-dimensional image Gk based on the viewpoint designation signal, the work W, and the three-dimensional model.

Similar to the virtual 3D image G2 shown in FIG. 3B, the basic virtual 3D image Gk includes main objects such as a support base T', a work W', a robot body 1', and a mark member M'. It is a virtual image that is displayed but is not associated with the actual image G1. Therefore, in the basic virtual three-dimensional image Gk, the positions of the main objects such as the support base T', the work W', the robot body 1'and the mark member M'are determined by the work W, the robot body 1 and the mark member in the actual image G1. It is a virtual image whose positional relationship does not match with M and the like.

The virtual image generation unit 2a generates the above-mentioned basic virtual three-dimensional image Gk at the initial stage of the work using the remote control device, and synthesizes the basic virtual three-dimensional image Gk by superimposing the real image G1 as shown in FIG. The video Gg is output to the display unit 2b. Further, the virtual image generation unit 2a adjusts the position of the basic virtual three-dimensional image Gk so that the mark member M of the real image G1 and the mark member M'of the basic virtual three-dimensional image Gk in the composite image Gg overlap. The telop to be prompted is displayed on the synthetic video Gg.

The operator first changes the direction of his / her head to align the viewpoint of the basic virtual three-dimensional image Gk with the viewpoint of the real image G1, that is, the position of the camera 1d. Alternatively, by generating an image that should be visible from the position of the camera 1d by the virtual image generation unit 2a and displaying it on the display unit 2b in the form of a small window, the viewpoint of the basic virtual three-dimensional image Gk can be changed to the viewpoint of the real image G1. Align with the position of camera 1d.

Then, when such viewpoint adjustment is completed, the operator operates the operation unit 2c so that the mark member M of the real image G1 and the mark member M'of the basic virtual three-dimensional image Gk overlap each other. Adjust the position of the 3D image Gk. Note that, in FIG. 4, only the mark member M of the actual image G1 and the mark member M'of the basic virtual three-dimensional image Gk are shown for convenience.

By overlapping the mark member M of the real image G1 and the mark member M'of the basic virtual three-dimensional image Gk in this way, the association of the coordinate system of the robot body 1 with the coordinate system of the work site is completed. .. When the alignment process between the basic virtual three-dimensional image Gk and the real image G1 is completed, the virtual image generation unit 2a generates the virtual three-dimensional image G2 and outputs it to the display unit 2b.

In the remote control of the robot body 1 based on the virtual three-dimensional image G2, when the operator changes the direction of the head, this change is detected by the motion sensor 2d of the display unit 2b, and the virtual image generation unit is used as a viewpoint designation signal. It is input to 2a. As a result, the virtual image generation unit 2a generates a virtual three-dimensional image (virtual reality image) of the viewpoint corresponding to the new head orientation. As a result, the work W'and the robot body 1'of the virtual three-dimensional image G2 displayed on the display unit 2b are viewed from a new viewpoint according to the direction of the head.

According to this embodiment, the robot body 1 is adjusted by adjusting the position of the basic virtual three-dimensional image Gk so that the mark member M of the real image G1 and the mark member M'of the basic virtual three-dimensional image Gk overlap each other. Since the coordinate system of the robot body 1 is associated with the coordinate system of the work site, it is possible to associate the coordinate system of the robot body 1 with the coordinate system of the work site without using a distance measuring device.

Further, according to the present embodiment, the operator can more accurately grasp the positional relationship between the robot body 1'and the work W'by changing the viewpoint of the virtual three-dimensional image G2. Therefore, according to the present embodiment, it is possible to provide a remote control device having better workability than the conventional one.

The present invention is not limited to the above embodiment, and for example, the following modifications can be considered.
(1) In the above embodiment, the mark member M is a common object, but the present invention is not limited to this. A main object other than the mark member M may be a common object. For example, the support base T or the work W may be a common object.

(2) In the above embodiment, the position of the basic virtual three-dimensional image Gk is adjusted by the operator operating the operation unit 2c, but the present invention is not limited to this. For example, the virtual image generation unit 2a extracts a mark member M having a peculiar shape by performing image processing on the real image G1, and the mark member M'of the basic virtual three-dimensional image Gk with respect to the extracted mark member M. The positions of the basic virtual three-dimensional video Gk may be automatically adjusted so that they overlap.

(3) In the above embodiment, the head-mounted display (HMD) is adopted as the display unit 2b, but the present invention is not limited to this. Further, the viewpoint designation unit of the present invention is not limited to the motion sensor 2d. For example, the operator may specify the viewpoint of the virtual three-dimensional image (virtual reality image) by operating the operation unit 2d.

1 Robot body 1a Mobile trolley 1b Manipulator 1c Work tool 1d Camera 1e Robot controller 2 Remote control device 2a Virtual image generator 2b Display 2c Operation unit 2d Motion sensor G1 Real image G2 Virtual 3D image F floor W work T support

Claims (5)

An imaging unit that captures the actual image of the work site of the mobile robot,
A virtual image generation unit that generates a virtual three-dimensional image of the work site based on the actual image and the three-dimensional model of the work site.
A display unit that displays the virtual three-dimensional image and
It is provided with an operation unit that generates an operation signal for remotely controlling the mobile robot.
The virtual image generation unit adjusts the position of the basic virtual three-dimensional image so that a common object common to the basic virtual three-dimensional image generated based on the three-dimensional model and the real image overlaps with the virtual image. A remote control device characterized by generating a three-dimensional image. The remote control device according to claim 1, wherein the common object is a mark member provided at the work site. The remote control device according to claim 1 or 2, wherein the mobile robot is a self-propelled robot. The remote control device according to any one of claims 1 to 3, wherein the display unit is a head-mounted display. The remote control device according to any one of claims 1 to 4, wherein the work site is a place where the radiation level is higher than the normal level.

Images