KR100786351B1 - System and method for teaching work-robot based on ar - Google Patents

Abstract

A system and a method for teaching a working robot using an augmented reality scheme are provided to maximize work efficiency by teaching the working robot using real and virtual environments. A system for teaching a working robot using an AR(Augmented Reality) scheme includes an image input device(200), a calculation unit(210), an AR image generating unit(220), an output device(230), and a correcting unit(240). The image input device generates real image information concerning with real work environments. The calculation unit generates spatial coordinates, which a virtual object is positioned, with respect to a reference marker. The AR image generating unit generates an AR image in which an image relating to the virtual object is inserted, and the AR image is inserted into the real image information based on the spatial coordinates. The output unit outputs the generated AR image to a display. The correcting unit calculates correcting information on the work positions of the working robot based on the output AR image.

Description

System and method for teaching work-robot based on AR using AR

1 is a conceptual diagram illustrating a concept of continuity between a real environment and a virtual environment;

Figure 2 is a block diagram showing the configuration of a preferred embodiment for a work robot teaching system using an AR according to the present invention,

3 is a flowchart illustrating a process of a preferred embodiment of a work robot teaching method using an AR according to the present invention;

4 is a view showing an embodiment of a marker as a reference for generating coordinates according to the present invention;

FIG. 5 is a diagram illustrating an embodiment of a correction marker used for calculating a distortion between an imaging device and an actual image; FIG.

6 is a diagram illustrating a marker installed in an actual working environment in order to set coordinates of a virtual work target;

7 is a diagram illustrating an image in which a real image and a virtual image are matched according to the present invention.

<Description of the symbols for the main parts of the drawings>

100 ... Video Input 102 ...

104 ... matching part 110 ... operating part

120 ... AR image generator 130 ... Output unit

140 ... Government 150 ... Object information storage

The present invention is a working robot teaching system and method using AR, and more specifically, for working robot teaching to apply the AR system in correcting the working position, teaching point, etc. of the working robot performing a line working process such as a vehicle System and method.

Today's markets are facing serious dynamic changes due to short life cycles of products, diversification of product types, various new technologies and customer needs.

Manufacturing global competition and rapidly developing new technologies require manufacturers to design production systems that can adapt quickly to market demands in order to survive intense market competition.

Key factors in the design of a production system are to meet the requirements of the new system in a short time, with a flexible strategy that can be easily adapted to the new system using predefined characteristics or geometries. Companies need a lot of time and investment, as well as their own research.

Various methods have been tried to increase the efficiency of such a production system. In particular, in the manufacturing sites for mass production that perform line work processes by specialized robots, the robots can use work machines or teaching points of many robots that perform work line processes using automated machines. have.

However, the workpiece moving by the hanger along the line cannot have the same loading position or condition. Therefore, the above-mentioned calibration work requires stopping the line and teaching calibration data for each work piece to the work robot one by one. There is a hassle.

In addition, since the work is used around the actual work object, there is a problem in that it is not possible to perform the simulation or the like for the work efficiency or work performance.

In order to solve the above problems of the present invention, there is provided a teaching system for a work robot that can apply the AR system to the teaching process of the work robot.

In addition, another object of the present invention to provide a teaching method of the working robot applying the AR system to the teaching process of the working robot.

Other objects and advantages of the invention will be described below and will be appreciated by the embodiments of the invention. Furthermore, the objects and advantages of the present invention can be realized by means and combinations indicated in the claims.

Work robot teaching system using AR to achieve the above object, the input image unit for generating real image information that is image information about the actual work environment; A calculator configured to generate spatial coordinate data in which the virtual work object is located in the working environment from the real image information and a marker that is a reference for generating coordinates installed in the actual working environment; An AR image generation unit generating an AR image in which an image relating to a virtual work object is inserted into the real image information based on the spatial coordinate data; An output unit which outputs the generated AR image to a predetermined screen display unit; And a calibration unit configured to calculate calibration information on a work position of a work robot corresponding to a work target based on the output AR image.

Through the above configuration, it is possible to realize a system that can teach a work robot to perform an effective work using the AR image.

Work robot teaching method using AR to achieve another object of the present invention, the real image generating step of generating real image information that is image information about the actual working environment; A spatial coordinate data generation step of generating spatial coordinate data in which a virtual work object is located in the working environment from a marker which is a reference for generating coordinates installed in the real image information and the actual working environment; An AR image generation step of generating an AR image in which an image relating to a virtual work object is inserted into the real image information based on the spatial coordinate data; An output step of outputting the generated AR image to a predetermined screen display means; And calculating calibration information on the work position of the work robot corresponding to the work object based on the output AR image.

Through the above step configuration, it is possible to realize a method for teaching a work robot so that an effective work can be operated using the AR image.

Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the accompanying drawings.

Prior to this, terms or words used in the specification and claims should not be construed as having a conventional or dictionary meaning, and the inventors should properly explain the concept of terms in order to best explain their own invention. It should be interpreted as meanings and concepts corresponding to the technical idea of the present invention based on the principle of definition.

Therefore, the embodiments described in the specification and the drawings shown in the drawings are only the most preferred embodiment of the present invention and do not represent all of the technical idea of the present invention, various modifications that can be replaced at the time of the present application It should be understood that there may be equivalents and variations.

1 is a conceptual diagram illustrating a concept of continuity between a real environment and a virtual environment.

Prior to the detailed description of the present invention, a conceptual description of AR (Augmented Reality) is as follows.

Real AR is part of Mixed Reality, a mixture of real and virtual worlds, where Reality was defined by Paul Milgram of the University of Toronto, Canada, in 1994.

Paul Milgram argues that the real and virtual worlds are contiguous with each other, where the real world refers to the surroundings that we can see visually and physically, and the virtual world is a computer that makes us feel that we are in a new environment. Refers to the virtual space created by

Mixed Reality aims to improve the reality of the system by complementing each of the shortcomings because the real and virtual environments exist at the same time in the design of the production system.

Mixed Reality is divided into Augmented Reality (AR) and Augmented Virtuality (AV) depending on whether it is based on real or virtual.

AR enhances the reality of the virtual environment by synthesizing a virtual object in the real environment, while AV enhances the reality of the virtual environment by synthesizing the real object in the virtual environment.

That is, as shown in FIG. 1, the real environment and the virtual environment may have a kind of continuous line spectrum, and the main technical configuration employed in the present invention is an Augmented Reality environment existing in an intermediate region spectrum between real and virtual. That's what happens.

Hereinafter, a working robot teaching system and method using AR according to the present invention will be described with reference to FIGS. 2 and 3.

The working robot teaching system using the AR according to the present invention includes an input image unit 200, an operation unit 210, an AR image generation unit 220, an output unit 230, a calibration unit 240, and a target information storage unit 250. ).

First, the input image unit 200 generates real image information, which is image information about an actual working environment (S300).

This means the actual image of the actual working environment of the factory automatic line is generated by data input image through a variety of imaging devices, cameras, vision systems, and the like.

In generating the real picture information, the visual device such as a camera may recognize the data and may be somewhat different from the real environment. It is desirable to.

That is, the actual environment image may be somewhat distorted due to physical and optical causes due to the visual resolution of the camera, the bending of the lens, the resolution of the camera, and the shape of the lens.

In order to effectively solve the above problems, the input image unit 200 preferably includes a correction unit 202 and a matching unit 204.

The correction unit 202 receives an image of a correction marker having a plurality of areas of the same size black and white, and calculates correction data by calculating a distortion degree of the input image (S302).

A calibration marker that can be applied to one embodiment of the calibration marker is shown in FIG. 5. As shown in FIG. 5, the checkerboard-shaped square marker has areas of the same size of black and white.

Since the correction markers have the same left and right lengths, the imager can easily determine the left and right symmetry, the bendability, and the like, so that the correction markers are first recognized and the necessary correction data are calculated.

In addition, in calculating the correction data, the calculation of the degree of distortion is not performed once, but after about 20 to 30 times, the correction data is calculated using an average value obtained by averaging the results, so that more substantial correction is performed. It is preferable.

In addition, it is more preferable to calculate the correction data centering on the average value of the remaining calculation values except for the calculation value which is far from the general calculation value due to a calculation error or an input error.

After the correction data is calculated by the correction unit 202 (S302), the matching unit 204 corrects and generates the real image information based on the correction data (S304).

Through the corrected data as described above, it is possible to generate an image related to a more realistic working environment by generating real image information by minimizing errors due to distortion, bending, and perspective of the real image information.

Next, the operation unit 210 generates spatial coordinate data at which the virtual work object is located in the working environment from the real image information and a marker that is a reference for generating coordinates installed in the actual working environment (S310).

The marker of the above process is installed in the actual working environment, and becomes a kind of mark which is a reference for generating coordinates.

An embodiment of the marker is shown in FIG. 4, which will be described in more detail as follows.

The space defined by the AR system refers to the actual environment seen through the camera or monitor of the computer. The data about the object obtained through tracking consists of X, Y, Z representing the position of the object and Rx, Ry, Rz representing the direction (or posture) of the object. (6DOF: 6 Degree of Freedom).

There are various measuring principles such as mechanical, magnetic, optical, inertial, and acoustical tracking, but optical tracking with high precision is desirable for this research and development. Optical tracking requires a marker of a particular shape to set the position of the object and the coordinate system.

The markers represent the coordinate system for the virtual model in the real environment when tracked through a visual device such as a camera. In the AR system, the real environment and the virtual model are not just synthesized, but are organically connected to each other through the definition of each space.

Data about the space is represented by reading the data of the marker tracked through the camera to the computer. The markers show various shapes depending on the system used. In the present invention, it is preferable to employ a square marker that has less tracking error and can easily display various codes.

Square markers consist of a combination of several large and small squares. The square located at the edge of the marker represents the minimum boundary at which the tracking module can recognize the marker. The small square inside is a 3x3 matrix that separates the shape of the markers.

Black means 1, white means 0, and a marker like FIG. 4 means a matrix of {(1, 0, 0), (0, 0, 0), (0, 0, 0)}. By changing the shape of these matrices, various marker shapes can be created.

Based on these markers, the spatial coordinate data that the virtual work target is located in the real environment is generated.

In generating the spatial coordinate data, the spatial coordinate data in which the virtual work object is located is calculated by calculating coordinates independent from each of the markers based on the plurality of markers, and superimposing respective spatial coordinate data calculated from the plurality of markers. It is desirable to precisely determine the position.

After the spatial coordinate data of the virtual work object is generated as described above, the AR image generator 220 generates an AR image in which an image of the virtual work object is inserted into the real image information based on the spatial coordinate data. (S320).

When generating an AR image, data about a work target, for example, when the work target is a vehicle, stores image information about each vehicle in a storage space such as a target information storage unit 250 and stores the image information. It is preferable to configure the data to read the data on the work object and to perform the process of matching the read work object data with the work environment image based on the calculated spatial coordinate data.

In addition, the generated AR image is to be stored in the target information storage unit 250, such as a predetermined storage space, and can be effectively used to calculate the error, performance, etc. of the work after the work on the work is completed. It is desirable to.

The operation unit 210 uses the main marker for determining the direction of the real image information and the coordinate axis and the three auxiliary markers rotated to have a coordinate system in the same direction with the main marker. It is possible to generate spatial coordinate data further located.

Also, the AR image generator 220 may generate an AR image in which an image of the virtual work tool is further inserted based on the spatial coordinate data of the virtual work tool generated by the calculator 210.

The virtual work tool that performs the work through the configuration as described above may be included in the AR image to perform the work by organically correlating the position of the work tool with the position of the work target and the work environment more precisely.

Table 1 below shows an example of a marker used as an embodiment for the marker, the main marker, and the auxiliary marker, and their respective origin distances and rotation angles.

 Coordinate system  Marker  Origin distance (mm)  Rotation angle (°)   Work target

 (-257.0, 0, -44.0)  (0, 0, 180)

 (770.0, 0, -44.0)  (0, 0, 180)      Work tool

 (0, 197.5, -192.5)  (0, 0, 0)

 (18.5, 197.5, -99.0)  (0, -90, 0)

 (-17.5, 197.5, -99.0)  (0, 90, 0)

 (0, 197.5, -157.5)  (0, 180, 0)

Multiple markers for a virtual tool can be designed in a three-dimensional shape consisting of four markers. This is so that the marker can always set the coordinate system on the display image even if the camera illuminates the marker at any arbitrary angle.

The marker configuration of the three-dimensional shape has a different configuration from the method of setting the coordinate system of the foregoing virtual vehicle. First, select the main marker that determines the direction of the coordinate axis among the four markers, and set the coordinate system so that the other three markers have the same coordinate system direction as the main marker. Rotate

Thereafter, similarly to the method of setting the coordinate system of the virtual vehicle, the center coordinates of the markers are converted into points corresponding to the actual tools.

When the AR image is generated, the output unit 230 of the present invention outputs the generated AR image to a predetermined screen display means so as to be visually output and used visually in the teaching work of the work robot (S330).

If the screen display means is a device capable of visually representing a data image, various applications such as a TV, a monitor, an LCD, a PDP are possible.

Thereafter, the calibration unit 240 calculates calibration information on the work position of the work robot corresponding to the work target based on the output AR image (S240).

Briefly summarizing the above process, the coordinates generated from the markers are synthesized at the position calculated by the tracking module (operation unit) and transmitted to the display device. The vehicle is overlaid.

Since the virtual vehicle added to the actual working environment has the same dimensions and shape as the actual vehicle, it acts as a vehicle for the robot teaching process, and the operator can perform the process while viewing the virtual vehicle.

In addition, the AR image generation unit 220 is configured to display the extension guide line of the most work tool in the AR image so as to overlap the virtual work object, and to display the extension guide line in a different color for each predetermined interval. desirable.

The above configuration is to more accurately operate the problems related to the visible part of the operator in the process of the robot teaching simulation by applying the AR system and the teaching data using the conventional robot teaching data and the AR system in the comparison process.

First, while the robot teaching process, the operator may not easily access the virtual tool and the virtual vehicle while looking at the display monitor. That is, since perspective perception of the virtual objects may not be easy, it may not be easy to determine the exact position of the virtual tool. Therefore, it is preferable to configure the virtual tool to be displayed by adding additional elements.

At first, the guide line is extended at the end of the virtual tool so that the guide line overlaps with the vehicle, and the guide line can be visually recognized by the distance between the virtual objects by changing the color in 50 mm units. A configuration that can overcome the difficulties can be realized.

On the other hand, in generating the spatial coordinate data in the calculation unit 210, the position of the virtual work object by compensating the difference between the height of the position of the actual work object and the height of the virtual work object corresponding to the center coordinate of the virtual work object It is preferable to generate the spatial coordinate data by determining.

This is configured to generate more precise spatial coordinate data by correcting this because when the actual work object is moved to the line by a hanger or the like, the actual height of each work object may be different.

As shown in Table 2 below, the result of teaching the AR system applied robot according to the present invention can be evaluated by comparing with the teaching point of the actual vehicle, and the comparison results are shown in Table 2.

The actual teaching data of the vehicle and the robot teaching data using the AR system showed an error of up to 3 mm in the wheelbase and the longitudinal direction of the vehicle, and an error of 12 mm in the height direction. This shows a high accuracy compared to a large error range within a teaching point radius of 50 mm, which is the first target error range.

When the actual vehicle is mounted on the hanger, considering the error of any position can be said to represent a value that can ignore the above degree of error.

Teaching point Actual (mm) AR system (mm) Error (mm)

One T (wheelbase of vehicle) -485.0 -488.0 3 L (vehicle length) -500.0 -497.0 -3 H (height direction of the vehicle) 12.1 13.3 -1.2 2 T (wheelbase of vehicle) -485.0 -488.0 3 L (vehicle length) 500.0 503.0 -3 H (height direction of the vehicle) 12.1 13.3 -1.2 3 T (wheelbase of vehicle) 315.0 318.0 -3 L (vehicle length) -315.0 -312.0 -3 H (height direction of the vehicle) -75.0 -63.0 12 4 T (wheelbase of vehicle) 315.0 318.0 -3 L (vehicle length) 315.0 318.0 -3 H (height direction of the vehicle) -75.0 -63.0 -12 5 T (wheelbase of vehicle) 433.0 436.0 -3 L (vehicle length) -424.0 -421.0 -3 H (height direction of the vehicle) -76.3 -64.3 -12 6 T (wheelbase of vehicle) 433.0 436.0 -3 L (vehicle length) 424.0 427.0 -3 H (height direction of the vehicle) -76.3 -64.3 -12

As described above, although the present invention has been described by way of limited embodiments and drawings, the present invention is not limited thereto and is described by the person of ordinary skill in the art to which the present invention pertains. Various modifications and variations are possible without departing from the scope of the appended claims.

Working robot teaching system and method using AR according to the present invention provides the following effects.

First, since the teaching work of the work robot can be performed using the virtual work object and the external work environment, more effective operation is possible.

Second, it can reflect errors and corrections in various real situations, so it can be efficiently applied to work systems that require precision.

Third, by using AR images, work efficiency can be enhanced, and the work performance can be saved by storing data. Therefore, work rate and future work plan can be effectively established.

Fourth, since the virtual work target can be used, more rapid work can be performed since there is no need to interrupt or interrupt the operation line for actual teaching.

Claims (14)

An input image unit generating real image information, which is image information about an actual working environment; A calculator configured to generate spatial coordinate data in which the virtual work object is located in the working environment from the real image information and a marker that is a reference for generating coordinates installed in the actual working environment; An AR image generation unit generating an AR image in which an image relating to a virtual work object is inserted into the real image information based on the spatial coordinate data; An output unit which outputs the generated AR image to a predetermined screen display unit; And And a calibration unit configured to calculate calibration information on a work position of a work robot corresponding to a work object based on the output AR image. The method of claim 1, wherein the input image unit, A correction unit configured to receive an image of a correction marker having a plurality of areas of the same size black and white, and calculate correction data by calculating a distortion degree of the input image; And And a matching unit for calibrating and generating the real image information based on the correction data. The method of claim 2, wherein the correction unit, The operation robot teaching system using the AR, characterized in that for performing the calculation of the distortion degree for the input image in a range of 20 to 30 times and calculating the correction data using the average value of the performance result. The method of claim 1, wherein the operation unit, And calculating coordinates independent from each of the markers, and calculating spatial coordinates at which the virtual work object is located by overlapping respective spatial coordinate data calculated from two markers. The method of claim 1, wherein the operation unit, The virtual work tool further includes spatial coordinate data in which the virtual work tool is located in the working environment by using the main marker for determining the direction of the real image information and the coordinate axis and three auxiliary markers rotated to have a coordinate system in the same direction as the main marker. Create, The AR image generation unit, Work robot teaching system using an AR, characterized in that for generating an AR image further inserted into the image of the virtual work tool based on the spatial coordinate data of the virtual work tool generated by the calculating unit. The method of claim 5, wherein the AR image generating unit, In the AR image, the extension guide line of the work tool is displayed so as to overlap with the virtual work object, and the extension guide line is displayed differently in a color distinguished every 50 mm intervals. The method of claim 6, wherein the calculation unit, Compensating for the difference between the height of the position of the actual work object and the height of the virtual work object corresponding to the center coordinates of the virtual work object to determine the position of the virtual work object to generate spatial coordinate data using AR Work robot teaching system. A real image generation step of generating real image information which is image information on an actual working environment; A spatial coordinate data generation step of generating spatial coordinate data in which a virtual work object is located in the working environment from a marker which is a reference for generating coordinates installed in the real image information and the actual working environment; An AR image generation step of generating an AR image in which an image relating to a virtual work object is inserted into the real image information based on the spatial coordinate data; An output step of outputting the generated AR image to a predetermined screen display means; And And a calibration information calculation step of calculating calibration information on a work position of a work robot corresponding to a work object based on the output AR image. The method of claim 8, wherein the real image generating step, A correction data calculation step of receiving an image of a correction marker having a plurality of areas of the same size black and white and calculating correction data by calculating a distortion degree of the input image; And And a matching step of correcting and generating the real image information based on the correction data. The method of claim 9, wherein the calculating of the correction data, And calculating the correction data using the average value of the result of performing the calculation of the distortion degree in the range of 20 to 30 times with respect to the input image. The method of claim 8, wherein the spatial coordinate data generation step, And calculating coordinates independent from each of the markers, and calculating spatial coordinates at which the virtual work object is located by overlapping respective spatial coordinate data calculated from two markers. The method of claim 8, wherein the spatial coordinate data generation step, The virtual work tool further includes spatial coordinate data in which the virtual work tool is located in the working environment by using the main marker for determining the direction of the real image information and the coordinate axis and three auxiliary markers rotated to have a coordinate system in the same direction as the main marker. Create, The AR image generation step, And an AR image in which an image related to the virtual work tool is further inserted based on the spatial coordinate data of the virtual work tool generated in the spatial coordinate data generation step. The method of claim 12, wherein the AR image generation step, In the AR image, the extension guide line of the work tool is displayed so as to overlap with the virtual work object, and the extension guide line is displayed differently in a color distinguished every 50 mm intervals. The method of claim 13, wherein the spatial coordinate data generation step, Compensating for the difference between the height of the position of the actual work object and the height of the virtual work object corresponding to the center coordinates of the virtual work object to determine the position of the virtual work object to generate spatial coordinate data, characterized in that How to teach robots.

Images