KR102420462B1 - Collaborative learning method using digital twin - Google Patents

Abstract

A collaborative learning method using a digital twin is disclosed. The cooperative learning method performed in a physical robot comprises steps of: classifying a physical object detected in a physical space using a pre-learned classification model; requesting class information on the physical object to a virtual robot twin that exists in a virtual space corresponding to the physical space and corresponds to the physical robot if the classification fails; and re-learning the classification model by receiving class information about a virtual object twin that exists in the virtual space and corresponds to the physical object from the virtual robot twin. The present invention is to provide the cooperative learning method using a digital twin that can receive training data from a virtual space.

Description

A collaborative learning method using digital twins {COLLABORATIVE LEARNING METHOD USING DIGITAL TWIN}

The present invention relates to a collaborative learning method using digital twins.

Metaverse is a compound word of meta, meaning fictitious transcendence, and universe, meaning world. And recently, along with the metaverse, digital twins are also attracting attention.

A digital twin refers to a technology that generates digital twins of objects in the real world in the virtual world, and predicts results in advance by simulating situations that may occur in the real world in the virtual world. The virtual world created by the digital twin is synchronized with the real world, and the real and virtual worlds interact with each other. Through this synchronization, the image in the real world is projected onto the virtual world as it is, and the problem of the real clock can be solved through simulation in the virtual world.

As related prior documents, there are Korean Patent Registration Nos. 10-2237449, 10-2127655, and Korean Patent Publication Nos. 2021-0108044 and 2021-0134133.

An object of the present invention is to provide a cooperative learning method using a digital twin that can receive training data from a virtual space.

Another object of the present invention is to provide a cooperative learning method using a digital twin, in which learning can be efficiently performed in a situation where the resources of the physical robot are insufficient.

According to an embodiment of the present invention for achieving the above object, in a cooperative learning method using a digital twin performed in a physical robot, classifying a physical object detected in a physical space using a pre-learned classification model step; when the classification fails, requesting class information about the physical object to a virtual robot twin that exists in a virtual space corresponding to the physical space and corresponds to the physical robot; and receiving class information about a virtual object twin existing in the virtual space and corresponding to the physical object from the virtual robot twin, and performing re-learning on the classification model. A method is provided.

In addition, according to another embodiment of the present invention for achieving the above object, there is provided a cooperative learning method using a digital twin performed in a physical robot, the method comprising: detecting a physical object in a physical space; requesting learning of a virtual object twin existing in a virtual space corresponding to the physical space and corresponding to the physical object to a virtual robot twin existing in the virtual space and corresponding to the physical robot; and receiving the classification model from the virtual robot twin that performs learning on the classification model by using the virtual object twin.

According to an embodiment of the present invention, by receiving additional training data from the virtual robot twin and performing learning on the classification model by the physical robot, learning efficiency may be improved.

In addition, according to an embodiment of the present invention, since learning is performed in a virtual robot twin driven in an environment in which more computing resources than a physical robot are provided, smooth learning can be performed regardless of the limited resources of the physical robot.

1 is a diagram for explaining a cooperative learning system using a digital twin according to an embodiment of the present invention.
2 is a flowchart illustrating a cooperative learning method using a digital twin according to an embodiment of the present invention.
3 is a diagram illustrating a physical space and a virtual space according to an embodiment of the present invention.
4 is a flowchart illustrating a cooperative learning method using a digital twin according to another embodiment of the present invention.

Since the present invention can have various changes and can have various embodiments, specific embodiments are illustrated in the drawings and described in detail in the detailed description. However, this is not intended to limit the present invention to specific embodiments, and should be understood to include all modifications, equivalents and substitutes included in the spirit and scope of the present invention. In describing each figure, like reference numerals have been used for like elements.

In the physical space, which is the space of the real world, there are many restrictions in acquiring training data for learning. For example, it is the space where learning takes place, manpower necessary for learning, and the like. However, in the virtual space, which is the space of the virtual world, the limitations of the physical space do not apply, so a large amount of training data can be acquired more efficiently. In particular, since the virtual space built through the digital twin is synchronized with the physical space, training data acquired through the virtual space in the digital twin environment may replace the training data acquired in the physical space.

The present invention proposes a method of performing learning in cooperation with a physical space, which is a real space, and a virtual space, using a digital twin, focusing on the above points.

An embodiment of the present invention proposes a cooperative learning method in which learning is performed in a physical space using training data acquired in a virtual space or learning is performed in a virtual space using training data acquired in a virtual space.

The cooperative learning method according to an embodiment of the present invention may be performed in a computing device including a processor and a memory. Hereinafter, a cooperative learning method performed in a physical robot will be described as an embodiment.

Hereinafter, embodiments according to the present invention will be described in detail with reference to the accompanying drawings.

1 is a diagram for explaining a cooperative learning system using a digital twin according to an embodiment of the present invention.

Referring to FIG. 1 , the cooperative learning system according to an embodiment of the present invention includes a physical robot 110 , a digital twin server 120 , and a virtual space 130 .

The physical robot 110 includes a pre-trained classification model, and may include sensors such as an image sensor, a radar sensor, a lidar sensor, an ultrasonic sensor, a GPS sensor, and an inertial sensor to detect a physical object in the real world. . The physical robot 110 detects a physical object existing in the physical space while moving in the physical space, and classifies the detected physical object using a classification model. Then, the classification model is retrained using the classified physical object. The classification model may be an artificial neural network-based classification model as an embodiment, and the physical robot 110 may include, as an embodiment, all robots that recognize surrounding objects while moving, such as drones, service robots, and autonomous vehicles. have.

The digital twin server 120 provides a virtual space 130 synchronized with the physical space. The digital twin server 120 creates a virtual space based on data collected about the physical space. For example, if the target physical space is a restaurant, a virtual space for the restaurant may be generated through sensing data of various physical objects existing in the restaurant. Various sensors, such as an image sensor, a radar sensor, and an ultrasonic sensor, for collecting data on the physical space and physical objects may be installed in the physical space, and the digital twin server 120 provides all information about the virtual space 130, for example. , a class of a virtual object existing in the virtual space 130 , location information, characteristic information, and the like are stored. All information on the virtual space 130 corresponds to information on the physical space.

A virtual object exists in the virtual space 130 generated by the digital twin server 120, and the virtual object is a virtual twin of a physical object existing in the physical space, and moves in synchronization with the movement of the physical object. The virtual space 130 includes a virtual object twin corresponding to a physical object and a virtual robot twin 131 corresponding to the physical robot 110 .

The physical robot 110 requests training data from the virtual robot twin 131 and receives the requested training data to learn a classification model. Alternatively, the physical robot 110 requests learning from the virtual robot twin 131 and receives the learned classification model from the virtual robot twin 131 that has performed the requested learning.

According to an embodiment of the present invention, by receiving additional training data from the virtual robot twin and performing learning on the classification model by the physical robot, learning efficiency may be improved.

In addition, according to an embodiment of the present invention, learning is performed in a virtual robot twin driven by a digital twin server that provides more computing resources than a physical robot, so that smooth learning can be performed regardless of the limited resources of the physical robot. have.

2 is a flowchart illustrating a cooperative learning method using a digital twin according to an embodiment of the present invention, and FIG. 3 is a diagram illustrating a physical space and a virtual space according to an embodiment of the present invention. In FIG. 2 , a cooperative learning method performed in a physical robot is described as an embodiment.

2 and 3 , a physical robot 311 according to an embodiment of the present invention classifies a physical object 312 detected in a physical space 310 using a pre-learned classification model (S210) do. When the classification is successful, the physical robot 311 performs re-learning on the classification model using the class for the physical object 312 .

For example, when the physical robot 311 obtains an image of the physical object 312 that is a soccer ball, inputs the image to the classification model, and the classification model classifies the physical object 312 into a soccer ball class, the physical robot 311 ) may retrain the classification model using the image of the physical object 312 labeled with the soccer ball class.

When the classification fails, the physical robot 311 requests class information on the physical object 312 to the virtual robot twin 321 ( S220 ). The virtual robot twin 321 together with the virtual object twin 322 exists in the virtual space 320 and corresponds to the physical object 312 and the physical robot 311, respectively. In addition, the virtual object twin 322 and the virtual robot twin 321 are synchronized with the movements of the physical object 312 and the physical robot 311 to move in the virtual space 320 .

Failure to classify means that class information on the physical object 312 has not been obtained, so the physical robot 311 sends the virtual robot twin 321 to the physical object ( 312) for class information. For example, if the confidence value for the classification result output by the classification model is smaller than the threshold value, it may be determined that the classification has failed. In this case, the physical robot 311 sends the virtual robot twin 312 to the physical object 312 . Request class information for

In step S220, the physical robot 311 may request class information by using at least one of location information and characteristic information about the physical object 312, and the location information is, as an embodiment, the physical robot 311 and the physical object ( 312 ) may be information on the separation distance and separation direction or coordinate values of the physical object 312 . The physical robot 311 may generate information on the separation distance and separation direction between the physical robot 311 and the physical object 312 on the basis of the physical robot using a radar sensor or the like, and through this, the physical object 312 . The coordinate values of can also be calculated.

As described above, since the virtual space is synchronized with the physical space in the digital twin environment, the relative position between the physical robot 311 and the physical object 312 is between the virtual robot twin 321 and the virtual object twin 322 . is the same as the relative position of , and therefore, the virtual robot twin 321 may identify the virtual object twin 322 from the position information transmitted from the physical robot 311 . In addition, since all information about the virtual space is stored in the digital twin server, the virtual robot twin 321 provides information about the virtual object twin 322 identified using the location information or feature information about the detected physical object 312 . Classes are also recognizable.

In step S220, the physical robot 311 allows the virtual robot twin 321 to more accurately identify the virtual object twin 322 corresponding to the physical object 312 together with the location information on the physical object 312, Alternatively, feature information obtained from sensing data of the physical object 312 may be separately transmitted to the virtual robot twin 322 . For example, the sensing data may be an image of the physical object 312 , a radar signal, a lidar signal, an ultrasound signal, or the like, and the feature information may be a feature vector obtained from the sensing data. The virtual object twin 322 is a virtual twin for the physical object 312 , and since the characteristic information of the virtual object twin 322 is the same as that of the physical object 312 , the virtual robot twin 321 is a physical object 312 . A virtual object twin 322 having characteristic information corresponding to the characteristic information of may be identified in the virtual space 320 .

Since the virtual object twin 322 is a twin for the physical object 312, the classes are the same, so the virtual robot twin 321 provides class information about the virtual object twin 322 corresponding to the physical object 312. It is transmitted to the physical robot 311 .

The physical robot 311 receives class information on the virtual object twin 322 from the virtual robot twin 321, and performs re-learning on the classification model (S230).

According to an embodiment of the present invention, since class information on a physical object that the physical robot cannot classify can be provided from the virtual robot twin, training data for the classification model can be additionally secured.

4 is a flowchart illustrating a cooperative learning method using a digital twin according to another embodiment of the present invention. In FIG. 4 , a cooperative learning method performed in a physical robot is described as an embodiment.

3 and 4 , a physical robot 311 according to an embodiment of the present invention detects a physical object 312 in a physical space 310 ( S410 ), and a virtual corresponding to the physical object 312 . Learning of the object twin 322 is requested to the virtual robot twin 321 (S420).

In step S420 , the physical robot 311 may request learning by using at least one of location information and characteristic information about the physical object 312 . As in the above-described embodiment, the location information may be information about the separation distance and separation direction between the physical robot 311 and the physical object 312 or may be a coordinate value of the physical object 312, and the virtual robot twin 321. may identify a virtual object twin 322 corresponding to the physical object 312 using the location information on the physical object 312 and recognize a class for the virtual object twin 322 . In addition, the characteristic information may be characteristic information obtained from sensing data of the physical object 312 .

In addition, when the classification of the detected physical object 312 using the classification model fails in step S420, the physical robot 311 may request the virtual robot twin 321 to learn about the virtual object twin 322, The sensed value of the object 312 may be transmitted to the virtual robot twin 321 together with location information.

As another embodiment, when the available resources of the physical robot 311 are less than or equal to a threshold resource, the physical robot 311 may request the virtual robot twin 321 to learn about the virtual object twin 322 . Since the virtual object twin 322 is driven on a digital twin server supporting powerful computing resources, learning may be performed using the resources of the digital twin server.

The virtual robot twin 321 uses the class for the virtual object twin 322 to learn the classification model, and transmits the learned classification model to the physical robot 311 . The virtual robot twin 321 is an embodiment, and by using the image for the virtual object twin 322 and the class for the virtual object twin 322, learning for the classification model can be performed, and the virtual object twin ( 322) may be provided from a digital twin server.

The physical robot 311 may receive the classification model learned from the virtual robot twin 321 ( S430 ), and classify the physical object 312 using the received classification model. When the physical robot 311 includes a separate classification model, the physical robot 311 may update the separate classification model by using the learning model received from the virtual robot twin 321 .

According to an embodiment of the present invention, by learning the classification model using the resources of the digital twin server in which the virtual twin robot is implemented, rather than the resources of the physical robot, the classification model is efficiently learned even in an environment where the resources of the physical robot are insufficient. can do.

The technical contents described above may be implemented in the form of program instructions that can be executed through various computer means and recorded in a computer-readable medium. The computer-readable medium may include program instructions, data files, data structures, etc. alone or in combination. The program instructions recorded on the medium may be specially designed and configured for the embodiments, or may be known and available to those skilled in the art of computer software. Examples of the computer-readable recording medium include magnetic media such as hard disks, floppy disks and magnetic tapes, optical media such as CD-ROMs and DVDs, and magnetic such as floppy disks. - includes magneto-optical media, and hardware devices specially configured to store and execute program instructions, such as ROM, RAM, flash memory, and the like. Examples of program instructions include not only machine language codes such as those generated by a compiler, but also high-level language codes that can be executed by a computer using an interpreter or the like. A hardware device may be configured to operate as one or more software modules to perform the operations of the embodiments, and vice versa.

As described above, the present invention has been described with specific matters such as specific components and limited embodiments and drawings, but these are only provided to help a more general understanding of the present invention, and the present invention is not limited to the above embodiments. , various modifications and variations are possible from these descriptions by those of ordinary skill in the art to which the present invention pertains. Therefore, the spirit of the present invention should not be limited to the described embodiments, and not only the claims to be described later, but also all those with equivalent or equivalent modifications to the claims will be said to belong to the scope of the spirit of the present invention. .

Claims (13)

In a cooperative learning method using a digital twin performed in a physical robot,
classifying a physical object detected in a physical space using a pre-trained classification model;
when the classification fails, requesting class information on the physical object to a virtual robot twin that exists in a virtual space corresponding to the physical space and corresponds to the physical robot; and
Receiving class information about a virtual object twin existing in the virtual space and corresponding to the physical object from the virtual robot twin, and performing re-learning on the classification model,
The step of requesting the class information to the virtual robot twin
Requesting the class information by using at least one of location information and characteristic information about the physical object,
The virtual robot twin
Recognizing a class for the virtual object twin, identified using at least one of the location information and the feature information
A collaborative learning method using digital twins.
delete The method of claim 1,
The location information is
Information on the separation distance and separation direction between the physical robot and the physical object or a coordinate value of the physical object
A collaborative learning method using a digital twin that includes
The method of claim 1,
The characteristic information is
Characteristic information obtained from sensing data for the physical object
A collaborative learning method using digital twins.
delete The method of claim 1,
The virtual object twin and the virtual robot twin are
Synchronized with the movement of the physical object and the physical robot, moving in the virtual space
A collaborative learning method using digital twins.
In a cooperative learning method using a digital twin performed in a physical robot,
detecting a physical object in physical space;
requesting learning of a virtual object twin existing in a virtual space corresponding to the physical space and corresponding to the physical object to a virtual robot twin existing in the virtual space and corresponding to the physical robot; and
Receiving the classification model from the virtual robot twin that performs learning on the classification model by using the virtual object twin,
The step of requesting learning of the virtual object twin as a virtual robot twin
When the available resources of the physical robot are less than or equal to a threshold resource,
A collaborative learning method using digital twins.
8. The method of claim 7,
The step of requesting learning of the virtual object twin as a virtual robot twin
requesting learning by using at least one of location information and characteristic information about the physical object
A collaborative learning method using digital twins.
9. The method of claim 8,
The location information is
Information on the separation distance and separation direction between the physical robot and the physical object or a coordinate value of the physical object
A collaborative learning method using a digital twin that includes
9. The method of claim 8,
The characteristic information is
Characteristic information obtained from sensing data for the physical object
A collaborative learning method using digital twins.
9. The method of claim 8,
The virtual robot twin
Recognizing a class for the virtual object twin by using at least one of the location information and the feature information, and learning the classification model
A collaborative learning method using digital twins.
delete 8. The method of claim 7,
The virtual object twin and the virtual robot twin are
Synchronized with the movement of the physical object and the physical robot, moving in the virtual space
A collaborative learning method using digital twins.

Images