JP2025502763A - METHOD FOR OPTIMIZING LEARNING MODEL FOR TARGET DEVICE AND SYSTEM THEREFOR - Google Patents

Abstract

The method and system for optimizing a learning model for a target device may include a step in which a learning model generation device generates a learning model by training a network function based on a predetermined framework using learning data, a step in which the learning model generation device encodes the learning model via a general-purpose encoding module so that it can be applied to a plurality of inference engines corresponding to a plurality of different operating environments or frameworks, a step in which the learning model generation device passes on the encoded learning model to the target device, a step in which the target device acquires site data related to the applied process, and a step in which the target device uses the site data to perform inference based on the learning model via an inference engine among the inference engines that is suitable for the operating environment of the target device.
[Selected figure] Figure 3

Description

The technical idea of this disclosure relates to a method and system for optimizing a learning model for a target device.

Recently, there has been a demand for adding artificial intelligence (AI)-based functions to computing devices, particularly embedded devices, that are used in fields such as security, transportation, manufacturing, medicine, autonomous driving, and smart homes. However, there has not yet been sufficient effort to develop development tools such as application programs for efficiently installing deep learning models formed using artificial intelligence in such embedded devices.

The efficiency of developing AI-based programs for embedded devices remains low, and is considered to be the biggest bottleneck impeding the development of related fields. In particular, existing AI development tools are primarily designed for the development of servers or cloud platforms, and there are currently insufficient AI development tools suited to embedded devices such as Internet of Things (IoT) devices.

In addition, to effectively install deep learning models on embedded devices with various operating environments, various optimization processes are required, but the technologies that have been used up until now have limitations in that they cannot guarantee optimal operation for deep learning models.

The technical problem that the method and system for optimizing a learning model for a target device based on the technical concept of this disclosure aims to solve is to provide a method and system that can optimize and install a learning model to suit various operating environments of the target device.

The technical problems that the method and system according to the technical ideas of this disclosure are intended to solve are in no way limited to the above-mentioned technical problems, and other technical problems not mentioned will be clearly understood by those skilled in the art from the following description.

According to one aspect of the technical idea of this disclosure, a method for optimizing a learning model for a target device may include a step in which a learning model generation device generates a learning model by training a network function based on a predetermined framework using learning data, a step in which the learning model generation device encodes the learning model via a general-purpose encoding module so that the learning model can be applied to a plurality of inference engines corresponding to a plurality of different operating environments or frameworks, a step in which the learning model generation device transfers the encoded learning model to the target device, a step in which the target device acquires on-site data related to the applied process, and a step in which the target device uses the on-site data to perform inference based on the learning model via an inference engine among the inference engines that is suitable for the operating environment of the target device.

According to an exemplary embodiment, the operating environment may be differentiated based on at least one of the operating system and hardware of the target device.

According to an exemplary embodiment, the operating environment that is differentiated by the hardware may include at least one of a general-purpose central processing unit (CPU) environment, a general-purpose graphics processing unit (GPU) environment, and an embedded environment.

According to this exemplary embodiment, the field data input to the learning model and the output data of the learning model may be converted into a predetermined material structure via at least one buffer.

According to an exemplary embodiment, the method for optimizing the learning model for the target device may further include a step in which the learning model generation device retrains the learning model based on the on-site data and the output data.

According to one aspect of the technical idea of this disclosure, a learning model optimization system for a target device may include a learning model generation device that generates a learning model by training a network function based on a predetermined framework using learning data, the learning model generation device encodes the learning model via a general-purpose encoding module so that the learning model can be applied to a plurality of inference engines corresponding to a plurality of different operating environments or frameworks, and the learning model generation device passes the encoded learning model to the target device, and the target device that acquires on-site data related to a process and uses the on-site data to perform inference based on the learning model via an inference engine among the inference engines that is suitable for the operating environment.

According to an embodiment based on the technical idea of this disclosure, the learning model generated in the learning model generation device (server) can be converted and operated to suit various operating environments of the target device.

The effects obtained by the method and system based on the technical ideas of this disclosure are in no way limited to the effects described above, and other effects not mentioned will be clearly understood by a person having ordinary skill in the technical field to which this disclosure pertains from the following description.

A brief description of each drawing is provided to provide a more complete understanding of the drawings referenced in this disclosure.

FIG. 1 is a simplified block diagram of a system for optimizing a learning model for a target device according to an embodiment of the present disclosure. 1 is a flowchart illustrating a method for optimizing a learning model for a target device according to an embodiment of the present disclosure. A block diagram conceptually showing the configuration of a learning model generation module of a learning model generation device and an inference module of a target device according to an embodiment of the present disclosure. 1 is a flowchart illustrating a method for optimizing a learning model for a target device according to an embodiment of the present disclosure. 1 is a block diagram showing a simplified configuration of a learning model generation device according to an embodiment of the present disclosure. FIG. 2 is a block diagram illustrating a simplified configuration of a target device according to an embodiment of the present disclosure.

The technical idea of this disclosure can be modified in various ways and can have various embodiments, so a specific embodiment is illustrated in the drawings and described in detail. However, this is not intended to limit the technical idea of this disclosure to a specific embodiment, and includes all modifications, equivalents, and alternatives that fall within the scope of the technical idea of this disclosure.

When explaining the technical ideas of this disclosure, detailed explanations of known technologies related to the present invention will be omitted if it is deemed that such explanations may obscure the gist of this disclosure. Note that numbers used in the explanation of this disclosure (e.g., "first", "second", etc.) are merely identification codes for distinguishing one component from another.

In addition, in this disclosure, when a certain component is referred to as being "coupled" or "connected" to another component, the certain component may be directly coupled or connected to the other component, but unless otherwise specified in this specification or clearly contradicted by the context, there may be another component between them and the components may be coupled or connected via the other component.

Furthermore, the terms "part", "device", "child", "module" and the like used in this disclosure refer to a unit that processes at least one function or operation, and this includes a processor, a microprocessor, a microcontroller, a central processing unit (CPU: Central Processing Unit), a graphics processing unit (GPU: Graphics Processing Unit), an accelerated processing unit (APU: Accelerated Processing Unit), a digital signal processor (DSP: Digital signal Processors), an application specific integrated circuit (ASIC: Application Specific Integrated Circuits), a field programmable gate array (FPGA: Field Programmable Gate Array), and the like. It may be realized by hardware, software, or a combination of hardware and software, such as Programmable Gate Arrays.

And we would like to clarify that the distinctions made between components in this disclosure are merely distinctions made according to the main functions that each component is responsible for. That is, two or more components described below may be combined into one component, or one component may be divided into two or more components for further subdivided functions. And it goes without saying that each of the components described below may perform some or all of the functions that are the responsibility of the other components in addition to the main function that it is responsible for, or some of the functions of the main function that each component is responsible for may be exclusively performed by the other components.

The method according to the embodiment of this disclosure may be performed in a personal computer, a workstation, a server computer device, or the like, that has computing power, or may be performed in a separate device for this purpose.

The method may also be performed in one or more computing devices. For example, at least one or more steps of the method according to the embodiments of the present disclosure may be performed in a client device, and other steps may be performed in a server device. In such a case, the client device and the server device may be connected via a network to transmit and receive the computation results. Alternatively, the method may be performed using distributed computing techniques.

Also, throughout this specification, network function may be used synonymously with neural network. Here, a neural network may generally be composed of a collection of interconnected computational units that may be referred to as nodes, and such nodes may also be referred to as neurons. A neural network is generally composed of a plurality of nodes. The nodes that make up a neural network may be connected to each other by one or more links.

Some of the nodes that make up the neural network may form a layer based on their distance from the first input node. For example, a set of nodes that are a distance n from the first input node may form an n-th layer.

The neural networks described in this specification may include a deep neural network (DNN) that includes multiple hidden layers in addition to the input and output layers.

The following describes the embodiments of this disclosure in detail.

Figure 1 is a simplified block diagram of a learning model optimization system for a target device according to an embodiment of the present disclosure.

The system may include a learning model generating device 110 and a target device 120.

In an embodiment, the learning model generation device 110 may be a cloud server.

The learning model generating device 110 may generate a learning model and pass it to the target device 120. To this end, the learning model generating device 110 may include a learning model generation module. The learning model generation module may be a software module.

Specifically, the learning model generation module may generate a network function based on a predetermined framework (e.g., TensorFlow), and if learning data is input from the outside, may learn the network function based on the input data, and generate a learning model as a result of the learning. The learning model generation module may also encode the generated learning model via a general-purpose encoding module into a format that can be run in multiple operating environments and/or frameworks. The learning model may be passed to the target device 120 in the form of an encoded file.

In addition, in an embodiment, the learning model generation device 110 may retrain the learning model using the field data transferred from the target device 120 and the output data output by the learning model based on the field data.

The target device 120 refers to a target device on which a learning model is to be realized, and includes all kinds of computing devices such as servers, personal computers (PCs), smart phones, smart pads, and tablet PCs. In an embodiment, the target device 120 may further include an IoT terminal, an edge device, an embedded board, and the like that operates in an embedded environment.

The target device 120 may receive at least one execution code, program, library, etc. for generating an encoded learning model and an inference module from the learning model generation device 110, and drive the learning model via the generated inference module. In this case, the inference module may be a software module.

The inference module may include multiple inference engines and at least one buffer (input/output buffer) corresponding to different operating environments. The inference module may run the learning model through an inference engine suitable for the operating environment, such as the operating system and/or hardware of the target device 120, and perform inference using field data acquired by the target device 120.

At this time, the field data input as the learning model and the output data of the learning model may be converted into a specified data structure via a buffer.

Figure 2 is a flowchart for explaining a method for optimizing a learning model for a target device according to an embodiment of the present disclosure, and Figure 3 is a block diagram conceptually illustrating the configuration of a learning model generation module of a learning model generation device according to an embodiment of the present disclosure and an inference module of a target device.

In step S210, the learning model generation device 110 may generate a learning model 312 by inputting learning data into the network function 311 of the learning model generation module 310 and performing learning.

At this time, the network function 311 may be generated based on a predetermined framework. The framework refers to a type of package configured to allow a developer to easily use multiple verified libraries and modules, pre-trained algorithms, etc., and may include, for example, Tensorflow, Keras, Pytorch, Cafe, MXnet, etc., but is not limited to these, and a wide variety of frameworks can be used to generate the network function 311.

In this embodiment, the learning model generation module 310 may generate the learning model 312 using a network function based on tensorflow.

In an embodiment, the training data may be a plurality of images. For example, the training data may be process images acquired (or photographed) during any process such as the production, manufacturing, or processing of a product, and may include information about the correct answer (label) to be detected or read.

In step S220, the learning model generation device 110 may encode the learning model 312 via a generic encoding module 313 of the learning model generation module 310. Through step S220, the learning model 312 can be converted into a format that can be run in multiple different operating environments and/or frameworks.

In an embodiment, the encoding module 313 may convert the learning model 312 into a neural network data format such as Neural Network Exchange Format (NNEF) or Open Neural Network Exchange (ONNX), but is not limited thereto.

In step S230, the learning model generation device 110 may pass the encoded learning model 312 to the target device 120.

Here, the target device 120 is provided to control the process or to measure or acquire specific data within the process environment and perform diagnosis or prediction related to the process based on the data, and may be provided with an inference module 320 for driving a learning model to perform inference.

In step S240, the target device 120 may acquire on-site data related to the process. For example, the target device 120 may be provided as a single component, or the on-site data may be acquired from a sensing device (at least one sensor, camera, etc.) connected to the target device 120 via wired or wireless communication.

In an embodiment, the on-site data may be at least one image (e.g., a process image).

In step S250, the target device 120 may perform inference using the on-site data via the inference module 320.

In the S250 embodiment, the inference module 320 may be provided with multiple inference engines 321, each suitable for a different operating environment, and may perform inference based on field data by driving the learning model 312 through an inference engine 321 suitable for the operating environment of the target device 120 among the multiple inference engines 321 installed.

Here, the operating environment may be distinguished based on at least one of the operation system and hardware (or hardware architecture) of the target device 120.

In an embodiment, the operating environment that is differentiated by the operating system may include at least one of a Linux environment and a window environment. However, this is merely an example, and different operating environments may be defined according to a wide variety of operating systems, such as IOS, Android, and Mac OS.

In addition, in an embodiment, the operating environment distinguished by hardware may include at least one of a general-purpose central processing unit (CPU) environment, a general-purpose graphics processing unit (GPU) environment, and an embedded environment. However, this is merely an example, and different operating environments may be defined according to a wide variety of hardware architectures, such as Nvidia, Jetson, Raspberry, ARM Cortex-A Family, Qcam Family, Hisilicon Family, X86-CPU, and X86-GPU.

In an embodiment, the inference engine 321 may include a first inference engine suitable for a general-purpose GPU environment and/or embedded environment and a second inference engine suitable for a general-purpose CPU environment.

In an embodiment, the input and output data of the learning model 312 driven via the inference engine 321 may be converted into a predetermined material structure via at least one buffer 322.

For example, the buffer 322 may include an input buffer and an output buffer, and the target device 120 may assign the field data to the input buffer and convert it into a material structure suitable for driving the learning model 312, and may assign the output data generated based on the inference result of the learning model 312 to the output buffer and convert it into a specified material structure. The material structure may be a structure corresponding to a program language that the target device 120 can parse.

Figure 4 is a flowchart illustrating a method for optimizing a learning model for a target device according to an embodiment of the present disclosure.

Steps S410 to S450 in method 400 are similar to steps S210 to S250 in method 200 described above with reference to FIG. 2, so duplicated explanations will be omitted.

The method 400 may further include a step of retraining the learning model 313.

Specifically, in step S460, the target device 120 may input the on-site data and the on-site data, and pass the output data obtained via the inference engine 321 and the learning model 313 to the learning model generation device 110, and the learning model generation device 110 may update the learning model 313 by re-learning the learning model 313 based on the received on-site data and output data.

The updated learning model 313 may be encoded via a general-purpose encoding module 313 and transmitted to the target device 120.

In an embodiment, step S460 may be performed periodically at regular time intervals.

Figure 5 is a block diagram showing a simplified configuration of a learning model generation device according to an embodiment of this disclosure.

The communication unit 510 may transmit and receive data to and from the outside. The communication unit 510 may include a wired or wireless communication unit. When the communication unit 510 includes a wired communication unit, the communication unit 510 may include one or more components that communicate via a local area network (LAN), a wide area network (WAN), a value-added network (VAN), a mobile radio communication network, a satellite communication network, and combinations thereof. When the communication unit 510 includes a wireless communication unit, the communication unit 510 may transmit and receive data or signals wirelessly using cellular communication, wireless LAN (e.g., Wi-Fi), etc. In an embodiment, the communication unit may send and receive data or signals to and from an external device or an external server under the control of the processor 540.

The input unit 520 may receive a wide variety of user commands through external operations. To this end, the input unit 520 may include one or more input devices or may be connected thereto. For example, the input unit 520 may be connected to an interface for a wide variety of inputs, such as a keypad or a mouse, to receive user commands. To this end, the input unit 520 may include not only a USB port but also an interface such as Thunderbolt. In addition, the input unit 520 may include a wide variety of input devices, such as a touch screen or a button, or may be coupled to these to receive external user commands.

Memory 530 may store programs and/or program commands for operation of processor 540 and may temporarily or permanently store input/output data. The memory 530 may be a flash memory type, a hard disk type, a multimedia card micro type, a card type memory (e.g., SD or XD memory, etc.), a random access memory (RAM), a static random access memory (SRAM), a read only memory (ROM), an electrically erasable programmable read-only memory (EEPROM), a programmable read-only memory (PROM), or a programmable read-only memory (PROM). The device may be equipped with at least one type of storage medium: read-only memory, magnetic memory, magnetic disk, or optical disk.

The memory 530 may also store various network functions, algorithms, libraries, etc., and may store a wide variety of data, programs (one or more instructions), applications, software, instructions, code, etc. for driving and controlling the device 500.

The processor 540 may run at least one program and/or instruction to control the overall operation of the device 500. The processor 540 may run one or more programs stored in the memory 530. The processor 540 may refer to a central processing unit (CPU), a graphics processing unit (GPU), or a dedicated processor in which the methods according to the technical ideas of this disclosure are performed.

In an embodiment, the processor 540 may generate the learning model generation module 310 based on at least one program and/or instruction. The learning model generation module 310 may be a software module.

In an embodiment, the processor 540 may generate a learning model by training a network function based on a predetermined framework using training data, and the learning model generation device may encode the learning model via a general-purpose encoding module so that the learning model can be applied to multiple inference engines corresponding to multiple different operating environments or frameworks, and pass the encoded learning model to the target device.

In an embodiment, the processor 540 may retrain the learning model based on the field data received from the target device and the output data of the inference engine and the learning model.

Figure 6 is a block diagram showing a simplified configuration of a target device according to an embodiment of this disclosure.

The communication unit 610 may receive data from the outside. The communication unit 610 may include a wired or wireless communication unit. When the communication unit 610 includes a wired communication unit, the communication unit 610 may include one or more components that communicate via a local area network (LAN), a wide area network (WAN), a value-added network (VAN), a mobile radio communication network, a satellite communication network, or a combination thereof. When the communication unit 610 includes a wireless communication unit, the communication unit 610 may transmit and receive data or signals wirelessly using cellular communication, a wireless LAN (e.g., Wi-Fi), or the like. In an embodiment, the communication unit may send and receive data or signals to and from an external device or an external server under the control of the processor 640.

The input unit 620 may receive a wide variety of user commands through external operations. To this end, the input unit 620 may include or be connected to one or more input devices. For example, the input unit 620 may be connected to an interface for a wide variety of inputs, such as a keypad or a mouse, to receive user commands. To this end, the input unit 620 may include not only a USB port but also an interface such as Thunderbolt. In addition, the input unit 620 may include a wide variety of input devices, such as a touch screen or a button, or may be coupled to these to receive external user commands.

Memory 630 may store programs and/or program commands for operation of processor 640 and may temporarily or permanently store input/output data. The memory 630 may be a flash memory type, a hard disk type, a multimedia card micro type, a card type memory (such as an SD or XD memory), a random access memory (RAM), a static random access memory (SRAM), a read only memory (ROM), an electrically erasable programmable read-only memory (EEPROM), a programmable read-only memory (PROM), or a programmable read-only memory (PROM). The device may be equipped with at least one type of storage medium: read-only memory, magnetic memory, magnetic disk, or optical disk.

Memory 630 may also store various network functions, algorithms, libraries, etc., and may store a wide variety of data, programs (one or more instructions), applications, software, instructions, code, etc. for driving and controlling device 600.

The processor 640 may run at least one program and/or instruction to control the overall operation of the device 600. The processor 640 may run one or more programs stored in the memory 630. The processor 640 may refer to a central processing unit (CPU), a graphics processing unit (GPU), or a dedicated processor in which the methods according to the technical ideas of this disclosure are performed.

In this embodiment, the processor 640 may generate the inference module 320 based on the learning model, at least one program and/or instructions received from the learning model generation device. The inference module 320 may be a software module.

In an embodiment, the processor 540 may acquire on-site data associated with the process and use the on-site data to perform inference based on the learning model via an inference engine among the inference engines that is suitable for the operating environment of the device 600.

In an embodiment, the processor 540 may retrain the learning model based on the field data received from the target device and the output data of the inference engine and the learning model.

On the other hand, although not shown, the device 600 may further include a sensing unit and/or an image capturing unit for acquiring on-site data. In this case, the sensing unit may be configured with at least one sensing module, and the image capturing unit may be configured with a camera module.

Methods according to embodiments of this disclosure may be embodied in a variety of types of program commands that may be executed via a variety of computer means and recorded on a computer readable medium. The computer readable medium may include program commands, data files, data structures, and the like, alone or in combination. The program commands recorded on the medium may be those specifically designed and constructed for this disclosure, or may be those known and available to those skilled in the art of computer software. Examples of computer readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tape, optical media such as compact discs (CDs), digital versatile discs (DVDs), magneto-optical media such as floptical disks, and hardware devices specifically configured to store and execute program commands, such as read only memory (ROM), random access memory (RAM), flash memory, etc. Examples of program commands include not only machine code, such as that produced by a compiler, but also high-level language code that can be executed by a computer using an interpreter or the like.

In addition, the methods according to the disclosed embodiments may be provided in a computer program product. The computer program product may be traded as a commodity between a seller and a buyer.

The computer program product may include a S/W program and a computer-readable storage medium on which the S/W program is stored. For example, the computer program product may include a product in the form of a S/W program (e.g., a downloadable application) that is electronically distributed by a manufacturer of electronic devices or via an online market (e.g., Google Play Store, Application Store). For electronic distribution, at least a portion of the S/W program may be stored in the storage medium or may be temporarily generated. In this case, the storage medium may be a storage medium of a manufacturer's server, an online market server, or a relay server that temporarily stores the S/W program.

In a system consisting of a server and a client device, the computer program product may include a storage medium of the server or a storage medium of the client device. Alternatively, when there is a third device (e.g., a smartphone) connected to the server or the client device via communication, the computer program product may include a storage medium of the third device. Alternatively, the computer program product may include the S/W program itself that is transmitted from the server to the client device or the third device, or transmitted from the third device to the client device.

In this case, one of the server, the client device, and the third device may run the computer program product to perform the method according to the disclosed embodiment. Alternatively, two or more of the server, the client device, and the third device may run the computer program product to perform the method according to the disclosed embodiment in a distributed manner.

For example, a server (e.g., a cloud server or an artificial intelligence server) may launch a computer program product stored on the server to control a client device communicatively connected to the server to perform a method according to the disclosed embodiments.

Although the embodiments have been described in detail above, the scope of the disclosure is in no way limited thereto, and various modifications and improvements made by those skilled in the art using the basic concept of this disclosure as defined in the appended claims also fall within the scope of the disclosure.

Claims (6)

1. A method for optimizing a learning model for a target device, comprising:
A learning model generating device generates a learning model by training a network function based on a predetermined framework using training data;
The learning model generation device encodes the learning model via a general-purpose encoding module so that the learning model can be applied to a plurality of inference engines corresponding to a plurality of different operating environments or frameworks;
The learning model generation device transmits the encoded learning model to the target device;
the target device acquiring in situ data associated with the applied process;
The method includes a step in which the target device uses the on-site data to perform inference based on the learning model via an inference engine among the inference engines that is suitable for the operating environment of the target device. The method of claim 1 , wherein the operating environments are differentiated based on at least one of an operation system and hardware of the target device. 3. The method of claim 2, wherein the operating environment differentiated by the hardware includes at least one of a general purpose central processing unit (CPU) environment, a general purpose graphics processing unit (GPU) environment, and an embedded environment. The method of claim 1 , wherein the field data input to the learning model and the output data of the learning model are converted to a predetermined material structure via at least one buffer. The method according to claim 4 , further comprising the step of: the learning model generation device retraining the learning model based on the field data and the output data. In a system for optimizing a learning model for a target device,
a learning model generation device that generates a learning model by training a network function based on a predetermined framework using training data, encodes the learning model via a general-purpose encoding module so that the learning model can be applied to a plurality of inference engines corresponding to a plurality of different operating environments or frameworks, and passes the encoded learning model to the target device;
and the target device acquires on-site data related to the process, and uses the on-site data to perform inference based on the learning model via an inference engine among the inference engines that is suitable for the operating environment.

Images