KR102701117B1 - Electronic apparatus capable of image super-resolution based on deep learning and the operating method thereof - Google Patents

Abstract

The present invention provides an electronic device capable of image super-resolution based on deep learning and an operating method thereof, thereby enabling a user to more conveniently obtain a high-resolution image from a low-resolution image.

Description

{ELECTRONIC APPARATUS CAPABLE OF IMAGE SUPER-RESOLUTION BASED ON DEEP LEARNING AND THE OPERATING METHOD THEREOF}

The present invention relates to an electronic device capable of image super-resolution based on deep learning and an operating method thereof.

Recently, various technologies have emerged to improve the resolution of images. The technology that converts low-resolution images into high-resolution images is called super-resolution technology, and this super-resolution technology can be used in various fields such as games, movies, CCTV security, and vehicle number recognition.

The technology that is mainly used to improve the resolution of images is image interpolation, which fills in empty pixels based on the pixel values of surrounding pixels.

In addition, since machine learning-based artificial intelligence technology has recently emerged that can create a learning model to predict results based on a certain amount of training data, the use of such artificial intelligence technology can also be considered for technologies that improve image resolution.

In this regard, a method can be considered of creating a learning model for improving the resolution of images by constructing training data with low-resolution images and high-resolution images corresponding to each of the images, designating the low-resolution images as input, matching the high-resolution images corresponding to each image with the corresponding correct answer value, and performing machine learning on a given deep learning model.

The present invention proposes an electronic device capable of image super-resolution based on deep learning and an operating method thereof, thereby enabling a user to more conveniently obtain a high-resolution image from a low-resolution image.

An electronic device according to one embodiment of the present invention comprises: a training data storage unit storing a plurality of pre-collected training images and a pre-produced high-resolution converted image corresponding to each of the plurality of training images; a model generation unit selecting one of the plurality of training images stored in the training data storage unit and a high-resolution converted image corresponding thereto, and passing the selected training image through a neural network model to generate a super-resolution prediction image and a low-resolution prediction image obtained by downsampling the super-resolution prediction image; and a model generation unit repeatedly performing machine learning for each of the plurality of training images to minimize an error between the super-resolution prediction image and the selected high-resolution converted image and an error between the low-resolution prediction image and the selected training image, thereby generating a super-resolution model; and after the generation of the super-resolution model is completed, when a first real image is input by a user and a super-resolution command is input, the first real image is input to the super-resolution model, and the first real image is generated by applying a machine learning command to the first real image. It includes an image generation unit that generates a corresponding super-resolution image.

In addition, an operating method of an electronic device according to an embodiment of the present invention comprises the steps of: maintaining a training data storage unit storing a plurality of pre-collected training images and a pre-produced high-resolution converted image corresponding to each of the plurality of training images; selecting one of the plurality of training images stored in the training data storage unit and a high-resolution converted image corresponding thereto, passing the selected training image through a neural network model to generate a super-resolution prediction image and a low-resolution prediction image obtained by downsampling the super-resolution prediction image; and repeatedly performing machine learning for each of the plurality of training images so that an error between the super-resolution prediction image and the selected high-resolution converted image and an error between the low-resolution prediction image and the selected training image are minimized, thereby generating a super-resolution model; and after the generation of the super-resolution model is completed, when a super-resolution command is input by a user while a first real image is input, the first real image is input to the super-resolution model to generate a super-resolution model corresponding to the first real image. It includes a step of generating a super-resolution image.

The present invention provides an electronic device capable of image super-resolution based on deep learning and an operating method thereof, thereby enabling a user to more conveniently obtain a high-resolution image from a low-resolution image.

FIG. 1 is a drawing illustrating the structure of an electronic device according to one embodiment of the present invention.
FIGS. 2 to 4 are drawings for explaining the operation of an electronic device according to one embodiment of the present invention.
FIG. 5 is a flowchart illustrating an operating method of an electronic device according to an embodiment of the present invention.

Hereinafter, embodiments according to the present invention will be described in detail with reference to the attached drawings. This description is not intended to limit the present invention to specific embodiments, but should be understood to include all modifications, equivalents, and substitutes included in the spirit and technical scope of the present invention. In describing each drawing, similar reference numerals are used for similar components, and unless otherwise defined, all terms used in this specification, including technical or scientific terms, have the same meaning as generally understood by a person of ordinary skill in the art to which the present invention belongs.

In this document, when a part is said to "include" a certain component, this does not mean that other components are excluded, but rather that other components may be included, unless otherwise specifically stated. In addition, in various embodiments of the present invention, each of the components, functional blocks or means may be composed of one or more subcomponents, and the electrical, electronic and mechanical functions performed by each component may be implemented by various known elements such as electronic circuits, integrated circuits, ASICs (Application Specific Integrated Circuits), or mechanical elements, and each may be implemented separately or two or more may be integrated into one.

Meanwhile, the blocks of the attached block diagram or the steps of the flow chart may be interpreted as computer program instructions that are loaded into a processor or memory of a device capable of data processing, such as a general-purpose computer, a special-purpose computer, a portable notebook computer, or a network computer, and perform designated functions. Since these computer program instructions may be stored in a memory equipped in a computer device or a computer-readable memory, the functions described in the blocks of the block diagram or the steps of the flow chart may be produced as a manufactured product that includes a command means for performing the same. In addition, each block or each step may represent a module, segment, or part of code that includes one or more executable instructions for performing a specific logical function(s). In addition, it should be noted that in some alternative embodiments, the functions mentioned in the blocks or steps may be performed in a different order from the specified order. For example, two blocks or steps illustrated in succession may be performed substantially simultaneously or in reverse order, and in some cases, some blocks or steps may be performed with some blocks or steps omitted.

FIG. 1 is a drawing illustrating the structure of an electronic device according to one embodiment of the present invention.

Referring to FIG. 1, an electronic device (110) according to the present invention includes a training data storage unit (111), a model generation unit (112), and an image generation unit (113).

The training data storage unit (111) stores a plurality of pre-collected training images and a pre-produced high-resolution converted image corresponding to each of the plurality of training images. Here, it is preferable that the training images stored in the training data storage unit (111) are composed of low-resolution images, and at this time, it is preferable that the high-resolution converted image corresponding to each training image is composed of an image that implements each training image in high resolution.

The model generation unit (112) selects one training image and its corresponding high-resolution conversion image from among the plurality of training images stored in the training data storage unit (111), passes the selected training image through a neural network model, thereby generating a super-resolution prediction image and a low-resolution prediction image obtained by downsampling the super-resolution prediction image, and then repeatedly performs machine learning for each of the plurality of training images so that the error between the super-resolution prediction image and the selected high-resolution conversion image and the error between the low-resolution prediction image and the selected training image are minimized, thereby generating a super-resolution model.

At this time, according to one embodiment of the present invention, the model generation unit (112) may include an output map generation unit (114), a forward output unit (115), a reverse output unit (116), a super-resolution prediction unit (117), a low-resolution prediction unit (118), and a learning execution unit (119). Hereinafter, the detailed configuration of the model generation unit (112) will be described based on the framework illustrated in FIGS. 2 to 4.

When the machine learning process for the first training image (LR) is to be performed in order as the first training image (LR) is selected among the plurality of training images, the output map generation unit (114) passes the first training image (LR) through T (T is a natural number greater than or equal to 3) convolution layers (211) to generate T output maps.

In this regard, when T is 3, the output map generation unit (114) can generate three output maps by passing the first training image (LR) through three convolution layers (211), as shown in the figure in FIG. 2, as shown in the figure at reference numeral 211.

The forward output unit (115) sequentially applies each of the T output maps as input to each of the T first cross-learning feedback blocks (CFBs) (212) configured to receive each output map as input, and when the t-th output map is applied to the t-th first CFB, the output of the t-1-th first CFB is also applied as input to the t-th first CFB, thereby generating the output of each of the T first CFBs (212), thereby generating the T forward outputs.

In this regard, as in the example described above, when T is 3, in the present invention, as shown in the drawing reference numeral 212, there may be three first CFBs (212), and at this time, each first CFB may be configured to receive each of the three output maps as input. At this time, when the three output maps are generated through the forward output unit (115) through the output map generation unit (114), as shown in the drawing reference numeral 212, each output map is sequentially applied as input to the first CFB one by one, but when the t-th output map is applied to the t-th first CFB, the output of the t-1-th first CFB is applied together as input to the t-th first CFB, thereby generating the output of each of the three first CFBs (212), thereby generating three forward outputs.

Specifically, the forward output unit (115) can apply the first output map as input to the first CFB, as indicated by '(1)' in Fig. 2. At this time, since the CFB of the previous order of the first CFB does not exist, the forward output unit (115) can apply only the first output map as input to the first CFB. Then, the forward output unit (115) can apply the second output map as input to the second CFB, as indicated by '(2)' in Fig. 2, and at the same time apply the output of the first CFB as input to the second CFB. Finally, the forward output unit (115) can input the third output map to the third first CFB as an input, as indicated by '(3)' in Fig. 2, and simultaneously input the output of the second first CFB to the third first CFB. Through this, the forward output unit (115) can generate three forward outputs by generating outputs from each of the first, second, and third first CFBs (212).

The reverse output unit (116) applies each of the T forward outputs as inputs to each of the T second CFBs (213) configured to receive each forward output as inputs, one by one, in an order opposite to the order of application of inputs to the T first CFBs (212), but when the t-th forward output is applied to the t-th second CFB, the output of the t+1-th second CFB and the t-th output map are applied together as inputs to the t-th second CFB, thereby generating the outputs of each of the T second CFBs (213), thereby generating the T reverse outputs.

In relation to this, as in the example described above, when T is 3, in the present invention, as shown in the drawing reference numeral 213, three second CFBs (213) may exist, and at this time, each second CFB may be configured to receive the forward outputs of the three first CFBs (212) as inputs, respectively. At this time, when three forward outputs are generated through the forward output unit (115), as shown in the drawing symbol 213, each forward output is sequentially applied as an input to each of the second CFBs in an order opposite to the order of application of inputs to the three first CFBs (212), but when the t-th forward output is applied to the t-th second CFB, the output of the t+1-th second CFB and the t-th output map are applied together as inputs to the t-th second CFB, thereby generating the output of each of the three second CFBs (213), thereby generating three reverse outputs.

Specifically, the reverse output unit (116) can apply the third forward output as input to the third second CFB, as indicated by '(3)' in Fig. 2. At this time, since there is no CFB in the order following the third second CFB, the forward output unit (115) can apply only the third forward output and the third output map as input to the third second CFB. Then, the reverse output unit (116) can apply the second forward output as input to the second second CFB, as indicated by '(2)' in Fig. 2, and at the same time apply the output of the third second CFB and the second output map together as input to the second second CFB. Finally, the reverse output unit (116) can input the first forward output to the first CFB as an input, as indicated by '(1)' in Fig. 2, and simultaneously input the output of the second CFB and the first output map to the first CFB as input. Through this, the reverse output unit (116) can generate three reverse outputs by generating outputs from each of the first, second, and third second CFBs (213).

At this time, according to one embodiment of the present invention, each of the T first CFBs and the T second CFBs may be a block composed of a first connection block (311) in which five residual attention blocks (RABs) are connected to each other, and a second connection block (312) in which three RABs are connected to each other, as illustrated in the diagram of FIG. 3.

At this time, the CFB may be configured so that the input (F _in ) is applied to the first RAB constituting the first connection block (311), and at the same time, the CFB may be configured so as to produce the output (F _out ) of the fifth RAB constituting the first connection block (311) as the final output.

At this time, the CFB may be configured so that when an output is produced through the first RAB constituting the first connection block (311), the produced output is applied as an input to the second RAB constituting the first connection block (311), and at the same time, the produced output is upsampled and applied as an input to the first RAB constituting the second connection block (312).

At this time, the 2nd, 3rd, 4th, and 5th RABs constituting the first connection block (311) are configured to receive the output of the RAB of the previous order, but the 3rd and 4th RABs are configured to additionally receive the results of downsampling the outputs of the 1st and 2nd RABs constituting the second connection block (312), and the 5th RAB may be configured to additionally receive the results of downsampling the outputs of the 2nd and 3rd RABs and the output of the 3rd RAB constituting the second connection block (312).

In addition, each of the second and third RABs constituting the second connection block (312) may be configured to receive the output of the RAB of the previous order, and additionally receive the upsampled output of the second and third RABs constituting the first connection block (311).

At this time, according to one embodiment of the present invention, the RAB may be a block configured to sequentially pass the input (F _in ) through a convolution layer, a channel attention module, and a spatial attention module to generate an output, as illustrated in FIG. 4, and then produce the final output (F _out ) through a residual connection between the generated output and the input. Here, channel attention refers to a model that calculates the relationship between feature map channels by focusing on the channel and emphasizing it, and spatial attention refers to a model that calculates the relationship between surrounding pixels by convolution, unlike channel attention, and performs emphasis through an element-by-element multiplication operation with the feature map extracted through this process.

The super-resolution prediction unit (117) performs upsampling by combining the T forward outputs and the T reverse outputs, passing them through a deconvolution layer (214) by combining outputs of the same order, thereby generating T upsampling maps, and then combining the results of combining the T upsampling maps and passing them through a convolution layer (215) with the results of performing bicubic interpolation (216) on the first training image, thereby generating a first super-resolution prediction image (SR).

In this regard, as shown in the diagram in FIG. 2, when three forward outputs and three reverse outputs are generated, the super-resolution prediction unit (117) can perform upsampling by combining the first forward output and the first reverse output and passing them through a deconvolution layer, as indicated by '(1)', and can perform upsampling by combining the second forward output and the second reverse output and passing them through a deconvolution layer, as indicated by '(2)', and can perform upsampling by combining the third forward output and the third reverse output and passing them through a deconvolution layer, as indicated by '(3)'.

Then, the super-resolution prediction unit (117) can combine three upsampling maps as illustrated in the reference numeral 215 and pass them to the convolution layer (215), and perform bicubic interpolation (216) on the first training image as illustrated in the reference numeral 216. Here, bicubic interpolation means a methodology for estimating new pixel values of an image based on surrounding pixel values.

Thereafter, the super-resolution prediction unit (117) can generate a first super-resolution prediction image (SR) by combining the result of passing through the convolution layer (215) and the result of bicubic interpolation (216).

The low-resolution prediction unit (118) generates the first super-resolution prediction image (SR) through the super-resolution prediction unit (117), and then, as shown in the drawing reference numeral 217, passes the first super-resolution prediction image (SR) through the convolution layer (217) to perform downsampling, thereby generating the first low-resolution prediction image (LR').

In this way, when the first super-resolution prediction image (SR) and the first low-resolution prediction image (LR') are generated, the learning performing unit (119) performs machine learning so that a weighted sum of a first error between the first super-resolution prediction image (SR) and the first high-resolution converted image corresponding to the first training image stored in the training data storage unit (111) and a second error between the first low-resolution prediction image (LR') and the first training image (LR) is minimized.

In relation to this, the learning execution unit (119) can perform machine learning so that the loss value is minimized after calculating the weighted sum according to the following mathematical expression 1 as a loss value.

Here, L ₁ represents the first error, L ₂ represents the second error, SR represents the first super-resolution prediction image, HR represents the first high-resolution converted image, LR' represents the first low-resolution prediction image, LR represents the first training image, and θ represents a pre-specified weight.

At this time, according to one embodiment of the present invention, the learning execution unit (119) can perform backpropagation processing so that the loss value as in the mathematical expression 1 is minimized.

According to the method described so far, the model generation unit (112) can generate a super-resolution model for improving the resolution of an image by repeatedly performing machine learning based on the plurality of training images stored in the training data storage unit (111) and the corresponding high-resolution converted images.

In this way, after the generation of the super-resolution model is completed through the model generation unit (112), when a super-resolution command is entered by the user while a first real image is input, the image generation unit (113) inputs the first real image as an input to the super-resolution model and generates a super-resolution image corresponding to the first real image.

At this time, according to one embodiment of the present invention, when a super-resolution command is input by a user while the first real image is input after the generation of the super-resolution model is completed, the image generation unit (113) inputs the first real image as an input to the super-resolution model, thereby passing the first real image through each of the T convolution layers (211), as illustrated in the drawing reference numeral 211, to generate T real output maps.

Thereafter, the image generating unit (113) sequentially applies each of the T actual output maps as input to each of the T first CFBs (212), as illustrated in the drawing symbol 212, but when the t-th actual output map is applied to the t-th first CFB, the output of the t-1-th first CFB is also applied as input to the t-th first CFB, thereby generating the output of each of the T first CFBs (212), thereby generating the T actual forward outputs.

Then, the image generating unit (113) applies each of the T actual forward outputs as inputs to each of the T second CFBs (213) one by one in an order opposite to the order of application of inputs to the T first CFBs (212), as illustrated in the drawing reference numeral 213, but when the t-th actual forward output is applied to the t-th second CFB, the output of the t+1-th second CFB and the t-th actual output map are applied together as inputs to the t-th second CFB, thereby generating the outputs of each of the T second CFBs (213), thereby generating the T actual reverse outputs.

Thereafter, the image generation unit (113) performs upsampling by combining the T actual forward outputs and the T actual reverse outputs, as shown in the drawing reference numeral 214, by passing them through the deconvolution layer (214) by combining outputs of the same order, thereby generating T actual upsampling maps, and then combining the results of combining the T actual upsampling maps and passing them through the convolution layer (215) with the results of performing bicubic interpolation (216) on the first actual image, thereby generating a super-resolution image corresponding to the first actual image.

According to one embodiment of the present invention, the electronic device (110) may further include a configuration related to user authentication so that only a predetermined authenticated user can obtain a super-resolution image with improved resolution from a predetermined image using the super-resolution model for which learning has been completed.

To this end, according to one embodiment of the present invention, the electronic device (110) may further include an authentication value storage unit (120), an authentication completion unit (121), and a message display unit (122).

First, in order to perform user authentication in the present invention, it is assumed that a first user identification number of p (where p is a natural number greater than or equal to 2) digits and a first password of q (where q is a natural number greater than or equal to 2) digits have been pre-issued to a user who has been pre-designated as a person who can use the super-resolution model.

At this time, the authentication value storage unit (120) may store a pre-issued authentication value for authenticating that the user is capable of using the super-resolution model.

Here, the authentication value is a result value generated by performing a convolution operation between a first vector composed of a result value obtained by performing a modulo-2 operation on each of the p numbers constituting the first user identification number, and a second vector composed of a result value obtained by performing a modulo-2 operation on each of the q numbers constituting the first password.

For example, if the first user identification number is '12345' and the first password is '567', the authentication value may be a result value generated by performing a convolution operation between a first vector '[1 0 1 0 1]' composed of '1', '0', '1', '0', '1' which are the operation results of performing a modulo-2 operation on '1', '2', '3', '4', '5' which constitute '12345', and a second vector '[1 0 1]' composed of '1', '0', '1' which are the operation results of performing a modulo-2 operation on '5', '6', '7' which constitute '567'.

The authentication completion unit (121) displays a message on the screen instructing the user to enter a user identification number and password for user authentication when a user authentication request command for use of the super-resolution model is authorized from the user after the generation of the super-resolution model is completed through the model generation unit (120), and then, when the first user identification number and the first password are entered by the user, a first vector formed by the result values of performing a modulo-2 operation on each of the p numbers constituting the first user identification number and a second vector formed by the result values of performing a modulo-2 operation on each of the q numbers constituting the first password is performed to generate a verification value, and then verifies whether the verification value matches the verification value, and if it is confirmed that they match, the user authentication is completed.

In this regard, if a first user identification number, say '12345', and a first password, say '567', are authorized by the user, the authentication completion unit (121) can generate the verification value by performing a convolution operation between a first vector, say '[1 0 1 0 1]', which is composed of '1', '0', '1', '0', '1', which are the results of a modulo-2 operation performed on each of '1', '2', '3', '4', and '5' constituting the first user identification number, '12345', and a second vector, say '[1 0 1]', which is composed of '1', '0', and '1', which are the results of a modulo-2 operation performed on each of '5', '6', and '7' constituting the first password, '567'.

Then, the authentication completion unit (121) checks whether the verification value matches the authentication value stored in the authentication value storage unit (120), and if they match, it determines that the first user identification number and the first password entered by the user have been entered correctly, and can complete the user authentication process.

When user authentication is completed, the message display unit (122) displays a message on the screen indicating that user authentication is completed, and at the same time displays a message indicating that an actual image for use of the super-resolution model should be input.

Through this, after the user confirms the message, the user can apply a predetermined real image to the electronic device (110), and at this time, when the first real image is applied as input by the user, the image generation unit (113) can generate a super-resolution image corresponding to the first real image based on the super-resolution model.

FIG. 5 is a flowchart illustrating an operating method of an electronic device according to an embodiment of the present invention.

In step (S510), a training data storage unit is maintained in which a plurality of pre-collected training images and a pre-produced high-resolution converted image corresponding to each of the plurality of training images are stored.

In step (S520), one training image and its corresponding high-resolution conversion image are selected from among the plurality of training images stored in the training data storage, and the selected training image is passed through a neural network model to generate a super-resolution prediction image and a low-resolution prediction image obtained by downsampling the super-resolution prediction image. Then, machine learning is repeatedly performed on each of the plurality of training images to minimize the error between the super-resolution prediction image and the selected high-resolution conversion image and the error between the low-resolution prediction image and the selected training image, thereby generating a super-resolution model.

In step (S530), after the generation of the super-resolution model is completed, when a super-resolution command is entered by the user while a first real image is input, the first real image is input to the super-resolution model to generate a super-resolution image corresponding to the first real image.

At this time, according to one embodiment of the present invention, in step (S520), when the first training image among the plurality of training images is selected, and the machine learning process for the first training image is to be performed in an order, the first training image is passed through T (T is a natural number greater than or equal to 3) convolution layers to generate T output maps, sequentially applying each of the T output maps as an input to each of the T first CFBs configured to receive each output map as an input, but when the t-th output map is applied to the t-th first CFB, the output of the t-1-th first CFB is applied together as an input to the t-th first CFB, thereby generating the output of each of the T first CFBs, thereby generating T forward outputs, applying each of the T forward outputs to each of the T second CFBs configured to receive each forward output as an input to the T first CFBs. A step of generating T reverse outputs by applying inputs one by one in an order opposite to the input order of the inputs, but when the t-th forward output is applied to the t-th second CFB, the output of the t+1-th second CFB and the t-th output map are applied together as inputs to the t-th second CFB, thereby generating outputs of each of the T second CFBs, a step of generating T reverse outputs, a step of combining outputs of the same order of the T forward outputs and the T reverse outputs and passing them through a deconvolution layer to perform upsampling, thereby generating T upsampling maps, and then a step of combining the results of passing the T upsampling maps through a convolution layer and the results of performing bicubic interpolation on the first training image to generate a first super-resolution prediction image, a step of generating a first low-resolution prediction image by passing the first super-resolution prediction image through a convolution layer to perform downsampling, and a step of storing the first super-resolution prediction image and the training data storage in the training data storage, The method may include a step of performing machine learning so that a weighted sum of a first error between a first high-resolution converted image corresponding to the first training image and a second error between the first low-resolution predicted image and the first training image is minimized.

At this time, according to one embodiment of the present invention, each of the T first CFBs and the T second CFBs is configured as a block consisting of a first connection block in which five RABs are connected to each other and a second connection block in which three RABs are connected to each other, and is configured so that an input is applied to the 1st RAB constituting the first connection block, and at the same time, the output of the 5th RAB constituting the first connection block is output as the final output, and at this time, when the output through the 1st RAB constituting the first connection block is output, the output produced is applied as an input to the 2nd RAB constituting the first connection block, and at the same time, the output produced is upsampled and applied as an input to the 1st RAB constituting the second connection block, and at this time, the 2nd, 3rd, 4th, and 5th RABs constituting the first connection block are configured to receive the output of the RAB of the previous order, while the 3rd and 4th RABs are configured to receive the output of the second connection block, respectively. The outputs of the 1st and 2nd RABs constituting the block are configured to additionally receive downsampled results, the 5th RAB is configured to additionally receive downsampled results of the outputs of the 2nd and 3rd RABs and the output of the 3rd RAB constituting the second connection block, and the 2nd and 3rd RABs constituting the second connection block may each be configured to receive outputs of RABs of a previous order, and at the same time, may be configured to additionally receive upsampled results of the outputs of the 2nd and 3rd RABs constituting the first connection block.

At this time, according to one embodiment of the present invention, the RAB may be a block configured to sequentially pass an input through a convolutional layer, a channel attention module, and a spatial attention module to generate an output, and then produce a final output through a residual connection between the generated output and the input.

In addition, according to one embodiment of the present invention, in step (S530), after the generation of the super-resolution model is completed, when the super-resolution command is input by the user while the first real image is input, the first real image is input to the super-resolution model, thereby generating T real output maps by passing the first real image through each of the T convolution layers, and sequentially applying each of the T real output maps to each of the T first CFBs one by one as an input, but when the t-th real output map is applied to the t-th first CFB, the output of the t-1-th first CFB is applied together as an input to the t-th first CFB, thereby generating the output of each of the T first CFBs, thereby generating T real forward outputs, and then applying each of the T real forward outputs to each of the T second CFBs in an order opposite to the order of application of inputs to the T first CFBs. By applying inputs one by one, but when the t-th actual forward output is applied to the t-th second CFB, the output of the t+1-th second CFB and the t-th actual output map are applied together as inputs to the t-th second CFB, thereby generating outputs of each of the T second CFBs, and then performing upsampling by combining outputs of the same order between the T actual forward outputs and the T actual reverse outputs and passing them through the deconvolution layer respectively, thereby generating T actual upsampling maps, and then combining the results of passing the T actual upsampling maps through the convolution layer and the results of performing bicubic interpolation on the first real image, it is possible to generate a super-resolution image corresponding to the first real image.

In addition, according to one embodiment of the present invention, the operating method of the electronic device comprises: a step of maintaining an authentication value storage unit in which a pre-issued authentication value for authenticating a user who can use the super-resolution model (the authentication value is a result value generated by performing a convolution operation between a first vector composed of a result value of performing a modulo-2 operation on each of p numbers constituting a first user identification number of p (wherein p is a natural number greater than or equal to 2) digits pre-issued to the user and a second vector composed of a result value of performing a modulo-2 operation on each of q numbers constituting a first password of q (wherein q is a natural number greater than or equal to 2) digits pre-issued to the user) is stored; step (S520), after the generation of the super-resolution model is completed, when a user authentication request command for using the super-resolution model is authorized from the user, a message instructing to input a user identification number and password for user authentication is displayed on a screen, and when the first user identification number and the first password are input by the user, The method may further include a step of performing a convolution operation between a first vector formed of a result value of performing a modulo-2 operation on each of p numbers constituting the first user identification number and a second vector formed of a result value of performing a modulo-2 operation on each of q numbers constituting the first password, thereby generating a verification value, and then confirming whether the verification value matches the authentication value, and completing user authentication if they are confirmed to match, and when the user authentication is completed, displaying a message indicating that user authentication is completed on a screen and simultaneously displaying a message indicating to input an actual image for use of the super-resolution model.

Above, the operating method of the electronic device according to one embodiment of the present invention has been described with reference to FIG. 5. Here, the operating method of the electronic device according to one embodiment of the present invention can correspond to the configuration for the operation of the electronic device (110) described using FIGS. 1 to 4, and therefore, a more detailed description thereof will be omitted.

The operating method of an electronic device according to one embodiment of the present invention can be implemented as a computer program stored in a storage medium for execution through combination with a computer.

In addition, the operating method of an electronic device according to an embodiment of the present invention may be implemented in the form of a program command that can be executed through various computer means and recorded on a computer-readable medium. The computer-readable medium may include program commands, data files, data structures, etc., alone or in combination. The program commands recorded on the medium may be those specially designed and configured for the present invention or may be those known to and usable by those skilled in the art of computer software. Examples of the computer-readable recording medium include magnetic media such as hard disks, floppy disks, and magnetic tapes, optical media such as CD-ROMs and DVDs, magneto-optical media such as floptical disks, and hardware devices specially configured to store and execute program commands such as ROMs, RAMs, and flash memories. Examples of the program commands include not only machine language codes generated by a compiler, but also high-level language codes that can be executed by a computer using an interpreter, etc.

Although the present invention has been described with specific details such as specific components and limited examples and drawings, these have been provided only to help a more general understanding of the present invention, and the present invention is not limited to the above examples, and those with common knowledge in the field to which the present invention pertains can make various modifications and variations from this description.

Therefore, the idea of the present invention should not be limited to the described embodiments, and all things that are equivalent or equivalent to the claims described below as well as the claims are included in the scope of the idea of the present invention.

110: Electronic devices
111: Training data storage unit 112: Model generation unit
113: Image generation section 114: Output map generation section
115: Forward output section 116: Reverse output section
117: Super-resolution prediction unit 118: Low-resolution prediction unit
119: Learning execution unit 120: Authentication value storage unit
121: Authentication completion section 122: Message display section

Claims (14)

A training data storage unit storing a plurality of pre-collected training images and a pre-produced high-resolution converted image corresponding to each of the plurality of training images;
A model generation unit that generates a super-resolution model by selecting one training image and its corresponding high-resolution conversion image from among the plurality of training images stored in the training data storage unit, passing the selected training image through a neural network model, thereby generating a super-resolution prediction image and a low-resolution prediction image obtained by downsampling the super-resolution prediction image, and then repeatedly performing machine learning on each of the plurality of training images so that an error between the super-resolution prediction image and the selected high-resolution conversion image and an error between the low-resolution prediction image and the selected training image are minimized; and
After the generation of the above super-resolution model is completed, when a super-resolution command is entered by a user while a first real image is entered as input, an image generation unit that inputs the first real image as input to the super-resolution model and generates a super-resolution image corresponding to the first real image
Including,
The above model generation part
When the machine learning process for the first training image is to be performed in an order in which the first training image is selected from among the plurality of training images, an output map generation unit that generates T output maps by passing the first training image through T (T is a natural number greater than or equal to 3) convolution layers, respectively;
A forward output unit that sequentially inputs each of the T output maps as an input to each of the T first cross-learning feedback blocks (CFBs) configured to receive each output map as an input, and when the t-th output map is input to the t-th first CFB, the output of the t-1-th first CFB is also input to the t-th first CFB, thereby generating T forward outputs by generating the output of each of the T first CFBs;
A reverse output unit configured to generate T reverse outputs by applying each of the T forward outputs as inputs to each of the T second CFBs configured to receive each forward output as input, one by one, in an order opposite to the order of application of inputs to the T first CFBs, but when the t-th forward output is applied to the t-th second CFB, applying the output of the t+1-th second CFB and the t-th output map together as inputs to the t-th second CFB, thereby generating the output of each of the T second CFBs;
A super-resolution prediction unit that generates a first super-resolution prediction image by combining the T forward outputs and the T reverse outputs, passing them through a deconvolution layer by combining outputs of the same order, performing upsampling, generating T upsampling maps, and then combining the results of passing the T upsampling maps through a convolution layer and performing bicubic interpolation on the first training image;
A low-resolution prediction unit that generates a first low-resolution prediction image by performing downsampling by passing the first super-resolution prediction image through a convolution layer; and
A learning performing unit that performs machine learning so that a weighted sum of a first error between the first super-resolution prediction image and the first high-resolution converted image corresponding to the first training image stored in the training data storage unit and a second error between the first low-resolution prediction image and the first training image is minimized.
An electronic device comprising: delete In the first paragraph,
Each of the T first CFBs and the T second CFBs,
A block comprising a first connection block in which five residual attention blocks (RABs) are connected to each other, and a second connection block in which three RABs are connected to each other, wherein the input is configured to be applied to the first RAB constituting the first connection block, and the output of the fifth RAB constituting the first connection block is output as the final output, and at this time, when the output through the first RAB constituting the first connection block is output, the output thus output is applied as an input to the second RAB constituting the first connection block, and at the same time, the output thus output is upsampled and applied as an input to the first RAB constituting the second connection block, and at this time, the 2nd, 3rd, 4th, and 5th RABs constituting the first connection block are configured to receive the output of the RAB of the previous order, while the 3rd and 4th RABs are configured to receive the output of the 1st and 2nd RABs constituting the second connection block, respectively. An electronic device characterized in that the output is configured to additionally receive a downsampled result, the 5th RAB is configured to additionally receive a downsampled result of the outputs of the 2nd and 3rd RABs and the output of the 3rd RAB constituting the second connection block, and the 2nd and 3rd RABs constituting the second connection block are each configured to receive an output of a RAB of a previous order, and at the same time, the outputs of the 2nd and 3rd RABs constituting the first connection block are configured to additionally receive an upsampled result. In the third paragraph,
The above RAB is
An electronic device characterized by being a block configured to sequentially pass an input through a convolutional layer, a channel attention module, and a spatial attention module to generate an output, and then produce a final output through a residual connection between the generated output and the input. In the first paragraph,
The above image generating unit
After the generation of the super-resolution model is completed, when a super-resolution command is input by the user while the first real image is input, the first real image is input to the super-resolution model, thereby generating T real output maps by passing the first real image through each of the T convolution layers, and sequentially applying each of the T real output maps as an input to each of the T first CFBs, such that when the t-th real output map is applied to the t-th first CFB, the output of the t-1-th first CFB is applied together as an input to the t-th first CFB, thereby generating the output of each of the T first CFBs, and then applying each of the T real forward outputs as an input to each of the T second CFBs, one by one, in an order opposite to the order of application of the inputs to the T first CFBs, such that the t-th real forward output is the t-th An electronic device characterized in that when applied to a second CFB, the output of the t+1th second CFB and the tth actual output map are applied together as input to the tth second CFB, thereby generating the output of each of the T second CFBs, thereby generating T actual reverse outputs, and then performing upsampling by combining outputs of the same order between the T actual forward outputs and the T actual reverse outputs and passing them through the deconvolution layer respectively, thereby generating T actual upsampling maps, and then combining the results of passing the T actual upsampling maps through a convolution layer and the results of performing bicubic interpolation on the first real image, thereby generating a super-resolution image corresponding to the first real image. In the first paragraph,
An authentication value storage unit storing a pre-issued authentication value for authenticating a user who can use the above super-resolution model, wherein the authentication value is a result value generated by performing a convolution operation between a first vector composed of a result value obtained by performing a modulo-2 operation on each of p numbers constituting a first user identification number of p digits (p is a natural number greater than or equal to 2) pre-issued for the user and a second vector composed of a result value obtained by performing a modulo-2 operation on each of q numbers constituting a first password of q digits (q is a natural number greater than or equal to 2) pre-issued for the user;
Through the above model generation unit, after the generation of the super-resolution model is completed, if a user authentication request command for using the super-resolution model is authorized from the user, an authentication completion unit which displays a message instructing the user to enter a user identification number and password for user authentication on a screen, and then, when the first user identification number and the first password are entered by the user, performs a convolution operation between a first vector formed by the result values of performing a modulo-2 operation on each of p numbers constituting the first user identification number and a second vector formed by the result values of performing a modulo-2 operation on each of q numbers constituting the first password, thereby generating a verification value, and then verifies whether the verification value matches the authentication value, and if it is confirmed that they match, completes the user authentication; and
When user authentication is completed, a message display section displays a message indicating that user authentication is completed on the screen, and at the same time displays a message indicating that an actual image should be input for use of the super-resolution model.
An electronic device further comprising: A step of maintaining a training data storage unit storing a plurality of pre-collected training images and a pre-produced high-resolution converted image corresponding to each of the plurality of training images;
A step of selecting one training image and its corresponding high-resolution conversion image from among the plurality of training images stored in the training data storage, passing the selected training image through a neural network model to generate a super-resolution prediction image and a low-resolution prediction image obtained by downsampling the super-resolution prediction image, and then repeatedly performing machine learning for each of the plurality of training images so that an error between the super-resolution prediction image and the selected high-resolution conversion image and an error between the low-resolution prediction image and the selected training image are minimized, thereby generating a super-resolution model; and
After the generation of the above super-resolution model is completed, when a super-resolution command is entered by a user while a first real image is entered as input, a step of entering the first real image as input to the super-resolution model to generate a super-resolution image corresponding to the first real image
Including,
The steps for generating the above super-resolution model are:
When a machine learning process for the first training image is to be performed in an order in which a first training image among the plurality of training images is selected, a step of generating T output maps by passing the first training image through T (T is a natural number greater than or equal to 3) convolution layers, respectively;
A step of generating T forward outputs by sequentially applying each of the T output maps as an input to each of the T first cross-learning feedback blocks (CFBs) configured to receive each output map as an input, such that when the t-th output map is applied to the t-th first CFB, the output of the t-1-th first CFB is also applied as an input to the t-th first CFB, thereby generating the output of each of the T first CFBs;
A step of generating T reverse outputs by applying each of the T forward outputs as inputs to each of the T second CFBs configured to receive each forward output as input, one by one, in an order opposite to the order of application of inputs to the T first CFBs, such that when the t-th forward output is applied to the t-th second CFB, the output of the t+1-th second CFB and the t-th output map are applied together as inputs to the t-th second CFB, thereby generating the output of each of the T second CFBs;
A step of generating T upsampling maps by combining outputs of the same order and passing each of the T forward outputs and the T reverse outputs through a deconvolution layer to perform upsampling, and then combining the results of combining the T upsampling maps and passing them through a convolution layer and the results of performing bicubic interpolation on the first training image to generate a first super-resolution prediction image;
A step of generating a first low-resolution prediction image by performing downsampling by passing the first super-resolution prediction image through a convolution layer; and
A step of performing machine learning so that a weighted sum of a first error between the first super-resolution prediction image and the first high-resolution converted image corresponding to the first training image stored in the training data storage unit and a second error between the first low-resolution prediction image and the first training image is minimized.
A method of operating an electronic device comprising: delete In Article 7,
Each of the T first CFBs and the T second CFBs,
A block comprising a first connection block in which five residual attention blocks (RABs) are connected to each other, and a second connection block in which three RABs are connected to each other, wherein the input is configured to be applied to the first RAB constituting the first connection block, and the output of the fifth RAB constituting the first connection block is output as the final output, and at this time, when the output through the first RAB constituting the first connection block is output, the output thus output is applied as an input to the second RAB constituting the first connection block, and at the same time, the output thus output is upsampled and applied as an input to the first RAB constituting the second connection block, and at this time, the 2nd, 3rd, 4th, and 5th RABs constituting the first connection block are configured to receive the output of the RAB of the previous order, while the 3rd and 4th RABs are configured to receive the output of the 1st and 2nd RABs constituting the second connection block, respectively. An operating method of an electronic device, characterized in that the output is configured to additionally receive a downsampled result, the 5th RAB is configured to additionally receive a downsampled result of the outputs of the 2nd and 3rd RABs and the output of the 3rd RAB constituting the second connection block, and the 2nd and 3rd RABs constituting the second connection block are each configured to receive an output of a RAB of a previous order, and at the same time, the outputs of the 2nd and 3rd RABs constituting the first connection block are additionally configured to receive an upsampled result. In Article 9,
The above RAB is
An operating method of an electronic device, characterized in that the block is configured to sequentially pass an input through a convolutional layer, a channel attention module, and a spatial attention module to generate an output, and then produce a final output through a residual connection between the generated output and the input. In Article 7,
The steps for generating the above super-resolution image are
After the generation of the super-resolution model is completed, when a super-resolution command is input by the user while the first real image is input, the first real image is input to the super-resolution model, thereby generating T real output maps by passing the first real image through each of the T convolution layers, and sequentially applying each of the T real output maps as an input to each of the T first CFBs, such that when the t-th real output map is applied to the t-th first CFB, the output of the t-1-th first CFB is applied together as an input to the t-th first CFB, thereby generating the output of each of the T first CFBs, and then applying each of the T real forward outputs as an input to each of the T second CFBs, one by one, in an order opposite to the order of application of the inputs to the T first CFBs, such that the t-th real forward output is the t-th An operating method of an electronic device, characterized in that when applied to a second CFB, the output of the t+1th second CFB and the tth actual output map are applied together as input to the tth second CFB, thereby generating the output of each of the T second CFBs, thereby generating T actual reverse outputs, and then performing upsampling by combining outputs of the same order between the T actual forward outputs and the T actual reverse outputs and passing them through the deconvolution layer respectively, thereby generating T actual upsampling maps, and then combining the results of passing the T actual upsampling maps through a convolution layer and the results of performing bicubic interpolation on the first real image, thereby generating a super-resolution image corresponding to the first real image. In Article 7,
A step for maintaining an authentication value storage unit in which a pre-issued authentication value for authenticating a user who can use the above super-resolution model is stored - the authentication value is a result value generated by performing a convolution operation between a first vector composed of a result value obtained by performing a modulo-2 operation on each of p numbers constituting a first user identification number of p digits (p is a natural number greater than or equal to 2) pre-issued for the user, and a second vector composed of a result value obtained by performing a modulo-2 operation on each of q numbers constituting a first password of q digits (q is a natural number greater than or equal to 2) pre-issued for the user;
Through the step of generating the super-resolution model, after the generation of the super-resolution model is completed, if a user authentication request command for using the super-resolution model is authorized from the user, a message instructing to input a user identification number and password for user authentication is displayed on the screen, and then, when the first user identification number and the first password are entered by the user, a step of performing a convolution operation between a first vector formed by a result value of performing a modulo-2 operation on each of p numbers constituting the first user identification number and a second vector formed by a result value of performing a modulo-2 operation on each of q numbers constituting the first password, thereby generating a verification value, and then checking whether the verification value matches the authentication value, and if it is confirmed that they match, completing the user authentication; and
When user authentication is completed, a step of displaying a message indicating that user authentication is completed on the screen and at the same time displaying a message indicating that an actual image for use of the super-resolution model is to be input
A method of operating an electronic device further comprising: A computer-readable recording medium having recorded thereon a computer program for executing the method of any one of claims 7 and 9 to 12 through combination with a computer. A computer program stored in a storage medium for executing the method of any one of claims 7 and 9 to 12 through combination with a computer.

Images