KR20230046913A - Apparatus and method for predicting compression quality of image in electronic device - Google Patents

Abstract

Disclosed are a method and device for predicting the compression quality of an image during image correction (e.g., image quality enhancement) in an electronic device and processing the image based on such predictions. The electronic device according to various embodiments comprises a display module, a memory, and a processor. The processor may be configured to: display an image through the display module; extract a plurality of designated blocks from the image in a designated manner; estimate the reliability of each block for the plurality of blocks; identify, based on the reliability estimation, a first block corresponding to an outlier block excluded from quality prediction and a second block for which quality prediction is possible among the plurality of blocks; exclude the first block from quality prediction targets among the plurality of blocks; and classify the compression quality of the image using the remaining second blocks, excluding the first blocks among the plurality of blocks. Various other embodiments are possible. Accordingly, the present invention can accurately analyze compression quality.

Description

Method and apparatus for predicting compression quality of image in electronic device

Embodiments of the present disclosure disclose a method and apparatus for estimating the compression quality of an image when correcting an image (eg, improving image quality) in an electronic device and processing the image based thereon.

With the development of digital technology, mobile communication terminals, PDAs (personal digital assistants), electronic notebooks, smart phones, tablet PCs (personal computers), wearable devices and / or laptops Various types of electronic devices such as personal computers are widely used. In order to support and increase functions of these electronic devices, hardware parts and/or software parts of electronic devices are continuously being developed.

As electronic devices diversify their functions, for example, multimedia devices equipped with complex functions such as taking pictures or videos, playing music or video files, playing games, receiving broadcasts, or making calls. ) is implemented in the form of Such an electronic device has a display and can display screens related to functions based on the display.

The electronic device may display an image stored in the electronic device or an image acquired from an external device (eg, a server and/or other electronic device) through a display. In recent years, as the resolution and/or physical size (e.g., screen size of a display) of an electronic device increases, high-definition images are required in electronic devices, and the need for image quality improvement accordingly increases. is being raised For example, the importance of image quality for images provided by electronic devices or external devices is also increasing.

Meanwhile, an image provided by an external device may be compressed and transmitted. For example, an image may be provided after being compressed with a certain compression quality (or compression rate) in order to save storage in a cloud environment and minimize delay in image transmission. In this case, the compression method information (eg, resolution information) and compression quality information (eg, compression bitrate information) of the original video are lost, and thus, in the video display device, the image related to the compression method information and compression quality information of the original video is lost. Quality determination has become difficult, making it difficult to set a picture quality suitable for an original video, and in particular, it has become difficult to implement the maximum performance of a picture quality algorithm in an image display device.

Image compression, for example, can result in compression artifacts in the image. For example, image compression may introduce at least one artifact (eg, ringing artifacts (or mosquito artifacts), blocking artifacts, blur artifacts, color distortion, and/or texture deviation). In general, more compression artifacts may occur as the compression quality (or compression rate) of an image is higher. Therefore, in an external device (eg, a service provider such as a content server or other external device), when an image is provided to an electronic device, an issue of minimizing artifacts to the user's eyes while compressing the image size as much as possible may be raised. can In addition, as electronic devices recently support a display of a large screen, an image that is optimized for a small screen and has hidden artifacts can be easily exposed by a user as it is enlarged and provided on a large screen. For example, artifacts that were not identified on a small screen may be enlarged on a large screen and identified by the user, and may be perceived as deterioration of image quality by the user.

In various embodiments, a method and apparatus capable of accurately and quickly predicting the compression quality of an image in an operation of determining the compression quality of an image when improving image quality (eg, image correction) in an electronic device are disclosed.

In various embodiments, a method and apparatus capable of predicting the compression quality of an image based on block-by-block confidence estimation when determining the compression quality of a given image in an electronic device are disclosed.

In various embodiments, when estimating block-by-block reliability for a given image in an electronic device, an outlier, which is a region in which compression quality prediction is difficult to predict (eg, a region to be excluded from quality prediction) is removed from the image, and the image is compressed. Disclosed is a method and apparatus capable of accurately and quickly predicting compression quality based on a region enabling quality prediction.

In various embodiments, a method and apparatus capable of determining compression quality of a given image and correcting an image based on a denoising model learned to correspond to the compression quality are disclosed in order to improve image quality in an electronic device. do.

An electronic device according to an embodiment of the present disclosure includes a display module, a memory, and a processor operatively connected to the display module and the memory, wherein the processor displays an image through the display module, and displays the image. extracts a plurality of blocks specified in a designated method (eg, a uniform method or a random method), estimates reliability for each block with respect to the plurality of blocks, and based on the reliability estimation, quality prediction is excluded from among the plurality of blocks. A first block corresponding to an outlier and a second block for which quality can be predicted are identified, the first block among the plurality of blocks is excluded from the quality prediction target, and the first block among the plurality of blocks is identified. It is possible to classify the compression quality of the image using the remaining second blocks except for .

An operating method of an electronic device according to an embodiment of the present disclosure includes an operation of displaying an image through a display module of the electronic device, an operation of extracting a plurality of designated blocks from the image in a designated manner, and each of the plurality of blocks. An operation of estimating reliability for each block; an operation of identifying a first block corresponding to an outlier whose quality cannot be predicted and a second block whose quality can be predicted among the plurality of blocks based on the reliability estimation; and An operation of excluding the first block among blocks from a quality prediction target, and an operation of classifying the compression quality of the image using second blocks other than the first block among the plurality of blocks. .

In various embodiments of the present disclosure to solve the above problems, a computer-readable recording medium in which a program for executing the method in a processor may be included.

Additional scope of applicability of the present disclosure will become apparent from the detailed description that follows. However, since various changes and modifications within the spirit and scope of the present disclosure may be clearly understood by those skilled in the art, it should be understood that the detailed description and specific embodiments such as the preferred embodiments of the present disclosure are given by way of example only.

According to the electronic device and method of operation thereof according to the present disclosure, when the electronic device analyzes an image, compression quality analysis is performed on a partial area rather than the entire area of the image using a deep neural network (DNN). According to this, it is possible to accurately and quickly process the compression quality calculation of the image. According to various embodiments, when determining (or analyzing) the compression quality of an image to improve image quality (eg, image correction) in an electronic device, the compression quality of an image is determined based on block-by-block confidence estimation. You can make accurate and fast predictions. According to various embodiments, an electronic device removes outliers from an image based on block-by-block reliability estimation for a given image, and analyzes compression quality based on a region in the image that enables compression quality prediction. By doing so, the compression quality can be more accurately analyzed.

According to various embodiments, an electronic device may accurately determine the compression quality of a given image and remove compression artifacts (or compression noise) of the image based on a denoising model learned to correspond to the determined compression quality. can Accordingly, the electronic device can provide a user with an image close to the original quality by increasing the quality of the compressed image.

In addition to this, various effects identified directly or indirectly through this document may be provided.

In the description of the drawings, the same or similar reference numerals may be used for the same or similar elements.
1 is a block diagram of an electronic device in a network environment according to various embodiments.
2 is a diagram schematically illustrating a configuration of an electronic device according to an exemplary embodiment.
3 is a flowchart illustrating examples of learning, classification, and removal operations for image correction in an electronic device according to an exemplary embodiment.
4 is a diagram illustrating an example of image correction in an electronic device according to an exemplary embodiment.
5 is a flowchart illustrating a method of operating an electronic device according to an exemplary embodiment.
6 is a diagram illustrating an example of an operation learned in an electronic device according to an embodiment.
7 is a flowchart illustrating an operation method of predicting compression quality in an electronic device according to an embodiment.
8 is a diagram illustrating an example of an inference operation for predicting quality in an electronic device according to an exemplary embodiment.
9 is a flowchart illustrating an operation method of removing artifacts in an electronic device according to an exemplary embodiment.
10 is a flowchart illustrating a method of compressing an image in an electronic device according to an exemplary embodiment.
11 is a flowchart illustrating an operation method of providing information on image correction in an electronic device according to an exemplary embodiment.
FIG. 12 is a diagram illustrating a user interface providing information about image correction and an operation example thereof in an electronic device according to an exemplary embodiment.

1 is a block diagram of an electronic device 101 within a network environment 100 according to various embodiments.

Referring to FIG. 1 , in a network environment 100, an electronic device 101 communicates with an electronic device 102 through a first network 198 (eg, a short-range wireless communication network) or through a second network 199. It may communicate with at least one of the electronic device 104 or the server 108 through (eg, a long-distance wireless communication network). According to an embodiment, the electronic device 101 may communicate with the electronic device 104 through the server 108 . According to an embodiment, the electronic device 101 includes a processor 120, a memory 130, an input module 150, an audio output module 155, a display module 160, an audio module 170, a sensor module ( 176), interface 177, connection terminal 178, haptic module 179, camera module 180, power management module 188, battery 189, communication module 190, subscriber identification module 196 , or the antenna module 197 may be included. In some embodiments, in the electronic device 101, at least one of these components (eg, the connection terminal 178) may be omitted or one or more other components may be added. In some embodiments, some of these components (eg, sensor module 176, camera module 180, or antenna module 197) are integrated into a single component (eg, display module 160). It can be.

The processor 120, for example, executes software (eg, the program 140) to cause at least one other component (eg, hardware or software component) of the electronic device 101 connected to the processor 120. It can control and perform various data processing or calculations. According to one embodiment, as at least part of data processing or operation, processor 120 transfers instructions or data received from other components (e.g., sensor module 176 or communication module 190) to volatile memory 132. , processing commands or data stored in the volatile memory 132 , and storing resultant data in the non-volatile memory 134 . According to an embodiment, the processor 120 may include a main processor 121 (eg, a central processing unit (CPU) or an application processor (AP)) or a secondary processor (which may be operated independently of or together with the main processor 121). 123) (e.g., graphic processing unit (GPU), neural processing unit (NPU), image signal processor (ISP), sensor hub processor, or communication processor (CP, communication processor)). For example, when the electronic device 101 includes the main processor 121 and the auxiliary processor 123, the auxiliary processor 123 may use less power than the main processor 121 or be set to be specialized for a designated function. can The secondary processor 123 may be implemented separately from or as part of the main processor 121 .

The secondary processor 123 may, for example, take the place of the main processor 121 while the main processor 121 is in an inactive (eg, sleep) state, or the main processor 121 may At least one of the components of the electronic device 101 (eg, the display module 160, the sensor module 176, or At least some of the functions or states related to the communication module 190) may be controlled. According to one embodiment, the auxiliary processor 123 (eg, an image signal processor or a communication processor) may be implemented as part of other functionally related components (eg, the camera module 180 or the communication module 190). there is. According to an embodiment, the auxiliary processor 123 (eg, a neural network processing device) may include a hardware structure specialized for processing an artificial intelligence model. AI models can be created through machine learning. Such learning may be performed, for example, in the electronic device 101 itself where the artificial intelligence model is performed, or may be performed through a separate server (eg, the server 108). The learning algorithm may include, for example, supervised learning, unsupervised learning, semi-supervised learning or reinforcement learning, but in the above example Not limited. The artificial intelligence model may include a plurality of artificial neural network layers. Artificial neural networks include deep neural networks (DNNs), convolutional neural networks (CNNs), recurrent neural networks (RNNs), restricted boltzmann machines (RBMs), deep belief networks (DBNs), bidirectional recurrent deep neural networks (BRDNNs), It may be one of deep Q-networks or a combination of two or more of the foregoing, but is not limited to the foregoing examples. The artificial intelligence model may include, in addition or alternatively, software structures in addition to hardware structures.

The memory 130 may store various data used by at least one component (eg, the processor 120 or the sensor module 176) of the electronic device 101 . The data may include, for example, input data or output data for software (eg, program 140) and commands related thereto. The memory 130 may include volatile memory 132 or non-volatile memory 134 .

The program 140 may be stored as software in the memory 130, and may include, for example, an operating system (OS) 142, middleware 144, or an application 146. there is.

The input module 150 may receive a command or data to be used by a component (eg, the processor 120) of the electronic device 101 from the outside of the electronic device 101 (eg, a user). The input module 150 may include, for example, a microphone, a mouse, a keyboard, a key (eg, a button), or a digital pen (eg, a stylus pen).

The sound output module 155 may output sound signals to the outside of the electronic device 101 . The sound output module 155 may include, for example, a speaker or a receiver. The speaker can be used for general purposes such as multimedia playback or recording playback. A receiver may be used to receive an incoming call. According to one embodiment, the receiver may be implemented separately from the speaker or as part of it.

The display module 160 may visually provide information to the outside of the electronic device 101 (eg, a user). The display module 160 may include, for example, a display, a hologram device, or a projector and a control circuit for controlling the device. According to an embodiment, the display module 160 may include a touch sensor configured to detect a touch or a pressure sensor configured to measure the intensity of force generated by the touch.

The audio module 170 may convert sound into an electrical signal or vice versa. According to an embodiment, the audio module 170 acquires sound through the input module 150, the sound output module 155, or an external electronic device connected directly or wirelessly to the electronic device 101 (eg: Sound may be output through the electronic device 102 (eg, a speaker or a headphone).

The sensor module 176 detects an operating state (eg, power or temperature) of the electronic device 101 or an external environmental state (eg, a user state), and generates an electrical signal or data value corresponding to the detected state. can do. According to one embodiment, the sensor module 176 may include, for example, a gesture sensor, a gyro sensor, an air pressure sensor, a magnetic sensor, an acceleration sensor, a grip sensor, a proximity sensor, a color sensor, an infrared (IR) sensor, a bio sensor, It may include a temperature sensor, humidity sensor, or light sensor.

The interface 177 may support one or more designated protocols that may be used to directly or wirelessly connect the electronic device 101 to an external electronic device (eg, the electronic device 102). According to one embodiment, the interface 177 may include, for example, a high definition multimedia interface (HDMI), a universal serial bus (USB) interface, a secure digital (SD) card interface, or an audio interface.

The connection terminal 178 may include a connector through which the electronic device 101 may be physically connected to an external electronic device (eg, the electronic device 102). According to one embodiment, the connection terminal 178 may include, for example, an HDMI connector, a USB connector, an SD card connector, or an audio connector (eg, a headphone connector).

The haptic module 179 may convert electrical signals into mechanical stimuli (eg, vibration or motion) or electrical stimuli that a user may perceive through tactile or kinesthetic senses. According to one embodiment, the haptic module 179 may include, for example, a motor, a piezoelectric element, or an electrical stimulation device.

The camera module 180 may capture still images and moving images. According to one embodiment, the camera module 180 may include one or more lenses, image sensors, image signal processors, or flashes.

The power management module 188 may manage power supplied to the electronic device 101 . According to one embodiment, the power management module 188 may be implemented as at least part of a power management integrated circuit (PMIC), for example.

The battery 189 may supply power to at least one component of the electronic device 101 . According to one embodiment, the battery 189 may include, for example, a non-rechargeable primary cell, a rechargeable secondary cell, or a fuel cell.

The communication module 190 is a direct (eg, wired) communication channel or a wireless communication channel between the electronic device 101 and an external electronic device (eg, the electronic device 102, the electronic device 104, or the server 108). Establishment and communication through the established communication channel may be supported. The communication module 190 may include one or more communication processors that operate independently of the processor 120 (eg, an application processor) and support direct (eg, wired) communication or wireless communication. According to one embodiment, the communication module 190 may be a wireless communication module 192 (eg, a cellular communication module, a short-range wireless communication module, or a global navigation satellite system (GNSS) communication module) or a wired communication module 194 (eg, a : a local area network (LAN) communication module or a power line communication module). Among these communication modules, a corresponding communication module is a first network 198 (eg, a short-range communication network such as Bluetooth, wireless fidelity (WiFi) direct, or infrared data association (IrDA)) or a second network 199 (eg, a legacy communication module). It may communicate with the external electronic device 104 through a cellular network, a 5G network, a next-generation communication network, the Internet, or a computer network (eg, a telecommunications network such as a LAN or wide area network (WAN)). These various types of communication modules may be integrated as one component (eg, a single chip) or implemented as a plurality of separate components (eg, multiple chips). The wireless communication module 192 uses subscriber information (eg, International Mobile Subscriber Identifier (IMSI)) stored in the subscriber identification module 196 within a communication network such as the first network 198 or the second network 199. The electronic device 101 may be identified or authenticated.

The wireless communication module 192 may support a 5G network after a 4G network and a next-generation communication technology, for example, NR access technology (new radio access technology). NR access technologies include high-speed transmission of high-capacity data (eMBB, mobile broadband), minimization of terminal power and access of multiple terminals (mMTC, massive machine type communications), or high-reliability and low-latency (URLLC, ultra-reliable and low-latency communications) can be supported. The wireless communication module 192 may support a high frequency band (eg, mmWave band) to achieve a high data rate, for example. The wireless communication module 192 uses various technologies for securing performance in a high frequency band, such as beamforming, massive multiple-input and multiple-output (MIMO), and full-dimensional multiplexing. Technologies such as input/output (FD-MIMO, full dimensional MIMO), array antenna, analog beam-forming, or large scale antenna may be supported. The wireless communication module 192 may support various requirements defined for the electronic device 101, an external electronic device (eg, the electronic device 104), or a network system (eg, the second network 199). According to one embodiment, the wireless communication module 192 may be used to realize peak data rate (eg, 20 Gbps or more) for realizing eMBB, loss coverage (eg, 164 dB or less) for realizing mMTC, or U-plane latency (for realizing URLLC). Example: downlink (DL) and uplink (UL) each of 0.5 ms or less, or round trip 1 ms or less) may be supported.

The antenna module 197 may transmit or receive signals or power to the outside (eg, an external electronic device). According to an embodiment, the antenna module 197 may include an antenna including a radiator formed of a conductor or a conductive pattern formed on a substrate (eg, PCB). According to one embodiment, the antenna module 197 may include a plurality of antennas (eg, an array antenna). In this case, at least one antenna suitable for a communication method used in a communication network such as the first network 198 or the second network 199 is selected from the plurality of antennas by the communication module 190, for example. can be chosen A signal or power may be transmitted or received between the communication module 190 and an external electronic device through the selected at least one antenna. According to some embodiments, other components (eg, a radio frequency integrated circuit (RFIC)) may be additionally formed as a part of the antenna module 197 in addition to the radiator.

According to various embodiments, the antenna module 197 may form a mmWave antenna module. According to one embodiment, the mmWave antenna module includes a printed circuit board, an RFIC disposed on or adjacent to a first surface (eg, a bottom surface) of the printed circuit board and capable of supporting a designated high frequency band (eg, mmWave band); and a plurality of antennas (eg, array antennas) disposed on or adjacent to a second surface (eg, a top surface or a side surface) of the printed circuit board and capable of transmitting or receiving signals of the designated high frequency band. can do.

At least some of the components are connected to each other through a communication method between peripheral devices (eg, a bus, general purpose input and output (GPIO), serial peripheral interface (SPI), or mobile industry processor interface (MIPI)) and signal ( e.g. commands or data) can be exchanged with each other.

According to an embodiment, commands or data may be transmitted or received between the electronic device 101 and the external electronic device 104 through the server 108 connected to the second network 199 . Each of the external electronic devices 102 or 104 may be the same as or different from the electronic device 101 . According to an embodiment, all or part of operations executed in the electronic device 101 may be executed in one or more external electronic devices among the external electronic devices 102 , 104 , or 108 . For example, when the electronic device 101 needs to perform a certain function or service automatically or in response to a request from a user or another device, the electronic device 101 instead of executing the function or service by itself. Alternatively or additionally, one or more external electronic devices may be requested to perform the function or at least part of the service. One or more external electronic devices receiving the request may execute at least a part of the requested function or service or an additional function or service related to the request, and deliver the execution result to the electronic device 101 . The electronic device 101 may provide the result as at least part of a response to the request as it is or additionally processed. To this end, for example, cloud computing, distributed computing, mobile edge computing (MEC), or client-server computing technology may be used. The electronic device 101 may provide an ultra-low latency service using, for example, distributed computing or mobile edge computing. In another embodiment, the external electronic device 104 may include an internet of things (IoT) device. Server 108 may be an intelligent server using machine learning and/or neural networks. According to one embodiment, the external electronic device 104 or server 108 may be included in the second network 199 . The electronic device 101 may be applied to intelligent services (eg, smart home, smart city, smart car, or health care) based on 5G communication technology and IoT-related technology.

Electronic devices according to various embodiments disclosed in this document may be devices of various types. The electronic device may include, for example, a portable communication device (eg, a smart phone), a computer device, a portable multimedia device, a portable medical device, a camera, a wearable device, or a home appliance. An electronic device according to an embodiment of the present document is not limited to the aforementioned devices.

Various embodiments of this document and terms used therein are not intended to limit the technical features described in this document to specific embodiments, but should be understood to include various modifications, equivalents, or substitutes of the embodiments. In connection with the description of the drawings, like reference numbers may be used for like or related elements. The singular form of a noun corresponding to an item may include one item or a plurality of items, unless the relevant context clearly dictates otherwise. In this document, "A or B", "at least one of A and B", "at least one of A or B", "A, B or C", "at least one of A, B and C", and "A Each of the phrases such as "at least one of , B, or C" may include any one of the items listed together in that phrase, or all possible combinations thereof. Terms such as "first", "second", or "first" or "secondary" may simply be used to distinguish that component from other corresponding components, and may refer to that component in other respects (eg, importance or order) is not limited. A (eg, first) component is said to be "coupled" or "connected" to another (eg, second) component, with or without the terms "functionally" or "communicatively." When mentioned, it means that the certain component may be connected to the other component directly (eg by wire), wirelessly, or through a third component.

The term "module" used in various embodiments of this document may include a unit implemented in hardware, software, or firmware, and is interchangeable with terms such as, for example, logic, logical blocks, parts, or circuits. can be used as A module may be an integrally constructed component or a minimal unit of components or a portion thereof that performs one or more functions. For example, according to one embodiment, the module may be implemented in the form of an application-specific integrated circuit (ASIC).

Various embodiments of this document provide one or more instructions stored in a storage medium (eg, internal memory 136 or external memory 138) readable by a machine (eg, electronic device 101). It may be implemented as software (eg, the program 140) including them. For example, a processor (eg, the processor 120 ) of a device (eg, the electronic device 101 ) may call at least one command among one or more instructions stored from a storage medium and execute it. This enables the device to be operated to perform at least one function according to the at least one command invoked. The one or more instructions may include code generated by a compiler or code executable by an interpreter. The device-readable storage medium may be provided in the form of a non-transitory storage medium. Here, 'non-temporary' only means that the storage medium is a tangible device and does not contain a signal (e.g. electromagnetic wave), and this term refers to the case where data is stored semi-permanently in the storage medium. It does not discriminate when it is temporarily stored.

According to one embodiment, the method according to various embodiments disclosed in this document may be provided by being included in a computer program product. Computer program products may be traded between sellers and buyers as commodities. A computer program product is distributed in the form of a device-readable storage medium (eg compact disc read only memory (CD-ROM)), or through an application store (eg Play Store ^TM ) or on two user devices ( It can be distributed (eg downloaded or uploaded) online, directly between smart phones. In the case of online distribution, at least part of the computer program product may be temporarily stored or temporarily created in a device-readable storage medium such as a manufacturer's server, an application store server, or a relay server's memory.

According to various embodiments, each component (eg, module or program) of the above-described components may include a single object or a plurality of entities, and some of the plurality of entities may be separately disposed in other components. there is. According to various embodiments, one or more components or operations among the aforementioned corresponding components may be omitted, or one or more other components or operations may be added. Alternatively or additionally, a plurality of components (eg modules or programs) may be integrated into a single component. In this case, the integrated component may perform one or more functions of each of the plurality of components identically or similarly to those performed by a corresponding component of the plurality of components prior to the integration. . According to various embodiments, the operations performed by a module, program or other component are executed sequentially, in parallel, iteratively, or heuristically, or one or more of the operations are executed in a different order. may be added, omitted, or one or more other actions may be added.

2 is a diagram schematically illustrating a configuration of an electronic device according to an exemplary embodiment.

Referring to FIG. 2 , an electronic device 101 according to an embodiment may include a communication module 190, a display module 160, a memory 130, and a processor 120. In one embodiment, processor 120 includes image processing module 210, sample patch extraction module 220, compression quality classification module 230, denoising model 260, model selection module 270, and/or At least one of the score calculation module 280 may be included.

The communication module 190 may support legacy networks (eg, 3G networks and/or 4G networks), 5G networks, out of band (OOB), and/or next-generation communication technologies (eg, new radio (NR) technologies). According to one embodiment, the communication module 190 may correspond to the wireless communication module 192 as illustrated in FIG. 1 . According to an embodiment, the electronic device 101 communicates with an external device (eg, the server 201 of FIG. 1 and/or other electronic devices 102 and 104) through a network using the communication module 190. can be done According to an embodiment, the electronic device 101 may receive an image and/or content including the image (eg, a web page) from an external device through the communication module 190 .

The display module 160 can visually provide various information to the outside of the electronic device 101 (eg, a user). According to an embodiment, the display module 160 includes a touch sensing circuit (or touch sensor) (not shown), a pressure sensor capable of measuring the strength of a touch, and/or a touch panel (eg, detecting a magnetic stylus pen). : digitizer) may be included. According to an embodiment, the display module 160 is a touch sensing circuit, a pressure sensor, and/or a signal for a specific position of the display module 160 based on the touch panel (eg, voltage, light intensity, resistance, electromagnetic signal, and/or Alternatively, a touch input and/or a hovering input (or proximity input) may be sensed by measuring a change in charge amount. According to an embodiment, the display module 160 may include a liquid crystal display (LCD), an organic light emitted diode (OLED), or an active matrix organic light emitted diode (AMOLED). According to some embodiments, the display module 160 may be configured as a flexible display.

The display module 160 may visually provide an image and/or content including the image under the control of the processor 120 . According to an embodiment, the display module 160 may provide various information (eg, a user interface) related to image processing (eg, image correction) corresponding to at least one displayed image.

The memory 130 may store various data used by the electronic device 101 . The data may include, for example, input data or output data for an application (eg, the program 140 of FIG. 1 ) and a command related thereto. According to one embodiment, the memory 130 may store instructions that, when executed, cause the processor 120 to operate. For example, the application may be stored as software (eg, the program 140 of FIG. 1 ) on the memory 130 and may be executable by the processor 120 . According to an embodiment, the application may be various applications (eg, gallery application, image editing application) capable of providing images in the electronic device 101 .

According to an embodiment, the memory 130 performs a function (or operation) of processing block-by-block confidence estimation-based compression quality prediction and image correction for each compression quality, which may be performed by the processor 120. ) and at least one component (or module) related to it. For example, the memory 130 includes an image processing module 210 of the processor 120, a sample patch extraction module 220, a compression quality classification module 230, a noise removal model 260, and a model selection module 270. , and/or at least some of the score calculation module 280 may be included in software form (or instruction form).

The processor 120 may control related operations for processing block-by-block confidence estimation-based compression quality prediction and image correction for each compression quality in the electronic device 101 . According to an embodiment, the processor 120 determines the compression quality related to the image stored in the memory 130 of the electronic device 101 and/or the image received from the external device based on block-by-block reliability estimation, and determines Based on the compression quality obtained, it is possible to control the actions associated with processing the image correction.

In one embodiment, the processor 120 may predict the compression quality of the image based on the block-by-block reliability estimation when determining (or analyzing) the compression quality of the image to improve image quality (eg, image correction). . In one embodiment, the processor 120 removes outliers from the image based on block-by-block reliability estimation for a given image, and enables compression quality prediction in a region in the image, eg, of the image. By analyzing the compression quality based on a partial region rather than the entire region, the classification speed of the compression quality may be improved. Here, the outlier is a specific block in which reliability estimation cannot be properly performed (or compression quality prediction is difficult or impossible), and may indicate, for example, a target region excluded from quality prediction.

According to an embodiment, the processor 120 may control the display module 160 to display a screen including at least one image through the display module 160 . According to an embodiment, the processor 120 may predict (or classify) the compression quality (eg, compression rate) of a given image when displaying the image or while displaying the image. According to an embodiment, the processor 120 may use various denoising models 260 (or denoising models, denoisers) learned (or modeled) for various compression qualities stored in the memory 130. Alternatively, among artifact reducers, a denoising model 260 trained to correspond to the predicted compression quality for a given image may be selected. According to one embodiment, the processor 120 processes image correction (eg, removing compression artifacts from a compressed image to restore it to an original quality image) based on the selected denoising model 260, and converts the corrected image to a specified value. It can be processed according to the action. For example, the processor 120 may operate to display the corrected image through the display module 160, store it in the memory 130, or transmit the corrected image to the outside (eg, a server or other electronic device).

According to an embodiment, the processor 120 may include at least one component (or module) for processing block-by-block reliability estimation-based compression quality prediction and image correction for each compression quality. For example, the processor 120 may include an image processing module 210, a sample patch extraction module 220, a compression quality classification module 230, a denoising model 260, a model selection module 270, and/or a score. At least one of the calculation modules 280 may be included. According to one embodiment, the image processing module 210, the sample patch extraction module 220, the compression quality classification module 230, the denoising model 260, the model selection module 270, and/or the score calculation module ( At least some of 280) may be included in processor 120 as hardware modules (eg, circuitry) and/or may be implemented as software including one or more instructions executable by processor 120. For example, operations performed by the processor 120 may be executed by instructions stored in the memory 130 and causing the processor 120 to operate when executed.

The image processing module 210 may include an image encoder and an image decoder. According to an embodiment, the image processing module 210 may process encoding of an image through an image encoder. For example, the image processing module 210 converts an image into a file of a specified compression rate (or compression level) and a specified format (eg, mpeg, jpeg, gif, and/or png) through image encoding using an image encoder. can be compressed with According to an embodiment, the image processing module 210 may process decoding of an image compressed by a designated encoding through an image decoder. For example, the image processing module 210 may restore (or restore) an image by using an image decoder to decompress an image file compressed by encoding.

The sample patch extraction module 220 (eg, sample patch extractor) may represent a module that extracts a predetermined number (eg, M) of patches (eg, 16x16 patches) in order to classify the compression quality of an image. According to one embodiment, the sample patch extraction module 220 extracts M (eg, about 256) samples (eg, about 256 16x16 patches) from a given image in a specified manner, and the processor ( 120) (eg, the compression quality classification module 230) performs compression quality classification (eg, quality prediction and reliability estimation) and integration on at least some of the extracted 256 16x16 patches to obtain a final quality (Q) for the image. It can operate to determine. According to an embodiment, when extracting patches, the sample patch extraction module 220 may equally or randomly extract a plurality of patches according to a designated method (eg, a uniform method or a random method). For example, the processor 120 may perform analysis through a partial region (eg, a patch unit) rather than the entire region in a given image. For example, the processor 120 may perform analysis based on a designated region of interest (ROI) rather than the entire region in the image.

According to an embodiment, the sample patch extraction module 220 may extract a plurality of regions (eg, uniform extraction or random extraction) from a given image in units of patches. In one embodiment, a patch may mean a minimum image unit capable of understanding the compression quality of an image. For example, when a compression method for compressing an image divides and compresses an image into 8x8 block units, a patch is a 10x10 block of a larger area that can include not only the 8x8 block but also the surrounding relationships of the 8x8 block as a patch. can be set According to an embodiment, an area of 16x16 size may be defined as the size of the patch by adding about 4 pixels of the adjacent block so that the patch includes the center of the adjacent block of the 8x8 block.

The compression quality classification module 230 (eg, compression quality classifier) divides the compression quality into a certain level (eg, N level) (eg, 13 level, 16 level, 50 level, or 100 level). module can be displayed. For example, the compression quality classification module 230 may classify N compression qualities corresponding to N compression qualities (or compression rates or compression levels) related to an image. According to an embodiment, the compression quality classification module 230 may be set to various levels (eg, 13 levels, 16 levels, 50 levels, or 100 levels) based on services or applications. For example, the compression quality classification module 230 may set the first application (eg, image editing application) to level X (eg, level 16), and the second application (eg, gallery application) to level Y (eg, level 16). : 100 steps). For example, the compression quality classification module 230 may prepare N quality compressed images by setting various quality options of an image encoder to N classes. For example, a library application implementing JPEG format encoding may provide encoding quality, for example, 100 compression steps from 1 to 100, and an image editing application may provide encoding quality, for example, For example, 13 compression stages from 1 to 13 can be provided. According to one embodiment, the input of the compression quality classification module 230 may be, for example, an image having a size of 16x16 patches. According to an embodiment, the compression quality classification operation of the compression quality classification module 230 will be described with reference to the following drawings.

According to an embodiment, the compression quality classification module 230 may predict the compression quality of a given image based on the reliability estimation of the patch extracted by the sample patch extraction module 220 . For example, the compression quality classification module 230 may include a reliability estimation module 240 (eg, a confidence estimator) operable to estimate the confidence in a given image on a block-by-block basis (eg, patch-by-patch) and predict the compression quality in the given image. It may include a quality prediction module 250 (eg, quality predictor) operable to do so.

The reliability estimation module 240 may estimate reliability of an input patch (eg, a patch extracted by the sample patch extraction module 220). According to an embodiment, the reliability estimation module 240 may estimate the reliability based on a specific region of interest (ROI) rather than the entire region in the image. For example, the reliability estimation module 240 may estimate the reliability by designating a range to be estimated in the image. According to an embodiment, the reliability estimation module 240 may perform reliability estimation on a plurality of (eg, designated M) patches extracted from an image. In an embodiment, the reliability estimation module 240 estimates reliability of the extracted patches based on various reliability estimation methods, and selects at least one patch (eg, K patches) corresponding to an outlier having a reliability equal to or less than a specified threshold. ) can be estimated. According to an embodiment, while the quality prediction module 250 is being trained, the reliability estimation module 240 continuously measures the quality prediction performance of the input patch at the same time, so that the quality prediction module 250 can determine the specific type of input patch. It can be trained together with the quality prediction module 250 so that a phenomenon in which quality measurement performance is degraded can be statistically grasped.

The quality prediction module 250 may measure the quality of an input patch. According to an embodiment, the quality prediction module 250 includes a patch (eg, a designated M number) corresponding to an outlier estimated based on the reliability estimation module 240 among a plurality of (eg, designated M) patches extracted from an image. : K patches) are excluded from the quality prediction target, and the compression quality can be determined using the rest of the input patches except for the outlier patches (eg, a region enabling compression quality prediction). According to an embodiment, the quality prediction module 250 excludes at least one patch (eg, K patches) corresponding to an outlier whose reliability is equal to or less than a specified threshold value from an average operation of compression quality for each patch, and determines the reliability level. By analyzing the compression quality (e.g. calculating the average of the compression quality for each remaining patch) based on the remaining patches (e.g. (M-K) patches) for which is greater than a specified threshold, the compression quality (e.g. the final quality for the image (e.g. Q)) can be classified (or determined).

For example, the quality prediction module 250 can more accurately and quickly predict the quality of an image by removing outliers from the image and integrating only prediction results obtained from image patches enabling meaningful compression quality prediction. According to an embodiment, if the quality prediction module 250 determines that all of the extracted plural (eg, designated M) patches are outliers, the quality prediction module 250 determines a plurality of (eg, designated M) patches from other regions. In a way of re-extracting , it may operate to predict the quality adaptively.

The denoising model 260 may include a denoising model, a denoiser, or an artifact remover. According to an embodiment, the denoising model 260 may represent, for example, a model to be used as a post-processing filter of the image processing module 210 (eg, an image decoder). According to an embodiment, the denoising model 260 may include a plurality (eg, n, where n is a natural number of 2 or more) for each of various compression qualities (or compression rates or compression levels) related to an image. According to an embodiment, the noise removal model 260 is learned by using various compressed images corresponding to various compression qualities in the electronic device 101, and a plurality of noise removal models 260 respectively corresponding to various compression qualities are learned. , may be stored in the memory 130.

The model selection module 270 may select an optimal denoising model 260 to apply to image correction (eg, compression artifact removal based on compression quality) of a given image based on at least one specified criterion. there is. According to an embodiment, when selecting the denoising model 260, the model selection module 270 may select the denoising model 260 learned to correspond to the compression quality based on reliability estimation. According to some embodiments, in selecting the denoising model 260, the model selection module 270 may, in addition to the compression quality, the user intention (or preference or taste), the type of service or application providing the image, and/or the actual A noise removal model 260 for applying (or setting) another correction strength may be selected in consideration of at least one of the physical sizes (eg, screen size of the display module 160) of the display module 160 on which an image is to be displayed. there is.

The score calculation module 280 may generate one output (eg, a number) by calculating a mean or a median value of the inference results of the input patch based on reliability. According to an embodiment, the score calculation module 280 may learn to output quality information (eg, a number) about the compression quality (eg, final quality (Q)) of an image. According to an embodiment, the score calculation module 280 may learn quality information about compression quality based on a predefined score calculation criterion or range (eg, average or median value).

Various embodiments described in the present disclosure may be implemented in a recording medium readable by a computer or similar device using software, hardware, or a combination thereof. According to hardware implementation, the operations described in various embodiments are ASICs (Application Specific Integrated Circuits), DSPs (Digital Signal Processors), DSPDs (Digital Signal Processing Devices), PLDs (Programmable Logic Devices), FPGAs (Field Programmable Gates) arrays), processors, controllers, micro-controllers, microprocessors, and/or electrical units for performing other functions. there is.

In various embodiments, an operation of displaying a screen including an image on a recording medium through the display module 160, an operation of determining the compression quality of the image, and an operation of selecting a noise removal model learned to correspond to the determined compression quality. , a computer-readable program recording a program for executing an operation of processing an image correction based on a selected denoising model, and an operation of processing the corrected image according to a specified operation (eg, a display, storage, and/or transmission operation). A recording medium may be included.

The electronic device 101 according to an embodiment of the present disclosure includes a display module (eg, the display module 160 of FIG. 1 or 2), a memory (eg, the memory 130 of FIG. 1 or 2), and the A display module and a processor (eg, the processor 120 of FIG. 1 or 2) operatively connected to the memory, wherein the processor displays an image through the display module and a plurality of blocks specified in the image. is extracted in a designated manner, and reliability is estimated for each block with respect to the plurality of blocks, and based on the reliability estimation, a first block and quality prediction corresponding to an outlier, which is a quality prediction exclusion target, among the plurality of blocks A possible second block is identified, the first block among the plurality of blocks is excluded from the quality prediction target, and the remaining second blocks other than the first block among the plurality of blocks are used to determine the quality of the image. It may operate to classify compression quality.

According to one embodiment, the image may be a compressed image compressed with a specified compression quality.

According to an embodiment, the processor estimates at least one first block corresponding to an outlier of a predetermined threshold value or less based on block-by-block reliability estimation for the plurality of extracted blocks, and the at least one Excluding one first block from the average operation for compression quality classification, performing an average operation for compression quality classification based on at least one second block exceeding a specified threshold value, and displaying the result of the average operation in the image. It can operate to classify by compression quality for

According to an embodiment, the processor selects a denoising model learned to correspond to the classified compression quality of the image, processes image quality improvement for the image based on the selected denoising model, and It can act to process images based on specified actions.

According to an embodiment, the processor stores in the memory a plurality of denoising models pre-learned for various compression qualities, and denoises learned corresponding to classification of the compression quality of the image among the plurality of denoising models. You can act to select a model.

According to an embodiment, the processor may operate to process the improved image through at least one designated operation of displaying through the display module, storing in the memory, or transmitting the image to the outside.

According to an embodiment, the processor receives a user input requesting information related to image quality of the image, and based on receiving the user input, the processor displays correction information for the image based on the image. It can operate to control the display module 160 to do so.

According to an embodiment, the processor identifies at least one block having a compression quality corresponding to the representative quality of the image, based on receiving the user input, and a designated notification object based on the identified block. and provide detailed information about image correction.

According to an embodiment, the processor identifies a high-reliability block from the image, measures quality and reliability of the identified block, compresses quality for each block according to the measured quality and reliability, and represents a quality of the image. Comparing, clustering blocks having a compression quality corresponding to the representative quality among the compression qualities of each block and having a relatively high reliability compared to other blocks, and corresponding to the blocks clustered in the image Provide the specified notification object based on the part, and control the display module to display detailed information including a description of a total score and classification for the image correction together with the notification object.

According to an embodiment, the processor may operate to remove artifacts generated during the first compression of the image, and then to perform the second compression during the second compression of the image.

According to an embodiment, the processor receives a user input for compressing the image, and based on receiving the user input, a noise removal model corresponding to a first compression quality upon first compression of the image. Based on, removing artifacts generated during the first compression, determining a second compression quality for the second compression based on the first compression quality related to the first compression, and based on the second compression quality and perform a second compression on the image.

According to an embodiment, the processor may operate to determine the second compression quality corresponding to the first compression quality.

According to an embodiment, the processor may include a quality estimation module and a reliability estimation module for classifying the compression quality, and operate to learn the reliability estimation module together while learning the quality estimation module.

According to an embodiment, the processor may operate to classify the compression quality of an image using a learning model or trained model learned using an artificial intelligence algorithm.

Hereinafter, an operating method of the electronic device 101 according to various embodiments will be described in detail. According to an embodiment, operations performed by the electronic device 101 described below include various processing circuitry and/or executable program elements of the electronic device 101. may be executed by the processor 120 . According to an embodiment, operations performed by the electronic device 101 may be executed by instructions stored in the memory 130 and causing the processor 120 to operate when executed.

3 is a flowchart illustrating examples of learning, classification, and removal operations for image correction in an electronic device according to an exemplary embodiment.

Referring to FIG. 3 , in operation 301, the processor 120 of the electronic device 101 may generate a dataset composed of compression qualities of a plurality of classes. According to one embodiment, the processor 120 may generate a dataset consisting of N classes of compression quality. For example, a dataset of images may be required to create a machine learning model in the electronic device 101 . In one embodiment, a dataset of images may be prepared using, for example, the image processing module 210 of FIG. 2 (eg, an image encoder). For example, the processor 120 may set the quality options of the image encoder in various ways to N classes to prepare N types of compressed images. For example, a library application implementing JPEG format encoding may provide encoding quality, for example, 100 compression steps from 1 to 100, and an image editing application may provide encoding quality, for example, For example, 13 compression stages from 1 to 13 can be provided.

According to one embodiment, the processor 120 learns, for example, the compression quality classification module 230 (eg, the reliability estimation module 240 and the quality prediction module 250) of FIG. 2 based on the dataset. can do. According to one embodiment, the processor 120 takes images compressed with various qualities as “input X”, and the quality used when generating each compressed image is “ground truth output Y” (eg, label or sign). ) can be set. According to an embodiment, while the quality prediction module 250 is being trained, the processor 120 simultaneously measures quality prediction performance for an input patch so that the quality prediction module 250 measures the quality of a specific type of input patch. The reliability estimation module 240, which can statistically grasp the phenomenon of deterioration in performance, can also operate to learn together. For example, the processor 120 may not only learn how to classify the corresponding compression quality in the training data, but also learn how to estimate the corresponding reliability. Based on this, the processor 120 may apply reliability estimation suitable for the corresponding compression quality when a specific image is given.

According to an embodiment, the processor 120 may learn, for example, the noise removal model 260 (eg, an artifact reducer) of FIG. 2 . According to an embodiment, the processor 120 may set an image compressed with various qualities as “input X” and an original image as “ground truth output Y”. According to one embodiment, unlike the compression quality classification module 230, the denoising model 260 may indicate an image whose label is not an integer.

At operation 303, the processor 120 may classify the compression quality class. According to an embodiment, the processor 120 may perform learning to classify compression quality classes. According to one embodiment, the processor 120 may use various machine learning methods as a method for classifying the compression quality class, for example, the compression quality classification module 230 (eg, convolutional neural network (CNN)) : The quality prediction module 250) may be implemented. For example, the processor 120 trains the CNN model with data decoded by the image processing module 210 (eg, an image decoder) of various compressed image qualities on the dataset generated in operation 301, and the electronic device 101 Given an arbitrary image on the image, it is possible to predict the compression quality of the image. A machine learning method according to an embodiment is not limited to a CNN model, and various machine learning methods that can be used for classification and improvement of compression quality may be used. For example, as a machine learning method, a deep belief network (DBN) based on an unsupervised learning method and / or a deep autoencoder (deep autoencoder), two-dimensional data processing such as images It may include various matrix learning methods, such as a convolutional neural network (CNN) for data processing, and/or a recurrent neural network (RNN) for time-series data processing.

According to an embodiment, the processor 120 may infer the compression quality of an image. According to an embodiment, when the electronic device 101 classifies the compression quality, the processor 120 does not check all patches of the image, for example, in FIG. 2 Using the sample patch extraction module 220, for example, about 256 16x16 patches are extracted (e.g. uniform or random extraction), and the extrusion quality (e.g. final quality (Q)) can be predicted. According to an embodiment, when predicting the final quality Q based on the extracted regions, the processor 120 estimates the reliability of the extracted regions, and identifies an outlier whose reliability (or score) is less than or equal to a specified threshold. For , the final quality (Q) can be predicted by excluding it from the average operation of the compression quality. According to various embodiments, the processor 120 may rapidly improve classification speed by analyzing a part of an image rather than the whole. According to an embodiment, a compression quality classification operation will be described with reference to the following drawings.

In operation 305, the processor 120 may remove artifacts based on the application of the denoising model learned to correspond to the compression quality. According to an embodiment, the processor 120 may remove artifacts by applying a denoising strength learned to correspond to a corresponding compression quality to a given image. According to an embodiment, the processor 120 may perform learning to determine a denoising model corresponding to compression quality. According to one embodiment, the processor 120 may generate, for example, N denoising models 260, such as the number of categories of compression quality. According to some embodiments, in order to reduce the number of denoising models 260, the processor 120 may generate the denoising models 260 by quantizing them to a number smaller than N, for example. As an example, the processor 120 may approximate, for example, about 8 qualities out of about 100 compression qualities (eg, compression qualities 20, 30, 40, 50, 60, 70, 80, and 90). . According to an embodiment, the processor 120 may generate a learning model so that the compressed image generated through approximation can be corrected in units of 8x8 patches with the original image. According to an embodiment, the processor 120 may set the input size to a patch having a size of 16x16 pixels so that at least some areas of neighboring blocks are also considered. For example, the processor 120 may prevent sudden changes in the texture and/or color of a specific block with neighboring blocks.

According to an embodiment, the processor 120 determines an appropriate denoising model 260 based on the quality predicted through the compression quality classification module 230 (eg, the final quality Q), and the determined denoising model. (260) may be applied to image correction to remove compression artifacts resulting from image lossy compression.

4 is a diagram illustrating an example of image correction in an electronic device according to an exemplary embodiment.

In the example of FIG. 4 , screen <401> may indicate an example in which a compressed image 410 (eg, a given image) before image correction according to compression quality is provided. In the example of FIG. 4 , screen <403> may indicate an example in which a corrected image 420 (eg, a restored image) after image correction according to compression quality is provided.

According to one embodiment, in the example of FIG. 4 , the example screen <401> indicates that the compression quality of a given image (eg, the compressed image 410 compressed with a specified compression quality) is, for example, a JPEG quality level (quality level). ) (e.g., about 80). According to one embodiment, in the example of FIG. 4 , example screen <403> classifies (or predicts) that the electronic device 101 is a JPEG quality level (eg, about 80) in a given image 410 and .

According to an embodiment, most of the images in the electronic device 101 may be compressed with a certain compression quality (or compression rate) and provided. As illustrated in the example screen <401>, such a compressed image may generate unique artifacts (eg, element 430 in the example screen <401>) depending on the compression quality. For example, an image may be subjected to various sampling (e.g. chroma subsampling), block size, and/or approximation (e.g. discrete cosine transform (DCT)) to reduce compression quality (e.g. bit rate). , discrete cosine transform coefficient approximation (coefficient quantization) can be used, and can be further compressed with more various compression quality control methods (eg, bit rate control method) according to the compression codec.

According to an embodiment, the electronic device 101 learns (or trains) patterns of unique artifacts (or noises) appearing in corresponding compression quality in training data (eg, images for each compression quality) composed of various compression qualities. and a plurality of models (eg, denoising models) to remove them may be learned through a deep neural network. In one embodiment, the deep neural network may represent an artificial neural network (ANN) including multiple hidden layers between an input layer and an output layer. According to an embodiment, a deep neural network can learn various nonlinear relationships including multiple hidden layers, and is used as a core model for deep learning. According to an embodiment, the deep neural network may include a deep trust neural network (DBN), a deep autoencoder, a convolutional neural network (CNN), and/or a recurrent neural network (RNN) according to an algorithm. According to an embodiment, the electronic device 101 receives a pre-learned (or trained) model (eg, a noise cancellation model) for patterns of artifacts (or noises) from another electronic device or obtains (eg, a noise cancellation model) from a server. can be downloaded).

In various embodiments, a denoising model optimized for each compression quality may be matched, and an artifact (or noise) (eg, element 430) may be removed by applying a denoising model corresponding to the compression quality. According to an embodiment, the electronic device 101 can classify artifacts according to compression quality through learning data, automatically analyze patterns of artifacts, and remove even unique artifacts corresponding to the corresponding compression quality. .

According to an embodiment, the electronic device 101 determines the compression quality of a given image (eg, the compressed image 410 of the example screen <401>), and learns according to the artifact (or noise) of the compression quality. An image from which artifacts are removed (eg, the restored image 420 of the example screen <403>) may be provided to the user by applying the denoising model.

5 is a flowchart illustrating a method of operating an electronic device according to an exemplary embodiment.

Referring to FIG. 5 , in operation 501, the processor 120 of the electronic device 101 may display a screen including an image. According to an embodiment, the processor 120 may visually provide an image and/or content including the image through the display module 160 . According to an embodiment, the processor 120 displays a screen including an image stored in the memory 130 of the electronic device 101 and/or an image received from an external device based on a user input, the display module 160 ) can be controlled. According to one embodiment, the image may represent a compressed image compressed with a specified compression quality (or compression rate or compression level).

In operation 503, the processor 120 may determine (or classify) the compression quality (or compression rate or compression level) of the given image. According to an embodiment, when the processor 120 determines the compression quality, the processor 120 determines the compression quality, not the entire area of the given image, but a partial area, for example, a patch (eg, 8x8 pixel, 16x16 pixel, or 64x64 pixel unit). , compression quality can be predicted for each patch by extracting designated M (eg, about 256) patches in a designated manner (eg, uniformly or randomly). For example, image compression may be performed by dividing an image into small blocks and collectively applying a compression mechanism (eg, chroma subsampling, DCT, quantization) of the same compression quality level to all blocks.

For example, a compressed image may be subjected to various sampling (eg, chroma subsampling), block size, and/or approximation (eg, discrete cosine transformation (eg, Various compression mechanisms such as DCT, discrete cosine transform, and coefficient approximation (coefficient quantization) may be used, and further various compression quality control methods (eg, bitrate control method) may be used depending on the compression codec. A compressed image may be compressed (eg, by applying a similar compression mechanism) to a certain compression quality (or compression rate) for all blocks to meet the target compression quality for the entire image. Thus, even if at least some blocks in the image are checked, the overall compression quality can be predicted for the image.

According to an embodiment, when determining the compression quality of the image, the processor 120 may predict the compression quality of the image based on reliability estimation in block units (eg, patch units). For example, the processor 120 may remove outliers from the image based on block-by-block reliability estimation for a given image, and analyze the compression quality based on a region in the image that enables compression quality prediction. can For example, the processor 120 excludes the patch corresponding to the outlier from the quality prediction target through reliability estimation in the input patch, and excludes the patch of the outlier among the input patches (e.g., enables compression quality prediction). area) can be used to determine the compression quality.

According to an embodiment, when determining the compression quality, the processor 120 extracts a plurality of (eg, designated M) patches from a given image in a designated method (eg, a uniform method or a random method), and extracts the patch. Reliability estimation of patched patches can be performed. For example, the processor 120 estimates the reliability of the extracted patches based on various reliability estimation methods, and for at least one patch (eg, K patches) corresponding to an outlier having a reliability equal to or less than a specified threshold. , Excluding from the average computation of compression quality per patch, analysis of the compression quality based on the remaining patches (eg, (M-K) patches) whose reliability exceeds a specified threshold (eg, computation of the average compression quality per remaining patch) In this way, the compression quality (eg, the final quality (Q) for the image) can be classified (or determined). For example, the processor 120 may more accurately and quickly predict the quality of an image by removing outliers from the image and integrating only prediction results obtained from image patches enabling meaningful compression quality prediction.

According to an embodiment, the processor 120 may determine the compression quality through classification learning and prediction of the compressed image in patch units (eg, 8x8 patch units, 16x16 patch units, or 64x64 patch units). For example, since image compression may be performed in patch units, classification may be possible in 8x8 patch units, 16x16 patch units, or at most 64x64 (eg, HEVC) patch units. Through this, the processor 120 can prevent calculation errors due to overall image characteristics by concentrating on a corresponding patch, which is a unit in which compression occurs. In addition, since the size of the input image of the operation for determining the compression quality is small, the processor 120 unnecessarily widens the receptive field and improves accuracy by using a conventional pooling layer. You can avoid actions that could cause you to fall.

According to an embodiment, the processor 120 may perform compression regardless of compression, such as an area (eg, an outlier) that does not clearly reflect the compression rate due to the nature of the image (eg, a black area and/or an area of low frequency components in an image). Since an area having similar characteristics) may occur, an outlier may be removed from an image when determining compression quality in consideration of this. For example, when calculating the average value of each compression quality (or score) of the extracted regions, the processor 120 analyzes the reliability of the extracted regions, and calculates the average value for outliers whose reliability is less than or equal to a specified threshold. can be excluded from

In operation 505, the processor 120 may select a model (eg, a denoising model or a denoising model) learned to correspond to the compression quality. For example, the processor 120 may predict the compression quality in advance and use the noise removal model 260 learned with the corresponding compression quality in the electronic device 101 .

According to an embodiment, the processor 120 selects a denoising model 260 learned to correspond to the compression quality of an image from among a plurality of denoising models 260 pre-learned for each compression quality in the memory 130. can According to an embodiment, the processor 120 may select a first denoising model corresponding to the classification of the first compression quality when classifying the given image as the first compression quality, and classifying it as the second compression quality. In this case, a second noise reduction model corresponding to the classification of the second compression quality may be selected, and in the case of classification into the third compression quality, a third noise reduction model corresponding to the classification of the third compression quality may be selected. According to an embodiment, when selecting a denoising model learned to correspond to the compression quality, the processor 120 determines the user's intention (or preference or taste), the type of service or application providing the image, and/or the actual image to be displayed. A noise cancellation model may be selected by further considering at least one of the physical sizes of the display module 160 (eg, the screen size of the display module 160).

In operation 507, the processor 120 may perform image processing based on the selected model. According to an embodiment, the processor 120 may process image correction using a noise removal model selected from among a plurality of noise removal models 260 based on compression quality of an image. According to an embodiment, the processor 120 removes compression artifacts (or noise) from the given image according to the correction strength corresponding to the selected denoising model, and restores the given image (eg, the compressed image) to an original image before compression. can do.

At operation 509, the processor 120 may process the designated operation. According to an embodiment, the processor 120 may control the display module 160 to display the corrected image. According to an embodiment, the processor 120 may control the display module 160 to display a corrected state based on a noise removal model corresponding to a given image. According to some embodiments, the designated operation of operation 509 may include an operation of transmitting the corrected image to the outside or storing it in the memory 130 of the electronic device 101 .

6 is a diagram illustrating an example of an operation learned in an electronic device according to an embodiment.

According to an embodiment, FIG. 6 may show an example of a pre-learning operation performed by the electronic device 101 . According to an embodiment, the pre-learning operation performed by the electronic device 101 as illustrated in FIG. 6 is performed, for example, in another environment (eg, a developer workstation and/or a cloud environment), and the electronic device It can also be downloaded from (101). According to an embodiment, for a model (eg, a classification model) according to a pre-learning operation of the electronic device 101, a pre-trained (or trained) model is acquired (eg, received or downloaded) by another electronic device or server. )You may.

Referring to FIG. 6 , in block 610, the processor 120 of the electronic device 101 may generate a dataset composed of N qualities. According to an embodiment, the processor 120 subsamples channels of images in N steps, transforms and /or can perform coefficients quantization.

At block 620, processor 120 may learn a model based on the compressed image. According to an embodiment, the processor 120 may perform training (eg, block 630) on the compressed image. According to one embodiment, learning in this disclosure represents learning that allows the target denoising model to analyze the artifacts of the transformed patch to determine the compression quality and the reliability of the compression quality, as illustrated in block 430. can

At block 640, processor 120 may generate one classification model according to learning (block 630). According to an embodiment, the processor 120 may use a classification model generated based on learning for an operation of predicting the compression quality of FIG. 7 to be described later.

7 is a flowchart illustrating an operation method of predicting compression quality in an electronic device according to an embodiment.

According to an embodiment, FIG. 7 may show an example of an operation of classifying compression quality performed by the electronic device 101 .

Referring to FIG. 7 , in operation 701, the processor 120 of the electronic device 101 may input an image. According to an embodiment, the processor 120 may input an image of compressed quality to be improved as an image of a learning target.

In operation 703, the processor 120 may extract about M patches at equal intervals from the image. According to an embodiment, the processor 120 may equally or randomly extract, for example, about 256 16x16 patches from an input image.

In operation 705, the processor 120 analyzes the artifact with a classification model and classifies it into one compression quality among N stages (eg, quality information (eg, number) according to the characteristics of the artifact). According to an embodiment, the processor 120 analyzes the corresponding artifact based on the classification model generated according to learning (eg, block 430) in FIG. 6 and classifies it into one compression quality among N levels of compression quality. can do

At operation 707, the processor 120 may remove about K outliers using the confidence level. According to an embodiment, the processor 120 may estimate the reliability in units of patches and remove outliers from the image based on the reliability estimation in units of patches. For example, through reliability estimation, the processor 120 may exclude a patch corresponding to an outlier having a reliability equal to or less than a specified threshold value from a quality prediction target. An example of this is illustrated in FIG. 8 .

In operation 709, the processor 120 may predict a representative quality (eg, final quality (Q)) of an image by integrating quality results of Y (eg, Y = M - K) patches. According to an embodiment, the processor 120 removes at least one patch (eg, K patches) corresponding to an outlier from the patches (eg, M patches) extracted from the image, and removes the remaining patches (eg, Y patches). The representative quality (or final quality (Q)) of an image can be predicted (or derived) by averaging the results (eg, compression quality for each Y number of patches) output from (eg, M - K) patches. . According to an embodiment, the processor 120 may use the predicted representative quality of the image as an input of the compression quality of the image in a compression artifact removal and/or image correction operation described below.

8 is a diagram illustrating an example of an inference operation for predicting quality in an electronic device according to an exemplary embodiment.

According to an embodiment, FIG. 8 may show an example of an operation of predicting reliability-based compression quality in a patch unit in the electronic device 101 .

Referring to FIG. 8 , the electronic device 101 may perform learning to estimate the reliability of each patch. According to an embodiment, the electronic device 101 may use various machine learning methods as a method for estimating reliability in units of patches, and for example, reliability estimation may be implemented with a convolutional neural network (CNN).

For example, when an arbitrary image 800 (eg, a compressed image) is given, the electronic device 101 uses the reliability recognition CNN 820 for each patch 810 of the image 800 to obtain the image ( 800) to predict the compression quality level. According to one embodiment, in the example of FIG. 8 , it may be assumed that only nine input patches 810 are inspected in the entire area of the image 800 . According to one embodiment, each input patch 810 may include a compressed coding block. In one embodiment, patches 810 may be uniformly sampled, for example, via sample patch extraction module 220 . The sampled patches can be input to CNN 820 for batch prediction, and through CNN 820, for example, 9 pairs of quality results (eg, quality 830) and confidence (confidence) 840) can be calculated.

According to an embodiment, the electronic device 101 may exclude patches (eg, two low-confidence patches) corresponding to outliers whose reliability is less than or equal to a specified threshold (eg, about 80%) from 9 pairs of quality results. can According to an embodiment, the electronic device 101 selects remaining patches (eg, patches corresponding to outliers are removed) whose reliability exceeds a specified threshold value (eg, about 80%) in the 9 pairs of quality results. For example, the quality results of 7 patches with high reliability) can be averaged. For example, in the example of FIG. 8 , the electronic device 101 averages output results from 7 patches, and converts the average value (eg, '85') to the representative quality (or final quality (or final quality) of the image 800). Q)) can be predicted (or derived).

A machine learning method according to an embodiment is not limited to a CNN model, and various machine learning methods that can be used for classification of compression quality may be used. For example, as a machine learning method, various methods such as a deep belief network (DBN), a deep autoencoder, a convolutional neural network (CNN), and/or a recurrent neural network (RNN) It may include machine learning methods.

According to an embodiment, when the electronic device 101 performs compression quality prediction (or compression quality class classification) for an image, the electronic device 101 does not check all patches of the image, for example, a sample Using the patch extraction module 220, for example, about 256 16x16 patches are extracted, and from the extracted 256 patches, patches corresponding to outliers are removed through reliability estimation, and compression quality can be predicted. The final quality (Q) for an image can be accurately and quickly predicted by classifying and integrating significant patches that For example, when predicting the final quality Q based on the extracted regions, the electronic device 101 estimates the reliability of the extracted regions, and excludes regions whose reliability is less than or equal to a specified threshold value in calculating the average value. The final quality (Q) can be predicted.

According to an embodiment, in the present disclosure, as illustrated in FIG. 8 , the quality result of the CNN 820 is determined by the quality 830 (eg, the prediction result by learning of the quality prediction module 250) and the reliability ( 840) (eg, a prediction result by learning of the reliability estimation module 240). For example, the electronic device 101 may predict the compression quality based on the reliability of the quality prediction of each patch based on the difference between the quality prediction result and the ground truth result. For example, according to the present disclosure, while the quality prediction module 250 is being trained, quality prediction performance for an input patch is continuously measured, and the quality prediction module 250 measures the quality of a specific type of input patch. The reliability estimation module 240, which can statistically grasp the phenomenon of poor performance, can also be learned.

For example, in the case of CNN(820) training, images (e.g. raw images) can be compressed to some degree to provide ground truth labels of compressed quality, but not true confidence (or scores) for predicting quality. does not exist. Hereinafter, an example of substantially estimating and providing reliability of a predicted quality level for each candidate patch (eg, an input patch) according to the present disclosure will be described.

)을 정확하게 예측할 수 있다고 가정하면, 주어진 입력 패치 xi의 실제 신뢰도 ci는 아래 <수학식 1>과 같이 근사화될 수 있다.For example, if a trained CNN model G(·) is given the compression quality of an input patch x, e.g.

) can be accurately predicted, the actual reliability ci of a given input patch xi can be approximated as shown in Equation 1 below.

는 주어진 i번째 입력 패치 xi의 예측된 품질(predicted quality)을 나타내고, qi는 실제 품질(true quality)을 나타내며, m은 최대 품질 레벨(maximum quality level)을 나타낼 수 있다. 예를 들면, m은 libjpeg-turbo의 경우 100으로 설정될 수 있고, FFmpeg의 경우 51로 설정될 수 있다. In <Equation 1>,

denotes the predicted quality of a given ith input patch xi, qi denotes true quality, and m denotes the maximum quality level. For example, m can be set to 100 for libjpeg-turbo and 51 for FFmpeg.

Accordingly, another reliability estimation module 240 may be trained using the CNN model G(·) (eg, the quality prediction module 250) and <Equation 1>.

는 입력 패치 x에 대해 추정된 신뢰도(estimated confidence)를 나타낼 수 있다. 예를 들면, L₁ 손실(loss)이 CNN 모델 H(·)(예: 신뢰도 추정 모듈(240)) 훈련에 사용되는 것을 가정하면, H(·)은 각 후보 패치(예: 입력 패치)에 대한 신뢰도 측정 값에서 평균 편차(average deviation)가 가장 작은 중간 값(median number) z를 추정할 수 있다.In <Equation 2>

may represent the estimated confidence for the input patch x. For example, assuming that L ₁ loss is used for training a CNN model H(·) (eg, reliability estimation module 240), H(·) is for each candidate patch (eg, input patch). A median number z having the smallest average deviation from the reliability measurement value for can be estimated.

However, the conventional method using two networks of G(·) and H(·) may be inefficient in that it requires two learning stages and two inferences. Therefore, in the present disclosure, since the two tasks of quality prediction and reliability estimation can share many common features, as shown in Equation 4 below, the quality 830 and reliability in one network A network F(·) can be implemented to output all (840).

In <Equation 4>, F(x) ₁ and F(x) ₂ may represent first and second output units of F(x), respectively. As a result, a loss function such as Equation 5 below may be adopted.

In <Equation 5>, F _θ may represent a confidence-aware CNN having a parameter set θ. In this disclosure, reliability estimation classifies inputs into two categories: reliable patches (e.g., outlier regions in an image) and unreliable patches (e.g., regions in an image where meaningful compression quality predictions are possible). can be used to Also, quality prediction may require more accurate quality output. In one embodiment, the parameter λ, which reflects the relative importance of reliability rather than quality, is experimentally set to 0.5, and the result can be expressed as follows. According to another embodiment, it can be confirmed experimentally that similar results are obtained even when the parameter λ is smaller than 0.5, such as 0.25.

According to an embodiment, <Equation 1> to <Equation 5> represent a background for classifying the output unit of the CNN 820 into quality 830 and reliability 840, and a loss function for learning. May indicate an example of how to design, and this example is only to help the understanding of the present disclosure, but does not limit the embodiments of the present disclosure.

According to an embodiment, <Table 1> and <Table 2> below may show examples of experimental results using the quality prediction method according to the present disclosure. For example, <Table 1> shows an example of experimental results when using the JPEG codec, and <Table 2> shows an example when using the H.264 codec. According to the quality prediction method according to the present disclosure (eg, the fifth method in the examples of <Table 1> and <Table 2>), when the compression quality is measured by estimating the reliability of an input patch, another existing method (eg, the first method) It can be seen that a more accurate compression quality prediction result is obtained in a similar or faster time than the first to fourth methods). In one embodiment, the first method may represent the MobileNetV2 algorithm, the second method may represent the EfficientNet algorithm, the third method may represent the block-based Naive algorithm, and the fourth method may represent the block-based Sobel. algorithm, and the fifth method may represent an algorithm of the quality prediction method (eg, named 'Q1Net') of the present disclosure.

<Table 1> and <Table 2> show the performance comparison for the data set, the mean absolute error (MAE), standard deviation of errors (SDE), and latency of each method. ), and/or model size to represent a summary of the main experimental results with a total of 224x224 or 256x256 input pixels. For example, in the task of predicting quality levels from 1 to 100, the fifth method achieves good prediction performance with an MAE of about 0.43 and high reliability with an SDE of about 0.42 with a processing time of about 10 ms for JPEG. It can be shown to achieve good prediction performance with a MAE of about 0.48 dml with a processing time of about 18 ms for H.264 and high reliability with an SDE of about 0.51. As illustrated in <Table 1> and <Table 2>, the fifth method has a relatively superior performance compared to the existing methods 1 to 4, and can derive more accurate and faster quality prediction results. can

Method Method 1 Method 2 Method 3 Method 4 Method 5 MAE 1.16 2.24 1.28 0.46 0.43 SDE 1.43 2.29 5.79 0.50 0.42 Time (ms) 7 12 9 14 10 #Params 2.32M 4.13M 125K 125K 125K

Method Method 1 Method 2 Method 3 Method 4 Method 5 MAE 0.58 0.56 1.01 0.77 0.48 SDE 0.52 0.6 2.16 0.87 0.51 Time (ms) 7 12 18 21 18 #Params 2.32M 4.13M 208K 208K 208K

According to the quality prediction method according to the present disclosure, the CNN input can be reduced to the size of a compressed coding block. The input size used for conventional image classification is about 224x224, but the size of a coding block used for compression can be much smaller than that. For example, in case of JPEG, it is 8x8, and in case of H.264, the largest coding block size may be 16x16. Therefore, according to the present disclosure, the input size of the CNN can be effectively reduced, and since the CNN size is reduced, the number of parameters of the CNN can also be reduced. According to an embodiment, since the number of parameters of the machine learning model is reduced, the amount of calculation is also reduced, so when based on a single block, algorithm latency is much lower than that of conventional image classification networks during inference. can have In addition, according to the present disclosure, the compression quality can be effectively and accurately measured even in an old electronic device or a low-cost electronic device having relatively weak computing power by variably accepting the number of blocks (or patches) to be inferred. In addition, according to the present disclosure, in an image such as super resolution, it is used before improving the quality of the image to prevent amplification of artifacts (or noise) in advance, or it can be applied to a browser to prevent an external server (eg : There is an effect of quickly identifying the need for improvement and the intensity of improvement of high-compression and low-resolution images received from the Internet).

According to an embodiment, artifact removal for improving the quality of a compressed image may be in a 'trade-off' relationship with detail preservation. For example, if the artifact removal is perfect, the detail of the image may disappear, and if the detail is maintained, the artifact may not be removed. Therefore, as artifacts are removed from the image, details may disappear and a blurrier image may be obtained. Therefore, the performance of selecting a noise removal model for artifact removal is improved using the image quality value (eg, the final quality (Q) predicted based on block-by-block reliability estimation) inferred according to the quality prediction method according to the present disclosure. can be improved (e.g. by being able to select the correct denoising model corresponding to the correct compression quality) and consequently improve the overall quality of the image.

9 is a flowchart illustrating an operation method of removing artifacts in an electronic device according to an exemplary embodiment.

According to an embodiment, FIG. 9 may show an example of an operation of removing an artifact based on a noise removal model learned to correspond to the compression quality classified by the electronic device 101 .

Referring to FIG. 9 , in operation 901, the processor 120 of the electronic device 101 may input an image. According to an embodiment, the processor 120 may input an image having a compression quality to be improved (eg, an image of a learning target of FIG. 7 ) as an image of a compression artifact removal target.

At operation 903, the processor 120 may input the predicted representative quality. According to an embodiment, the processor 120 may input the representative quality predicted in FIG. 7 as the compression quality of an image to be de-artifacted.

At operation 905, the processor 120 may select one of the N denoising models based on the representative quality. According to an embodiment, the processor 120 may select a denoising model (eg, translation model i) learned to correspond to the representative quality of a given image from among N transformation models pre-trained for various compression qualities. According to an embodiment, the processor 120 may select a first denoising model corresponding to the classification of the first compression quality when classifying the given image as the first compression quality, and classifying it as the second compression quality. In this case, a second noise removal model corresponding to the classification of the second compression quality may be selected, and in the case of classification into the third compression quality, a third noise removal model corresponding to the classification of the third compression quality may be selected.

In operation 907, the processor 120 may perform image processing based on the selected noise removal model. According to an embodiment, the processor 120 may process image correction using a denoising model selected from among a plurality of denoising models based on compression quality (eg, representative quality) of an image. According to an embodiment, the processor 120 compresses the given image (eg, the compressed image) by removing compression artifacts (or noise) from the given image (eg, the compressed image) according to the correction strength corresponding to the selected denoising model. It can be restored to the previous original image. For example, the processor 120 may restore the compressed image to an original quality image (eg, an original image).

10 is a flowchart illustrating a method of compressing an image in an electronic device according to an exemplary embodiment.

According to one embodiment, FIG. 10 may show an example of an operation of re-compressing (or encoding) (eg, secondary compression) a compressed image.

Referring to FIG. 10 , in operation 1001, the processor 120 of the electronic device 101 may provide an image. According to an embodiment, the processor 120 may display a compressed image corresponding to a user selection or an image list including at least one compressed image.

In operation 1003, the processor 120 may receive a user input for image compression. According to an embodiment, the processor 120 may detect a user input causing compression to be performed on the provided image (or at least one image selected by the user). In one embodiment, the user input may be input based on a save command of an image, a menu-based compression command, and/or a designated object-based selection for compression. For example, a user can edit an image (e.g. crop, rotate, adjust brightness, adjust resolution, adjust color, adjust sharpness, resize, change format) through a designated application (e.g. gallery application or image editing application). and/or effect setting), and may want to save the edited image.

In operation 1005, the processor 120 may remove artifacts in the first compression of the image. According to an embodiment, the processor 120 may receive a user input related to image compression and identify a first compression quality related to previous compression (eg, first compression) of the image based on the user input. According to an embodiment, the processor 120 may remove an artifact (eg, an artifact of a high frequency component region in an image) generated during the first compression, based on a noise removal model corresponding to the first compression quality.

At operation 1007, the processor 120 may determine a second compression quality based on the first compression quality associated with the first compression. According to an embodiment, the processor 120 may determine a quality parameter for secondary encoding using the quality prediction module 250. For example, when a compressed image is edited and then re-encoded (eg, second compression), the processor 120 may re-determine the compression quality (or compression level) of the image. For example, even if an image once encoded with low quality is encoded again with high quality, degraded details may not be restored again. For example, even if the second compression quality related to the second compression is set higher than the first compression quality related to the first compression, information already lost by the first compression (eg, high frequency component information) cannot be restored. Accordingly, the processor 120 may set the maximum value of the second compression quality to a compression quality similar to that of the first compression quality. For example, when the first compression quality of the image is first compressed to about 70, the second compression quality may not be set to about 80 or higher. Here, the higher the number of compression quality, the better picture quality can be derived.

According to an embodiment, the processor 120 encodes the image with a compression quality similar to that of the original (eg, first compression quality) (eg, a slightly higher compression quality than the first compression quality) during the second compression of the image. , you can reduce the file size of the image, and reduce the archive size.

In operation 1009, the processor 120 may perform a second compression on the image based on the second compression quality. For example, a high-frequency component may occur in an image due to a compression artifact generated in the first compression, and another artifact may occur in the process of compressing the corresponding high-frequency component. Accordingly, the processor 120 may perform the second compression after removing compression artifacts in the first compression before the second compression.

In operation 1011, the processor 120 may store an image. For example, the processor 120 may store an image generated according to the second compression in the memory 130 . According to an embodiment, when storing an image, the processor 120 may replace the original image and save it, or may store the original image as a separate image.

11 is a flowchart illustrating an operation method of providing information on image correction in an electronic device according to an exemplary embodiment.

12 is a diagram illustrating a user interface for providing information on image correction in an electronic device according to an exemplary embodiment and an operation example thereof.

According to one embodiment, FIGS. 11 and 12 show correction information (eg, compression quality score (or level), correction area notification object, and/or correction information for an image provided from a designated application (eg, a gallery application or an image editing application)). Detailed information) may be shown as an example of an operation of providing.

Referring to FIG. 11 , in operation 1101, the processor 120 of the electronic device 101 may display a screen including an image. According to an embodiment, the processor 120 may execute a designated application and visually provide an image through the display module 160 based on the designated application. For example, as illustrated in FIG. 12 , the processor 120 may control the display module 160 to display a screen including an image 1200 selected by a user through a gallery application. According to one embodiment, the image 1200 may represent a compressed image compressed with a specified compression quality (or compression rate or compression level).

In operation 1103, the processor 120 may receive a user input requesting information related to image quality. According to one embodiment, the processor 120 may detect user input instructing it to provide correction information (eg, compression quality score (or level), correction area notification object, and/or correction details) for the provided image. there is. In one embodiment, the user input may be input based on a menu-based request command and/or selection based on a designated object for providing correction information. For example, as illustrated in FIG. 12 , the processor 120 uses a function execution object 1210 designated to execute a function (eg, an AI remaster function) providing correction information for an image in a gallery application. input can be detected. In one embodiment, the function of providing correction information for an image (eg, an AI remaster function) may refer to a function of automatically correcting an image with AI (eg, improving image quality) and providing correction information for the automatic correction. there is.

According to an embodiment, the electronic device 101 provides a designated area for executing the remaster function through a designated area (eg, the upper right area of the screen, the upper left area of the screen, or the lower center area of the screen) of the screen on which the image 1200 is displayed. A function execution object 1210 may be provided. According to an embodiment, the processor 120 may execute a remaster function based on a user input using the function execution object 1210 and provide a user interface as illustrated in FIG. 12 . According to an embodiment, the processor 120 may detect a user input selecting an option specified to execute a function of providing correction information on an image through menu entry in the gallery application.

In operation 1105, the processor 120 may identify a patch having high confidence in a classification operation. According to an embodiment, the processor 120 may identify a patch (eg, a location or region of a patch) extracted from a given image in an operation of classifying compression quality (eg, a block-by-block reliability-based compression quality prediction operation). can

In operation 1107, the processor 120 may additionally extract a specified number of samples around the identified patch. According to an embodiment, the processor 120 may perform quality (eg, quality 830 of FIG. 8 ) and reliability (eg, reliability 840 of FIG. 8 ) around the extracted patch (eg, the location or region of the patch). ))), a specified number of samples (e.g., about A) may be additionally extracted.

At operation 1109, processor 120 may determine quality and reliability. According to an embodiment, the processor 120 may measure quality and reliability based on the extracted patch and samples additionally extracted from the vicinity of the extracted patch.

In operation 1111, the processor 120 may cluster patches corresponding to the representative quality (eg, final quality (Q)) of the image. According to an embodiment, the processor 120 compares the compression quality according to the quality and reliability measured in operation 1109 with the representative quality of the image, and has the compression quality corresponding to the representative quality of the image among the extracted patches, Compared to other patches, patches having relatively high reliability can be clustered.

In operation 1113, the processor 120 may provide a designated notification object based on a clustered part (or region) in the image. According to an embodiment, the processor 120 may provide a dotted line and/or an overlay mask of a specified color through the clustered parts in the image. For example, as illustrated in FIG. 12 , the processor 120 provides a notification object 1230 (eg, a yellow dotted line object) for a clustered portion in a given image 1200, and provides a notification object 1230 for an image correction portion. Information can be provided to users. In one embodiment, the notification object 1230 may represent an object for intuitively notifying a user of a clustered part (eg, a corrected area) in a given image.

In operation 1115, the processor 120 may provide detailed information about image correction (eg, information related to compression quality of an image (or quality notification information)). According to an embodiment, the processor 120 may provide detailed information on image correction (eg, an overall score and a description of why the image is classified as the current quality) along with an image-based notification object. For example, as illustrated in FIG. 12, the processor 120 provides detailed information 1250 on the correction result of the image 1200 (eg, 'The compression quality of this picture is 50. Accuracy 99 through the yellow part) It was judged as %. As a result of the judgment, compression noise was removed before image enlargement.') can be provided. For example, a user may be guided as to why the image 1200 was classified as a corresponding image quality while displaying an overall score (eg, '50') for the compression quality of the image 1200 .

According to one embodiment, the processor 120 may include correction information (eg, compression quality score (or level), correction area notification object, and / or correction detailed information) may be provided in various ways according to various specified conditions. According to an embodiment, the processor 120 determines the type of the designated application and/or the depth of the menu, and based on the type of the designated application and/or the depth of the menu, the processor 120 provides correction target information and corresponding correction information. may be provided differently. For example, the processor 120 determines which target information (eg, resolution, brightness, color, and/or sharpness) is to be provided in detail according to the type of the designated application and/or the depth of the menu, Correction information may be provided in a different manner according to the determination result.

According to an embodiment, when the processor 120 executes a remaster function for a selected image in a first application (eg, a gallery application), the depth of the menu corresponds to the minimum first depth (eg, about 1 depth) , correct all target information (eg, resolution, brightness, color tone, and/or sharpness), and provide correction information (eg, correction information based on all target information) for the correction result. For example, the processor 120 may provide correction information (eg, compression quality score, correction area) for how much resolution, brightness, color tone, and/or sharpness have been corrected in each part corresponding to the correction information in the image. notification object and/or correction detailed information) may be provided.

According to another embodiment, when the processor 120 executes the remastering function for the selected image in the second application (eg, an image editing application), the second depth of the menu is greater than the first depth (eg, about 2 to about 3 depth), and corrects for some of the target information (eg, brightness, color, sharpness) specified among all target information (eg, resolution, brightness, color tone, and/or sharpness), and determines the result of the correction. Correction information (eg, correction information based on some target information) may be provided. For example, the processor 120 may provide correction information (e.g., compression quality score, correction area notification object) for how much brightness, color, and/or sharpness are corrected in each part corresponding to the correction information in the image. and/or calibration details).

According to another embodiment, when the processor 120 executes a remaster function for a selected image in a third application (eg, a web page application), the third depth of the menu is greater than the second depth (eg, about 3 to about 4 depth), corrects for the specified minimum target information (eg, brightness, color) among all target information (eg, resolution, brightness, color tone, and / or sharpness), and provides information about the correction result. Correction information (eg, correction information based on minimum target information) may be provided. For example, the processor 120 may provide correction information (e.g., compression quality score, correction area notification object and/or calibration details).

According to one embodiment, when the processor 120 provides detailed information 1250 on image correction, as illustrated in FIG. 12, the display position of the image 1200 is changed and a separate image is displayed based on screen division. Detailed information 1250 may be provided through a window. According to another embodiment, the processor 120 may provide detailed information 1250 through a pop-up window or overlay on the image 1200 while the display position of the image 1200 is fixed.

According to the present disclosure, when the electronic device 101 provides an image through a designated application (eg, a gallery application), AI automatically analyzes the image and, for example, automatically corrects a low-quality image to a high-quality image. may be provided (e.g., remaster function provided). According to an embodiment, when providing the remaster function, the electronic device 101 may provide detailed information (or explanation) on why the image has been improved. Through this, the user can provide a more detailed guide on why the quality of the image has been evaluated (or image quality has been improved). For example, the electronic device 101 determines whether a given image has a specific degradation due to which part of the image by explicitly displaying the score of the corresponding image and providing a score of a part with high reliability. can be provided to the user. For example, the electronic device 101 intuitively provides a notification object 1230 on an image, a score of the image, and detailed information 1250 including notification information on determination of degradation in the image, so that the user can improve the image. We can provide a more detailed understanding of why.

An operating method performed by the electronic device 101 according to an embodiment of the present disclosure includes an operation of displaying an image through a display module of the electronic device, an operation of extracting a plurality of designated blocks from the image in a designated manner, and the plurality of blocks. Operation of estimating reliability for each block for the number of blocks, based on the reliability estimation, identifying a first block corresponding to an outlier to be excluded from quality prediction and a second block capable of quality prediction among the plurality of blocks An operation of excluding the first block among the plurality of blocks from a quality prediction target, and an operation of classifying the compression quality of the image using second blocks other than the first block among the plurality of blocks. can include

According to an embodiment, the operation of classifying the compression quality may include selecting at least one first block corresponding to an outlier equal to or less than a specified threshold based on block-by-block reliability estimation for the plurality of extracted blocks. estimating, excluding the at least one first block from an average operation for compression quality classification, performing an average operation for compression quality classification based on at least one second block exceeding a specified threshold; An operation of classifying a result of the averaging operation into a compression quality for the image may be included.

According to an embodiment, the operation method performed by the electronic device includes an operation of storing a plurality of denoising models previously learned for each of various compression qualities, and corresponding to a compression quality classified for the image among the plurality of denoising models. An operation of selecting a denoising model that has been learned to be denoising, and an operation of processing image quality improvement for the image based on the selected denoising model.

According to an embodiment, the operating method performed by the electronic device may include receiving a user input requesting information related to image quality of the image, and correction information for the image based on receiving the user input. It may include an operation of controlling the display module to display based on the image.

According to an embodiment, the operation method performed by the electronic device includes an operation of identifying at least one block having a compression quality corresponding to the representative quality of the image, based on receiving the user input, in the image , An operation of providing a designated notification object based on a part corresponding to the identified block, and an operation of displaying detailed information including a description of the overall score and classification for the image correction together with the notification object. can

According to an embodiment, the operating method performed by the electronic device may include receiving a user input for compressing the image, and based on receiving the user input, artifacts generated during the first compression of the image. Determining a second compression quality for the second compression based on the first compression quality related to the first compression, and performing a second compression on an image based on the second compression quality. can include

Various embodiments of the present disclosure disclosed in the present specification and drawings are only presented as specific examples to easily explain the technical content of the present disclosure and help understanding of the present disclosure, and are not intended to limit the scope of the present disclosure. Therefore, the scope of the present disclosure should be construed as including all changes or modified forms derived based on the technical spirit of the present disclosure in addition to the embodiments disclosed herein.

101: electronic device
120: processor
130: memory
160: display module
190: communication module
210: image processing module
220: sample patch extraction module
230: compression quality classification module
240: reliability estimation module
250: quality prediction module
260: denoising model
270: model selection module
280: score calculation module

Claims (20)

In electronic devices,
display module;
Memory; and
a processor operatively coupled with the display module and the memory, the processor comprising:
displaying an image through the display module;
Extracting a plurality of designated blocks from the image in a designated manner;
Estimating reliability for each block with respect to the plurality of blocks;
Based on the reliability estimation, among the plurality of blocks, a first block corresponding to an outlier to be excluded from quality prediction and a second block from which quality prediction is possible are identified;
Excluding the first block among the plurality of blocks from the quality prediction target, and
An electronic device configured to classify the compression quality of the image using second blocks other than the first block among the plurality of blocks.
The method of claim 1, wherein the image,
An electronic device characterized in that it is a compressed image compressed with a specified compression quality.
The method of claim 1, wherein the processor,
Estimating at least one first block corresponding to an outlier of a predetermined threshold value or less based on block-by-block reliability estimation for the plurality of extracted blocks;
Excluding the at least one first block from an average operation for classification of compression quality,
Performing an average operation for classification of compression quality based on at least one second block exceeding a specified threshold;
An electronic device configured to classify a result of averaging operation into a compression quality for the image.
The method of claim 1, wherein the processor,
Selecting a denoising model trained to correspond to the classified compression quality for the image;
Process quality improvement for the image based on the selected noise removal model;
An electronic device configured to process enhanced images based on specified actions.
The method of claim 4, wherein the processor,
Storing a plurality of pre-learned denoising models for various compression qualities in the memory;
An electronic device configured to select a denoising model learned to correspond to the classification of the compression quality of the image from among the plurality of denoising models.
The method of claim 4, wherein the processor,
An electronic device configured to process an improved image through at least one designated operation of displaying the improved image through the display module, storing the image in the memory, or transmitting the image to the outside.
The method of claim 1, wherein the processor,
Receive user input requesting information related to image quality of the image;
An electronic device configured to control the display module 160 to display correction information for the image based on the image, based on receiving the user input.
The method of claim 7, wherein the processor,
Based on receiving the user input, identify at least one block having a compression quality corresponding to a representative quality of the image;
An electronic device configured to provide specified notification objects and detailed information on image correction based on the identified block.
The method of claim 8, wherein the processor,
identify a high confidence block from the image;
measure the quality and reliability of identified blocks;
Comparing the compression quality of each block according to the measured quality and reliability with the representative quality of the image,
Clustering blocks having a compression quality corresponding to the representative quality among the compression qualities of each block and having a relatively high reliability compared to other blocks,
Providing the specified notification object based on a part corresponding to the clustered block in the image;
An electronic device configured to control the display module to display detailed information including an overall score for image correction and a description of classification, together with the notification object.
The method of claim 1, wherein the processor,
An electronic device configured to, in the second compression of the image, remove artifacts generated in the first compression of the image, and then perform the second compression.
The method of claim 10, wherein the processor,
Receiving a user input for compressing the image;
Based on receiving the user input, based on a noise removal model corresponding to a first compression quality of the first compression of the image, removing artifacts generated during the first compression,
determine a second compression quality for the second compression based on the first compression quality associated with the first compression;
An electronic device configured to perform a second compression on an image based on the second compression quality.
The method of claim 11, wherein the processor,
An electronic device configured to determine the second compression quality corresponding to the first compression quality.
The method of claim 1, wherein the processor,
A quality prediction module and a reliability estimation module for classifying the compression quality,
An electronic device configured to learn the reliability estimation module together while learning the quality prediction module.
The method of claim 2, wherein the processor,
An electronic device configured to classify the compression quality of an image using a learning model learned using an artificial intelligence algorithm.
In the operating method of the electronic device,
displaying an image through a display module of an electronic device;
extracting a plurality of designated blocks from the image in a designated manner;
estimating reliability for each block with respect to the plurality of blocks;
based on the reliability estimation, identifying a first block corresponding to an outlier that is to be excluded from quality prediction and a second block from which quality prediction is possible, among the plurality of blocks;
excluding the first block among the plurality of blocks from a quality prediction target; and
and classifying a compression quality of the image using second blocks other than the first block among the plurality of blocks.
16. The method of claim 15, wherein the classification of the compression quality comprises:
estimating at least one first block corresponding to an outlier equal to or less than a specified threshold based on block-by-block reliability estimation for the plurality of extracted blocks;
Excluding the at least one first block from an average operation for classification of compression quality;
An operation of performing an average operation for classification of compression quality based on at least one second block exceeding a specified threshold;
and classifying a result of the averaging operation as a compression quality for the image.
According to claim 15,
An operation of storing a plurality of pre-trained denoising models for various compression qualities;
Selecting a denoising model learned to correspond to the classified compression quality of the image from among the plurality of denoising models;
and processing image quality enhancement of the image based on the selected noise removal model.
According to claim 15,
Receiving a user input requesting information related to image quality of the image;
and controlling a display module to display correction information for the image based on the image, based on receiving the user input.
According to claim 18,
Based on receiving the user input, identifying at least one block having a compression quality corresponding to the representative quality of the image;
In the image, an operation of providing a specified notification object based on a part corresponding to the identified block;
and displaying, together with the notification object, detailed information including an overall score for the image correction and a description of the classification.
According to claim 15,
Receiving a user input for compressing the image;
Based on receiving the user input, removing artifacts generated during the first compression of the image;
determining a second compression quality for the second compression based on the first compression quality related to the first compression;
and performing a second compression on the image based on the second compression quality.

Images