KR20240031246A - Selectively increases depth of field in scenes with multiple regions of interest - Google Patents

Abstract

The present disclosure provides multi-frame depth of field (ROI) coverage of background regions of interest (ROI), such as background planes that may be blurred due to the large aperture size or other characteristics of the camera used to capture the image frame. Systems, devices, methods, and computer-readable media supporting multi-frame depth-of-field (MF-DOF) are provided. Processing may include the use of two image frames acquired at two different focus points corresponding to multiple ROIs within the image frame. The corrected image frame may be determined by deblurring one or more ROIs of the first image frame using an AI-based model and/or local gradient information. MF-DOF is the ability to selectively increase the depth of field (DOF) of an image, without causing background blur that may be required for photography or reducing the aperture (and subsequently the amount of light available for photography). May provide focused capture of areas of interest.

Description

Selectively increases depth of field in scenes with multiple regions of interest

Cross-reference to related applications

This application is related to U.S. Patent Application No. 17/645,688, filed December 22, 2021, and entitled “SELECTIVELY INCREASING DEPTH-OF-FIELD IN SCENES WITH MULTIPLE REGIONS OF INTEREST,” and filed on July 6, 2021. Claims the benefit of U.S. Provisional Patent Application No. 63/218,804, filed and entitled "SELECTIVELY INCREASING DEPTH-OF-FIELD IN SCENES WITH MULTIPLE REGIONS OF INTEREST," both of which are incorporated by reference in their entirety. are expressly incorporated into the specification.

Technology field

Aspects of the present disclosure generally relate to image signal processing. Some features may enable and provide improved communication, including deblurring operations that may compensate for image characteristics such as blur introduced by large aperture sizes.

Image capture devices are devices capable of capturing one or more digital images, whether still images for photos or sequences of images for videos. Capture devices can be integrated into a wide variety of devices. By way of example, image capture devices may include standalone digital cameras or camera-equipped wireless communication devices such as digital video camcorders, mobile phones, cellular or satellite radio phones, handsets, personal digital assistants (PDAs), panels or It may include tablets, gaming devices, computer devices such as webcams, video surveillance cameras, or other devices with digital imaging or video capabilities.

Image processing techniques can be used to improve the quality of photos and videos captured by image capture devices. For example, algorithms can change the appearance of colors in a photograph, increase the range or dynamic range of colors in a photograph, increase detail in a photograph, reduce noise in a photograph, and Blur can be reduced, which can improve the quality of the image the user sees, among many other processing techniques.

Some aspects of the disclosure are summarized below to provide a basic understanding of the techniques discussed. This summary is not a comprehensive overview of all considered features of the disclosure, and is not intended to identify key or critical elements of all aspects of the disclosure or to delineate the scope of any or all aspects of the disclosure. The sole purpose of this summary is to present some concepts of one or more aspects of the disclosure in summary form as a prelude to the more detailed description that is presented later.

Embodiments of the image processing technique described herein may produce a first image based on the characteristics of the second image frame without combining (e.g., merging, fusing, or blending) the data of the first and second image frames. The appearance of the frame can be improved. This processing eliminates the possibility of adding artifacts to the first image frame resulting from merging of data. For example, discontinuities and shifts in objects between image frames can cause ghosting lines or broken lines when a first image frame is merged with a second image frame. An operation on a first image frame may be performed based on an analysis of the second image frame, including analysis comparing corresponding regions of interest between the first and second image frames.

Image processing techniques according to embodiments herein may be beneficial for increasing the apparent depth of focus of an image. Limitations inherent in image capture devices result in parts of the image frame being blurrier than others due to the camera's limited ability to focus. For example, an image may have the side closer to the camera in the foreground being in focus while the side farther from the camera in the background is out of focus. The user may want both sides in the photo to be in-focus. Image processing techniques described herein include multi-frame depth of field to improve the appearance of a photograph by reducing blur in out-of-focus areas using a second image frame captured at a different focus distance than the first image frame. multi-frame depth-of-field (MF-DOF) processing can be performed.

In one aspect of the disclosure, a method for image processing includes receiving a first image frame representing a captured scene at a first focus distance; Receiving a second image frame representing a scene captured at a second focus distance different from the first focus distance; determining a corresponding set of regions of interest including a first region of interest in a first image frame and a second region of interest in a second image frame; determining a value based on a comparison of the first region of interest and the second region of interest; and determining a corrected first image frame based on the first image frame and the value.

In a further aspect of the disclosure, an apparatus includes at least one processor, and a memory coupled to the at least one processor. The at least one processor is configured to perform operations, the operations comprising: receiving a second image frame representing a scene captured at a second focus distance that is different from the first focus distance; determining a corresponding set of regions of interest including a first region of interest in a first image frame and a second region of interest in a second image frame; determining a value based on a comparison of the first region of interest and the second region of interest; and determining a corrected first image frame based on the first image frame and the value.

In a further aspect of the disclosure, an apparatus includes means for receiving a first image frame representing a scene captured at a first focus distance; means for receiving a second image frame representing a scene captured at a second focus distance different from the first focus distance; means for determining a corresponding set of regions of interest including a first region of interest in a first image frame and a second region of interest in a second image frame; means for determining a value based on a comparison of the first region of interest and the second region of interest; and means for determining a corrected first image frame based on the first image frame and the value.

In a further aspect of the disclosure, a non-transitory computer-readable medium stores instructions that, when executed by a processor, cause the processor to perform operations. The operations may include receiving a first image frame representing a scene captured at a first focus distance; Receiving a second image frame representing a scene captured at a second focus distance different from the first focus distance; determining a corresponding set of regions of interest including a first region of interest in a first image frame and a second region of interest in a second image frame; determining a value based on a comparison of the first region of interest and the second region of interest; and determining a corrected first image frame based on the first image frame and the value.

Image capture devices, devices capable of capturing one or more digital images, whether still image photographs or sequences of images for videos, can be integrated into a wide variety of devices. By way of example, image capture devices may include stand-alone digital cameras or camera-equipped wireless communication devices such as digital video camcorders, mobile phones, cellular or satellite radio phones, handsets, personal digital assistants (PDAs), panels or It may include tablets, gaming devices, computer devices such as webcams, video surveillance cameras, or other devices with digital imaging or video capabilities.

In general, this disclosure describes image processing techniques involving digital cameras with image sensors and image signal processors (ISPs). The ISP may be configured to control the capture of image frames from one or more image sensors and to process one or more image frames from one or more image sensors to generate a view of the scene in a calibrated image frame. A calibrated image frame may be part of a sequence of image frames forming a video sequence. The video sequence may include other image frames received from the image sensor or other image sensors and/or other calibrated image frames based on input from the image sensor or other image sensors. In some embodiments, processing of one or more image frames may be performed within the image sensor, such as in a binning module. Image processing techniques described in embodiments disclosed herein may include circuitry, such as a binning module, within an image sensor, within an image signal processor (ISP), within an application processor (AP), or a combination or two or all of these components. It can be performed by .

In one example, an image signal processor may receive instructions from an image capture device to capture a sequence of image frames in response to loading software, such as a camera application, to generate a preview display. An image signal processor may be configured to generate a single stream of output frames based on image frames received from one or more image sensors. A single flow of output frames may include raw image data from an image sensor, binned image data from an image sensor, or corrected image frames processed by one or more algorithms within an image signal processor, such as a binning module. there is. For example, an image frame acquired from an image sensor, which may have performed some processing on the data before being output to an image signal processor, can be imaged to perform one or more of the following: tone mapping, portrait lighting, contrast enhancement, gamma correction, etc. The image signal processor may be processed by processing the image frame through an image post-processing engine (IPE) and/or other image processing circuitry.

After the output frame representing the scene is determined by the image signal processor using image correction, such as binning, as described in various embodiments herein, the output frame may be displayed on the device display as a single still image and/or as part of a video sequence. may be displayed on, stored on a storage device as a picture or video sequence, transmitted over a network, and/or printed on an output medium. For example, an image signal processor obtains input frames of image data (e.g., pixel values) from different image sensors and then captures corresponding output frames of image data (e.g., preview display frames, still image capture). , frames for video, frames for object tracking, etc.). In other examples, an image signal processor may output frames of image data for further processing, e.g., 3A parameter synchronization (e.g., autofocus (AF), automatic white balance (AWB), and automatic exposure control (AEC)). Frames can be output to various output devices and/or camera modules for creating video files, organizing frames for display, organizing frames for storage, sending frames over a network connection, etc. You can. That is, the image signal processor can acquire incoming frames from one or more image sensors each coupled to one or more camera lenses, and then generate a flow of output frames and output them to various output destinations. You can.

In some aspects, the corrected image frame combines aspects of image correction of the present disclosure with other computational photography techniques, such as high dynamic range (HDR) photography or multi-frame noise reduction (MFNR). It can be created by combining. With HDR photography, the first image frame and the second image frame can have different exposure times, different apertures, different lenses, and/or improved dynamic range of the fused image when the two image frames are combined. It is captured using other characteristics that are present. In some aspects, the method includes the first image frame and the second image frame being captured using the same or different exposure times and producing a corrected first image frame with reduced noise compared to the captured first image frame. This can be performed to photograph the MFNRs that are fused to produce.

In some aspects, the device includes a processor (e.g., an application processor) that includes specific functionality for processing and/or camera controls to enable or disable the binning module or otherwise control aspects of image correction. ) or may include an image signal processor. The methods and techniques described herein may be performed entirely by an image signal processor or processor, or the various operations may be split between the image signal processor and the processor, and in some aspects to additional processors. there is.

The device may include one, two, or more image sensors, such as including a first image sensor. When multiple image sensors are present, the first image sensor may have a larger field of view (FOV) than the second image sensor, or the first image sensor may have a different sensitivity or a different dynamic range than the second image sensor. You can have it. In one example, the first image sensor may be a wide-angle image sensor, and the second image sensor may be a telephoto image sensor. In another example, the first sensor is configured to acquire an image through a first lens having a first optical axis, and the second sensor is configured to acquire an image through a second lens having a second optical axis different from the first optical axis. It is composed. Additionally or alternatively, the first lens may have a first magnification and the second lens may have a second magnification that is different from the first magnification. This configuration may appear as a lens cluster on a mobile device, for example, when multiple image sensors and associated lenses are located at offset locations on the front or back of the mobile device. Additional image sensors with larger, smaller, or identical fields of view may be included. Image correction techniques described herein may be applied to image frames captured from any of the image sensors in a multi-sensor device.

In a further aspect of the disclosure, a device configured for image processing and/or image capture is disclosed. The device includes means for capturing image frames. The device includes image sensors (including charge-coupled devices (CCDs), Bayer-filter sensors, infrared (IR) detectors, ultraviolet (UV) detectors, and complementary metal oxide semiconductor (CMOS) sensors), flight It further includes one or more means for capturing data representative of the scene, such as time detectors. The device may further include one or more means (including simple lenses, compound lenses, spherical lenses, and aspherical lenses) for accumulating and/or focusing light rays onto one or more image sensors. These components may be controlled to capture first and/or second image frames input to the image processing techniques described herein.

Other aspects, features, and implementations will become apparent to those skilled in the art upon review of the following description of specific example embodiments in conjunction with the accompanying drawings. Although features may be discussed with respect to specific aspects and figures below, various aspects may include one or more of the advantageous features discussed herein. In other words, although one or more aspects may be discussed as having certain advantageous features, one or more of such features may also be used in accordance with various aspects. In a similar manner, although example aspects may be discussed below as device, system, or method aspects, example aspects may be implemented in a variety of devices, systems, and methods.

The method may be implemented on a computer-readable medium as computer program code containing instructions that cause a processor to perform the method steps. In some embodiments, the processor includes: a first network adapter configured to transmit data, such as images or videos, as recorded or streaming data over a first of a plurality of network connections; and a processor coupled to a first network adapter and memory. The processor may cause transmission of the corrected image frames described herein via a wireless communication network, such as a 5G NR communication network.

The foregoing has outlined rather broadly the features and technical advantages of examples according to the present disclosure in order that the detailed description that follows may be better understood. Additional features and advantages will be described later. The disclosed concepts and specific examples can be readily utilized as a basis for modifying or designing other structures to carry out the same purposes of the disclosure. Such equivalent structures do not depart from the scope of the appended claims. BRIEF DESCRIPTION OF THE DRAWINGS The characteristics of the concepts disclosed herein, i.e., both their construction and method of operation, along with their associated advantages, will be better understood from the following description when considered in conjunction with the accompanying drawings. Each of the drawings is provided for purposes of illustration and description and not as a definition of the limitations of the claims.

Although aspects and implementations are described herein as examples of some examples, those skilled in the art will understand that additional implementations and use cases may be implemented in many different devices and scenarios. The innovations described herein can be implemented across many different platform types, devices, systems, shapes, sizes, and packaging arrangements. For example, aspects and/or uses may apply to integrated chip implementations and other non-module-component based devices (e.g., end user devices, vehicles, communication devices, computing devices, industrial equipment, retail /purchase devices, medical devices, artificial intelligence (AI) enabled devices, etc.). Although some examples may or may not be specifically related to use cases or applications, broad applicability of the described innovations may result. Implementations range from chip-level or modular components to non-modular, non-chip-level implementations and can be integrated, distributed, or original equipment manufacturer (OEM), including one or more aspects of the innovations further described. ) can range from devices or systems. In some practical settings, devices incorporating the described aspects and features may also necessarily include additional components and features for implementing and practicing the claimed and described aspects. For example, the transmission and reception of wireless signals involves a number of components for analog and digital purposes (e.g., antennas, radio frequency (RF)-chains, power amplifiers, modulators, buffers, processor(s) , hardware components including interleavers, adders/summers, etc.). It is intended that the innovations described herein can be implemented in a wide range of devices, chip-level components, systems, distributed devices, end-user devices, etc. of various sizes, shapes, and configurations.

A further understanding of the properties and advantages of the present disclosure can be realized with reference to the drawings below. In the accompanying drawings, similar components or features may have the same reference label. Additionally, various components of the same type may be distinguished by a dash symbol following the reference label and a second label that distinguishes similar components. If only the first reference label is used in the specification, the description is applicable to any one of the similar components having the same first reference label, regardless of the second reference label.
1 shows a block diagram of an example device for performing image capture from one or more image sensors.
Figure 2 is an example showing a scene where two sides are at different focus distances.
3 is a block diagram illustrating a method for processing an image with two or more ROIs, in accordance with one or more aspects.
4 is a block diagram illustrating processing of multiple image frames to increase depth of focus in an image frame, in accordance with one or more aspects.
5 is a flow diagram illustrating a method for processing an image with two or more ROIs, in accordance with one or more aspects.
6 is a flow diagram illustrating a method for processing an image with two or more ROIs using a contrast operation, in accordance with one or more aspects.
7 is a block diagram illustrating a method for processing an image with two or more ROIs with AI-based deblurring, in accordance with one or more aspects.
Like reference numerals and designations in the various drawings indicate like elements.

The detailed description set forth below in conjunction with the accompanying drawings is intended as an illustration of various configurations and is not intended to limit the scope of the disclosure. Rather, the detailed description includes specific details for the purpose of providing a thorough understanding of the claimed subject matter. It will be apparent to those skilled in the art that these specific details are not required in every case, and in some cases, for clarity of presentation, well-known structures and components are shown in block diagram form.

The present disclosure provides multi-frame depth of field (ROI) coverage of background regions of interest (ROI), such as background planes that may be blurred due to the large aperture size or other characteristics of the camera used to capture the image frame. Provided are systems, devices, methods, and computer-readable media supporting MF-DOF). Processing may include the use of two image frames acquired at two different focus points corresponding to multiple ROIs within the image frame. The corrected image frame may be one or more of the first image frame using an AI-based model and/or local gradient information, for example, using a CV-based deblurring method by local gradient information. It can be determined by deblurring the ROIs. MF-DOF is the ability to selectively increase the depth of field (DOF) of an image, without causing background blur or reducing the aperture (and subsequently the amount of light available for photography) that may be required for photography. May provide focused capture of areas of interest.

Certain implementations of the subject matter described in this disclosure can be implemented to realize one or more of the following potential advantages or advantages. In some aspects, the present disclosure reduces the computational complexity involved in fusing multiple image frames, e.g., reduces artifacts that can be created when fusing multiple image frames (e.g., a halo effect), e.g. , provides techniques for image signal processing to reduce ghosting effects). An improved image frame can still be obtained through image processing techniques according to embodiments of the disclosure herein by deblurring background ROIs using multi-frame processing.

An exemplary device, such as a smartphone, for capturing image frames using one or more image sensors may have one, two on the back (e.g., the opposite side of the user's display) or the front (e.g., the same side as the user's display) of the device. , may include a configuration of three, four or more cameras. Devices with multiple image sensors may be equipped with one or more image signal processors (ISPs), computer vision processors (CVPs) (e.g., AI engines), or for processing images captured by the image sensors. Includes other suitable circuitry. One or more image signal processors may store the processed image frames in memory and/or a processor (e.g., an application processor, image front end (IFE)) for further processing, such as encoding, storage, transmission, or other manipulation. , an image post-processing engine (IPE), or other suitable processing circuitry).

As used herein, an image sensor refers to the image sensor itself and storage in memory, whether a short-term buffer or long-term non-volatile memory, or any device used to generate image frames for processing by an image signal processor or other logic circuitry. It may refer to any other components coupled to the image sensor. For example, the image sensor may include other components of the camera, including a shutter, buffer, or other readout circuitry to access individual pixels of the image sensor. An image sensor may also refer to an analog front end, or other circuitry for converting analog signals into digital representations of an image frame that are provided to digital circuitry coupled to the image sensor.

In the following description, numerous specific details are set forth, such as examples of specific components, circuits, and processors, to provide a thorough understanding of the disclosure. As used herein, the term “coupled” means connected or directly connected through one or more intervening components or circuits. Additionally, in the following description, and for purposes of explanation, certain terminology is developed to provide a thorough understanding of the present disclosure. However, it will be apparent to those skilled in the art that these specific details may not be required to practice the teachings disclosed herein. In other instances, well-known circuits and devices are shown in block diagram form to avoid obscuring the teachings of the disclosure.

Certain portions of the detailed description that follows are presented in terms of procedures, logic blocks, processing, and other symbolic representations of operations on data bits within a computer memory. In this disclosure, a procedure, logic block, process, etc. is recognized as being a self-consistent sequence of steps or instructions that lead to a desired result. The steps are those requiring physical manipulations of physical quantities. Typically, but not necessarily, these quantities take the form of electrical or magnetic signals that can be stored, transmitted, combined, compared and otherwise manipulated in a computer system.

In the drawings, a single block may be described as performing a function or functions. The function or functions performed by such a block may be performed in a single component or across multiple components and/or may be performed using hardware, software, or a combination of hardware and software. To clearly illustrate this interchangeability of hardware and software, various illustrative components, blocks, modules, circuits and steps are described below generally in terms of their functionality. Whether this functionality is implemented in hardware or software depends on the specific application and design constraints imposed on the overall system. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present disclosure. Additionally, example devices may include components other than those shown, including well-known components such as a processor, memory, and the like.

Aspects of the present disclosure are applicable to any suitable electronic device that includes or is coupled to two or more image sensors capable of capturing image frames (or “frames”). Additionally, aspects of the disclosure may be implemented in devices having or coupled to image sensors of the same or different capabilities and characteristics (eg, resolution, shutter speed, sensor type, etc.). Additionally, aspects of the present disclosure may include processing devices capable of retrieving stored images for processing, including processing devices residing in cloud computing systems, whether the device includes or is coupled to image sensors. , can be implemented in devices for processing image frames.

As is clear from the following discussion and unless specifically stated otherwise, throughout this application the terms “accessing,” “receiving,” “transmitting,” “using,” and “selecting.” , “determining”, “normalizing”, “multiplying”, “averaging”, “monitoring”, “comparing”, “applying”, “updating”, “ Discussions utilizing terms such as “measuring,” “deducing,” “settling,” “generating,” etc. are expressed in terms of physical (electronic) quantities within the registers and memories of a computer system. Actions and processes of a computer system, or similar electronic computing device, that manipulate and convert data into other data similarly represented as physical quantities within the registers or memories of the computer system or other such information storage, transmission, or display device. refers to them.

The terms “device” and “apparatus” are not limited to one or a specific number of physical objects (eg, one smartphone, one camera controller, one processing system, etc.). As used herein, a device may be any electronic device having one or more parts capable of implementing at least some portions of the disclosure. Although the following description and examples use the term “device” to describe various aspects of the disclosure, the term “device” is not limited to a particular configuration, type, or number of objects. As used herein, apparatus may include a device or portion of a device for performing the described operations.

1 shows a block diagram of an example device 100 for performing image capture from one or more image sensors. Device 100 includes an image signal processor 112 for processing image frames from one or more image sensors, such as first image sensor 101, second image sensor 102, and depth sensor 140. or may otherwise be coupled thereto. In some implementations, device 100 also includes or is coupled to a processor 104 and a memory 106 that stores instructions 108. Device 100 may also include or be coupled to a display 114 and input/output (I/O) components 116. I/O components 116, such as a touch screen interface and/or a physical button interface, may be used to interact with the user. I/O components 116 may also include a wide area network (WAN) adapter 152, a local area network (LAN) adapter 153, and/or a personal area network. It may include network interfaces for communicating with other devices, including a PAN) adapter 154. Examples of WAN adapters 152 include 4G LTE or 5G NR wireless network adapters. An example LAN adapter 153 is an IEEE 802.11 WiFi wireless network adapter. An exemplary PAN adapter 154 is a Bluetooth wireless network adapter. Each of adapters 152, 153, and/or 154 may be coupled to an antenna, configured for primary and diversity reception, and/or configured to receive specific frequency bands. there is. Device 100 may also include or be coupled to a power supply 118 for device 100, such as a battery or components for coupling device 100 to an energy source. Device 100 may also include or be coupled to additional features or components not shown in FIG. 1 . In one example, one or more transceivers and a baseband processor may be coupled to or included in WAN adapter 152 for a wireless communication device. In a further example, an analog front end (AFE) for converting analog image frame data to digital image frame data may be coupled between the image sensors 101 and 102 and the image signal processor 112. .

The device includes a sensor hub 150 for interfacing with sensors to receive data about the movement of device 100, data about the environment around device 100, and/or data from other sensors other than cameras. It can be done or coupled to it. One example sensor other than a camera is a gyroscope, a device configured to generate motion data by measuring rotation, orientation, and/or angular velocity. Another exemplary non-camera sensor is an accelerometer, i.e., a device configured to measure acceleration, which can also be used to determine distance traveled and velocity by appropriately integrating the measured acceleration, among acceleration, velocity, and distance. One or more may be included in the generated motion data. In some aspects, a gyroscope may be coupled to a sensor hub or directly to the image signal processor 112 in an electronic image stabilization system (EIS). In another example, a sensor other than a camera may be a global positioning system (GPS) receiver. Data from sensor hub 150 may be used by image signal processor 112 to generate corrected image frames, such as by applying EIS and/or digital image stabilization (DIS).

Image signal processor 112 may receive image data from one or more cameras in the form of image frames. In one embodiment, a local bus connection couples the image signal processor 112 to the respective image sensors 101 and 102 of the first and second cameras. In another embodiment, a wired interface couples image signal processor 112 to an external image sensor. In a further embodiment, a wireless interface couples image signal processor 112 to image sensors 101 and 102.

The first camera may include a first image sensor 101 and a corresponding first lens 131. The second camera may include a second image sensor 102 and a corresponding second lens 132. Each of the lenses 131 and 132 may be controlled by an associated autofocus (AF) algorithm 133 running on the ISP 112, which adjusts the lenses 131 and 132 to correspond to a predetermined focus position. Focuses on a specific focus plane. AF algorithm 133 may be assisted by depth sensor 140 by using depth data proximate to the focus position.

The first image sensor 101 and the second image sensor 102 are configured to capture one or more image frames. Lenses 131 and 132 may include one or more apertures to receive light, one or more shutters to block light when outside the exposure window, and one or more color filter arrays to filter light outside specific frequency ranges ( Light is focused at image sensors 101 and 102, respectively, via a color filter array (CFA), one or more analog front ends for converting analog measurements to digital information, and/or other suitable components for imaging. First lens 131 and second lens 132 may have different fields of view to capture different representations of the scene. For example, the first lens 131 may be an ultra-wide-angle (UW) lens, and the second lens 132 may be a wide-angle (W) lens. The plurality of image sensors may include a combination of ultra-wide (wide field of view (FOV)), wide-angle, telephoto, and ultra-telephoto (narrow FOV) sensors. That is, each image sensor may be configured through hardware configuration and/or software settings to obtain different, but overlapping, fields of view. In one configuration, the image sensors are comprised of different lenses with different magnifications resulting in different fields of view. The sensors may be configured such that the UW sensor has a larger FOV than the W sensor, the W sensor has a larger FOV than the T sensor, and the T sensor has a larger FOV than the UT sensor. For example, a sensor configured for a wide FOV may capture fields of view ranging from 64 to 84 degrees, a sensor configured for an ultra-wide FOV may capture fields of view ranging from 100 to 140 degrees, and a sensor configured for a telephoto FOV may capture fields of view ranging from 100 to 140 degrees. can capture fields of view ranging from 10 to 30 degrees, and a sensor configured for a supertelephoto FOV can capture fields of view ranging from 1 to 8 degrees.

Image signal processor 112 processes image frames captured by image sensors 101 and 102. 1 illustrates device 100 as including two image sensors 101 and 102 coupled to image signal processor 112, any number (e.g., 1, 2, 3) may be used. , 4, 5, 6, etc.) image sensors may be coupled to the image signal processor 112. In some aspects, depth sensors, such as depth sensor 140, may be coupled to image signal processor 112, and output from the depth sensors may be processed in a manner similar to the output of image sensors 101, 102. Corrected image frames may be generated based on the image frame captured by the depth sensor 140. Depth sensor 140 may also provide a first image frame captured from one of the image sensors 101, 102, for example, foreground and image frames from sensors 101 or 102 when determining a corrected first image frame. By using depth data to segment into background areas and processing the foreground and background areas separately, it can be used to apply correction. 1 may reflect the configuration of some embodiments of the disclosed image signal processing techniques and methods, but may include any number of additional image sensors while still implementing aspects of the disclosed image signal processing techniques and methods. or image signal processors may be included in other embodiments of device 100.

In some embodiments, image signal processor 112 may receive instructions from memory, such as instructions 108 from memory 106, instructions stored in a separate memory coupled to or included in image signal processor 112. , or instructions provided by the processor 104 may be executed. Additionally, or alternatively, image signal processor 112 may include special hardware (eg, one or more integrated circuits (ICs)) configured to perform one or more operations described in this disclosure. For example, image signal processor 112 may include one or more image front ends (IFE) 135, one or more image post-processing engines 136 (IPE), one or more automatic exposure control (AEC) 134. engines, and/or one or more autofocus (AF) 133 engines. AF 133, AEC 134, IFE 135, and IPE 136 may each include application-specific circuitry, software code executed by ISP 112, and/or hardware within ISP 112. It can be implemented as a combination of and software code running on it.

In some implementations, memory 106 includes a non-transitory or non-transitory computer-readable medium that stores computer-executable instructions 108 for performing all or a portion of one or more operations described in this disclosure. can do. In some implementations, instructions 108 include a camera application (or other suitable application) to be executed by device 100 to generate images or videos. Instructions 108 may also include other applications or programs executed by device 100, such as operating systems and specific applications other than those for image or video creation. Execution of a camera application, such as by processor 104, may cause device 100 to generate images using image sensors 101 and 102 and image signal processor 112. Memory 106 may also be accessed by image signal processor 112 to store processed frames, or by processor 104 to obtain processed frames. In some embodiments, device 100 does not include memory 106. For example, device 100 may be a circuit that includes image signal processor 112, and the memory may be external to device 100. Device 100 may be coupled to an external memory and configured to access the memory to record output frames for display or long-term storage. In some embodiments, device 100 is a system that integrates image signal processor 112, processor 104, sensor hub 150, memory 106, and input/output components 116 into a single package. It is a system on chip (SoC).

In some embodiments, at least one of image signal processor 112 or processor 104 is configured to perform various operations described herein, including MF-DOF operations as described in embodiments herein. Commands can be executed. For example, execution of instructions may direct image signal processor 112 to begin or end capturing an image frame or sequence of image frames, wherein the capture is as described in embodiments herein. Includes MF-DOF operations. In some embodiments, processor 104 may include one or more general purpose processor cores 104A capable of executing scripts or instructions of one or more software programs, such as instructions 108 stored within memory 106. You can. For example, processor 104 may include one or more application processors configured to execute a camera application (or other suitable application for generating images or video) stored in memory 106.

When executing a camera application, processor 104 may be configured to instruct image signal processor 112 to perform one or more operations with reference to image sensors 101 or 102 . For example, a camera application may receive a command to initiate a video preview display in which video comprising a sequence of image frames is captured and processed from one or more image sensors 101 or 102. Image correction may be applied to one or more image frames within a sequence. Execution of instructions 108 outside the camera application by processor 104 may also cause device 100 to perform any number of functions or operations. In some implementations, processor 104 may include ICs or other hardware (ICs) in addition to the ability to execute software to cause device 100 to perform multiple functions or operations, such as those described herein. For example, it may include an artificial intelligence (AI) engine 124). In some other embodiments, device 100 does not include processor 104, such as when all of the described functionality is comprised in image signal processor 112.

In some embodiments, display 114 may include one or more suitable displays to allow user interaction and/or present items to the user, such as a preview of image frames captured by image sensors 101 and 102. Or it may include screens. In some aspects, display 114 is a touch-sensitive display. I/O components 116 may be or include any suitable mechanism, interface, or device for receiving input (such as commands) from a user and providing output to the user via display 114. . For example, I/O components 116 may include a graphical user interface (GUI), keyboard, mouse, microphone, speakers, a compressible bezel, one or more buttons (e.g., a power button), sliders, switches, etc. may (but is not limited to).

Although shown as coupled to each other via processor 104, components (e.g., processor 104, memory 106, image signal processor 112, display 114, and I/O components 116) ) may be coupled together in a variety of other arrangements, such as via one or more local buses not shown for simplicity. Although image signal processor 112 is shown as separate from processor 104, image signal processor 112 may be an application processor unit (APU), included in a system on chip (SoC), or otherwise included in processor 104. ) may be the core of the processor 104 included together. Although a device 100 is referenced in examples herein for performing aspects of the disclosure, some device components may not be shown in FIG. 1 to avoid obscuring aspects of the disclosure. Additionally, other components, numbers of components, or combinations of components may be included in a suitable device for performing aspects of the disclosure. As such, the present disclosure is not limited to the configuration of specific devices or components including device 100.

The quality of images captured by an image capture device, such as device 100, can be improved by using large aperture lenses on one or more image sensors of the image capture device. Larger aperture lenses have many advantages, including better low-light performance (which can be advantageous for indoor and night photography) and better bokeh blur (which can be advantageous for portrait photography). The advantages can be particularly advantageous in mobile devices where form factors are limited in available space for camera lenses and other equipment. However, large aperture lenses also have a short depth of focus (DOF), meaning that limited portions of the captured image are in-focus and the remainder of the image has varying degrees of blur. When a scene contains many objects at various distances, an image capture device with a large aperture lens will capture an image in which some of these objects are out-of-focus. This can be undesirable, especially when out-of-focus objects include other people. The shortcomings mentioned herein are representative only and are included to highlight problems that the inventors have identified with existing devices and sought to improve. Aspects of the devices described below may address some or all of the drawbacks as well as others known in the art. Aspects of the improved devices described below may present other benefits than those described above and may be used in other applications.

An image frame as shown in FIG. 2 may be captured by device 100. 2 is an example showing a scene where two sides are at different focus distances from device 100. Image frame 200 includes a first object 202 and a second object 204. Autofocus (AF) 133 may have focused the camera on a first object 202 in the foreground, while a second object 204 is out-of-focus in the background due to the limited depth of focus of lens 131 . The depth of focus of the image capture may be sufficiently shallow that neither the first object 202 nor the second object 204 may be in-focus in a single image frame. As described in embodiments herein, multi-frame depth-of-field (MF-DOF) processing may be performed when two or more regions of interest (ROIs) are determined to be present in an image frame. Detection of interest in the ROI may include detecting objects 202 and 204 in image frame 200. Exemplary embodiments of MF-DOF processing showing different aspects of MF-DOF processing are described with reference to FIGS. 3-7.

3 is a block diagram illustrating a method for processing an image with two or more ROIs, in accordance with one or more aspects. At block 302, a first image frame is captured, which may have an object in-focus in the foreground (e.g., near) and an object out of focus in the background (e.g., far). Additional image capture and/or image processing, such as multi-frame depth-of-field (MF-DOF) processing, may be triggered by determination of at least two regions of interest (ROIs) within the captured image frame at block 302. Identification may include object or face detection, for example, using machine learning algorithms, computer vision (CV)-based segmentation, deep learning, digital signal processing algorithms, depth estimation, or other algorithms. It can be performed through . Identification may include analysis to determine that at least one of the regions of interest is blurry or out-of-focus by measuring local contrast within the identified regions of interest. In some embodiments, MF-DOF combines two ROIs with object/face detection to determine situations when it is desirable to increase the acuity of the two faces with depth on the same image frame. Triggered when detected. In some embodiments, MF-DOF is triggered when two ROIs with at least one blurred ROI are detected. In some embodiments, MF-DOF may be triggered when three (or more alternatively, N) objects are detected by object/face detection. The types of objects and number of objects for triggering MF-DOF processing may be customized based on the configuration of the image capture device and/or user preferences. For example, a user can specify rules that specify certain conditions that, when met, can trigger MF-DOF processing. One example rule might be the detection of two faces in an outdoor scene. Another example rule could be detection of two faces separated by a distance above a set distance threshold.

When MF-DOF processing is triggered, multiple image frames may be processed to improve the representation of a scene captured by photography using an image capture device. Accordingly, after determining that the trigger condition has been met, a new first image frame may be acquired at block 306 and a second image frame may be acquired at block 308. In some embodiments, the first image frame of block 306 has the focus point at a location 320 between the foreground and background objects at a focus position that increases detail in the background objects while not causing significant blur to the foreground objects. Acquired in a set state. Position 320 may be midway between the foreground and background objects or somewhere else between the foreground and background objects, such as 20% from the foreground and background objects, but position 320 may be located halfway between the foreground and background objects. may be adjustable to trade off with detail in background objects. In some embodiments, the midpoint may be midway in the depth of field between objects 202 and 204 and/or midway in the lateral dimensions between objects 202 and 204 . In some embodiments, the first image frame at block 306 is acquired while focused on the foreground ROI and the second image frame at block 308 is acquired while focused on the background ROI. In some embodiments, the capture of a new image frame at block 306 or block 308 is skipped and the image frame at block 302 is not included in the first or second image frame at block 306 or block 308. It is reused as.

The first and second image frames from blocks 306 and 308 are capable of performing, at block 312, feature-based alignment between corresponding ROIs within the first and second image frames. It is provided in the MF-DOF processing algorithm of (310). For example, the surface features of a background surface can be determined and an alignment factor that matches the background surface features in a first image frame with corresponding features in a second image frame can be determined. The alignment factor can be expressed as a single vector corresponding to the distance difference between the first and second image frames. In some embodiments, the alignment factor may be determined as a function of the position within the image frame of block 308, allowing the alignment factor to be position dependent and the alignment difference to be expressed as a set of motion vectors. In some embodiments, the alignment factor may be determined as an average of a set of motion vectors.

An output image frame in which the improved characteristics span solely one of the first and/or second image frames may be generated based on either the first or second image frames, and an alignment factor may be used to determine can be used For example, MF-DOF processing performed on one of the image frames using an alignment factor and/or other parameters may reduce blur and/or increase detail in some of the regions of interest, thereby reducing the first or second image frames. An image frame can be created that has more “in-focus” areas than any one of the image frames alone. In some embodiments, MF-DOF processing may include applying artificial intelligence or machine learning algorithms to certain areas of interest to extend shallow depth of field.

In one embodiment, MF-DOF processing may include processing in blocks 314 and 316 to determine the output image frame 330. At block 314, a local gradient comparison is performed based on the aligned features of the ROIs. In some embodiments, local refers to a region of interest where the local gradient difference is the gradient difference within corresponding regions of interest within the first and second image frames. At block 316, a local enhancement operation is performed on the background plane based on local gradient information, such as the intensity value determined at block 314. An enhancement operation may improve image quality, for example, by improving the sharpness or sharpness in the ROI of an image frame that was not in-focus. For example, if a first image frame has a foreground surface that is in-focus and a background surface that is out-of-focus, an enhancement operation may use local gradient information to improve the sharpness or sharpness of the background surface. In some embodiments, the local enhancement may be a brightness preserving contrast enhancement to enhance detail in a background plane or background ROI. After local enhancement at block 316, output image frame 330 may appear to have both objects 202 and 204 in-focus.

In some embodiments, the gradient information in block 314 may be normalized prior to comparison, such as to normalize to a scale of 0 to 1. The comparison in block 314 may determine a local enhancement factor to be used as an input parameter for processing in block 316. The factor can be determined by comparing the gradients on both sides to a lookup table. In some embodiments, local enhancement may be controlled based on differences in gradients and human subjective evaluation, for example, by a lookup table configured to map relationships. In some embodiments, the table is displayed, e.g., on a scale of 1 to 4, where 1 represents a lower intensity enhancement operation and 4 represents a higher intensity enhancement operation (e.g., a stronger contrast operation) - block 316 ) may include an intensity value for the enhancement operation to be performed. A number of such intensity values can be determined for an image by having each intensity value associated with a portion of the image. In some embodiments, each ROI can have a constant intensity value. In some embodiments, some or all ROIs may be divided into subblocks of size NxN (eg, 2x2, 3x3, 4x4, etc.), with each subblock having a certain intensity value.

In some embodiments, a threshold may be used to assign intensity values to regions of the image. For example, the difference in gradients for first and second image frames with corresponding subblocks is compared. If the difference is less than the threshold, the contrast operation can be performed with parameter 1, if the difference is greater than the first threshold and less than the second threshold, the contrast operation can be performed with parameter 2, and the difference is greater than the second threshold. If it is less than the third threshold, the contrast operation can be performed with parameter 3. If the difference is greater than the third threshold, the contrast operation can be performed with parameter 4.

The image processing technique described in FIG. 3 improves the appearance of a first image frame based on characteristics of the second image frame without combining data from the first and second image frames. This processing eliminates the possibility of adding artifacts to the first image frame resulting from merging of data from multiple image frames. For example, discontinuities and shifts in objects between image frames can cause ghosting lines or broken lines when a first image frame is merged with a second image frame. Additionally, artifacts, such as luminance changes and different noise levels, that may be introduced during the fusion operation can be avoided. An operation on a first image frame may be performed based on an analysis of the second image frame, including analysis comparing corresponding regions of interest between the first and second image frames. Aspects of image processing techniques for correcting a first image frame without merging with data from a second image frame are described with reference to FIGS. 4 and 5 .

4 is a block diagram illustrating processing of multiple image frames to increase depth of focus in an image frame, in accordance with one or more aspects. The image signal processor 112 may receive a first image frame 410 and a second image frame 412. Two image frames 410, 412 may have a foreground object 202 and a background object 204. ISP 112 may recognize regions of interest 402 and 404 that correspond to faces on objects 202 and 204 . The first and second image frames 410, 412 may be input to the ROI analysis module 424, which analyzes (e.g., local gradient) based on the first and second image frames 410, 412. (by using comparison) to determine one or more parameters. One or more parameters may be input to an image correction module 422 that receives the first image frame 410 and performs operations on the first image frame 410 based on the one or more parameters. Accordingly, the image correction module 422 operates only on data from the first image frame 410 to determine the corrected first image frames 430 . Artifacts can be reduced. Although individual image frames are shown as input to ISP 112, ISP 112 may process a series of image frames corresponding to a video sequence and output the series of calibrated image frames as a calibrated video sequence. and where each image frame is processed in a similar manner to improve the characteristics of certain regions of interest (ROIs) within the image frames.

One example method for image signal processing by image signal processor 112 is described with reference to FIG. 5 . 5 is a flow diagram illustrating a method for processing an image with two or more ROIs, in accordance with one or more aspects. Method 500 may be implemented by another processor, such as processor 104, or other logic circuitry within an image capture device, or a combination of circuitry within an image capture device. Method 500 includes, at block 502, receiving first and second image frames at first and second focus distances, respectively. For example, ISP 112 may control the camera via autofocus (AF) module 133 to capture first and second image frames. At block 504, ISP 112 may recognize regions of interest in the first and second image frames and may match the regions of interest between the first and second image frames. At block 506, ISP 112 may determine a corrected first image frame based on analysis of the first image frame and corresponding regions of interest within the first and second image frames.

In some embodiments, the determination of block 506 performed in image correction module 422 may be dependent on the analysis performed by ROI analysis module 424 to determine local gradient differences between corresponding regions of interest. It may include a contrast operation performed based on the contrast operation. An exemplary method for such an embodiment is described with reference to FIG. 6 .

6 is a flow diagram illustrating a method for processing an image with two or more ROIs, in accordance with one or more aspects. Method 600 begins at block 602 with receiving a first image frame representing a scene captured at a first focus distance. At block 604, whether there are two regions of interest (ROIs) (e.g., faces) present within the first image frame is determined, such as by determining whether there are two regions of interest (ROIs) (e.g., faces) present within the first image frame. . If there are no more than two ROIs at block 604, the method 600 continues at block 614, which outputs a corrected first image frame based on the first image frame. The original captured first image with the blurred background is processed (e.g. cropped, tone mapped, etc.) by an image signal processor and output into a corrected first image frame that is displayed on a preview or saved as a photo. decide. In some embodiments, the original first image frame is output without any modification at block 614.

If two or more ROIs are identified at block 604, the method 600 continues at block 606 with a second focus distance representing the same scene as the first image frame but at a different focus distance than the first focus distance of the first image frame. Receive the captured second image frame. The second image frame may be configured, for example, by seeding the AF algorithm with a depth value corresponding to the ROI obtained when capturing the first image frame and/or by seeding the autofocus (AF) algorithm with a position to focus on. It can be captured by doing. In some embodiments, block 606 may include receiving a third image frame for use in place of the first image frame received at block 602.

At block 608, an alignment correction is determined to match features of the first image frame to the second image frame. At block 610, local gradient differences are determined for corresponding portions of the first and second image frames based on the determined alignment correction. At block 612, an enhancement operation is performed on the first image frame based on the local gradient difference of block 610 to produce a corrected first image frame.

In some embodiments, the enhancement operation of block 612 may be based on an artificial intelligence (AI) algorithm, such as a machine learning (ML) algorithm. 7 is a block diagram illustrating a method for processing an image with two or more ROIs with AI-based deblurring, in accordance with one or more aspects. A multi-frame depth-of-field (MF-DOF) operation 710 may perform AI-based image processing 714 using face feature-based alignment in block 712, similar to block 312 in FIG. 3. The alignment correction determined at block 712 is used by the AI-based deblurring algorithm 716 to apply an enhancement operation to the first image frame 410, e.g., to a region of interest within the first image frame. A first image frame 720 may be created. The corrected first image frame 720 is captured as part of a series of corrected first image frames 430 on a preview display within the camera application of the image capture device and/or is captured as a photo for storage or transmission by the image capture device. It is captured as.

The AI-based deblurring algorithm combines ground truth information (e.g., a focused background person/face/ROI) and a blurred image (e.g., a first image frame with a shallow depth of focus in which the background part/face/ROI is blurred). Can be trained based on In some embodiments, AI-based deblurring may be performed on a GPU, DSP, ASIC, or other logic circuitry rather than the CPU, which frees CPU computational resources for other tasks. In some embodiments, AI-based deblurring may have less latency and reduced artifacts compared to the local gradient comparison and enhancement methods of FIG. 3. Training of the parameters of the AI-based algorithm can be based on the Adam optimizer with loss function MSE:

Here, K is the epoch, N is the patch number, a is the ground truth PSF, a- is the estimated PSF at the n-th image within the patch and the k-th epoch. Training can be performed offline using target scores assigned by humans to pairs of blurred and non-blurred ROIs, and the training model can be configured as a static model on the image capture device.

In some embodiments, the AI-based deblurring algorithm may use a collected dataset of corresponding clear and blurry ROIs from image frames, using a model trained by a ResNet algorithm, such as the ResNet-34 algorithm. We can train the model by adopting ResNet-34 to consider the degradation problem of connecting blurry and sharp surface ROIs. Although ResNet-34 is provided as an example of a machine learning algorithm, other algorithms can be trained and used alternatively or in combination with ResNet-34.

As described herein, machine learning models may, in some embodiments, include logistic regression techniques, linear discriminant analysis, linear regression analysis, artificial neural networks, machine learning classifier algorithms, or classification/regression trees. You can. In some aspects, machine learning may include one or more artificial neural networks, which determine the relationship between parameters, e.g., between two image frames corresponding to in-focus and blurred representations of a face or other object. may include an interconnected group of artificial neurons (e.g., neuron models) for modeling relationships. In some aspects, machine learning may include one or more convolutional neural networks, which are a type of feed-forward artificial neural network. Convolutional neural networks may include sets of neurons, each with a receptive field, that collectively tile the input space. In some aspects, machine learning may include one or more deep learning architectures, such as deep belief networks and deep convolutional networks, where the output of a first layer of neurons is transmitted to a second layer of neurons. These are layered neural network architectures, where the output of the second layer of neurons becomes the input to the third layer of neurons, and so on. Deep neural networks can be trained to recognize hierarchies of features. In various aspects, machine learning systems may utilize variations of Naive Bayes predictive modeling analysis, learning vector quantization, or Adaboost or probabilistic methods to iteratively update weights to train a machine learning classifier. Employing an implementation of boosting algorithms, such as gradient boosting systems, to determine the relationship between influential attributes, such as local gradient level, and the in-focus or blurred aspect of an object or face within an image frame and/or such influences. properties can determine the extent to which they affect the final output pixels for correcting an image frame or the results of such a system.

Processing of image frames to deblur background ROIs, such as background planes, can overcome the disadvantages of using larger lenses and larger apertures (which increase blurring of the background), increase light influx in low-light conditions, and It has more natural bokeh performance than other computational techniques. Specifically, in dual-face in-focus embodiments, sharpness can be improved on the back side, the front side can also be made sharper, and/or hair on the back side (face) can be compared to the hair on the background side. It can be enhanced to improve its natural appearance.

In some embodiments, the corrected first image frame is determined without merging data in the first image frame with data in the second image frame. This eliminates the possibility of artifacts in the corrected first image frame resulting from merging two representations from two different image frames.

Note that one or more blocks (or operations) described with reference to FIGS. 3 to 5 may be combined with one or more blocks (or operations) described with reference to another of the drawings.

In one or more aspects, techniques for supporting image signal processing may be performed in conjunction with one or more other processes or devices described elsewhere herein or in any single aspect or combination of aspects described below. It may include additional aspects such as. In a first aspect, supporting image signal processing can include an apparatus configured to perform steps, the steps comprising: receiving a first image frame representing a captured scene at a first focus distance; Receiving a second image frame representing a scene captured at a second focus distance different from the first focus distance; determining a local gradient difference between a first portion of a first image frame and a second portion of a second image frame, wherein the first and second portions correspond to a region of interest (ROI) within the scene; determining a radiant difference; and/or performing a contrast enhancement operation on the first portion of the first image frame based on the local gradient difference to determine the corrected first image frame. Additionally, the apparatus may perform or operate according to one or more aspects as described below. In some implementations, the apparatus includes a wireless device, such as a UE. In some implementations, a device can include at least one processor, and memory coupled to the processor. A processor may be configured to perform the operations described herein on a device. In some other implementations, a device may include a non-transitory computer-readable medium having program code recorded thereon, the program code being used by a computer to cause the computer to perform the operations described herein with reference to the device. It may be feasible. In some implementations, a device may include one or more means configured to perform the operations described herein. In some implementations, a method of wireless communication may include one or more operations described herein with reference to a device.

In one or more aspects, techniques for supporting image capture and/or image processing include any single aspect or aspects described below or in conjunction with one or more other processes or devices described elsewhere herein. Additional aspects may be included, such as any combination of these. In a first aspect, supporting image capture and/or image processing can include an apparatus configured to perform steps, the steps comprising: receiving a first image frame representing a captured scene at a first focus distance; Receiving a second image frame representing a scene captured at a second focus distance different from the first focus distance; determining a corresponding set of regions of interest including a first region of interest in a first image frame and a second region of interest in a second image frame; determining a value based on a comparison of the first region of interest and the second region of interest; and determining a corrected first image frame based on the first image frame and the value. Additionally, the apparatus may perform or operate according to one or more aspects as described below. In some implementations, the apparatus includes a wireless device, such as a UE. In some implementations, a device can include at least one processor and memory coupled to the processor. A processor may be configured to perform the operations described herein on a device. In some other implementations, a device may include a non-transitory computer-readable medium having program code recorded thereon, the program code being used by a computer to cause the computer to perform the operations described herein with reference to the device. It may be feasible. In some implementations, a device may include one or more means configured to perform the operations described herein. In some implementations, a method of wireless communication may include one or more operations described herein with reference to a device.

In a second aspect, in combination with the first aspect, the apparatus is further configured to perform operations, the operations comprising: determining a first face in a first area and in a second area; and determining a second face in the third region of interest, wherein determining the first corrected image frame causes the first face and the second face to appear in-focus within the first corrected image frame. Includes an operation to increase detail of the first side.

In a third aspect, in combination with one or more of the first aspect or the second aspect, determining the value includes determining a local gradient difference between the first region of interest and the second region of interest; Determining the corrected first image frame includes performing a contrast operation on a first region of interest in the first image frame based on local gradient differences.

In a fourth aspect, in combination with one or more of the first to third aspects, performing a contrast operation preserves brightness of the first image frame.

In a fifth aspect, in combination with one or more of the first to fourth aspects, determining the corrected first image frame includes performing an artificial intelligence based operation.

In a sixth aspect, in combination with one or more of the first to fifth aspects, an apparatus is further configured to perform operations comprising determining an alignment difference between a first image region of interest and a second region of interest, Determining the value here is based on the alignment difference.

In a seventh aspect, in combination with one or more of the first to sixth aspects, determining an alignment difference comprises determining an alignment difference within a first image frame from a first location within the first image frame to a second location within the second image frame. It involves determining the motion of the recognized object.

In an eighth aspect, in combination with one or more of the first to seventh aspects, an apparatus is further configured to perform operations comprising determining a number of regions of interest in a first image frame, wherein the second image frame The performing of receiving, determining the value, and determining the corrected first image frame is based on the number of regions being more than one.

In a ninth aspect, in combination with one or more of the first to eighth aspects, determining the number of regions of interest within the first image frame includes detecting faces within the first image frame.

In a tenth aspect, in combination with one or more of the first to ninth aspects, the first focus distance is associated with the first face within the first area of interest and the second area of interest; The second focus distance is associated with a second, different plane within the third region of interest.

In an eleven aspect, in combination with one or more of the first to tenth aspects, the first focus distance is a location between the first face in the first area of interest and the second face in the third area of interest, and the second focus distance is the It is the focus distance of 2 sides.

In a twelfth aspect, in combination with one or more of the first to eleventh aspects, determining the corrected first image frame is not based on combining any portion of the first image frame and the second image frame.

In a thirteenth aspect, in combination with one or more of the first to twelfth aspects, the device further includes a camera including an image sensor and a lens.

In a fourteenth aspect, in combination with one or more of the first to thirteenth aspects, the at least one processor is coupled to the camera and configured to perform operations, the operations to capture a first image frame at a first focus distance. Actions to control the camera; determining the presence of a first side and a second side within a first image frame; and controlling the camera to capture a second image frame at a second focus distance based on determining the presence of a first side and a second side within the first image frame.

In a fifteenth aspect, in combination with one or more of the first to fourteenth aspects, the at least one processor includes an image signal processor (ISP).

Those skilled in the art will understand that information and signals may be represented using any of a variety of different techniques and techniques. For example, data, instructions, commands, information, signals, bits, symbols, and chips that may be referenced throughout the above description include voltages, currents, electromagnetic waves, magnetic fields or magnetic particles, It may be represented by optical fields or optical particles, or any combination of these.

Components, functional blocks, and modules described herein with respect to FIGS. 1-7 include processors, electronic devices, hardware devices, electronic components, logic circuits, memories, software codes, among other examples. , firmware codes, or any combination thereof. Software, whether referred to as software, firmware, middleware, microcode, hardware description language, or other terminology, includes instructions, instruction sets, code, code segments, program code, programs, subprograms, among other examples. should be construed broadly to mean fields, software modules, applications, software applications, software packages, routines, subroutines, objects, executables, threads of execution, procedures, and/or functions. Additionally, the features discussed herein may be implemented via special processor circuitry, via executable instructions, or combinations thereof.

Those skilled in the art will also recognize that the various illustrative logical blocks, modules, circuits, and algorithm steps described in connection with the disclosure herein may be implemented as electronic hardware, computer software, or combinations of the two. will be. To clearly illustrate this interchangeability of hardware and software, various illustrative components, blocks, modules, circuits, and steps have been described above generally in terms of their functionality. Whether this functionality is implemented in hardware or software depends on the specific application and design constraints imposed on the overall system. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be construed as causing a departure from the scope of the present disclosure. Those skilled in the art will also understand that the order or combination of components, methods or interactions described herein are examples only and that the various aspects of the components, methods or interactions of the present disclosure are those illustrated and described herein. It will be easy to recognize that it can be combined or performed in other ways.

The various illustrative logics, logical blocks, modules, circuits and algorithmic processes described in connection with the implementations disclosed herein may be implemented as electronic hardware, computer software, or combinations of the two. The interchangeability of hardware and software has been described generally in terms of functionality and is illustrated by the various example components, blocks, modules, circuits and processes described above. Whether such functionality is implemented in hardware or software depends on the specific application and design constraints imposed on the overall system.

The hardware and data processing devices used to implement the various illustrative logics, logical blocks, modules and circuits described in connection with the aspects disclosed herein include general-purpose single- or multi-chip processors, digital signal processing devices (DSPs), and processor), application specific integrated circuit (ASIC), field-programmable gate array (FPGA) or other programmable logic device, discrete gate or transistor logic, discrete hardware components, or any of these designed to perform the functions described herein. It may be implemented or performed in any combination. A general-purpose processor may be a microprocessor, or any conventional processor, controller, microcontroller, or state machine. In some implementations, the processor may be implemented as a combination of computing devices, such as a combination of a DSP and a microprocessor, a plurality of microprocessors, one or more microprocessors combined with a DSP core, or any other such configuration. In some implementations, specific processes and methods may be performed by circuitry specific to a given function.

In one or more aspects, the functions described may be implemented in hardware, digital electronic circuitry, computer software, firmware, including the structures disclosed herein and their structural equivalents, or any combination thereof. Implementations of the subject matter described herein may also include one or more computer programs, i.e., computer program instructions, encoded on computer storage media for execution by or to control the operation of a data processing device. It may be implemented as one or more modules.

If implemented in software, the functions may be stored on or transmitted over as one or more instructions or code on a computer-readable medium. The processes of a method or algorithm disclosed herein may be implemented in a processor-executable software module that may reside on a computer-readable medium. Computer-readable media includes both computer storage media and communication media including any medium that facilitates transfer of a computer program from one place to another. Storage media can be any available media that can be accessed by a computer. By way of example, and not limitation, such computer-readable media may include random-access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), CD-ROM or other optical disk storage, magnetic It may include disk storage or other magnetic storage devices, or any other medium that can be used to store desired program code in the form of instructions or data structures and that can be accessed by a computer. Additionally, any connection may be properly termed a computer-readable medium. Disk and disk as used herein include compact disc (CD), laser disc (disc), optical disc (disc), digital versatile disc (DVD), floppy disk (disk), and Blu-ray disc. Discs generally reproduce data magnetically, but discs reproduce data optically using lasers. Combinations of the above should also be included within the scope of computer-readable media. Additionally, the operations of the method or algorithm may reside as one or any combination or set of codes and instructions in a machine-readable medium or computer-readable medium that may be incorporated into a computer program product.

Various modifications to the implementations described in this disclosure may be apparent to those skilled in the art, and the general principles defined herein may be applied to some other implementations without departing from the spirit or scope of the disclosure. . Accordingly, the claims are not intended to be limited to the implementations described herein but are to be accorded the widest scope consistent with the disclosure, principles and novel features disclosed herein.

Additionally, those skilled in the art will recognize that the terms "top" and "bottom" are often used for ease of describing the drawings and indicate relative positions that correspond to the orientation of the drawing on a properly oriented page and that, when implemented, any It will be readily appreciated that this may not reflect the proper orientation of the device.

Certain features that are described herein in the context of separate implementations can also be implemented in combination in a single implementation. Conversely, various features described in the context of a single implementation may also be implemented in multiple implementations separately or in any suitable subcombination. Moreover, although features have been previously described and even initially claimed as operating in certain combinations, in some cases one or more features from a claimed combination can be removed from that combination and the claimed combination is a subcombination. Or it may be related to changes in subcombinations.

Similarly, although operations are depicted in the drawings in a particular order, this does not mean that, to achieve the desired results, such operations must be performed in the specific order or sequential order shown or that all illustrated operations must be performed. It shouldn't be understood. Additionally, the drawings may schematically depict one or more example processes in the form of a flow diagram. However, other operations not depicted may be incorporated into the example processes schematically illustrated. For example, one or more additional operations may be performed before, after, concurrently with, or between any of the illustrated operations. In certain environments, multitasking and parallel processing may be advantageous. Moreover, the separation of various system components in the implementations described above should not be construed as requiring such separation in all implementations, and that the program components and systems described are generally integrated together in a single software product or integrated into multiple It should be understood that it can be packaged into software products. Additionally, some other implementations are within the scope of the following claims. In some cases, the actions recited in the claims can be performed in a different order and still achieve desirable results.

As used herein, including the claims, the term "or", when used in a list of two or more items, means that any of the listed items may be employed alone or any two or more of the listed items may be employed alone. This means that a combination of can be adopted. For example, if a component is described as including component A, component B, or component C, then the component includes A alone; B alone; C alone; By combining A and B; By combining A and C; By combining B and C; Alternatively, it may include a combination of A, B, and C. Additionally, as used herein, including the claims, “or” as used in a list of items started by “at least one of” means, for example, “at least one of A, B, or C.” denotes a disjunctive list such that a list of means A or B or C or AB or AC or BC or ABC (i.e. A and B and C) or any of these in any combination thereof. The term “substantially” is defined as substantially specified, but not necessarily exclusively, as understood by those skilled in the art (and includes what is specified; e.g., substantially 90 degrees includes 90 degrees, substantially parallel refers to parallel includes). In any of the disclosed embodiments, the term “substantially” can be replaced with “within [percentage]” of that specified, where percentage includes .1, 1, 5, or 10%.

The preceding description of the disclosure is provided to enable any person skilled in the art to make or use the disclosure. Various modifications to the disclosure will be readily apparent to those skilled in the art, and the generic principles defined herein may be applied to other variations without departing from the spirit or scope of the disclosure. Thus, the present disclosure is not intended to be limited to the examples and designs described herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims (34)

As a method,
Receiving a first image frame representing a scene captured at a first focus distance;
Receiving a second image frame representing the scene captured at a second focus distance different from the first focus distance;
determining a corresponding set of regions of interest including a first region of interest in the first image frame and a second region of interest in the second image frame;
determining a value based on a comparison of the first region of interest and the second region of interest; and
Determining a corrected first image frame based on the first image frame and the value. According to paragraph 1,
determining a first surface within the first region of interest and the second region of interest; and
further comprising determining a second side within the third region of interest;
Determining the corrected first image frame increases detail of the first face such that the first face and the second face appear in-focus within the corrected first image frame. A method comprising the step of doing so. According to paragraph 1,
Determining the value includes determining a local gradient difference between the first region of interest and the second region of interest;
Wherein determining the corrected first image frame includes performing a contrast operation on the first region of interest in the first image frame based on the local gradient difference. According to paragraph 3,
Wherein performing the contrast operation preserves brightness of the first image frame. According to paragraph 1,
The method of claim 1, wherein determining the corrected first image frame includes performing an artificial intelligence-based operation. According to paragraph 1,
further comprising determining an alignment difference between the first region of interest and the second region of interest,
Wherein determining the value is based on the alignment difference. According to clause 6,
Determining the alignment difference includes determining motion of the recognized object within the first image frame from a first location within the first image frame to a second location within the second image frame. . According to paragraph 1,
further comprising determining a number of regions of interest within the first image frame,
Wherein receiving the second image frame, determining the value, and determining the corrected first image frame are based on the number of regions of interest being greater than 1. According to clause 8,
Wherein determining the number of regions of interest within the first image frame includes detecting faces within the first image frame. According to paragraph 1,
the first focus distance is associated with a first surface within the first region of interest and the second region of interest;
The method of claim 1, wherein the second focus distance is associated with a second, different plane within the third region of interest. According to clause 10,
The first focus distance is a position between the first face in the first area of interest and the second face in the third area of interest, and the second focus distance is a focus distance of the second face. According to paragraph 1,
Wherein determining the corrected first image frame is not based on combining any portion of the second image frame with the first image frame. As a device,
a memory storing processor-readable code; and
At least one processor coupled to the memory, the at least one processor configured to execute the processor readable code to cause the at least one processor to perform operations, the operations comprising:
Receiving a first image frame representing a scene captured at a first focus distance;
receiving a second image frame representing the scene captured at a second focus distance different from the first focus distance;
determining a corresponding set of regions of interest including a first region of interest in the first image frame and a second region of interest in the second image frame;
determining a value based on a comparison of the first region of interest and the second region of interest; and
Determining a corrected first image frame based on the first image frame and the value. According to clause 13,
The at least one processor is further configured to execute the processor readable code to cause the at least one processor to perform operations, the operations comprising:
determining a first face within the first region of interest and the second region of interest; and
determining a second face within a third region of interest;
Determining the corrected first image frame includes increasing detail of the first face so that the first face and the second face appear in-focus within the corrected first image frame. , Device. According to clause 13,
Determining the value includes determining a local gradient difference between the first region of interest and the second region of interest;
Wherein determining the corrected first image frame includes performing a contrast operation on the first region of interest in the first image frame based on the local gradient difference. According to clause 15,
Wherein performing the contrast operation preserves brightness of the first image frame. According to clause 13,
Wherein determining the corrected first image frame includes performing an artificial intelligence based operation. According to clause 13,
The at least one processor is configured to execute the processor readable code to cause the at least one processor to perform additional operations, the additional operations comprising:
determining an alignment difference between the first region of interest and the second region of interest,
The operation of determining a value is based on the alignment difference. According to clause 18,
wherein determining the alignment difference comprises determining motion of an object recognized within the first image frame from a first location within the first image frame to a second location within the second image frame. . According to clause 13,
The at least one processor is configured to execute the processor readable code to cause the at least one processor to perform additional operations, the additional operations comprising:
determining a number of regions of interest within the first image frame;
Wherein receiving the second image frame, determining the value, and determining the corrected first image frame are based on the number of regions of interest being greater than 1. According to clause 20,
Wherein determining the number of regions of interest within the first image frame includes detecting faces within the first image frame. According to clause 13,
the first focus distance is associated with a first surface within the first region of interest and the second region of interest;
and the second focus distance is associated with a second, different plane within the third region of interest. According to clause 22,
The first focus distance is a position between the first face in the first area of interest and the second face in the third area of interest, and the second focus distance is a focus distance of the second face. According to clause 13,
The apparatus of claim 1, wherein determining the corrected first image frame is not based on combining any portion of the second image frame with the first image frame. According to clause 13,
further comprising a camera including an image sensor and a lens;
The at least one processor is coupled to the camera and configured to perform operations, the operations comprising:
controlling the camera to capture the first image frame at the first focus distance;
determining the presence of a first side and a second side within the first image frame; and
Controlling the camera to capture the second image frame at the second focus distance based on determining the presence of the first side and the second side within the first image frame. According to clause 13,
The apparatus of claim 1, wherein the at least one processor includes an image signal processor (ISP). A non-transitory computer-readable storage medium storing instructions, comprising:
The instructions, when executed by a processor, cause the processor to perform operations, the operations comprising:
Receiving a first image frame representing a scene captured at a first focus distance;
receiving a second image frame representing the scene captured at a second focus distance different from the first focus distance;
determining a corresponding set of regions of interest including a first region of interest in the first image frame and a second region of interest in the second image frame;
determining a value based on a comparison of the first region of interest and the second region of interest; and
determining a corrected first image frame based on the first image frame and the value. According to clause 27,
The instructions, when executed by the processor, cause the processor to perform additional operations, the additional operations being:
determining a first face within the first region of interest and the second region of interest; and
determining a second face within a third region of interest;
Determining the corrected first image frame includes increasing detail of the first face so that the first face and the second face appear in-focus within the corrected first image frame. , a non-transitory computer-readable storage medium. According to clause 27,
Determining the value includes determining a local gradient difference between the first region of interest and the second region of interest;
Determining the corrected first image frame includes performing a contrast operation on the first region of interest in the first image frame based on the local gradient difference. media. According to clause 27,
The instructions, when executed by the processor, cause the processor to perform additional operations, the additional operations being:
determining a number of regions of interest within the first image frame;
performing the operations of receiving the second image frame, determining the value, and determining the corrected first image frame is based on the number of regions of interest being greater than 1. storage media. As a device,
A camera including an image sensor and lens;
a memory storing processor-readable code; and
At least one processor coupled to the memory and the camera, the at least one processor configured to execute the processor readable code to cause the at least one processor to perform operations, the operations comprising:
controlling the camera to acquire a first image frame representing a scene captured at a first focus distance;
determining that the first image frame includes a first side associated with the first focus distance and a second side associated with the second focus distance;
controlling the camera to obtain a second image frame representing the scene captured at a third focus distance between the first and second focus distances;
controlling the camera to obtain a third image frame representing the scene captured at the second focus distance;
determining a corresponding set of regions of interest including a first region of interest in the second image frame associated with the first side and a second region of interest in the third image frame associated with the first side;
determining a value based on a comparison of the first region of interest and the second region of interest; and
and determining a corrected image frame based on the second image frame and the value. According to clause 31,
Wherein determining the corrected image frame includes performing a contrast enhancement operation on a first region of interest in the second image frame. According to clause 32,
wherein performing the contrast enhancement operation on the first region of interest of the second image frame is based on an intensity corresponding to the value. According to clause 31,
Wherein determining the corrected image frame includes performing an artificial intelligence-based operation on a first region of interest in the second image frame.