KR20240114659A - Apparatus and method for obtaining image emplying color separation lens array - Google Patents

Abstract

An image acquisition device according to an embodiment includes an image sensor including a color separation lens array that separates and condenses incident light according to color, and a signal processing unit that processes an input image acquired from the image sensor. The signal processing unit includes a demosaicing unit that reconstructs the green signal to have full pixel resolution using the input image, a sharpening filter unit that sharpens the reconstructed green signal in each preset direction to generate a first image, and a white balance unit that converts the input image into a white balance unit. A direction image generator that generates a second image in which the base band of the white-balanced input image is removed for each preset direction and only the detail band is extracted using an unsharpening filtering algorithm, and white balances the input image. and a gray detection unit for detecting the gray area of the white-balanced input image, an edge detection unit for white-balancing the input image and detecting the edge direction of the white-balanced input image, and based on the detected gray area and edge direction. Thus, it includes a selection unit that generates a third image by blending the first image and the second image.

Description

Apparatus and method for obtaining image emplicating color separation lens array}

It relates to an image acquisition device and method using a color separation lens array.

In an image sensor, red, green, and blue color filters are arranged in a certain pattern on a light sensing cell for imaging purposes. For example, considering the visual/perceptual characteristics of humans who are sensitive to green color, 50% green, red, and blue are used. A Bayer Pattern structure in which are arranged crosswise so that each is 25% can be applied. Since the color filter absorbs light of other colors except for the light of the corresponding color, the efficiency of light use may decrease.

The color separation lens array is formed to have a phase distribution that focuses light of different wavelengths on adjacent photo-sensing cells, so that incident light can be separated and concentrated according to color. Because of this, the image sensor including the color separation lens array can receive additional light through the light path of the surrounding pixels, and thus light use efficiency can be increased.

However, an image sensor including a color separation lens array may experience blur, grid artifacts, and color mixing, thereby reducing spatial resolution.

An image acquisition device and method that can improve spatial resolution while using a color separation lens array are provided.

An image acquisition device according to an embodiment includes an image sensor including a color separation lens array that separates and focuses incident light according to color, and a signal processing unit that processes an input image acquired from the image sensor.

The signal processing unit includes a demosaicing unit that reconstructs the green signal to have full pixel resolution using the input image, a sharpening filter unit that sharpens the reconstructed green signal in each preset direction to generate a first image, and the input A direction image generator that white balances an image and uses an unsharpening filtering algorithm to generate a second image in which the base band of the white balanced input image is removed for each preset direction and only the detail band is extracted; A gray detector that white balances the input image and detects a gray area of the white balanced input image, an edge detector that white balances the input image and detects an edge direction of the white balanced input image, and a selection unit that generates a third image by blending the first image and the second image based on the detected gray area and the edge direction.

An image acquisition method according to an embodiment includes an image acquisition device including an image sensor including a color separation lens array that separates and focuses incident light according to color, using the input image to convert a green signal into a full pixel resolution. performing demosaicing to reconstruct the reconstructed green signal, sharpening the reconstructed green signal in each preset direction to generate a first image, white balancing the input image, and using an unsharpening filtering algorithm, A direction image generation step of generating a second image in which the base band of the white balance-processed input image is removed for each preset direction and only the detail band is extracted, white balance processing the input image, and the white balance-processed input A gray detection step of detecting a gray area of an image, an edge detection step of white balancing the input image and detecting an edge direction of the white balance processed input image, and based on the detected gray area and the edge direction , and a selection step of generating a third image by blending the first image and the second image.

According to an image acquisition device and method according to an embodiment, the spatial resolution of the image can be improved while maintaining the effect of improving light efficiency by performing a plurality of filtering processes on the captured raw image signal.

Figure 1 schematically shows the configuration of an image acquisition device according to an embodiment.
Figure 2 is a block diagram of an image sensor according to an embodiment.
Figure 3 is a diagram for explaining a Bayer pattern as a pixel arrangement of an image sensor's pixel array.
4 and 5 schematically show an image sensor according to an embodiment.
Figure 6 is a plan view schematically showing the arrangement of photo-sensing cells in the pixel array of an image sensor.
Figure 7 is a plan view exemplarily showing a plurality of nanoposts arranged in a plurality of regions of the color separation lens array corresponding to the arrangement of the photo-sensing cells of the sensor substrate of the image sensor.
Figure 8 is a block diagram for explaining the operation of a signal processor according to an embodiment.
FIG. 9 is a diagram showing a frequency region in which color can be expressed through the color filter array of the Bayer pattern of FIG. 3.
FIGS. 10A and 10B are diagrams for explaining the image processing operation performed in the direction image generator of FIG. 8
11 to 13 are drawings to explain the effect of the present invention.
Figure 14 is a diagram for explaining an embodiment of using a condition check unit during sharpening processing.
Figure 15 is a block diagram for performing an unsharpening filtering algorithm according to another embodiment.
Figure 16 is a diagram for explaining the chief ray angle of the image sensor.
Figure 17 is a flowchart explaining a resolution improvement method according to an embodiment.
Figure 18 is a block diagram schematically showing an electronic device including an image sensor according to embodiments.
FIG. 19 is a block diagram illustrating a camera module provided in the electronic device of FIG. 18.

Hereinafter, exemplary embodiments will be described in detail with reference to the attached drawings. In the following drawings, the same reference numerals refer to the same components, and the size of each component in the drawings may be exaggerated for clarity and convenience of explanation. The embodiments described below are merely illustrative, and various modifications are possible from these embodiments.

Hereinafter, the term “above” or “above” may include not only what is directly above in contact but also what is above without contact. Singular expressions include plural expressions unless the context clearly dictates otherwise. Additionally, when a part "includes" a certain component, this means that it may further include other components rather than excluding other components, unless specifically stated to the contrary. The use of the term “above” and similar referential terms may refer to both the singular and the plural.

Figure 1 schematically shows the configuration of an image acquisition device according to an embodiment.

Referring to FIG. 1, the image acquisition device 100 converts the optical image formed by the photographing lens unit 120, which forms an optical image by focusing the light reflected from the object OBJ, into an electrical signal. It may include an image sensor 200 that acquires image signals for each color, and a signal processing unit 250 that processes the image signals for each color acquired by the image sensor 200 to generate an output signal. It may further include a memory 180 that stores image signals for each color and images formed in the signal processor 250. Additional optical elements, such as an infrared cut-off filter, may be further disposed between the image sensor 200 and the photographing lens unit 120.

Figure 2 is a schematic block diagram of an image sensor according to an embodiment.

Referring to FIG. 2 , the image sensor 200 may include a pixel array 210, a timing controller 240, a row decoder 220, and an output circuit 230. The image sensor 200 may be a charge coupled device (CCD) image sensor or a complementary metal oxide semiconductor (CMOS) image sensor.

The pixel array 210 includes pixels arranged two-dimensionally along a plurality of rows and columns. The row decoder 220 selects one of the rows of the pixel array 210 in response to the row address signal output from the timing controller 240. The output circuit 230 outputs a light-sensing signal in column units from a plurality of pixels arranged along the selected row. To this end, the output circuit 230 may include a column decoder and an analog to digital converter (ADC). For example, the output circuit 230 may include a plurality of ADCs arranged for each column between the column decoder and the pixel array 210, or one ADC arranged at the output terminal of the column decoder. The timing controller 240, row decoder 220, and output circuit 230 may be implemented as one chip or as separate chips. The signal processing unit 250 of FIG. 1 for processing the image signal output through the output circuit 230 is implemented as a single chip together with the timing controller 240, the row decoder 220, and the output circuit 230. It may be possible.

The pixel array 210 may include a plurality of pixels that sense light of different wavelengths. Array of pixels can be implemented in various ways.

FIG. 3 is a diagram for explaining a Bayer pattern as a pixel arrangement of an image sensor.

Referring to FIG. 3, one unit pixel includes four quadrant regions, and the first to fourth quadrants are blue pixel (B), green pixel (G), red pixel (R), and green pixel, respectively. It can be a pixel (G). These unit pixels are repeatedly arranged two-dimensionally along the first direction (X direction) and the second direction (Y direction). In other words, within a unit pixel in a 2×2 array, two green pixels (G) are arranged diagonally on one side, and one blue pixel (B) and one red pixel (R) are placed diagonally on the other side. is placed. Looking at the overall pixel arrangement, a first row in which a plurality of green pixels (G) and a plurality of blue pixels (B) are alternately arranged along the first direction, and a plurality of red pixels (R) and a plurality of green pixels (G) are arranged alternately in the first direction. Second rows alternately arranged along the first direction are repeatedly arranged along the second direction.

4 and 5 schematically show an image sensor according to an embodiment. Figure 6 is a plan view schematically showing the arrangement of photo-sensing cells in the pixel array of an image sensor. Figure 7 is a plan view exemplarily showing a plurality of nanoposts arranged in a plurality of regions of the color separation lens array corresponding to the arrangement of the photo-sensing cells of the sensor substrate of the image sensor.

3 to 5, the pixel array 210 of the image sensor 200 is disposed on a sensor substrate 500 including an array of a plurality of photo-sensing cells for sensing light, and at the front of the sensor substrate 500. It includes a color separation lens array 300 that separates and focuses light according to color and makes it enter a plurality of photo-sensing cells.

The sensor substrate 500 of the image sensor 200 includes a first photo-sensing cell 511, a second photo-sensing cell 512, a third photo-sensing cell 513, and a fourth photo-sensing cell that converts light into an electrical signal. It may include a cell 514. Unit pixels of the first photo-sensing cell 511, the second photo-sensing cell 512, the third photo-sensing cell 513, and the fourth photo-sensing cell 514 may be repeatedly arranged in two dimensions. When the pixel array 210 of the image sensor 200 has a Bayer pattern, for example, the first photo-sensing cell 511 and the fourth photo-sensing cell 514 correspond to the green pixel (G), and the One of the two photo-sensing cells 512 and the third photo-sensing cell 513 may correspond to the red pixel (R) and the other may correspond to the blue pixel (B).

The color separation lens array 300 is provided with a microstructure in each of the plurality of regions (311, 312, 313, 314) facing each of the plurality of photo-sensing cells (511, 512, 513, and 514), so that different wavelengths are transmitted to the adjacent photo-sensing cells. It is provided to form a phase distribution that condenses light, so that incident light can be separated and concentrated according to color. As exemplarily shown in FIG. 5, the microstructure of the color separation lens array 300 has one or more nanoposts (NPs) distributed to form a phase distribution that focuses light of different wavelengths on adjacent photosensing cells. It may include a plurality of nanoposts arranged so as to be possible. As another example, the microstructure of the color separation lens array 300 may be patterned to form a phase distribution that focuses light of different wavelengths on adjacent photo-sensing cells. Hereinafter, a case where the microstructure is made of nanoposts will be described as an example.

The color separation lens array 300 may include a plurality of regions 311, 312, 313, and 314 that face and correspond to the plurality of photosensing cells 511, 512, 513, and 514 of the sensor substrate 500 in a one-to-one correspondence. For example, the color separation lens array 300 corresponds one-to-one with the first to fourth photo-sensing cells 511, 512, 513, and 514 of the sensor substrate 500, and has first to fourth regions (311, 312, 313, 314) facing each other. ), and the first to fourth regions 311, 312, 313, and 314 may include first to fourth microstructures to form a phase distribution that focuses light of different wavelengths on adjacent photosensing cells.

When the first to fourth microstructures each include one or more nanoposts as shown in FIG. 5, the first region 311 includes one or more first nanoposts, and the second region 312 includes one or more nanoposts. It includes 2 nanoposts, the third region 313 may include one or more third nanoposts, and the fourth region 314 may include one or more fourth nanoposts. Light of a predetermined wavelength passing through the color separation lens array 300 forms a phase in which the light is focused on a photo-sensing cell corresponding to one of the first to fourth photo-sensing cells 511, 512, 513, and 514, and the remaining The shape, size, and arrangement of the first to fourth nanoposts may be determined to form a phase that does not proceed to the photosensitive cell. Additionally, the plurality of regions provided in the color separation lens array 300 each include a plurality of sub-regions, and a plurality of nanoposts may be arranged in the plurality of sub-regions. Additionally, the plurality of regions provided in the color separation lens array 300 each include a plurality of sub-regions, and a plurality of nanoposts may be arranged at intersections between the plurality of sub-regions.

The image sensor 200 may have a pixel array structure in which unit pixels are repeatedly arranged. For example, the image sensor 200 has four quadrant regions in which unit pixels composed of a blue pixel, a green pixel, a red pixel, and a green pixel are arranged two-dimensionally along the first and second directions. It may have a repetitively arranged Bayer pattern structure. At this time, for example, among the plurality of nanoposts of the color separation lens array 300, the nanopost provided in the center of the area corresponding to the green pixel among the plurality of areas is provided in the center of the area corresponding to the pixel of another color. It can have a larger cross-sectional area than a nanopost. In addition, among the plurality of nanoposts, the nanoposts provided in the area corresponding to the green pixel among the plurality of regions may have a smaller cross-sectional area than the nanoposts arranged in the peripheral area than the nanoposts arranged in the center. In addition, the nanoposts provided in the areas corresponding to the green pixels among the plurality of areas have different distribution rules along the first and second directions, and the nanoposts provided in the areas corresponding to the blue pixels and the red pixels among the plurality of areas. Nanoposts may have a symmetrical distribution rule along the first and second directions. As another example, a nanopost provided at the center of an area corresponding to a blue or red pixel may have a larger cross-sectional area than a nanopost provided at the center of an area corresponding to a pixel of another color.

For example, the first photo-sensing cell 511 senses light of the first wavelength corresponding to the first pixel, and the second photo-sensing cell 512 senses light of the second wavelength corresponding to the second pixel. In addition, the third photo-sensing cell 513 can sense light of the third wavelength corresponding to the third pixel, and the fourth photo-sensing cell 514 can sense light of the fourth wavelength corresponding to the fourth pixel. there is. However, the examples are not limited to this. Although not shown, a separator for cell separation may be further formed at the boundary between cells.

In the image sensor 200, the first pixel, the second pixel, the third pixel, and the fourth pixel are a green pixel (G), a blue pixel (B), a red pixel (R), and a green pixel (G), respectively; When arranged in a Bayer pattern, the first photo-sensing cell 511, the second photo-sensing cell 512, the third photo-sensing cell 513, and the fourth photo-sensing cell 514 of the sensor substrate 500 are It can be arranged corresponding to the Bayer pattern arrangement.

The color separation lens array 300 is for separating and concentrating incident light according to color and making it incident on a plurality of photo-sensing cells, such as the first to fourth photo-sensing cells 511, 512, 513, and 514. As shown in FIG. 5, a plurality of nanoposts are used. (NP) may include an array.

For example, in the color separation lens array 300, light of a first wavelength is incident on the first photo-sensing cell 511 to form a green pixel (G), and light of the second wavelength is incident on the second photo-sensing cell 512. Light is incident to form a blue pixel (B), light of a third wavelength is incident to the third photo-sensing cell 513 to form a red pixel (R), and light of the third wavelength is incident to the fourth photo-sensing cell 514. Incident light can be separated and concentrated according to color so that light of one wavelength is incident to form a green pixel (G).

The color separation lens array 300 includes a nanopost array in which a plurality of nanoposts (NPs) are arranged in a predetermined rule. The nanopost array can be supported by a spacer layer. The spacer layer serves to maintain a constant gap between the sensor substrate 500 and the color separation lens array 300. The spacer layer may be designed to have an interval as necessary to secure the focal distance of the light passing through the color separation lens array 300 to each photo-sensing cell. Compared to the combination of a conventional microlens array and a color filter, when using the color separation lens array 300, the spacer layer is a sensor substrate 500 and a color separation lens array 300 to implement color separation and light collection. A thicker gap is required between them. Additionally, the spacer layer may be made of a material that is transparent to visible light. For example, the spacer layer has a lower refractive index than the refractive index of the nanoposts (NPs) of the color separation lens array 300, such as SiO2, silanol-based glass (SOG; siloxane-based spin on glass), and has an absorption rate in the visible light band. It may be made of low dielectric material. In FIG. 5 , the spacer layer is omitted for convenience in order to more clearly express the separation and concentration of light incident on the color of the color separation lens array 300.

Meanwhile, the color separation lens array 300 may be further provided with a protective layer that protects a plurality of nanoposts (NPs). The protective layer may be made of a dielectric material with a lower refractive index than the refractive index of the material forming the nanopost (NP). Additionally, the color separation lens array 300 may further include a color filter between the spacer layer and the sensor substrate 500. For example, if the first pixel, second pixel, third pixel, and fourth pixel are green pixel (G), blue pixel (B), red pixel (R), and green pixel (G), respectively, the sensor substrate ( 500), a green color filter is disposed on the first photo-sensing cell 511 and the fourth photo-sensing cell 514, a blue color filter is disposed on the second photo-sensing cell 512, and a third photo-sensing cell 514 is provided. A red color filter may be placed on the cell 513. Color filters can be used for various purposes, but for example, if a color filter is used, it can be easy to create an image using existing image signal processing technology for signals in each wavelength band that pass through the color filter.

The pixel array 210 of the image sensor 200 may have a two-dimensional array. That is, as shown in FIG. 6, the first row, the third photo-sensing cell 513, and the fourth photo-sensing cell 514 are arranged alternately with the first photo-sensing cell 511 and the second photo-sensing cell 512. A plurality of first photo-sensing cells 511, second photo-sensing cells 512, third photo-sensing cells 513, and 4 Photo-sensing cells 514 may be two-dimensionally arranged along the first direction (X direction) and the second direction (Y direction). At this time, when the pixel array 210 of the image sensor 200 is arranged in a Bayer pattern as shown in FIG. 3, the first photo-sensing cell 511 and the fourth photo-sensing cell 514 are connected to the green pixel (G). Correspondingly, the second photo-sensing cell 512 corresponds to the blue pixel (B), and the third photo-sensing cell 513 corresponds to the red pixel (R). In a one-to-one correspondence with the arrangement of the first to fourth photo-sensing cells 511, 512, 513, and 514 of the sensor substrate 500, the nanopost array of the color separation lens array 300 has a plurality of regions 311, 312, 313, and 314 corresponding thereto, as shown in FIG. 7. can be divided into

For example, when the pixel arrangement of the image sensor 200 is a Bayer pattern arrangement, one unit pixel includes four quadrant regions, and the first to fourth quadrants are each a blue pixel. (B), green pixel (G), red pixel (R), or green pixel (G). These unit pixels are arranged two-dimensionally and repeatedly along the first direction (X direction) and the second direction (Y direction).

In a unit pixel, the green pixel (G) corresponds to the first photo-sensing cell 511 and the corresponding first area 311 of the color separation lens array 300, and the blue pixel (B) corresponds to the second photo-sensing cell 511. The cell 512 corresponds to the area 312 of the color separation lens array 300, and the red pixel (R) includes the third photosensing cell 513 and the corresponding color separation lens array 300. The area 313 corresponds to the green pixel (G), and the fourth photo-sensing cell 514 and the corresponding fourth area 314 of the color separation lens array 300 correspond to the green pixel (G).

Referring to FIG. 7, the nanopost array of the color separation lens array 300 corresponds one-to-one with the first to fourth photo-sensing cells 511, 512, 513, and 514 and faces the first to fourth regions 311. , 312, 313, 314). One or more nanoposts (NPs) may be disposed in each of the first to fourth regions (311, 312, 313, and 314), and at least one of the shape, size, and arrangement of the nanoposts (NPs) may vary depending on the region. there is.

A plurality of nanoposts (NPs) of the color separation lens array 300 focus light of different wavelengths on the first photo-sensing cell 511 and the second photo-sensing cell 512 adjacent to each other of the sensor substrate 500. The shape, size, and arrangement can be determined to form a desired phase distribution. In addition, the plurality of nanoposts (NPs) of the color separation lens array 300 transmit light of different wavelengths to the third photo-sensing cell 513 and the fourth photo-sensing cell 514 adjacent to each other of the sensor substrate 500. The shape, size, and arrangement can be determined to form a phase distribution that concentrates light.

For example, when the pixel array 210 of the image sensor 200 is arranged in a Bayer pattern, as shown in FIG. 7, the area 311 of the color separation lens array 300 corresponds to the green pixel (G). , the area 312 of the color separation lens array 300 corresponds to the blue pixel (B), the area 313 of the color separation lens array 300 corresponds to the red pixel (R), and the green pixel (G) Since the area 314 of the color separation lens array 300 corresponds to the cross-sectional area at the center of the area of the color separation lens array 300 corresponding to the green pixel (G), blue pixel (B), and red pixel (R) These different nanoposts (NPs) are arranged, and the nanoposts (NPs) are placed at the intersection of the pixel borderline and the center on the borderline between pixels. The cross-sectional area of the nanopost (NP) disposed at the boundary between pixels may be smaller than that of the nanopost (NP) disposed at the center of the pixel.

For example, the cross-sectional area of the nanopost (NP) disposed in the center of the areas 311 and 314 of the color separation lens array 300 corresponding to the green pixel (G) is the color separation lens corresponding to the blue pixel (B). It is larger than the cross-sectional area of the nanopost (NP) disposed in the center of the area 312 of the array 300 or the area 313 of the color separation lens array 300 corresponding to the red pixel (R), and the blue pixel (B) The cross-sectional area of the nano-post (NP) disposed at the center of the region 312 corresponding to is larger than the cross-sectional area of the nano-post (NP) disposed at the center of the region 313 corresponding to the red pixel (R). However, it is not limited to this. Here, the cross-sectional area refers to the area of the cross-section perpendicular to the height direction (Z direction) of the nanopost (NP).

Meanwhile, nanoposts (NPs) provided in the areas 311 and 314 corresponding to the green pixel (G) may have different distribution rules along the first direction (X direction) and the second direction (Y direction). That is, the nanoposts (NP) provided in the areas 311 and 314 corresponding to the green pixel (G) may have an asymmetric size arrangement along the first direction (X direction) and the second direction (Y direction). As shown in FIG. 7, the nanopost is located at the boundary between the areas 311 and 314 corresponding to the green pixel (G) and the area 312 corresponding to the blue pixel (B) adjacent in the first direction (X direction). The cross-sectional area of the nanopost (NP) located at the border of the region 313 corresponding to the red pixel (R) adjacent in the second direction (Y direction) is different from each other.

On the other hand, the nanoposts provided in the areas 312 and 313 corresponding to the blue pixel (B) and the red pixel (R) have a symmetrical distribution rule along the first direction (X direction) and the second direction (Y direction). You can. As shown in FIG. 7, the area 312 corresponding to the blue pixel (B) is a region of nanoposts (NPs) lying at the boundary between adjacent pixels in the first direction (X direction) and the second direction (Y direction). The cross-sectional areas are the same, and the area 313 corresponding to the red pixel (R) is also a nanopost (NP) located at the boundary between adjacent pixels in the first direction (X direction) and the second direction (Y direction). The cross-sectional areas are the same.

This distribution is similar to that of the blue pixel (B) and red pixel (R) in the pixel arrangement of the Bayer pattern, while the pixels adjacent to each other in the first direction (X direction) and the second direction (Y direction) are the same as the green pixel (G). In the case of the green pixel (G), the pixel adjacent to it in the first direction (X direction) is the blue pixel (B), and the pixel adjacent to the second direction (Y direction) is the red pixel (R). Therefore, nanoposts (NPs) are arranged in the form of 4-fold symmetry in the second region 312 and the third region 313 corresponding to the blue pixel (B) and red pixel (R), respectively. In the first and fourth regions 311 and 314 corresponding to the green pixel (G), nanoposts (NPs) may be arranged in a two-fold symmetry. In particular, the first area 311 and the fourth area 314 may be rotated by 90 degrees with respect to each other.

In Figures 5 and 7, a plurality of nanoposts (NPs) are all shown as having a symmetrical circular cross-sectional shape, but the present invention is not limited thereto. Some nanoposts with an asymmetric cross-sectional shape may be included. For example, nanoposts having an asymmetric cross-sectional shape with different widths in the first direction (X direction) and the second direction (Y direction) are employed in the regions 311 and 314 corresponding to the green pixel (G), and the blue pixel (B), the second area 312 and third area 313 respectively corresponding to the red pixel (R) have a symmetrical cross-sectional shape with the same width in the first direction (X direction) and the second direction (Y direction). Nanoposts having can be employed.

The color separation lens array 300 is formed to have a phase distribution that focuses light of different wavelengths on adjacent photosensing cells 511, 512, 513, and 514, so that incident light can be separated and concentrated according to color. Because of this, the image sensor 200 including the color separation lens array 300 can receive additional light through the light path of the surrounding pixels, and thus light use efficiency can be increased.

However, the image sensor 200 including the color separation lens array 300 may experience blur, grid artifacts, and color mixing, thereby reducing spatial resolution.

Hereinafter, using FIGS. 8 to 13 , a method in which the signal processor 250 improves spatial resolution by filtering the image signal for each color obtained from the image sensor 200 will be described in detail.

Figure 8 is a block diagram for explaining the operation of a signal processor according to an embodiment. FIG. 9 is a diagram showing a frequency region in which color can be expressed through the color filter array of the Bayer pattern of FIG. 3. FIGS. 10A and 10B are diagrams for explaining an image processing operation performed in the direction image generator of FIG. 8 .

Referring to FIG. 8, the signal processing unit 250 according to one embodiment includes a demosaicing unit 251, a direction image generation unit 252, a gray detection unit 253, an edge detection unit 254, and a pacer unit 255. ), a sharpening filter unit 256, a selection unit 257, an adjustment unit 258, and a Bayer recombination unit 259. At this time, the signal processing unit 25 may be implemented as a software module or hardware module. Additionally, the signal processing unit 250 may be provided in the logic of the image sensor 200 or may be provided in a companion chip.

The demosaicing unit 251 may receive the input image (IMG_IN) from the memory 180. In order to reduce the capacity of the memory 180 used for calculation, the demosaicing unit 251 performs color interpolation only on the green signal of the input image (IMG_IN) and produces a green signal (IMG_G) reconstructed to have full pixel resolution. can be created.

At this time, the demosaicing algorithm can apply various methods. For example, various methods exist, such as bilinear interpolation, an approach using correlation between channels, and a method of increasing spatial resolution using edge directionality.

Afterwards, the sharpening filter unit 256 may sharpen the green signal (IMG_G) reconstructed to have full pixel resolution in each preset direction to generate the first image (IMG1).

According to one embodiment, the sharpening filter unit 256 may be equipped with a filter to perform a sharpening algorithm that changes the image to give it a sharp feel. For example, one or more high pass filters may be provided in the sharpening filter unit 256. Alternatively, when an unsharpning masking algorithm is applied, a low pass filter for passing the image signal through the low pass filter may be provided in the sharpening filter unit 256. there is.

For image sharpening effect through sharpening filtering algorithm (or unsharpening filtering algorithm), the image signal can be processed through a high pass filter, and the mask of the high pass filter is used. ), the sum of all coefficients can have a positive value. For example, the center coefficient of the mask may have a positive value, and the peripheral coefficients around the center coefficient may have a negative value. Additionally, the sum of all coefficients of the mask may have a value of 0 or more, and as an example, the sum of all coefficients of the mask may have a value of 1.

In addition, the unsharpening method can be applied as a method for the image sharpening effect, and the unsharpening method reduces the low-frequency component in the image by subtracting the original image signal and the image signal that has undergone low-frequency filtering. A sharpening effect can be achieved. In describing the sharpening operation according to an embodiment, either a sharpening operation based on the unsharpening filter algorithm as described above or a sharpening operation based on a sharpening filter (or high pass filter). Any one may be applied.

As in the above-described embodiment, the sharpening filter unit 256 may include one or more sharpening filters (and/or unsharpening filters; hereinafter, the sharpening filter is referred to as a concept including an unsharpening filter) for an image sharpening effect. there is. The sharpening filter may have a mask value to achieve an image sharpening effect. As an example, the mask of the sharpening filter may have N*N coefficients (N is an integer greater than or equal to 2).

Additionally, according to one embodiment, one sharpening filter unit 256 may include a plurality of directional filters. The sharpening filter unit 256 can be applied along the edge direction using the directional filters, and can adjust the degree of sharpness based on the noise level, flatness, luminance level, and texture amount of the image.

At this time, the preset directions for applying the directional filter may be horizontal, vertical, 45 degrees diagonal, and 135 degrees diagonal. That is, the first image (IMG1) obtained by sharpening the reconstructed green signal (IMG_G) is the first image (IMG1) filtered along the horizontal direction, the first image (IMG1) filtered along the vertical direction, and the 45-degree diagonal direction. It may include a first image (IMG1) filtered along a 135-degree diagonal direction and a first image (IMG1) filtered along a 135-degree diagonal direction.

The pacer unit 255 may detect the edge direction of the reconstructed green signal (IMG_G). For example, the phaser unit 255 can detect the edge direction of the image by calculating a differential value along the pixel in polar coordinates. If the edge strength of the reconstructed green signal (IMG_G) is greater than a preset value, the selection unit 257 may edge improve the first image (IMG1) corresponding to the edge direction detected by the pacer unit 255. . In other words, the selection unit 257 selects the first image (IMG1) filtered along the horizontal edge direction, the first image (IMG1) filtered along the vertical edge direction, and the first image (IMG1) filtered along the 45-degree diagonal edge direction. IMG1), and any first image (IMG1) corresponding to the edge direction detected by the pacer unit 255 among the first images (IMG1) filtered along a 135-degree diagonal edge direction. . The pacer unit 255 may provide edge direction information D1 of the reconstructed green signal IMG_G to the selection unit 257.

When the selection unit 257 outputs a signal by performing only the process described so far, the output signal may be the edge-enhanced first image (IMG1') rather than the third image (IMG3), which will be described later. Afterwards, the adjustment unit 258 adjusts the red signal (IMG_R) and the blue signal (IMG_B) based on the amount of change in the green signal of the edge-enhanced first image (IMG1'), and the Bayer recombiner 259 adjusts the edge-enhanced first image (IMG1'). When outputting an output signal by combining the green signal of the first image (IMG1'), the adjusted red signal (IMG_R'), and the adjusted blue signal (IMG_B'), the 4-1 image having a Bayer pattern array ( IMG4-1) can be generated. However, the 4-1 image (IMG4-1) is not the 4th image (IMG4), which will be described later.

Unlike the present invention, if the image sensor 200 does not include a color separation lens array (300 in FIG. 5), the signal processor 250 may output the 4-1 image (IMG4-1) having a Bayer pattern array. and sufficient spatial resolution improvement can be expected as an output signal. However, in the case of the image sensor 200 including the color separation lens array (300 in FIG. 5) of the present invention, there is a problem in improving spatial resolution due to blurring due to the characteristics of the color separation lens array (300 in FIG. 5) described above. Limitations may arise.

Looking in detail with reference to FIG. 9, area 910 briefly represents the resolution at which green can be expressed in Nyquist frequency. Area 910 is green and shows that it can display a resolution equal to 1/2 of the total pixels of the image sensor (200 in FIG. 5).

Area 920 briefly represents the resolution at which red or blue can be expressed in Nyquist frequency. Area 920 is red or blue and shows a resolution equal to 1/4 of the total pixels of the image sensor (200 in FIG. 5). Through area 920, it can be seen that the range of resolution at which red or blue can be expressed is 1/2 of the resolution at which green can be expressed.

On the other hand, area 930 simply represents the resolution at which a monochrome signal can be expressed in Nyquist frequency. Area 930 shows that the resolution of all pixels of the image sensor (200 in FIG. 5) can be expressed as a monochrome signal. Through area 930, it can be seen that the range of resolution in which a monochrome signal can be expressed is twice the resolution expressed in green.

Area 940 represents the frequency range of wasted pixels. That is, there may be a loss of resolution as much as area 940, excluding area 910, where green can be expressed, in area 930, which can be expressed as a monochrome signal. Therefore, even if the spatial resolution is improved based on the green signal (IMG_G) reconstructed to have full pixel resolution, resolution loss as much as area 940 may occur.

Meanwhile, even with the image sensor 200 having a Bayer pattern color filter array, for gray subjects in an image, resolution of all pixels (or entire band) can be obtained like a monochrome sensor. To this end, if the image acquired by the image sensor 200 having a Bayer pattern color filter array is subjected to white balance processing, the gray subject in the image can be regarded as a gray subject acquired by the monochrome sensor. White balance adjusts the gains of R and B with G as the center so that the RGB (Red, Green, Blue) ratio in the gray area is the same (e.g. R=G=B) under various color temperature lighting.

Hereinafter, a specific method of improving spatial resolution by extracting a wide signal band for a gray subject from an image acquired by the image sensor 200 having a Bayer pattern color filter array will be described.

Referring again to FIG. 8, the direction image generator 252 may perform white balance processing on the input image (IMG_IN). White balance may mean correcting color differences resulting from differences in color temperature. There may be differences in color expression depending on the light (or light source) when shooting an image. For example, if the color temperature is low, it appears reddish, and if the color temperature is high, it appears blue, so white balance can be performed to correct the color temperature of the light to display an ideal white color. White balance adjusts the gains of R and B with G as the center so that the RGB (Red, Green, Blue) ratio in the gray area is the same (e.g. R=G=B) under various color temperature lighting.

The direction image generator 252 uses an unsharpening filtering algorithm to generate a second image (IMG2) from which the base band of the white-balanced input image (IMG_WB) is removed for each preset direction and only the detail band is extracted. You can. At this time, the preset direction may be a horizontal direction, a vertical direction, a 45 degree diagonal direction, and a 135 degree diagonal direction. That is, the second image (IMG2) is a second image (IMG2) filtered along the horizontal direction, a second image (IMG2) filtered along a vertical direction, a second image (IMG2) filtered along a 45-degree diagonal direction, and a second image (IMG2) filtered along a 135-degree diagonal direction.

Looking specifically with reference to FIGS. 10A and 10B, FIG. 10A shows a block diagram for performing the unsharpening filtering algorithm, and FIG. 10B shows an embodiment of a configuration for performing the unsharpening filtering algorithm of FIG. 10A. .

The direction image generator 252 may include an unsharpening algorithm performing unit 252a and an operator 252b. The unsharpening algorithm performing unit 252a receives the white balance-processed input image (IMG_WB) and outputs the filtered result. At this time, filtering may use a first low-pass filter (LPF1) to pass the image signal through the low-pass filter.

The operator 252b may generate the second image IMG2 by calculating the white balance-processed input image IMG_WB and the output of the unsharpening algorithm performing unit 252a. For example, the calculator 252b may generate the second image IMG2 by subtracting the fifth image IMG5, which is the output of the unsharpening algorithm performing unit 252a, from the white balance-processed input image IMG_WB.

Meanwhile, as shown in FIG. 10B, the unsharpening algorithm performing unit 252a may include one or more directional filters (DF1 to DFm), one or more comparators (CP1 to CPm), and an adder (ADD). . At this time, one or more directional filters (DF1 to DFm) may be defined as a directional filter set.

To perform the unsharpening filtering algorithm, one or more directional filters (DF1 to DFm) may be set according to the mask coefficient of the unsharpening filter. For example, the unsharpening filter may have a predetermined size, for example, it may correspond to a filter of size A*A (A is an integer of 2 or more). The total sum of coefficients of one or more directional filters (DF1 to DFm) may be set to be equal to the mask coefficient of the unsharpening filter. Additionally, each of the one or more directional filters DF1 to DFm may have coefficient values for detecting edges in different directions. For example, one or more directional filters DF1 to DFm may have a filter type for detecting edges in at least one of the horizontal direction, vertical direction, 45 degree diagonal direction, and 135 degree diagonal direction.

The size of the unsharpening filter and the size of each of the one or more directional filters (DF1 to DFm) may be the same or different from each other. For example, each of the directional filters DF1 to DFm may correspond to a filter of size A*A (A is an integer equal to or greater than 2). Alternatively, each of the directional filters DF1 to DFm may be smaller than the size of the unsharpening filter. The positions of each of the directional filters DF1 to DFm may be set so that the total sum of coefficients of the directional filters DF1 to DFm is equal to the mask coefficient of the unsharpening filter.

Meanwhile, one or more comparators (CP1 to CPm) may compare output values of one or more directional filters (DF1 to DFm) with preset thresholds (Th1 to Thm) and generate a comparison result. For example, the output of the first directional filter (DF1) and the first threshold (Th1) are provided to the first comparator (CP1), and the first comparator (CP1) provides the output of the first directional filter (DF1) and the first threshold. The result according to the result of comparing the value (Th1) can be provided by an adder (ADD). As an example, the first threshold Th1 may be a value that corresponds to the standard by which the output of the directional filter can be considered sharpening noise, and the first comparator CP1 may be set to the output of the first directional filter DF1 and the first Depending on the result of comparing the threshold value Th1, a value corresponding to “0” may be output, or a value corresponding to the output of the first directional filter DF1 may be provided to the adder ADD. For example, when the output of the first directional filter (DF1) is greater than or equal to the first threshold (Th1), the first comparator (CP1) sets a value corresponding to the output of the first directional filter (DF1) to the adder (ADD). ) can be provided.

One or more directional filters (DF1 to DFm) provide results of filtering the white balanced input image (IMG_WB), respectively. Some of the outputs of the one or more directional filters (DF1 to DFm) may be above the respective thresholds, while other portions of the outputs of the one or more directional filters (DF1 to 410_N) may be above the respective thresholds. It may be smaller than the value. Accordingly, the adder ADD may generate a result of summing only the outputs of some of the one or more directional filters DF1 to DFm. At this time, the threshold values (Th1 to Thm) may be the same or different.

According to the above-described operation, in the directional filter set, the outputs of some directional filters may be selectively summed according to preset thresholds. The output of the calculator 252b (or the second image IMG2) may be generated as a detail band. At this time, the base band may be removed from the detail band, and only the minimum signal that can be sharpened may remain.

The direction image generator 252 according to an embodiment may further include a high-pass filter 252c for detecting the limit resolution of the white balance-processed input image (IMG_WB).

The gray detector 253 may perform white balance processing on the input image (IMG_IN) and detect the gray area of the white balance processed input image (IMG_WB). The gray detection unit 253 may provide gray area information (D2) to the selection unit 257.

The edge detection unit 254 may white balance the input image (IMG_IN) and detect the edge direction of the white balance processed input image (IMG_WB). The edge detection unit 254 according to one embodiment may detect the edge direction of the white balanced input image (IMG_WB) by sequentially calculating the maximum edge value for each preset direction. For example, the edge detection unit 254 determines that the edge direction is vertical when the maximum edge value is detected in the horizontal direction, and determines that the edge direction is horizontal when the maximum edge value is detected in the vertical direction. If the maximum edge value is detected, the edge direction can be determined to be 135 degrees, and if the edge maximum value is detected in the 135 degree direction, the edge direction can be determined to be 45 degrees. The edge detection unit 254 may provide edge direction information D3 of the white balance-processed input image (IMG_WB) to the selection unit 257.

The edge detector 254 according to an embodiment may further include a second low-pass filter (LPF2) to remove noise from the white-balanced input image (IMG_WB).

The selection unit 257 blends the first image (IMG1) and the second image (IMG2) based on the detected gray area information (D2) and the edge direction information (D3) of the white balance-processed input image (IMG_WB). Thus, the third image (IMG3) can be generated.

If the edge strength of the reconstructed green signal (IMG_G) is greater than or equal to a preset value, the selection unit 257 may determine the presence or absence of a gray area in the input image (IMG_WB) white balanced by the gray detection unit 253.

When a gray area exists in the white-balanced input image (IMG_WB), the selection unit 257 performs edge improvement using the second image (IMG2) corresponding to the edge direction determined by the edge detection unit 254, A third image (IMG3) may be generated by blending the first image (IMG1) and the edge-enhanced second image (IMG2).

Meanwhile, the selection unit 257 can determine the presence or absence of a gray area in the input image (IMG_WB) white balanced by the gray detection unit 253 even when the edge strength of the reconstructed green signal (IMG_G) is less than a preset value. there is.

When a gray area exists in the white-balanced input image (IMG_WB), the selection unit 257 performs edge improvement using the second image (IMG2) corresponding to the edge direction determined by the edge detection unit 254, A third image (IMG3) may be generated by blending the reconstructed green signal (IMG_G) and the edge-enhanced second image (IMG2).

The adjuster 258 may adjust the red signal (IMG_R) and the blue signal (IMG_B) based on the amount of change in the green signal of the third image (IMG3).

The Bayer recombiner 259 combines the green signal, the adjusted red signal (IMG_R'), and the adjusted blue signal (IMG_B') of the third image (IMG3) to generate a fourth image (IMG4) having a Bayer pattern array. can do.

According to one embodiment, the direction image generator 252, the gray detector 253, and the edge detector 254 may be implemented as hardware modules arranged in parallel to share memory with the demosaicing unit 251. . Because of this, the image acquisition device 100 can prevent a decrease in calculation speed for improving spatial resolution.

11 to 13 are drawings to explain the effect of the present invention. At this time, FIGS. 11A, 12A, and 13A correspond to the 4-1 image (IMG4-1) on which only the sharpening process was performed after the demosaicing described above in FIG. 8, and FIGS. 11B, 12B, and 13B corresponds to the fourth image (IMG4) that has been subjected to the white balance process described above in FIG. 8 followed by a filtering process using a monochrome signal. Both the 4-1st image (IMG4-1) and the 4th image (IMG4) were obtained by photographing the star illuminance using the image sensor 200 including the color separation lens array 300 of the present invention.

Referring to FIGS. 12A and 13A, which enlarge a partial area of FIG. 11A, it can be seen that in the 4-1 image (IMG4-1), lines arranged at narrow intervals along the circumference of the circle are not clearly distinguished. That is, it can be seen that the maximum spatial resolution area of the 4-1st image (IMG4-1) is suppressed.

On the other hand, referring to the circled areas of FIGS. 12B and 13B, which are enlarged portions of a portion of FIG. 11B, it can be seen that the lines arranged at narrow intervals around the circle are more clearly distinguished. That is, it can be confirmed that the maximum spatial resolution of the fourth image (IMG4) is restored using a monochrome signal.

Hereinafter, other embodiments will be described. In the following embodiments, the description of the same configuration as the already described embodiment will be omitted or simplified, and the differences will be mainly explained.

Figure 14 is a diagram for explaining an embodiment of using a condition check unit during sharpening processing.

Referring to FIGS. 8 and 14, the only difference between the embodiment shown in FIG. 14 is that it includes a condition check unit 255' instead of the facer unit 255 of the embodiment shown in FIG. 8, and the remaining configuration is are substantially the same.

The condition check unit 255' may perform a condition check to determine whether to apply the sharpening filter unit 256 to the reconstructed green signal (IMG_G). For example, the condition checker 255' may calculate the noise level of the reconstructed green signal (IMG_G). The selection unit 257 may apply the sharpening filter unit 256 to the reconstructed green signal (IMG_G) when the noise level calculated by the condition check unit 255' is below a preset threshold.

Figure 15 is a block diagram for performing an unsharpening filtering algorithm according to another embodiment.

Referring to FIGS. 10A and 15 , the embodiment shown in FIG. 15 is different from the embodiment shown in FIG. 10A in that it further includes a multiplier 252d between the unsharpening algorithm performing unit 252a and the operator 252b. There are only differences, but the rest of the configuration is substantially the same.

The direction image generator 252 may include an unsharpening algorithm performing unit 252a, a multiplier 252d, and an operator 252b. The unsharpening algorithm performing unit 252a receives the white balance-processed input image (IMG_WB) and outputs the fifth image (IMG5) as a result of filtering. At this time, filtering may use a first low-pass filter (LPF1) to pass the image signal through the low-pass filter.

The multiplier 252d may multiply the fifth image IMG5 by a gain. At this time, the gain may be a coefficient that determines the strength of the unsharpening mask filter. For example, if the gain is 2, this may mean an operation of doubling the effect of the unsharpening mask filter.

The operator 252b may generate the second' image (IMG2') by calculating the output of the unsharpening algorithm performing unit 252a, which is the white balance processed input image (IMG_WB) multiplied by the gain. For example, the operator 252b subtracts the fifth' image (IMG5'), which is the output of the gain-applied unsharpening algorithm performing unit 252a, from the white balance-processed input image (IMG_WB) to produce the second' image (IMG2'). ) can be created.

Figure 16 is a diagram for explaining the chief ray angle of the image sensor.

Referring to FIGS. 5 and 16 , the incident angle of light incident on the image sensor 200 is typically defined as the chief ray angle (CRA). The chief ray (CR) refers to a ray of light that passes from one point of the subject through the center of the objective lens (FL) (or module lens) and enters the image sensor 200.

The chief ray angle refers to the angle that the chief ray (CR) forms with the optical axis (OPA) of the objective lens (FL), and is generally the same as the angle of incidence of the chief ray (CR) incident on the image sensor 200. For example, the chief ray (CR) of light originating from a point on the optical axis of the objective lens (FL) is incident perpendicularly to the center of the image sensor 200, and in this case, the chief ray angle (CRA_1) is 0 degrees. As the starting point moves away from the optical axis (OPA) of the objective lens (FL), the principal ray angle (CRA_2) increases and enters the edge of the image sensor 200. From the perspective of the image sensor 200, the chief ray angle (CRA_1) of the light incident on the center of the image sensor 200 is 0 degrees, and as the distance from the center of the image sensor 200 increases, the principal ray angle (CRA_2) of the incident light increases. ) becomes larger.

However, the color separation lens array 300 described above may generally have directionality with respect to incident light. In other words, the color separation lens array 300 operates efficiently for light incident within a specific angle range, but when the incident angle moves away from the specific angle range, the color separation performance of the color separation lens array 300 may deteriorate. .

As such, since the luminous efficiency of the color separation lens array 300 decreases in the area where the chief ray angle increases, the image sensor 200 requires image processing (or signal processing) to overcome this. The selection unit (257 in FIG. 8) according to one embodiment may generate the third image IMG3 based on field information of the chief ray angle of the image sensor 200. For example, as the chief ray angle increases, the blur signal generated by the color separation lens array 300 becomes stronger, so the selection unit 257 can increase the degree of improvement in spatial resolution as the chief ray angle increases. The selection unit 257 may receive field information on the chief ray angle from an external device.

Figure 17 is a flowchart explaining a resolution improvement method according to an embodiment. At this time, since the flowchart shown in FIG. 17 corresponds to the operation of the signal processor block diagram shown in FIG. 8, the embodiments described through FIGS. 8 to 16 can be applied.

8 to 10B and 17, the image acquisition method according to one embodiment includes performing demosaicing to reconstruct the green signal to have full pixel resolution using the input image (IMG_IN) (S100). , a sharpening filtering step (S200) of generating a first image (IMG1) by sharpening the reconstructed green signal (IMG_G) in each preset direction, white balancing the input image (IMG_IN) (S300), and performing an unsharpening filtering algorithm. Using the direction image generation step (S400), the base band of the white-balanced input image (IMG_WB) for each preset direction is removed and only the detail band is extracted to generate a second image (IMG2), the input image (IMG_IN) White balance processing (S300), a gray detection step of detecting the gray area of the white balanced input image (IMG_WB) (S500), white balance processing of the input image (IMG_IN) (S300), and white balance processing of the input. Based on the edge detection step (S600) of detecting the edge direction of the image (IMG_WB) and the detected gray area and edge direction, the first image (IMG1) and the second image (IMG2) are blended to create a third image (IMG3) It may include a selection step (S700) for generating.

The image acquisition method according to one embodiment may include an adjustment step (S800) of adjusting the red signal (IMG_R) and the blue signal (IMG_B) based on the amount of change in the green signal of the third image (IMG3).

The image acquisition method according to one embodiment combines the green signal and the adjusted red signal (IMG_R') and blue signal (IMG_B') of the third image (IMG) to produce a fourth image (IMG4) having a Bayer pattern array. It may include a Bayer recombination step (S900).

In the step of performing demosaicing (S100), in order to reduce the capacity of the memory 180 used for calculation, color interpolation is performed only on the green signal of the input image (IMG_IN), and the green signal is reconstructed to have full pixel resolution. (IMG_G) can be created.

In the sharpening filtering step (S200), it includes detecting the edge direction of the reconstructed green signal (IMG_G), and if the edge strength of the reconstructed green signal (IMG_G) is more than a preset value, the edge direction corresponding to the detected edge direction is It may include edge-enhancing the first image (IMG1). At this time, the first image (IMG1) is a first image (IMG1) filtered along the horizontal direction, a first image (IMG1) filtered along a vertical direction, and a first image (IMG1) filtered along a 45-degree diagonal direction. , and a first image (IMG1) filtered along a 135-degree diagonal direction.

In the white balance processing step (S300), the gains of R and B are adjusted centered on G so that the RGB (Red, Green, Blue) ratio of the gray area is the same under various color temperature lighting (e.g., R=G=B). You can.

In the direction image creation step (S400), using an unsharpening filtering algorithm, the base band of the white-balanced input image (IMG_WB) is removed for each preset direction, and a second image (IMG2) is generated from which only the detail band is extracted. can do. At this time, the preset direction may be a horizontal direction, a vertical direction, a 45 degree diagonal direction, and a 135 degree diagonal direction. That is, the second image (IMG2) includes a second image (IMG2_H) filtered along the horizontal direction, a second image (IMG2_V) filtered along a vertical direction, a second image (IMG2_45) filtered along a 45-degree diagonal direction, and a second image (IMG2_135) filtered along a 135-degree diagonal direction.

In the gray detection step (S500), the gray area of the white balance-processed input image (IMG_WB) can be detected.

In the edge detection step (S600), the edge direction of the white balanced input image (IMG_WB) can be detected by sequentially calculating the maximum edge value for each preset direction. For example, the edge detection unit 254 determines that the edge direction is vertical when the maximum edge value is detected in the horizontal direction, and determines that the edge direction is horizontal when the maximum edge value is detected in the vertical direction. If the maximum edge value is detected, the edge direction can be determined to be 135 degrees, and if the edge maximum value is detected in the 135 degree direction, the edge direction can be determined to be 45 degrees.

In the selection step (S700), if the edge strength of the reconstructed green signal (IMG_G) is greater than a preset value, the presence or absence of a gray area in the white-balanced input image (IMG_WB) is determined, and the white-balanced input image (IMG_WB) If a gray area exists within the edge, the edge is improved using the second image (IMG2) corresponding to the determined edge direction, and the first image and the edge-enhanced second image (IMG2) are blended to create a third image (IMG3). It may include a step of generating (S710).

In the selection step (S700), if the edge strength of the reconstructed green signal (IMG_G) is less than a preset value, the presence or absence of a gray area in the white balanced input image (IMG_WB) is determined, and the white balanced input image (IMG_WB) If a gray area exists within the edge, the edge is improved using the second image (IMG2) corresponding to the determined edge direction, and the reconstructed green signal (IMG_G) and the edge-enhanced second image (IMG2) are blended to create a third image. It may include a step (S720) of generating (IMG3).

Figure 18 is a block diagram schematically showing an electronic device including an image sensor according to embodiments. Referring to FIG. 18, in a network environment (ED00), an electronic device (ED01) communicates with another electronic device (ED02) through a first network (ED98) (a short-range wireless communication network, etc.) or a second network (ED99). It is possible to communicate with another electronic device (ED04) and/or a server (ED08) through (a long-distance wireless communication network, etc.). The electronic device ED01 can communicate with the electronic device ED04 through the server ED08. The electronic device (ED01) consists of a processor (ED20), memory (ED30), input device (ED50), audio output device (ED55), display device (ED60), audio module (ED70), sensor module (ED76), and interface (ED77). ), a haptic module (ED79), a camera module (ED80), a power management module (ED88), a battery (ED89), a communication module (ED90), a subscriber identity module (ED96), and/or an antenna module (ED97). You can. In the electronic device ED01, some of these components (such as the display device ED60) may be omitted or other components may be added. Some of these components can be implemented as one integrated circuit. For example, the sensor module ED76 (fingerprint sensor, iris sensor, illumination sensor, etc.) may be implemented by being embedded in the display device ED60 (display, etc.).

The processor ED20 may execute software (program ED40, etc.) to control one or a plurality of other components (hardware, software components, etc.) of the electronic device ED01 connected to the processor ED20. , various data processing or calculations can be performed. As part of data processing or computation, the processor (ED20) loads commands and/or data received from other components (sensor module (ED76), communication module (ED90), etc.) into volatile memory (ED32) and volatile memory (ED32). Commands and/or data stored in ED32) can be processed, and the resulting data can be stored in non-volatile memory (ED34). The processor (ED20) includes the main processor (ED21) (central processing unit, application processor, etc.) and an auxiliary processor (ED23) that can operate independently or together (graphics processing unit, image signal processor, sensor hub processor, communication processor, etc.). It can be included. The auxiliary processor (ED23) uses less power than the main processor (ED21) and can perform specialized functions.

The auxiliary processor (ED23) acts on behalf of the main processor (ED21) while the main processor (ED21) is in an inactive state (sleep state), or as the main processor (ED21) while the main processor (ED21) is in an active state (application execution state). Together with the processor (ED21), it is possible to control functions and/or states related to some of the components of the electronic device (ED01) (display device (ED60), sensor module (ED76), communication module (ED90), etc.) You can. The auxiliary processor (ED23) (image signal processor, communication processor, etc.) may also be implemented as part of other functionally related components (camera module (ED80), communication module (ED90), etc.).

The memory ED30 can store various data required by components (processor ED20, sensor module ED76, etc.) of the electronic device ED01. Data may include, for example, input data and/or output data for software (such as program ED40) and instructions related thereto. The memory ED30 may include a volatile memory ED32 and/or a non-volatile memory ED34.

The program (ED40) may be stored as software in the memory (ED30) and may include an operating system (ED42), middleware (ED44), and/or applications (ED46).

The input device ED50 may receive commands and/or data to be used in components (processor ED20, etc.) of the electronic device ED01 from an external source (a user, etc.) of the electronic device ED01. The input device ED50 may include a microphone, mouse, keyboard, and/or digital pen (stylus pen, etc.).

The audio output device ED55 may output an audio signal to the outside of the electronic device ED01. The sound output device ED55 may include a speaker and/or a receiver. The speaker can be used for general purposes such as multimedia playback or recording playback, and the receiver can be used to receive incoming calls. The receiver can be integrated as part of the speaker or implemented as a separate, independent device.

The display device ED60 can visually provide information to the outside of the electronic device ED01. The display device ED60 may include a display, a hologram device, or a projector, and a control circuit for controlling the device. The display device ED60 may include a touch circuitry configured to detect a touch, and/or a sensor circuit configured to measure the intensity of force generated by the touch (such as a pressure sensor).

The audio module (ED70) can convert sound into electrical signals or, conversely, convert electrical signals into sound. The audio module (ED70) acquires sound through the input device (ED50), the sound output device (ED55), and/or another electronic device (electronic device (ED02), etc.) directly or wirelessly connected to the electronic device (ED01). ) can output sound through speakers and/or headphones.

The sensor module (ED76) detects the operating status (power, temperature, etc.) of the electronic device (ED01) or external environmental status (user status, etc.) and generates electrical signals and/or data values corresponding to the detected status. can do. The sensor module (ED76) includes a gesture sensor, gyro sensor, barometric pressure sensor, magnetic sensor, acceleration sensor, grip sensor, proximity sensor, color sensor, IR (Infrared) sensor, biometric sensor, temperature sensor, humidity sensor, and/or illuminance sensor. May include sensors.

The interface ED77 may support one or a plurality of designated protocols that can be used to directly or wirelessly connect the electronic device ED01 to another electronic device (such as the electronic device ED02). The interface ED77 may include a High Definition Multimedia Interface (HDMI), a Universal Serial Bus (USB) interface, an SD card interface, and/or an audio interface.

The connection terminal ED78 may include a connector through which the electronic device ED01 can be physically connected to another electronic device (such as the electronic device ED02). Connection terminals (ED78) may include an HDMI connector, USB connector, SD card connector, and/or audio connector (headphone connector, etc.)

The haptic module (ED79) can convert electrical signals into mechanical stimulation (vibration, movement, etc.) or electrical stimulation that the user can perceive through tactile or kinesthetic senses. The haptic module (ED79) may include a motor, a piezoelectric element, and/or an electrical stimulation device.

The camera module (ED80) can shoot still images and videos. The camera module ED80 may include a lens assembly including one or a plurality of lenses, the above-described image sensor 200, image signal processors, and/or flashes. The lens assembly included in the camera module (ED80) can collect light emitted from the subject that is the target of image capture.

The power management module ED88 can manage power supplied to the electronic device ED01. The power management module (ED88) may be implemented as part of a Power Management Integrated Circuit (PMIC).

Battery ED89 may supply power to components of electronic device ED01. The battery ED89 may include a non-rechargeable primary cell, a rechargeable secondary cell, and/or a fuel cell.

The communication module (ED90) establishes a direct (wired) communication channel and/or a wireless communication channel between the electronic device (ED01) and other electronic devices (electronic device (ED02), electronic device (ED04), server (ED08), etc.) and can support communication through established communication channels. The communication module (ED90) operates independently of the processor (ED20) (application processor, etc.) and may include one or more communication processors that support direct communication and/or wireless communication. The communication module (ED90) is a wireless communication module (ED92) (cellular communication module, short-range wireless communication module, GNSS (Global Navigation Satellite System, etc.) communication module) and/or a wired communication module (ED94) (LAN (Local Area Network) communication module, power line communication module, etc.). Among these communication modules, the corresponding communication module is the first network (ED98) (a short-range communication network such as Bluetooth, WiFi Direct, or IrDA (Infrared Data Association)) or the second network (ED99) (a cellular network, the Internet, or a computer network (LAN) , WAN, etc.) can communicate with other electronic devices. These various types of communication modules may be integrated into one component (such as a single chip) or may be implemented as a plurality of separate components (multiple chips). The wireless communication module (ED92) uses subscriber information (international mobile subscriber identifier (IMSI), etc.) stored in the subscriber identification module (ED96) to communicate within a communication network such as the first network (ED98) and/or the second network (ED99). You can check and authenticate the electronic device (ED01).

The antenna module (ED97) can transmit signals and/or power to or receive signals and/or power from the outside (such as other electronic devices). The antenna may include a radiator consisting of a conductive pattern formed on a substrate (PCB, etc.). The antenna module ED97 may include one or multiple antennas. When a plurality of antennas are included, an antenna suitable for a communication method used in a communication network such as the first network (ED98) and/or the second network (ED99) may be selected from among the plurality of antennas by the communication module (ED90). You can. Signals and/or power may be transmitted or received between the communication module ED90 and other electronic devices through the selected antenna. In addition to the antenna, other components (RFIC, etc.) may be included as part of the antenna module (ED97).

Some of the components are connected to each other through communication methods between peripheral devices (bus, GPIO (General Purpose Input and Output), SPI (Serial Peripheral Interface), MIPI (Mobile Industry Processor Interface), etc.) and signals (commands, data, etc.) ) can be interchanged.

Commands or data may be transmitted or received between the electronic device ED01 and an external electronic device ED04 through the server ED08 connected to the second network ED99. The other electronic devices ED02 and ED04 may be the same or different types of devices from the electronic device ED01. All or part of the operations performed in the electronic device ED01 may be executed in one or more of the other electronic devices ED02, ED04, and ED08. For example, when the electronic device (ED01) needs to perform a certain function or service, instead of executing the function or service itself, it performs part or all of the function or service to one or more other electronic devices. You can ask them to do it. One or more other electronic devices that have received the request may execute additional functions or services related to the request and transmit the results of the execution to the electronic device ED01. For this purpose, cloud computing, distributed computing, and/or client-server computing technologies may be used.

FIG. 19 is a block diagram illustrating the camera module ED80 provided in the electronic device of FIG. 18. Referring to FIG. 19, the camera module (ED80) includes a lens assembly 1170, a flash 1120, an image sensor 200, an image stabilizer 1140, an AF control unit 1130, and a memory 1150 (buffer memory, etc.) , may include an actuator 1180 and/or an image signal processor (ISP) 1160.

The lens assembly 1170 may collect light emitted from a subject that is the target of image capture. Lens assembly 1170 may include one or more optical lenses. The lens assembly 1170 may include a path switching member that bends the path of light and directs it toward the image sensor 200. Depending on the presence of a path switching member and the arrangement of the optical lens, the camera module ED80 may have a vertical shape or a folded shape. The camera module ED80 may include a plurality of lens assemblies 1170. In this case, the camera module ED80 may be a dual camera, a 360-degree camera, or a spherical camera. Some of the plurality of lens assemblies 1170 may have the same lens properties (angle of view, focal length, autofocus, F number, optical zoom, etc.) or may have different lens properties. The lens assembly 1170 may include a wide-angle lens or a telephoto lens.

The actuator 1180 may drive the lens assembly 1170. For example, at least some of the optical lenses and path switching members constituting the lens assembly 1170 can be moved by the actuator 1180. The optical lens may move along the optical axis, and the distance between adjacent lenses may be adjusted by moving at least a portion of the optical lens included in the lens assembly 1170, thereby adjusting the optical zoom ratio.

The actuator 1180 may adjust the position of one optical lens included in the lens assembly 1170 so that the image sensor 200 is located at the focal length of the lens assembly 1170. The actuator 1180 may drive the lens assembly 1170 according to the AF driving signal transmitted from the AF control unit 1130.

The flash 1120 may emit light used to enhance light emitted or reflected from a subject. Flash 1120 may emit visible light or infrared light. The flash 1120 may include one or a plurality of light emitting diodes (RGB (Red-Green-Blue) LED, White LED, Infrared LED, Ultraviolet LED, etc.), and/or a Xenon Lamp. The image sensor 200 may be the image sensor 200 described in FIG. 1, or one of the image sensors 1001-1012 described above, or a combination or modified form thereof. The image sensor 200 converts light emitted or reflected from a subject and transmitted through the lens assembly 1170 into an electrical signal, thereby obtaining an image corresponding to the subject.

The image sensor 200 is equipped with the color separation lens array 300 described above, and each pixel may include a plurality of photo-sensing cells forming a plurality of channels, for example, a plurality of photo-sensing cells arranged in 2X2. You can. Some of these pixels may be used as AF pixels, and the image sensor 200 may generate AF driving signals from signals of multiple channels within the AF pixels.

The image stabilizer 1140 moves one or more lenses or image sensors 200 included in the lens assembly 1170 in a specific direction in response to the movement of the camera module (ED80) or the electronic device (ED01) including it. Alternatively, the operation characteristics of the image sensor 200 can be controlled (adjustment of read-out timing, etc.) to compensate for the negative effects of movement. The image stabilizer 1140 detects the movement of the camera module (ED80) or the electronic device (ED01) using a gyro sensor (not shown) or an acceleration sensor (not shown) placed inside or outside the camera module (ED80). You can. The image stabilizer 1140 may be implemented optically.

The AF control unit 1130 may generate an AF driving signal from a signal value sensed from the AF pixel of the image sensor 200. The AF control unit 1130 may control the actuator 1180 according to the AF driving signal.

The memory 1150 may store part or all data of the image acquired through the image sensor 200 for the next image processing task. For example, when multiple images are acquired at high speed, the acquired original data (Bayer-Patterned data, high-resolution data, etc.) is stored in the memory 1150, only the low-resolution images are displayed, and then the selected (user selection, etc.) It can be used to ensure that the original data of the image is transmitted to the image signal processor 1160. The memory 1150 may be integrated into the memory ED30 of the electronic device ED01, or may be configured as a separate memory that operates independently.

The image signal processor (ISP) 1160 may perform image processing on images acquired through the image sensor 200 or image data stored in the memory 1150. Image processing includes depth map creation, 3D modeling, panorama creation, feature point extraction, image compositing, and/or image compensation (noise reduction, resolution adjustment, brightness adjustment, blurring, sharpening) , softening, etc.). The image signal processor 1160 may perform control (exposure time control, lead-out timing control, etc.) on components (such as the image sensor 200) included in the camera module ED80. Images processed by the image signal processor 1160 are stored back in the memory 1150 for further processing or stored in external components of the camera module (ED80) (memory (ED30), display device (ED60), electronic device (ED02) , electronic device (ED04), server (ED08), etc.). The image signal processor 1160 may be integrated into the processor ED20 or may be configured as a separate processor that operates independently from the processor ED20. When the image signal processor 1160 is configured as a separate processor from the processor ED20, the image processed by the image signal processor 1160 undergoes additional image processing by the processor ED20 and then is displayed on the display device ED60. It can be displayed through

The AF control unit 1130 may be integrated into the image signal processor 1160. The image signal processor 1160 processes signals from the autofocusing pixels of the image sensor 200 to generate an AF signal, and the AF control unit 1130 converts this into an actuator 1180 driving signal and transmits it to the actuator 1180. It may be possible.

The image sensor 200 according to embodiments may be applied to various electronic devices.

The image sensor 200 according to embodiments may be applied to a mobile phone or smart phone, a tablet or smart tablet, a digital camera or camcorder, a laptop computer, a television, or a smart television. For example, a smartphone or smart tablet may include a plurality of high-resolution cameras, each equipped with a high-resolution image sensor. Using high-resolution cameras, you can extract depth information of subjects in an image, adjust the outfocusing of an image, or automatically identify subjects in an image.

Additionally, the image sensor 200 may be applied to smart refrigerators, security cameras, robots, medical cameras, etc. For example, a smart refrigerator can automatically recognize the food in the refrigerator using an image sensor and inform the user of the presence or absence of specific food and the type of food received or shipped through the smartphone. Security cameras can provide ultra-high resolution images and, using high sensitivity, can recognize objects or people in the images even in dark environments. Robots are deployed in disaster or industrial sites that cannot be accessed directly by humans, providing high resolution. Medical cameras can provide high-resolution images for diagnosis or surgery and can dynamically adjust their field of view.

Additionally, the image sensor 200 may be applied to a vehicle. The vehicle may include a plurality of vehicle cameras arranged at various locations. Each vehicle camera may include an image sensor according to an embodiment. The vehicle uses a plurality of vehicle cameras to provide information about the interior or surroundings of the vehicle. Various information can be provided to the driver, and information necessary for autonomous driving can be provided by automatically recognizing objects or people in the video.

The image sensor having the above-described color separation lens array and the image acquisition device including the same have been described with reference to the embodiment shown in the drawings, but this is merely an example, and those skilled in the art will understand various It will be understood that variations and equivalent other embodiments are possible. Therefore, the disclosed embodiments should be considered from an illustrative rather than a restrictive perspective. The scope of rights is indicated in the patent claims, not the foregoing description, and all differences within the equivalent scope should be interpreted as being included in the scope of rights.

100: Image acquisition device
120: Shooting lens unit
200: image sensor
210: pixel array
250: signal processing unit
251: demosaicing unit
252: Direction image creation unit
253: Gray detection unit
254: Edge detection unit
255: Pacerbu
256: Sharpening filter unit
257: selection part
258: Control department
259: Bayer recombination unit
300: Color separation lens array
500: sensor substrate

Claims (20)

An image sensor including a color separation lens array that separates and focuses incident light according to color; and
Includes a signal processor that processes the input image obtained from the image sensor,
The signal processing unit,
a demosaicing unit that reconstructs the green signal to have full pixel resolution using the input image;
a sharpening filter unit that generates a first image by sharpening the reconstructed green signal in each preset direction;
Generating a direction image by white balancing the input image and using an unsharpening filtering algorithm to generate a second image in which the base band of the white balance processed input image is removed for each preset direction and only the detail band is extracted. wealth;
a gray detector that white balances the input image and detects a gray area of the white balanced input image;
an edge detection unit that white balances the input image and detects an edge direction of the white balanced input image; and
An image acquisition device comprising: a selection unit that generates a third image by blending the first image and the second image based on the detected gray area and the edge direction. According to claim 1,
The image sensor is,
A sensor substrate including a plurality of light-sensing cells that sense light; and
The color separation device is disposed at the front end of the sensor substrate and includes a microstructure that forms a phase distribution that focuses light of different wavelengths on adjacent photosensing cells in a plurality of areas facing each of the plurality of photosensing cells. An image acquisition device comprising a lens array. According to claim 1,
The image sensor is an image acquisition device having a Bayer pattern array. According to claim 1,
The signal processing unit is an image acquisition device including an adjusting unit that adjusts the red signal and the blue signal based on the amount of change in the green signal of the third image. According to clause 4,
An image acquisition device comprising a Bayer recombination unit that combines the green signal of the third image and the adjusted red and blue signals to generate a fourth image having a Bayer pattern array. According to claim 1,
The signal processing unit is an image acquisition device including a pacer unit that detects an edge direction of the reconstructed green signal. According to clause 6,
The selection unit, if the edge strength of the reconstructed green signal is greater than a preset value, edge improves the first image corresponding to the edge direction detected by the pacer unit. According to clause 7,
The selection unit determines the presence or absence of a gray area in the input image white balanced by the gray detection unit if the edge strength of the reconstructed green signal is greater than a preset value. According to clause 8,
If a gray area exists in the white balance-processed input image, the selection unit performs edge improvement using the second image corresponding to the edge direction determined by the edge detection unit, and selects the first image and the edge-enhanced image. An image acquisition device that generates the third image by blending the second image. According to clause 7,
The selection unit determines the presence or absence of a gray area in the input image white balanced by the gray detection unit if the edge strength of the reconstructed green signal is less than a preset value. According to claim 10,
If a gray area exists in the white balanced input image, the selection unit performs edge improvement using the second image corresponding to the edge direction determined by the edge detection unit, and the reconstructed green signal and the edge improvement An image acquisition device that generates the third image by blending the second image. According to claim 1,
The preset direction includes horizontal, vertical, 45 degrees, and 135 degrees. According to claim 1,
The direction image generator,
a low-pass filter that generates a fifth image by low-frequency filtering the white-balanced input image; and
An image acquisition device comprising: an operator that receives the white balance processed input image and the fifth image, and subtracts the fifth image from the white balance processed input image. According to claim 13,
The direction image generator is disposed between the low-pass filter and the operator and further includes a multiplier that multiplies the fifth image by a gain. According to claim 13,
The direction image generator further includes a high-pass filter for detecting a limiting resolution of the white balance-processed input image. According to claim 1,
The edge detection unit includes a low-pass filter to remove noise from the white-balanced input image. According to claim 1,
The selection unit is configured to generate the third image based on field information of the chief ray angle of the image sensor. According to claim 1,
The direction image generation unit, the gray detection unit, and the edge detection unit are arranged in parallel to share memory with the demosaicing unit. An image acquisition device including an image sensor including a color separation lens array that separates and focuses incident light according to color,
performing demosaicing to reconstruct the green signal to have full pixel resolution using the input image;
A sharpening filtering step of generating a first image by sharpening the reconstructed green signal in each preset direction;
Generating a direction image by white balancing the input image and using an unsharpening filtering algorithm to generate a second image in which the base band of the white balance processed input image is removed for each preset direction and only the detail band is extracted. step;
A gray detection step of white balancing the input image and detecting a gray area of the white balance processed input image;
An edge detection step of white-balancing the input image and detecting an edge direction of the white-balanced input image; and
A selection step of generating a third image by blending the first image and the second image based on the detected gray area and the edge direction. According to clause 19,
An image acquisition method comprising an adjustment step of adjusting a red signal and a blue signal based on the amount of change in the green signal of the third image.

Images