TWI599809B - Lens module array, image sensing device and fusing method for digital zoomed images - Google Patents

Description

Lens module array, image sensing device and digital zoom image fusion method

The invention relates to a lens module array, and in particular to a lens module array, an image sensing device and a digital zoom image fusion method with four lens modules.

With the development of technology, a variety of intelligent electronic devices, such as tablets and smart phones, have become indispensable tools for modern people. Among them, the image quality of the camera lens mounted on the high-end smart electronic device has been comparable to that of the traditional consumer camera, and can even be replaced. A few high-end smart electronic devices are equipped with camera lenses that are close to digital monocular pixels and image quality, even with dual lenses and the ability to capture 3D images.

For the above-mentioned electronic device equipped with a dual lens, a common application is to take a wide-angle image by using a wide-angle lens of a dual lens, and to take a narrow-angle image by using another telephoto lens, and then according to the magnification. The magnification selects a wide-angle image or a narrow-angle image as the target image, and performs digital zooming of the single-labeled image to simulate the optical zoom function. However, after the image is digitally enlarged, the details of the image are often not preserved, and the higher the magnification, the more blurred the image is. During the continuous zooming of the image, if the target image has to be replaced with a wide-angle image or a narrow-angle image, the image may be unsmooth or jump.

The invention provides a lens module array, an image sensing device and a digital zoom image fusion method, which are used to solve the problem that the traditional dual lens module cannot balance the image quality and the size.

The lens module array of the present invention is mounted on a portable device. The lens module array includes a wide-angle monocular lens module, a narrow-angle monochrome lens module, and two color lens modules. When the wide-angle monochrome lens module, the narrow-angle monochrome lens module and the color lens module are mounted on the portable device, the wide-angle monochrome lens module, the narrow-angle monochrome lens module and the color lens module are respectively located at one The four vertices of the quadrilateral, and the color lens module is located at the opposite two vertices.

The image sensing device of the present invention comprises an image processing module, a wide-angle monochrome lens module, a narrow-angle monochrome lens module and two color lens modules. When the wide-angle monochrome lens module, the narrow-angle monochrome lens module and the color lens module are mounted on the portable device, the wide-angle monochrome lens module, the narrow-angle monochrome lens module and the color lens module are respectively located at one The four vertices of the quadrilateral, and the color lens module is located at the opposite two vertices. The image processing module is used for digitally scaling the images captured by the wide-angle monochrome lens module, the narrow-angle monochrome lens module, and the color lens module, and merging into a composite image.

The digital zoom image fusion method of the present invention comprises the following steps. The image is captured by a wide-angle monochrome lens module, a narrow-angle monochrome lens module, and two color lens modules. The wide-angle monochrome lens module, the narrow-angle monochrome lens module and the two color lens modules are respectively located at four vertices of a quadrilateral, and the two color lens modules are located at two opposite vertices. Then, the image captured by the wide-angle monochrome lens module, the narrow-angle monochrome lens module, and the two color lens modules is integrated into a composite image.

In an embodiment of the digital zoom image fusion method of the present invention, the method of integrating the synthesized image includes the following steps. According to the scaling ratio, the monochrome image captured by the wide-angle monochrome lens module and the narrow-angle monochrome lens module is a digitally-scaled monochrome image. Then, the digital zoom monochrome image and the color image captured by the two color lens modules are merged into a color composite image conforming to the scale.

In an embodiment of the digital zoom image fusion method of the present invention, the method of fusing the digital zoom monochrome image with the color image captured by the color lens module includes the following steps. Pixel position calibration is performed on the digital image captured by the digital zoom monochrome image and the color lens module, respectively. Then, based on the digitally-scaled monochrome image, the color information in the color image captured by the color lens module is filled into the pixel at the corresponding position in the digitally-scaled monochrome image to obtain a composite image.

In an embodiment of the invention, the maximum angle of view of the wide-angle monochrome lens module is between 70 degrees and 80 degrees, and the maximum angle of the narrow angle monochrome lens module is between 35 degrees and 40 degrees. Between degrees.

In an embodiment of the invention, the color lens module has a maximum angle of view between 70 degrees and 80 degrees.

In an embodiment of the invention, the quadrilateral may be square, rectangular, rhombic, scorpion or the like.

In an embodiment of the invention, the wide-angle monochrome lens module has the same amount of pixels as the color lens module.

In an embodiment of the invention, the image processing module includes a storage unit and a processing unit, and the storage unit is coupled to the processing unit.

In an embodiment of the invention, the processing unit includes an image input module, an image pre-processing module, a feature point analysis module, an image scaling deformation module, an image fusion module, and a coloring mode. group.

Based on the above, the lens module array, the image sensing device, and the digital zoom image fusion method of the present invention adopt a wide-angle monochrome lens module and a narrow-angle monochrome lens module, so that the sensing elements of the same area can be used. Simulate a narrower angle optical zoom effect and avoid an increase in overall thickness.

The above described features and advantages of the invention will be apparent from the following description.

The components of the present invention will be described in detail in the following description in conjunction with the accompanying drawings. These examples are only a part of the invention and do not disclose all of the embodiments of the invention. Rather, these embodiments are merely examples of devices in the scope of the patent application of the present invention.

1 is a schematic diagram of a lens module array mounted on a portable device according to an embodiment of the present invention, but is for convenience of description and is not intended to limit the present invention. Wherein, the lens module array is presented in a manner indicating the physical position of the four lens modules, and the remaining portions are indicated by functional blocks.

Referring to FIG. 1 , the lens module array 100 of the present embodiment is mounted on the portable device 50 . In this embodiment, the portable device 50 is, for example, a digital camera, a monocular camera, a digital camera, a network camera, or other smart phone, tablet computer, personal digital assistant, head mounted display, etc. with image capturing function. The electronic device is not limited to the invention.

The lens module array 100 includes a wide-angle monochrome lens module 110, a narrow-angle monochrome lens module 120, and two color lens modules 130A and 130B. After the wide-angle monochrome lens module 110, the narrow-angle monochrome lens module 120, and the color lens modules 130A and 130B are mounted on the portable device 50, the wide-angle monochrome lens module 110 and the narrow-angle monochrome lens module 120 The color lens modules 130A and 130B are respectively located at four vertices S12 of a quadrilateral S10, and the color lens modules 130A and 130B are located at opposite vertices S12. Here, the maximum angle of view of the wide-angle monochrome lens module 110 is greater than the maximum angle of view of the narrow-angle monochrome lens module 120. On the other hand, the narrow-angle monochrome lens module 120 can also be referred to as a telephoto lens module.

All the pixels of the wide-angle monochrome lens module 110 are used to sense and record the total brightness of the light entering the pixel. All the pixels of the narrow-angle monochrome lens module 120 are also used for sensing and recording. The intensity of the total brightness of the light entering the pixel. In one embodiment, the wide-angle monochrome lens module 110 and the narrow-angle monochrome lens module 120 may be black and white lens modules, that is, the intensity of light recorded regardless of the wavelength range. On the contrary, all the pixels of the color lens modules 130A and 130B include pixels for sensing and recording the light intensity of various wavelength ranges, for example, some pixels are used to sense and record the intensity of red light, and some pixels are used for some pixels. Inductively and record the intensity of green light, some pixels are used to sense and record the intensity of blue light. Of course, the light used by the color lens modules 130A and 130B for sensing and recording is not limited to being divided into red light, green light, and blue light, and may be other combinations as long as it is suitable for providing full color image recording.

Compared with the conventional two-lens module, the wide-angle color lens module and the narrow-angle color lens module, the wide-angle monochrome lens module 110 and the narrow-angle monochrome lens module 120 of the present embodiment have more resolution. High because the pixels do not need to be grouped to sense different shades of light. On the other hand, the monochrome lens module has less color filter film than the color lens module, so the amount of light entering is also large, which can effectively suppress noise generation even in a low light source environment, and improve the signal-to-noise ratio to provide clarity. Image. Because of this, the narrow-angle monochrome lens module 120 can provide the same resolution as the conventional narrow-angle color lens module in a thinner size under the same area of the photosensitive element, thereby reducing the lens mode of the embodiment. The overall thickness of the array 100.

Other alternative designs of the lens module array 100 of the present embodiment are described below, but the present invention is not limited thereto. In this embodiment, the maximum angle of view of the wide-angle monochrome lens module 110 is between 70 degrees and 80 degrees, and the maximum angle of the narrow-angle monochrome lens module 120 is between 35 degrees and 40 degrees. between. The maximum viewable angle refers to the angle between the leftmost orientation in the horizontal direction and the rightmost orientation of the image that can be captured by the lens module. When the same pixel is used, the image captured by the wide-angle monochrome lens module 110 has a larger maximum angle of view and can capture a wider image. The image captured by the narrow-angle monochrome lens module 120 is captured. The maximum possible angle of view provides more image detail. By integrating the images captured by the wide-angle monochrome lens module 110 and the narrow-angle monochrome lens module 120, a digital zoom function can be provided to simulate the optical zoom. Later, we will briefly describe how to integrate images. In addition, the two color lens modules 130A and 130B function to provide color information to the integrated image. In order to provide the color information required for the widest image, the maximum angle of view of the color lens modules 130A and 130B can be selected to be the same as the maximum angle of the wide-angle monochrome lens module 110, for example, 70 degrees. Between 80 degrees. Furthermore, the pixel amount of the color lens modules 130A and 130B and the pixel amount of the wide-angle monochrome lens module 110 may be the same. In addition, in order to facilitate the integration of the images of the four lens modules, the above-mentioned quadrilateral may be a square, a rectangle, a diamond, a dome or the like.

The wide-angle monochrome lens module 110, the narrow-angle monochrome lens module 120, and the color lens modules 130A and 130B, respectively, may include a photosensitive element for sensing light intensity to generate an image. The photosensitive element is, for example, a Charge Coupled Device (CCD) or a Complementary Metal-Oxide Semiconductor (CMOS) element, and the present invention is not limited thereto. The wide-angle monochrome lens module 110, the narrow-angle monochrome lens module 120, and the color lens modules 130A and 130B are used to capture an image of a subject, wherein the subject may be a specific object or a scene.

Referring to FIG. 1 , the portable device 50 can include an image processing module 51 . The image processing module 51 is configured to integrate the images captured by the wide-angle monochrome lens module 110, the narrow-angle monochrome lens module 120, and the color lens modules 130A and 130B into a composite image. The image processing module 51 of this embodiment includes a storage unit 52 and a processing unit 54. The storage unit 52 is, for example, any type of fixed or removable random access memory (RAM), read-only memory (ROM), flash memory, and hard memory. Disc or other similar device or a combination of these devices. The storage unit 52 is coupled to the lens module array 100 for storing images captured by the wide-angle monochrome lens module 110, the narrow-angle monochrome lens module 120, and the color lens modules 130A and 130B.

The processing unit 54 can be a central processing unit (CPU), or other programmable general purpose or special purpose microprocessor (Microprocessor), digital signal processor (DSP), programmable Programmable Controller, Application Specific Integrated Circuits (ASIC), Programmable Logic Device (PLD) or other similar devices or a combination of these devices.

The storage unit 52 is coupled to the lens module array 100 and the processing unit 54 for obtaining complete depth information of the subject by using the images captured by the wide-angle monochrome lens module 110 and the narrow-angle monochrome lens module 120. The processing unit 54 includes, for example, an image input module 541, an image pre-processing module 542, a feature point analysis module 543, an image scaling deformation module 544, an image fusion module 545, and a coloring module 546, which can be loaded into the storage unit 52. , thus performing the function of digital zoom. The following is a detailed description of the detailed steps of performing a digital zoom method for the portable device 50.

FIG. 2 is a flowchart of a digital zoom image fusion method according to an embodiment of the present invention. The digital zoom image fusion is performed by using the electronic device of FIG. 1 as an example. Referring to FIG. 1 and FIG. 2 simultaneously, first, the storage unit 52 of the portable device 50 controls the wide-angle monochrome lens module 110, the narrow-angle monochrome lens module 120, and the color lens module 130A of the lens module array 100. 130B captures images of a scene to generate four images and inputs them to the image input module 541, step S110. In this embodiment, the monochrome main image captured by the wide-angle monochrome lens module 110 has a low image quality but covers a large viewing angle range. The monochrome sub-image captured by the narrow-angle monochrome lens module 120 has a higher image quality, but covers a smaller range of viewing angles, and is used as an auxiliary for digital zoom in subsequent steps.

Then, the wide-angle monochrome lens module 110, the narrow-angle monochrome lens module 120, and the color lens module capture the images of 130A and 130B into a composite image. For example, the image pre-processing module 542 performs image correction processing on the monochrome main image, the monochrome sub-image, and the two color images to generate a monochrome main corrected image, a monochrome secondary corrected image, and two color corrected images. In detail, the image pre-processing module 542 will respectively generate brightness, color and geometric position of the four images for the wide-angle monochrome lens module 110, the narrow-angle monochrome lens module 120 and the color lens modules 130A and 130B, respectively. The deviation on the correction is corrected.

In this embodiment, the image pre-processing module 542 obtains a plurality of correction parameters associated with the wide-angle monochrome lens module 110, the narrow-angle monochrome lens module 120, and the color lens modules 130A and 130B from the storage unit 52. The correction parameters may be parameters such as intrinsic parameters and extrinsic parameters of the lens module array 100 for image rectification. The internal parameters can be used to describe the conversion relationship between the camera module array 100 camera coordinates and the image coordinates, that is, the lens module array 100 coordinates are projected onto the imaging plane by the pinhole imaging principle ( Projective plane). For example, internal parameters include parameters such as focal length, image center, principal point, and lens distortion coefficient. The external parameters are used to describe the conversion relationship between the world coordinate system and the camera coordinate system of the lens module array 100. For example, the position and the shooting direction of the portable device 50 in the three-dimensional coordinates, including a rotation matrix and a translation vector, and the like, are related to the position of the portable device 50 and the shooting direction. In addition, the calibration parameter may also be related parameters such as illumination compensation or color correction, and the present invention is not limited thereto. The image pre-processing module 542 will correct the monochrome main image, the monochrome sub-image and the two color images according to the above-mentioned correction parameters for image correction.

Then, the feature point analysis module 543 performs feature point detection on the monochrome primary corrected image and the monochrome secondary corrected image, thereby detecting overlapping regions of the monochrome primary corrected image and the monochrome secondary corrected image, thereby obtaining the overlapping region. Pixel displacement characteristics and pixel depth information. In detail, the feature point analysis module 543 can utilize edge detection, corner detection, blob detection or other feature-points detection algorithms. To detect multiple feature points of the monochrome primary corrected image and the monochrome secondary corrected image. Next, the feature point analysis module 543 finds a corresponding combination of the same characteristic feature points from the feature points, and distinguishes overlapping regions in the monochrome main corrected image and the monochrome secondary corrected image, respectively. The feature point analysis module 543 can calculate the depth of field for each pixel of the overlapping area of the wide-angle monochrome lens module 110 and the narrow-angle monochrome lens module 120 to obtain depth information of the overlapping area, and is, for example, a depth map. Form record of (depth map).

Then, the image is scaled and deformed according to the command input by the user, that is, the zoom ratio is switched, step S120. When the zoom magnification is between 1 and the monochrome primary and secondary image magnifications, the image scaling deformation module 544 respectively corrects the image for the monochrome primary image according to the zoom magnification, each of the pixel displacement characteristics, and each of the pixel depth information. The color secondary correction image and the two color corrected images perform image zooming and image warping to generate a monochrome main deformation image, a monochrome secondary deformation image, and two color deformation images. Here, the "monochromatic primary and secondary image magnification" is the size ratio between the monochrome primary corrected image and the monochrome secondary corrected image, which is fixed and known in advance, and the zooming magnification is intended for the user to view or output. The size of the main screen varies, which may be set by the user or may be a preset value of the portable device 50. The image scaling deformation module 544 can utilize the relative displacement, skew and depth characteristics of the two overlapping regions, and performs image scaling processing on the monochrome primary corrected image, the monochrome secondary corrected image, and the two color corrected images respectively according to the zooming magnification. And image deformation processing to produce an image that meets the user's zoom magnification requirements and overlaps the area of view with the same appearance. In addition, the magnitude of the image deformation is related to the depth information of the two overlapping regions.

Thereafter, the image fusion module 545 performs image blending on the overlapping area of the monochrome main deformation image and the monochrome sub-deformation image according to the zoom magnification to generate a monochrome digital zoom image, step S130. In detail, the image fusion module 545 can set the weights required for image fusion in the overlapping regions of the monochrome main deformation image and the monochrome sub-deformation image according to the zoom magnification, and is respectively defined as "the first weight set (First Weight-set )" and "Second weight-set". Next, the image fusion module 545 will weight the mixture ratio of the pixel gray levels of the two overlapping regions with the first weight set and the second weight set, and perform image fusion processing on the two overlapping regions. Here, the result produced by the image fusion processing is defined as "fusion overlapping area". Thereafter, the image fusion module 545 replaces the overlapping regions in the original monochromatic main deformation image with the fused overlapping regions, thereby generating a high-quality digital zoom monochrome image.

In another embodiment, when the zoom magnification is less than 1, the digital reduction processing and the subsequent image deformation processing are performed only on the monochrome main corrected image and the two color corrected images to generate a monochrome main deformation image and two colors. Deform the image and directly set the monochrome main deformation image to the digital zoom monochrome image. On the other hand, when the zoom magnification is greater than the monochrome primary and secondary image magnifications, only the monochrome secondary correction image and the two color corrected images are subjected to digital amplification processing and subsequent image deformation processing to generate a monochrome secondary deformation image and two The color deformation image is displayed, and the monochrome sub-deformation image is directly set as the digital zoom monochrome image.

Next, the coloring module 546 first performs pixel position alignment (Pixel Alignment) on the two color deformation images and the fused digital zoom monochrome image targets. Then, based on the digital zoom monochrome image, the color information missing from each monochrome pixel is found from the two color deformation images, and the pixel color of the corresponding position is merged and filled in the monochrome pixel position to obtain A composite image, such as a digital zoom color image, is step S140. Finally, the digital zoom color image is output, for example, a screen (not shown) of the portable device 50 is used to display the digital zoom color image finally obtained after the foregoing steps are completed, step S150.

In summary, the present invention uses the image captured by the wide-angle monochrome lens module and the narrow-angle monochrome lens module as the basis of the digital zoom image, so that the image resolution and the signal-to-noise ratio can be greatly improved. At the same time, due to the improved resolution, a high-power optical zoom effect can be achieved with a thinner narrow-angle monochrome lens module under the condition of the same area of the sensing element. The color lens module is used to obtain the desired color information, and the digital zoom image can be colored in the coloring module. In addition, the present invention adopts two color lens modules and is located on both sides of the monochrome main and sub-lens. In the middle of the coloring module, the Occlusion problem often encountered by the conventional dual lens module can be solved or greatly improved.

Although the present invention has been disclosed in the above embodiments, it is not intended to limit the present invention, and any one of ordinary skill in the art can make some changes and refinements without departing from the spirit and scope of the present invention. The scope of the invention is defined by the scope of the appended claims.

50‧‧‧ portable device
51‧‧‧Image Processing Module
52‧‧‧ storage unit
54‧‧‧Processing unit
541‧‧‧Image input module
542‧‧•Image pre-processing module
543‧‧‧Feature Point Analysis Module
544‧‧‧Image scaling deformation module
545‧‧‧Image Fusion Module
546‧‧‧Coloring module
100‧‧‧ lens module array
110‧‧‧ Wide-angle monochrome lens module
120‧‧‧Narrow-angle monochrome lens module
130A, 130B‧‧‧ color lens module
S10‧‧‧ quadrilateral
S12‧‧‧ vertex
S110-S150‧‧‧Steps

1 is a schematic diagram of a lens module array mounted on a portable device according to an embodiment of the invention. FIG. 2 is a flowchart of a digital zoom image fusion method according to an embodiment of the present invention.

50‧‧‧ portable device

51‧‧‧Image Processing Module

52‧‧‧ storage unit

54‧‧‧Processing unit

541‧‧‧Image input module

542‧‧•Image pre-processing module

543‧‧‧Feature Point Analysis Module

544‧‧‧Image scaling deformation module

545‧‧‧Image Fusion Module

546‧‧‧Coloring module

100‧‧‧ lens module array

110‧‧‧ Wide-angle monochrome lens module

120‧‧‧Narrow-angle monochrome lens module

130A, 130B‧‧‧ color lens module

S10‧‧‧ quadrilateral

S12‧‧‧ vertex

Claims (15)

An array of lens modules for mounting on a portable device, the lens module array comprising: a wide-angle monochrome lens module; a narrow-angle monochrome lens module; and two color lens modules, wherein The wide-angle monochrome lens module, the narrow-angle monochrome lens module, and the color lens modules are respectively disposed on the portable device, and are respectively located at four vertices of a quadrilateral, and the color lens modules are Located at the opposite two vertices. The lens module array according to claim 1, wherein the wide-angle monochrome lens module has a maximum imageable angle of between 70 degrees and 80 degrees, and the maximum angle of the narrow-angle monochrome lens module is The angle is between 35 and 40 degrees. The lens module array of claim 2, wherein the color lens modules have a maximum angle of view between 70 degrees and 80 degrees. The lens module array of claim 1, wherein the quadrilateral may be square, rectangular, diamond or 鸢. The lens module array of claim 1, wherein the wide-angle monochrome lens module has the same amount of pixels as the color lens modules. An image sensing device includes: an image processing module; a wide-angle monochrome lens module; a narrow-angle monochrome lens module; and two color lens modules, wherein the wide-angle monochrome lens module, the When the narrow-angle monochrome lens module and the color lens module are mounted on a portable device, they are respectively located at four vertices of a quadrilateral, and the color lens modules are located at two opposite vertices, and the image processing is performed. The module is configured to integrate the wide-angle monochrome lens module, the narrow-angle monochrome lens module, and the images captured by the color lens modules into a composite image. The image sensing device of claim 6, wherein the wide-angle monochrome lens module has a maximum imageable angle between 70 degrees and 80 degrees, and the maximum angle of the narrow-angle monochrome lens module is The angle is between 35 and 40 degrees. The image sensing device of claim 7, wherein the color lens modules have a maximum angle of view between 70 degrees and 80 degrees. The image sensing device of claim 6, wherein the quadrilateral may be square, rectangular, rhombic or dome shaped. The image sensing device of claim 6, wherein the wide-angle monochrome lens module has the same amount of pixels as the color lens modules. The image sensing device of claim 6, wherein the image processing module comprises a storage unit and a processing unit, the storage unit being coupled to the processing unit. The image sensing device of claim 11, wherein the processing unit comprises an image input module, an image pre-processing module, a feature point analysis module, an image scaling deformation module, and an image fusion Module and a coloring module. A digital zoom image fusion method includes: capturing a video by a wide-angle monochrome lens module, a narrow-angle monochrome lens module, and two color lens modules, wherein the wide-angle monochrome lens module, the narrow angle The monochrome lens module and the color lens modules are respectively located at four vertices of a quadrilateral, and the color lens modules are located at two opposite vertices; and the wide-angle monochrome lens module is integrated, the narrow-angle monochrome The image captured by the lens module and the color lens modules is a composite image. The digital zoom image fusion method of claim 13, wherein the method for integrating the composite image comprises: combining the wide-angle monochrome lens module and the narrow-angle monochrome lens module according to a scaling ratio; The monochromatic image is a digitally scaled monochrome image; and the digitally-scaled monochrome image and the color image captured by the color lens modules are merged into the composite image of the color corresponding to the scaling. The digital zoom image fusion method of claim 14, wherein the method of merging the digitally-scaled monochrome image with the color image captured by the color lens modules comprises: respectively scaling the monochrome image of the digital image Pixel position calibration with the color images captured by the color lens modules; and color information in the color images captured by the color lens modules is filled in the digital position based on the digitally scaled monochrome images The pixels of the corresponding position in the monochrome image are scaled to obtain the composite image.