JP7113289B2 - Imaging device, imaging method and program - Google Patents

Description

The present disclosure relates to imaging devices and the like used in cameras and the like.

Conventionally, there has been proposed an imaging apparatus that generates a composite image by synthesizing image data obtained while shifting the relative positions of an imaging device and an optical image projected onto an imaging surface of the imaging device (see Patent Document 1). ).

JP 2015-226256 A

However, the imaging apparatus of Patent Document 1 has a problem that it is difficult to improve the image quality.

Accordingly, the present invention provides an imaging apparatus capable of improving image quality.

An imaging device according to an aspect of the present disclosure is arranged along one surface with a pixel pitch apart from each other, and receives light transmitted through a translucent portion of any one of RGB colors included in a color filter. an image pickup device having a plurality of light receiving units that are exposed to each other; a drive unit that changes a relative position within the image pickup plane with respect to an optical image projected on the image pickup surface of the image pickup device; a synthesizing unit that generates a synthesized image by synthesizing pixel data output from each of the plurality of light receiving units of the imaging element, and corresponds to a first color that is any one of the RGB colors. The light-receiving area of the first light-receiving unit is positioned within one pixel area of the optical image, and the driving unit aligns the light-receiving area of the first light-receiving unit within the one pixel area from the pixel pitch. are arranged in order at a plurality of positions spaced apart from each other by a short distance, and the synthesizing unit operates when the light receiving regions of the first light receiving unit are arranged in order at each of the plurality of positions in the pixel region. The composite image is generated based on the pixel data output from the first light receiving unit , and the pixel area in the optical image includes the entire light receiving area of one light receiving unit on the imaging surface and the surrounding area of the light receiving area. It has a size that overlaps with the non-light-receiving area and does not overlap with the light-receiving areas of other light-receiving sections .

In addition, in the imaging method according to an aspect of the present disclosure, the light transmitted through the translucent portions of any one of the colors of RGB included in the color filter is arranged apart from each other at every pixel pitch along one surface. An image pickup method using an image pickup device having a plurality of light receiving units that receive light, wherein the relative position in the image pickup plane is changed with respect to an optical image projected on the image pickup surface of the image pickup device, and the image pickup device is exposed. A synthesized image is generated by synthesizing pixel data output from each of the plurality of light-receiving units of the imaging element each time the image is captured, and corresponds to a first color that is any one of the RGB colors. The light-receiving area of the first light-receiving unit is positioned within one pixel area of the optical image, and the change in the relative position of the imaging surface changes the light-receiving area of the first light-receiving unit within the one pixel area. , and in the generation of the composite image, the light receiving regions of the first light receiving unit are positioned at each of the plurality of positions in the pixel region. The composite image is generated based on the pixel data output from the first light receiving units when arranged in order, and the pixel area in the optical image is the entire light receiving area of one light receiving unit on the imaging surface. , and has a size that overlaps with the non-light-receiving area surrounding the light-receiving area and does not overlap with the light-receiving areas of other light-receiving sections .

In addition, these general or specific aspects may be realized by a system, method, integrated circuit, computer program, or a recording medium such as a computer-readable CD-ROM. and any combination of recording media. Also, the recording medium may be a non-temporary recording medium.

The imaging device of the present disclosure can improve image quality.

FIG. 1 is a diagram showing the principle of interpolation processing. FIG. 2 is a diagram showing interpolation processing by an imaging device having a non-light-receiving area. FIG. 3 is a diagram showing an example of the positioning error of the imaging device. FIG. 4 is a block diagram showing the configuration of the imaging device according to the embodiment. FIG. 5 is a diagram showing a cross section of an imaging element in the embodiment. FIG. 6 is a diagram showing each region on the imaging surface of the imaging element in the embodiment. 7A and 7B are diagrams for explaining the processing of the memory, shifter, RGB decomposing unit, and synthesizing unit according to the embodiment. FIG. 8 is a diagram showing changes in the relative position of the imaging surface of the imaging element with respect to the optical image in the embodiment. 9A and 9B are diagrams showing transitions of operation modes of the sequencer and the image pickup device, and movement of the image pickup device, according to the embodiment. FIG. 10 is a diagram showing the position of each light-receiving region shifted with respect to the optical image by movement of the imaging element in the embodiment. FIG. 11 is a flow chart showing the processing operation of the imaging device according to the embodiment. FIG. 12 is a block diagram showing the configuration of an imaging device according to a modification of the embodiment; 13A and 13B are diagrams for explaining processing of the memory, the shifter, the RGB decomposing unit, and the synthesizing unit in the modified example of the embodiment; 14A and 14B are diagrams showing changes in the relative position of the imaging surface of the imaging element with respect to the optical image in the modification of the embodiment. 15A and 15B are diagrams showing the transition of the operation modes of the sequencer and the image pickup device and the movement of the image pickup device in the modified example of the embodiment. FIG. 16 is a diagram showing the displacement of the light receiving area with respect to the pixel area in this modification of the embodiment.

(Findings on which this disclosure is based)
2. Description of the Related Art An imaging apparatus having a Bayer array imaging element outputs signals of the three primary colors of light from each light receiving section included in the imaging element. That is, each light receiving section outputs an R (red) signal, a G (green) signal, or a B (blue) signal. In addition, in the Bayer array, the number of light receiving sections that output G signals is twice as large as the number of light receiving sections that output R signals or B signals. The signals output from these light receiving units are hereinafter referred to as pixel data.

However, such an imaging device can only obtain pixel data of R, G, or B at the position of one light receiving portion, and cannot obtain pixel data of all colors of RGB.

Therefore, it is conceivable to generate pixel data of a color that is not actually obtained in one light receiving portion from pixel data of another light receiving portion by electrical signal processing.

However, the pixel data generated by such electrical signal processing has low reliability because it is not actual pixel data.

Therefore, by mechanically changing the relative position of the imaging surface with respect to the optical image projected on the imaging surface of the imaging device, all of RGB can be obtained at the position of one light receiving unit, that is, in the same area within the optical image. Interpolation processing has been proposed to obtain color pixel data.

FIG. 1 is a diagram showing the principle of interpolation processing.

An optical image is projected onto the imaging surface of the imaging element by an optical system such as a lens. At this time, for example, as shown in (a) of FIG. 1, an optical image is projected only on the light receiving area of G on the imaging surface of the imaging element. The light-receiving area is an area where light is received by one light-receiving section through the light-transmitting section of the color filter. Of the light projected onto the light-receiving area, only light of R, G, or B predetermined for the light-transmitting portion is transmitted through the light-transmitting portion. Therefore, although R, G, and B lights are projected onto the light receiving area, only the light that has passed through the light transmitting section is received by the light receiving section. Since the G light-transmitting portion is arranged in the G light-receiving region, the light-receiving portion receives only the G light and outputs the G pixel data according to the received light.

Next, the imaging device moves to the left by one pixel pitch, as shown in FIG. 1(b). As a result, the optical image is projected onto the R light receiving area. Therefore, at this time, the light-receiving portion of the R light-receiving region outputs the R pixel data. Note that the one-pixel pitch is the distance between adjacent light receiving portions.

Next, the imaging element moves upward by one pixel pitch from the position shown in FIG. 1(b), as shown in FIG. 1(c). As a result, the optical image is projected onto the G light receiving area. Therefore, at this time, the light-receiving portion of the G light-receiving region outputs the G pixel data. It should be noted that the G light-receiving region on which the optical image is projected at this time is different from the G light-receiving region shown in FIG.

Further, the imaging element is moved to the right by one pixel pitch from the position shown in FIG. 1(c), as shown in FIG. 1(d). As a result, the optical image is projected onto the B light receiving area. Therefore, at this time, the light-receiving portion of the B light-receiving region outputs the B pixel data.

Through the processing shown in (a) to (d) of FIG. 1, pixel data of all RGB colors can be obtained at the position of the optical image.

However, in recent years, as the number of pixels of an image pickup device increases, the pixel pitch becomes finer, and it is becoming more and more difficult to move the image pickup device by exactly one pixel pitch. Therefore, if there is a positioning error in the moving imaging device, the balance of the three primary colors of light may be lost, resulting in color unevenness. In addition, in an actual imaging device, the light receiving areas are not arranged without gaps on the imaging surface, and there are non-light receiving areas that cannot receive light between the light receiving areas.

FIG. 2 is a diagram showing interpolation processing by an imaging device having a non-light-receiving area.

As shown in FIG. 2, non-light-receiving regions exist between the R light-receiving region, the G light-receiving region, and the B light-receiving region on the imaging surface of the image sensor.

Interpolation processing similar to that in FIG. 1 is also performed on such an image pickup device, as shown in (a) to (d) of FIG. 2, for example. Note that FIG. 2 shows an example in which the brightness and the like in the optical image are not uniform as a whole and, for example, edges are included.

However, even with such an image pickup device, color unevenness may occur due to a positioning error of the image pickup device.

FIG. 3 is a diagram showing an example of the positioning error of the imaging device.

For example, as shown in (d) of FIG. 2, the image pickup device cannot be moved to the right by one pixel pitch and stopped, and the position of the image pickup device shifts from the predetermined position as shown in FIG. Sometimes.

In such a case, if the optical image includes an edge or the like, light different from that of the other R and G light receiving regions is projected onto the B light receiving region. As a result, the balance of the three primary colors of light is lost, resulting in color unevenness. Note that, as described above, the positional deviation of the imaging element from the predetermined position that occurs when the imaging element cannot be arranged at the predetermined position is referred to as a positioning error in the present disclosure.

Further, in the technique disclosed in Japanese Patent Laid-Open No. 2002-200001, the imaging element is shifted to improve the resolution, but there is a possibility that the color unevenness described above may occur.

Therefore, an imaging device according to an aspect of the present disclosure includes a plurality of imaging devices that are arranged apart from each other along one surface and receive light that has passed through a translucent portion of any one of RGB colors included in a color filter. a driving unit for changing the relative position of the imaging surface with respect to the optical image projected on the imaging surface of the imaging device; and the plurality of the imaging devices each time the imaging device is exposed to light. a synthesizing unit that generates a synthetic image by synthesizing pixel data output from each of the light receiving units, and the driving unit changes the relative position of any one of the RGB. The light receiving areas of the first light receiving section corresponding to the first color are sequentially placed at a plurality of positions separated from each other by a distance shorter than the interval between the plurality of light receiving sections within the pixel region of the optical image. and the synthesizing unit adds the pixel data output from the first light receiving unit when the light receiving areas of the first light receiving unit are arranged in order at each of the plurality of positions in the pixel area. to calculate the pixel value of the first color in the pixel area as the pixel value included in the composite image. For example, the pixel region in the optical image overlaps the entire light-receiving region of one light-receiving unit on the imaging surface, the non-light-receiving region surrounding the light-receiving region, and overlaps the light-receiving region of another light-receiving unit. It may have a size that does not

This can equivalently increase the aperture ratio of the image sensor with respect to the pixel area. In other words, when a plurality of light receiving portions in the imaging device are arranged apart from each other, non-light receiving areas exist between the plurality of light receiving portions. An image pickup device having such a non-light-receiving region has a lower aperture ratio in one light-receiving portion than an image pickup device having no non-light-receiving region. As a result, in each of the plurality of pixel regions forming the optical image, light cannot be received over a wide range within the pixel region. Therefore, for example, when a positioning error of the imaging device as shown in FIG. 3 occurs, pixel data of one color cannot be obtained more appropriately than pixel data of other colors. However, in the imaging device according to the above aspect, the light receiving areas of the first light receiving section corresponding to the first color are sequentially arranged at a plurality of positions within the pixel area, and at those positions, the light from the first light receiving section is exposed by exposure. Pixel data is output. As a result, the light projected onto the non-light-receiving region can also be captured, and the substantial area of the non-light-receiving region can be reduced. In other words, even if the light receiving area is smaller than the pixel area, the light receiving area can be virtually enlarged. That is, the aperture ratio can be equivalently increased as described above. Therefore, even if the positioning error of the imaging element as described above occurs, the occurrence of color unevenness can be suppressed. Further, in the imaging device according to the above aspect, the light receiving regions corresponding to one color are arranged not only at one position within the pixel region but also at a plurality of positions in sequence. Therefore, it is possible to suppress the influence of a single positioning error on color unevenness. Further, even if positioning errors occur multiple times, if the positioning errors are random, the influence of the positioning errors on color unevenness can be suppressed. Note that the aperture ratio is, for example, the ratio of the size of the light receiving area to the size of the pixel area.

The imaging device further includes a memory for storing pixel data output from each of the plurality of light receiving units of the imaging device, and a shifter for shifting the pixel position of the pixel data stored in the memory. and the driving unit further changes the light receiving area of the second light receiving unit corresponding to a second color of the RGB, which is different from the first color, to the pixel area by changing the relative position. and the shifter is arranged at each of the plurality of positions in the second light receiving section stored in the memory when the pixel data output from the second light receiving section is transferred to the memory. to the pixel position of the pixel data of the first light receiving section, and the synthesizing section sequentially shifts the light receiving region of the second light receiving section to each of the plurality of positions within the pixel region. By adding the shifted pixel data output from the second light receiving unit when arranged, the pixel value of the second color in the pixel area is used as the pixel value included in the synthesized image. can be calculated.

As a result, not only the pixel value of the first color but also the pixel value of the second color can be appropriately calculated with respect to the pixel region while suppressing the occurrence of color unevenness.

For example, the arrangement of the plurality of translucent portions included in the color filter may be a Bayer arrangement. In this case, the color filter is composed of a plurality of sets each including an R light-transmitting portion, two G light-transmitting portions, and a B light-transmitting portion, and the plurality of positions are four positions. , the driving section, when arranging the light receiving areas of the first light receiving section and the second light receiving section, sets the light receiving area of the first light receiving section corresponding to R as the first color to the four light receiving sections. By sequentially arranging the light-receiving regions of the second light-receiving unit corresponding to B, which is the second color, at the four positions and changing the relative positions, the third The light-receiving areas of the third light-receiving section corresponding to G, which is the color of , are arranged in order at the four positions, and the light-receiving areas of the fourth light-receiving section corresponding to G, the fourth color, are arranged at the four positions When the pixel data output from the third light receiving section is transferred to the memory, the shifter shifts the pixel position of the pixel data of the third light receiving section stored in the memory. , to the pixel position of the pixel data of the first light receiving portion, and when the pixel data output from the fourth light receiving portion is transferred to the memory, the position of the pixel data of the fourth light receiving portion stored in the memory The pixel position of the pixel data is shifted to the pixel position of the pixel data of the first light receiving section, and the synthesizing section further shifts the light receiving region of the third light receiving section to each of the plurality of positions within the pixel region. A pixel value of the third color in the pixel area is calculated by adding the shifted pixel data output from the third light receiving section when arranged in order, and the pixel value of the third color in the pixel area is calculated. By adding the shifted pixel data output from the fourth light receiving unit when the light receiving regions of the fourth light receiving unit are arranged in order at each of the plurality of positions, the fourth pixel data in the pixel region is added. may be calculated as the pixel value of G included in the composite image.

As a result, the R pixel value, the two G pixel values, and the B pixel value can be appropriately calculated with respect to the pixel region while suppressing the occurrence of color unevenness.

Further, the arrangement of the plurality of translucent portions included in the color filter may be a delta arrangement.

Accordingly, it is possible to appropriately suppress the occurrence of color unevenness even for the delta-arrangement imaging device.

Hereinafter, embodiments will be specifically described with reference to the drawings.

It should be noted that the embodiments described below are all comprehensive or specific examples. Numerical values, shapes, materials, components, arrangement positions and connection forms of components, steps, order of steps, and the like shown in the following embodiments are examples and are not intended to limit the present invention. In addition, among the constituent elements in the following embodiments, constituent elements that are not described in independent claims representing the highest concept will be described as arbitrary constituent elements.

Each figure is a schematic diagram and is not necessarily strictly illustrated. Moreover, in each figure, the same code|symbol is attached|subjected about the same component.

(Embodiment)
FIG. 4 is a block diagram showing the configuration of the imaging device according to this embodiment.

Imaging apparatus 100 according to the present embodiment includes imaging element 101, lens 102, vertical actuator 103a, horizontal actuator 103b, sequencer 104, memory 105, shifter 106, RGB decomposing unit 107, and synthesizing unit 108. and

A lens 102 projects an image of a subject as an optical image onto an imaging surface of an imaging element 101 .

The imaging device 101 is, for example, a Bayer array imaging device (also referred to as an image sensor). Specifically, the imaging element 101 is, for example, a CCD (Charge Coupled Device) or a CMOS (Complementary Metal Oxide Semiconductor). On the image pickup surface of the image pickup device 101, respective light receiving areas of RGB are arranged in a matrix along each of the horizontal direction and the vertical direction. In the horizontal direction, each of the plurality of light receiving regions is arranged so as to be separated from the adjacent light receiving region by the horizontal pixel pitch, and in the vertical direction, each of the plurality of light receiving regions is separated from the adjacent light receiving region by the vertical pixel pitch. placed apart. A light receiving section is arranged in each light receiving region as described later.

The vertical actuator 103a and the horizontal actuator 103b are driving units that change the relative position of the imaging surface of the imaging element 101 with respect to the optical image projected on the imaging surface. Specifically, the vertical actuator 103a and the horizontal actuator 103b move the imaging element 101 within a plane perpendicular to the optical axis of the lens 102 using a driving source such as a voice coil motor or a piezoelectric element. The vertical actuator 103a moves the imaging device 101 in the vertical direction. The horizontal actuator 103b moves the imaging element 101 in the horizontal direction. The vertical direction and the horizontal direction are directions orthogonal to each other in the plane perpendicular to the optical axis.

The memory 105 is a recording medium for recording pixel data output from each of the plurality of light receiving units 101b of the image sensor 101. FIG.

The sequencer 104 controls exposure of the image pickup device 101 , movement of the image pickup device 101 by the vertical actuator 103 a and horizontal actuator 103 b , and writing of pixel data to the memory 105 .

A shifter 106 shifts the pixel position of the pixel data stored in the memory 105 .

The RGB decomposition unit 107 decomposes the pixel data output from each of the plurality of light receiving units of the image sensor 101 into R pixel data, G pixel data, and B pixel data.

The synthesis unit 108 generates a synthesized image by synthesizing pixel data output from each of the plurality of light receiving units of the image sensor 101 each time the image sensor 101 is exposed. Specifically, for each pixel region forming the optical image, the synthesizing unit 108 adds a plurality of R pixel data corresponding to the pixel region, and adds a plurality of G pixel data corresponding to the pixel region. is multiplied by 1/2, and a plurality of R pixel data corresponding to the pixel area are added. Thereby, the synthesizing unit 108 generates a synthetic image.

FIG. 5 is a diagram showing a cross section of the image sensor 101. As shown in FIG.

The imaging device 101 has a plurality of light receiving portions 101b which are arranged apart from each other along one surface and receive light transmitted through the light transmitting portions 111 of any one of RGB colors included in the color filter 101a. . Each of the plurality of light receiving portions 101b is, for example, a photodiode, and upon receiving light, outputs an electrical signal corresponding to the amount of light as pixel data. Moreover, as shown in FIGS. 4 and 5, the arrangement of the plurality of translucent portions 111 included in the color filter 101a is a Bayer arrangement. That is, the color filter 101a is composed of a plurality of sets each including an R light-transmitting portion 111, two G light-transmitting portions 111, and a B light-transmitting portion 111, respectively. That is, one set includes the R translucent portion 111 , two G translucent portions 111 , and the B translucent portion 111 .

FIG. 6 is a diagram showing each region on the imaging surface of the imaging element 101. As shown in FIG.

The pixel area in the optical image has a size that overlaps the entire light receiving area of one light receiving unit 101b on the imaging surface and the non-light receiving area surrounding the light receiving area, and does not overlap the light receiving areas of the other light receiving units 101b. have In this embodiment, the pixel region has a rectangular shape surrounding one light-receiving region, as indicated by the dashed line in FIG. 6, and the boundary of the pixel region (broken line in FIG. between the light-receiving regions. Also, in the present embodiment, the optical image consists of four pixel regions.

Here, as shown in FIGS. 5 and 6, the light receiving area of the image sensor 101 is smaller than the pixel area.

Therefore, the vertical actuator 103a and the horizontal actuator 103b in the present embodiment change the relative position of the imaging surface of the imaging element 101 with respect to the optical image in accordance with the control by the sequencer 104, thereby making the light receiving area smaller within the pixel area. shift. That is, the vertical actuator 103a and the horizontal actuator 103b move the light-receiving region of the first light-receiving unit 101b corresponding to the first color, which is one of RGB, to a plurality of light-receiving units within the pixel region of the optical image. 101b in sequence at each of a plurality of positions separated from each other by a distance less than the interval 101b. A distance shorter than the interval between the plurality of light receiving units 101b is a distance shorter than each of the horizontal pixel pitch and the vertical pixel pitch described above, and is, for example, 1/2 of each pitch in the present embodiment. Also, the first light receiving unit 101b is one of the plurality of light receiving units 101b that the imaging element 101 has.

Synthesizing unit 108 in the present embodiment combines the pixel data output from first light receiving unit 101b when the light receiving regions of first light receiving unit 101b are arranged in order at each of the plurality of positions in the pixel region. to add. As a result, the synthesizing unit 108 calculates the pixel values of the first color in the pixel area as the pixel values included in the synthesized image. This can equivalently increase the aperture ratio of the image sensor 101 with respect to the pixel area. Therefore, even if the positioning error of the imaging element as described above occurs, the occurrence of color unevenness can be suppressed.

FIG. 7 is a diagram for explaining the processing of memory 105, shifter 106, RGB decomposing unit 107, and synthesizing unit 108. FIG. FIG. 8 is a diagram showing changes in the relative position of the imaging surface of the imaging element 101 with respect to the optical image. FIG. 9 is a diagram showing the transition of the operation modes of the sequencer 104 and the image pickup device 101 and the movement of the image pickup device 101. As shown in FIG. In the example shown in FIG. 7, the image sensor 101 has M×N light receiving units 101b. Numbers 1 to 16 in FIGS. 8 and 9 indicate the order of imaging.

As shown in FIG. 9, the sequencer 104 repeatedly moves and stops the imaging device 101 by controlling the vertical actuator 103a and the horizontal actuator 103b. Thereby, the sequencer 104 sequentially moves the image sensor 101 to different imaging positions. For example, as shown in FIG. 8, the sequencer 104 sequentially moves the image sensor 101 to 16 image capturing positions and stops the image sensor 101 at those image capturing positions. Then, the sequencer 104 outputs pixel data from each of the M×N light receiving units 101b by exposing the image sensor 101 every time the image sensor 101 stops at the imaging position. At this time, the sequencer 104 stores the pixel data output from each of the M×N light receiving units 101 b in the storage area of the memory 105 .

Specifically, in the first imaging, the sequencer 104 stops the imaging device 101 at the first initial position, which is the first imaging position, and causes the imaging device 101 to perform imaging (that is, exposure) at this time. At the first initial position, for example, as shown in FIG. 8, the G light-receiving area is arranged at the upper left end of the optical image. Then, the sequencer 104 transfers the M×N pixel data of the image pickup device 101 obtained by the first image pickup to the first area 1a of the memory 105 while moving the image pickup device 101 to the second image pickup position. do. At this time, each of the M×N pieces of pixel data is stored in the first area 1a in association with the pixel position of the picked-up image composed of M×N pieces of pixels.

The RGB decomposing unit 107 decomposes the M×N pixel data into R pixel data, G pixel data, and B pixel data, and outputs the R pixel data, G pixel data, and B pixel data to the synthesizing unit 108 .

When the sequencer 104 moves the image sensor 101 to the second imaging position, as shown in FIGS. 8 and 9, the sequencer 104 moves the image sensor 101 to the left from the first initial position by the horizontal pixel pitch and stops. At the second imaging position, for example, as shown in FIG. 8, the B light receiving area is arranged at the upper left end of the optical image. Then, the sequencer 104 causes the imaging device 101 to perform the second imaging (that is, exposure). The sequencer 104 transfers the M×N pixel data of the image sensor 101 obtained by the second imaging to the second area 1b of the memory 105 while moving the image sensor 101 to the third imaging position. Also at this time, each of the M×N pieces of pixel data is stored in the second area 1b in association with the pixel position in the captured image.

The shifter 106 shifts each pixel position of the M×N pixel data stored in the second area 1b to the left by one pixel. As a result, the captured image represented by the M×N pixel data stored in the second area 1b is shifted to the left by one pixel. That is, the pixel positions of the M×N pieces of pixel data obtained by the second imaging are shifted to the pixel positions when the imaging device 101 is arranged at the first initial position. Therefore, the pixel position of the upper left corner of the optical image is associated with the G pixel data and the B pixel data.

The RGB decomposing unit 107 decomposes the shifted M×N pixel data into R pixel data, G pixel data, and B pixel data, and outputs them to the synthesizing unit 108 .

When the sequencer 104 moves the image pickup element 101 to the third image pickup position, as shown in FIGS. 8 and 9, the sequencer 104 moves the image pickup element 101 upward from the second image pickup position by the vertical pixel pitch and stops. At the third imaging position, for example, as shown in FIG. 8, the G light receiving area is arranged at the upper left end of the optical image. Then, the sequencer 104 causes the imaging device 101 to perform the third imaging (that is, exposure). The sequencer 104 transfers the M×N pixel data of the image sensor 101 obtained by the third imaging to the third area 1c of the memory 105 while moving the image sensor 101 to the fourth imaging position. Also at this time, each of the M×N pieces of pixel data is stored in the third area 1c in association with the pixel position in the captured image.

The shifter 106 shifts the pixel position of each of the M×N pixel data stored in the third area 1c to the left by one pixel and then shifts it upward by one pixel. As a result, the captured image represented by the M.times.N pixel data stored in the third area 1c is shifted leftward by one pixel and shifted upward by one pixel. That is, the pixel positions of the M×N pieces of pixel data obtained by the third imaging are shifted to the pixel positions when the imaging device 101 is arranged at the first initial position. Therefore, the pixel position of the upper left corner of the optical image is associated with the G pixel data, the B pixel data, and the G pixel data.

The RGB decomposing unit 107 decomposes the shifted M×N pixel data into R pixel data, G pixel data, and B pixel data, and outputs them to the synthesizing unit 108 .

When the sequencer 104 moves the image pickup device 101 to the fourth image pickup position, as shown in FIGS. 8 and 9, the sequencer 104 moves the image pickup device 101 from the third image pickup position to the right by the horizontal pixel pitch and stops. At the fourth photographing position, for example, as shown in FIG. 8, the R light-receiving area is arranged at the upper left end of the optical image. Then, the sequencer 104 causes the imaging device 101 to perform the fourth imaging (that is, exposure). The sequencer 104 transfers the M×N pixel data of the image pickup device 101 obtained by the fourth image pickup to the fourth area 1d of the memory 105 while moving the image pickup device 101 to the fifth image pickup position. Also at this time, each of the M×N pieces of pixel data is stored in the fourth area 1d in association with the pixel position in the captured image.

The shifter 106 shifts the pixel positions of the M×N pieces of pixel data stored in the fourth area 1d upward by one pixel. As a result, the captured image represented by the M×N pixel data stored in the fourth area 1d is shifted upward by one pixel. That is, the pixel positions of the M×N pieces of pixel data obtained by the fourth imaging are shifted to the pixel positions when the imaging device 101 is arranged at the first initial position. Therefore, the pixel position of the upper left corner of the optical image is associated with G pixel data, B pixel data, G pixel data, and R pixel data.

The RGB decomposing unit 107 decomposes the shifted M×N pixel data into R pixel data, G pixel data, and B pixel data, and outputs them to the synthesizing unit 108 .

By performing the first to fourth imaging, R pixel data associated with each of the plurality of pixel positions of the captured image when the image sensor 101 is arranged at the first initial position, and 2 The G pixel data and the B pixel data are output to the synthesizing unit 108 .

Here, the sequencer 104 moves the imaging element 101 to the second initial position, which is the fifth imaging position. At this time, as shown in FIGS. 8 and 9, the sequencer 104 moves the image sensor 101 from the fourth imaging position to the right by a distance less than the horizontal pixel pitch and further downward by the vertical pixel pitch. to stop. As shown in FIG. 8, when the image sensor 101 is at the second initial position, each light receiving area that overlaps the optical image when the image sensor 101 is at the first initial position is 1/2 of the horizontal pixel pitch, for example. shifted to the right. Further, when the image sensor 101 is at the second initial position, the B light receiving area is arranged at the upper right end of the optical image. Then, the sequencer 104 causes the imaging device 101 to perform the fifth imaging (that is, exposure). The sequencer 104 transfers the M×N pixel data of the image pickup device 101 obtained by the fifth image pickup to the fifth area 2 a of the memory 105 while moving the image pickup device 101 to the sixth image pickup position. At this time, each of the M×N pieces of pixel data is stored in the fifth area 2a in association with the pixel position of the picked-up image composed of M×N pieces of pixels.

The RGB decomposing unit 107 decomposes the M×N pixel data into R pixel data, G pixel data, and B pixel data, and outputs the R pixel data, G pixel data, and B pixel data to the synthesizing unit 108 .

In order to perform the sixth, seventh and eighth imaging, the sequencer 104 performs the same processing as the second, third and fourth imaging based on the second initial position instead of the first initial position. .

By performing the fifth to eighth imaging, R pixel data associated with each of the plurality of pixel positions of the captured image when the image sensor 101 is arranged at the second initial position, and 2 The G pixel data and the B pixel data are output to the synthesizing unit 108 .

Similarly, the sequencer 104 moves the imaging device 101 to the third initial position, which is the ninth imaging position. At this time, as shown in FIGS. 8 and 9, the sequencer 104 moves the image sensor 101 downward from the eighth imaging position by a distance equal to or greater than the vertical pixel pitch and less than twice the vertical pixel pitch. stop. As shown in FIG. 8, when the image sensor 101 is at the third initial position, each light receiving area that overlaps the optical image when the image sensor 101 is at the first initial position is 1/2 of the horizontal pixel pitch, for example. , and further shifted downward by 1/2 of the vertical pixel pitch. Further, when the image sensor 101 is at the third initial position, the G light receiving area is arranged at the lower right end of the optical image. Then, the sequencer 104 causes the imaging element 101 to perform the ninth imaging (that is, exposure). The sequencer 104 transfers the M×N pixel data of the image pickup device 101 obtained by the ninth image pickup to the ninth area 3 a of the memory 105 while moving the image pickup device 101 to the tenth image pickup position. At this time, each of the M×N pieces of pixel data is stored in the ninth area 3a in association with the pixel position of the picked-up image composed of M×N pieces of pixels.

The RGB decomposing unit 107 decomposes the M×N pixel data into R pixel data, G pixel data, and B pixel data, and outputs the R pixel data, G pixel data, and B pixel data to the synthesizing unit 108 .

In order to perform the 10th, 11th and 12th imaging, the sequencer 104 performs the same processing as the 2nd, 3rd and 4th imaging based on the 3rd initial position instead of the 1st initial position. .

Through the ninth to twelfth imaging, the R pixel data associated with each of the plurality of pixel positions of the captured image when the image sensor 101 is arranged at the third initial position, and the 2 The G pixel data and the B pixel data are output to the synthesizing unit 108 .

Similarly, the sequencer 104 moves the imaging device 101 to the fourth initial position, which is the 13th imaging position. At this time, as shown in FIGS. 8 and 9, the sequencer 104 moves the image sensor 101 from the twelfth imaging position to the left by a distance less than the horizontal pixel pitch, and further downward by the vertical pixel pitch. to stop. As shown in FIG. 8, when the image sensor 101 is at the fourth initial position, each light receiving region that overlaps the optical image when the image sensor 101 is at the first initial position is 1/2 of the vertical pixel pitch, for example. is shifted downwards. Further, when the imaging device 101 is at the fourth initial position, the R light receiving area is arranged at the lower left end of the optical image. Then, the sequencer 104 causes the imaging device 101 to perform the 13th imaging (that is, exposure). The sequencer 104 transfers the M×N pixel data of the image pickup device 101 obtained by the 13th image pickup to the 13th area 4 a of the memory 105 while moving the image pickup device 101 to the 14th image pickup position. At this time, each of the M×N pieces of pixel data is stored in the thirteenth area 4a in association with the pixel position of the picked-up image made up of M×N pieces of pixels.

The RGB decomposing unit 107 decomposes the M×N pixel data into R pixel data, G pixel data, and B pixel data, and outputs the R pixel data, G pixel data, and B pixel data to the synthesizing unit 108 .

In order to perform the 14th, 15th and 16th imaging, the sequencer 104 performs the same processing as the 2nd, 3rd and 4th imaging based on the 4th initial position instead of the 1st initial position. .

Through the 13th to 16th imaging, the R pixel data associated with each of the plurality of pixel positions of the captured image when the image sensor 101 is arranged at the fourth initial position, and 2 The G pixel data and the B pixel data are output to the synthesizing unit 108 .

FIG. 10 is a diagram showing the position of each light-receiving region shifted with respect to the optical image due to the movement of the imaging element 101 shown in FIGS. 8 and 9. FIG.

As shown in (a) of FIG. 10 , in the first, fifth, ninth, and thirteenth imaging, the G (third color) light-receiving region is the upper left pixel region in the optical image. , upper right, lower right, and lower left. Similarly, in the upper right pixel region, the light receiving regions for B are arranged in the order of upper left, upper right, lower right, and lower left, and an image is captured. , the lower right, and the lower left, and the images are captured. Also, in the lower left pixel area, the R light receiving areas are arranged in the order of the upper left, upper right, lower right, and lower left for imaging.

In this way, the vertical actuator 103a and the horizontal actuator 103b move the light receiving area of the light receiving section 101b corresponding to G (third color) of RGB to the vertical pixel pitch and the horizontal pixel area within the upper left pixel area of the optical image. They are arranged in order at each of a plurality of positions separated from each other by a distance shorter than the pixel pitch. Similarly, the vertical actuator 103a and the horizontal actuator 103b sequentially arrange the light receiving areas of the light receiving section 101b corresponding to B at a plurality of positions apart from each other in the upper right pixel area of the optical image. Similarly, the vertical actuator 103a and the horizontal actuator 103b sequentially arrange the light-receiving regions of the light-receiving unit 101b corresponding to G at a plurality of positions separated from each other in the lower right pixel region of the optical image. Similarly, the vertical actuator 103a and the horizontal actuator 103b sequentially arrange the light-receiving regions of the light-receiving unit 101b corresponding to R at a plurality of positions apart from each other in the lower left pixel region of the optical image. The multiple positions mentioned above are four positions.

When the light receiving regions for G (third color) are arranged at mutually different positions in the upper left pixel region as described above, the synthesizing unit 108 combines G and G output from the light receiving unit 101b corresponding to the light receiving regions. (3rd color) pixel data are added. Thereby, the synthesizing unit 108 calculates the pixel value of G (third color) in the upper left pixel area. Similarly, the synthesizing unit 108 calculates the B pixel value in the upper right pixel region, the G pixel value in the lower right pixel region, and the R pixel value in the lower left pixel region.

Further, as shown in (b) of FIG. 10 , in the second, sixth, tenth, and fourteenth imaging, the light receiving region of B (second color) in the upper left pixel region in the optical image are arranged in the order of upper left, upper right, lower right, and lower left. Similarly, in the upper right pixel region, the light receiving regions for G are arranged in the order of upper left, upper right, lower right, and lower left, and an image is captured. , the lower right, and the lower left, and the images are captured. Also, in the lower left pixel area, the light receiving areas for G are arranged in the order of the upper left, upper right, lower right, and lower left to perform imaging.

In this way, the vertical actuator 103a and the horizontal actuator 103b move the light receiving area of the light receiving section 101b corresponding to B (second color) of RGB to the vertical pixel pitch and the horizontal pixel area within the upper left pixel area of the optical image. They are arranged in order at each of a plurality of positions separated from each other by a distance shorter than the pixel pitch. The plurality of positions is four positions. When the pixel data output from the B light receiving unit 101b is transferred to the memory 105, the shifter 106 shifts the pixel position of the pixel data of the B light receiving unit 101b stored in the memory 105 to G ( (third color) to the pixel position of the pixel data of the light receiving unit 101b. That is, the pixel positions of the upper left, upper right, lower right, and lower left B pixel data in the upper left pixel area in the optical image are the upper left, upper right, lower right, and lower left pixel positions in the pixel area. It is shifted to the pixel position of each G (third color) pixel data. Also in the upper right, lower right, and lower left pixel regions of the optical image, the light receiving regions are shifted in the same manner as described above, and the pixel positions of the pixel data are shifted.

When the B (second color) light-receiving regions are arranged at different positions in the upper left pixel region as described above, the synthesizing unit 108 outputs from the light-receiving unit 101b corresponding to the light-receiving regions, and add the shifted B pixel data. Accordingly, the synthesizing unit 108 calculates the pixel value of B (second color) in the upper left pixel area. Similarly, the synthesizing unit 108 calculates the G pixel value in the upper right pixel region, the R pixel value in the lower right pixel region, and the G pixel value in the lower left pixel region.

Further, as shown in (c) of FIG. 10 , in the third, seventh, eleventh, and fifteenth imaging, the G (fourth color) light-receiving region in the upper left pixel region in the optical image are arranged in the order of upper left, upper right, lower right, and lower left. Similarly, in the upper right pixel region, the R light receiving regions are arranged in the order of upper left, upper right, lower right, and lower left, and the image is captured. , the lower right, and the lower left, and the images are captured. Also, in the lower left pixel area, the light receiving areas of B are arranged in the order of the upper left, upper right, lower right, and lower left for imaging.

In this way, the vertical actuator 103a and the horizontal actuator 103b move the light receiving area of the light receiving section 101b corresponding to G (fourth color) of RGB to the vertical pixel pitch and the horizontal pixel area within the upper left pixel area of the optical image. They are placed in turn at each of four positions separated from each other by a distance smaller than the pixel pitch. When the pixel data output from the G (fourth color) light receiving unit 101 b is transferred to the memory 105 , the shifter 106 shifts the G (fourth color) light received stored in the memory 105 to the memory 105 . The pixel position of the pixel data of the unit 101b is shifted to the pixel position of the pixel data of the G (third color) light receiving unit 101b. That is, the pixel positions of the G (fourth color) pixel data in the upper left, upper right, lower right, and lower left pixel regions in the upper left pixel region in the optical image are the upper left, upper right, and It is shifted to the pixel position of the G (third color) pixel data on the lower right and lower left respectively. Also in the upper right, lower right, and lower left pixel regions of the optical image, the light receiving regions are shifted in the same manner as described above, and the pixel positions of the pixel data are shifted.

When the G (fourth color) light-receiving regions are arranged at different positions in the upper left pixel region as described above, the synthesizing unit 108 outputs from the light-receiving unit 101b corresponding to the light-receiving regions, And the shifted G (fourth color) pixel data is added. Thereby, the synthesizing unit 108 calculates the pixel value of G (fourth color) in the upper left pixel area. Similarly, the synthesizing unit 108 calculates the R pixel value in the upper right pixel region, the G pixel value in the lower right pixel region, and the B pixel value in the lower left pixel region.

Further, as shown in (d) of FIG. 10 , in the fourth, eighth, twelfth, and sixteenth imaging, the R (first color) light-receiving region in the upper left pixel region in the optical image are arranged in the order of upper left, upper right, lower right, and lower left. Similarly, in the upper right pixel region, the light receiving regions for G are arranged in the order of upper left, upper right, lower right, and lower left, and an image is captured. , the lower right, and the lower left, and the images are captured. Also, in the lower left pixel area, the light receiving areas for G are arranged in the order of the upper left, upper right, lower right, and lower left to perform imaging.

In this way, the vertical actuator 103a and the horizontal actuator 103b move the light receiving area of the light receiving section 101b corresponding to R (first color) of RGB to the vertical pixel pitch and the horizontal pixel area within the upper left pixel area of the optical image. They are placed in turn at each of four positions separated from each other by a distance smaller than the pixel pitch. When the pixel data output from the R light receiving unit 101b is transferred to the memory 105, the shifter 106 shifts the pixel position of the pixel data of the R light receiving unit 101b stored in the memory 105 to G ( (third color) to the pixel position of the pixel data of the light receiving unit 101b. That is, the pixel positions of the upper left, upper right, lower right, and lower left R pixel data in the upper left pixel area in the optical image are the upper left, upper right, lower right, and lower left pixel positions in the pixel area. It is shifted to the pixel position of each G (third color) pixel data. Also in the upper right, lower right, and lower left pixel regions of the optical image, the light receiving regions are shifted in the same manner as described above, and the pixel positions of the pixel data are shifted.

When the R (first color) light-receiving regions are arranged at different positions in the upper left pixel region as described above, the synthesizing unit 108 outputs from the light-receiving unit 101b corresponding to the light-receiving regions, and add the shifted R pixel data. Thereby, the synthesizing unit 108 calculates the pixel value of R (first color) in the upper left pixel area. Similarly, the synthesizing unit 108 calculates the G pixel value in the upper right pixel region, the B pixel value in the lower right pixel region, and the G pixel value in the lower left pixel region.

By the above-described processing, the R pixel value and the B pixel value are calculated as the pixel values included in the combined image for each pixel region of the optical image. Further, the synthesizing unit 108 averages the two G pixel values calculated for the pixel region as described above, that is, the pixel value of the third color and the pixel value of the fourth color. As a result, one G pixel value for that pixel area is calculated as the pixel value included in the synthesized image.

In the above-described processing, the shifter 106 shifts the pixel position of the pixel data of colors other than G (third color) to the pixel position of the pixel data of G (third color). The method is not limited to this. For example, the shifter 106 may shift the pixel position of the pixel data of a color other than R (first color) to the pixel position of the pixel data of R (first color).

FIG. 11 is a flow chart showing the processing operation of the imaging device 100. As shown in FIG.

The sequencer 104 reads the movement point position information indicating the imaging position described above (step S11). Next, the sequencer 104 instructs the vertical actuator 103a and the horizontal actuator 103b to move the imaging element 101 to the imaging position indicated by the read movement point position information (step S12).

Next, the sequencer 104 determines whether or not the movement of the imaging element 101 to the imaging position has been completed (step S13). When it is determined that the movement has been completed (T in step S13), the sequencer 104 exposes the image sensor 101 (step S14). The sequencer 104 then determines whether or not the exposure has been completed (step S15). When it is determined that the exposure has been completed (T in step S15), the sequencer 104 transfers M×N pixel data to the memory 105 (step S16). The sequencer 104 determines whether or not the transfer of pixel data has been completed (step S17), and when it is determined that the transfer has been completed (T in step S17), exposure at all movement points, that is, all imaging positions is completed. It is determined whether or not (step S18).

Here, if it is determined that exposure has not been completed at all imaging positions (F of step S18), the sequencer 104 repeats the processing from step S11. On the other hand, when it is determined that exposure has been completed at all imaging positions, the synthesizing unit 108 generates a synthesized image from the pixel data (step S19).

As described above, in the present embodiment, the light-receiving region is shifted a plurality of times within the pixel region, and exposure is performed at the shifted positions. Therefore, it is possible to equivalently increase the aperture ratio of the image sensor 101 with respect to the pixel area. In other words, when the plurality of light receiving portions 101b in the image sensor 101 are arranged apart from each other, non-light receiving regions exist between the plurality of light receiving portions 101b. The image sensor 101 having such a non-light-receiving region has a lower aperture ratio in one light-receiving portion 101b than an image sensor having no non-light-receiving region. As a result, in each of the plurality of pixel regions forming the optical image, light cannot be received over a wide range within the pixel region. Therefore, for example, when a positioning error of the imaging device as shown in FIG. 3 occurs, pixel data of one color cannot be obtained more appropriately than pixel data of other colors. However, in the imaging device 100 according to the present embodiment, the light receiving regions of the first light receiving unit 101b corresponding to the first color such as G are sequentially arranged at a plurality of positions within the pixel region, and exposure is not possible at those positions. , the pixel data is output from the first light receiving unit 101b. As a result, the light projected onto the non-light-receiving region can also be captured, and the substantial area of the non-light-receiving region can be reduced. In other words, even if the light receiving area is smaller than the pixel area, the light receiving area can be virtually enlarged. That is, the aperture ratio can be equivalently increased as described above. Therefore, even if the positioning error of the imaging element as described above occurs, the occurrence of color unevenness can be suppressed. Further, in the imaging device 100 according to the present embodiment, the light receiving regions corresponding to one color are arranged not only at one position within the pixel region but also at a plurality of positions in sequence. Therefore, it is possible to suppress the influence of a single positioning error on color unevenness. Further, even if positioning errors occur multiple times, if the positioning errors are random, the influence of the positioning errors on color unevenness can be suppressed.

(Modification)
In the imaging device 101 of the imaging device 100 in the above embodiment, the plurality of translucent portions 111 of the color filter 101a are arranged according to the Bayer arrangement. However, the imaging device of the imaging device of the present disclosure is not limited to such a Bayer arrangement. In the imaging element of the imaging apparatus according to this modified example, the plurality of translucent portions 111 are arranged according to a delta arrangement.

FIG. 12 is a block diagram showing the configuration of an imaging device in this modified example.

The imaging apparatus 200 according to the present embodiment includes a delta array imaging element 201, a lens 102, a vertical actuator 103a, a horizontal actuator 103b, a sequencer 104, a memory 105, a shifter 106, an RGB decomposition unit 107, and a synthesizing unit 108 . In other words, the imaging device 200 in this modified example includes a delta array imaging device 201 instead of the Bayer array imaging device 101 in the imaging device 100 shown in FIG. In other words, the arrangement of the plurality of translucent portions 111 included in the color filter of the imaging element 201 is a delta arrangement. Further, in the image sensor 201, as shown in FIG. 12, the horizontal pixel pitch is longer than the vertical pixel pitch.

FIG. 13 is a diagram for explaining the processing of memory 105, shifter 106, RGB decomposing unit 107, and synthesizing unit 108. FIG. FIG. 14 is a diagram showing changes in the relative position of the imaging surface of the imaging element 201 with respect to the optical image. 15A and 15B are diagrams showing the transition of the operation modes of the sequencer 104 and the image sensor 201 and the movement of the image sensor 201. FIG. In the example shown in FIG. 13, the imaging device 201 has M×N light receiving units 101b. The numbers 1 to 18 in FIGS. 14 and 15 indicate the order of imaging.

As shown in FIG. 15, the sequencer 104 repeatedly moves and stops the imaging device 201 by controlling the vertical actuator 103a and the horizontal actuator 103b. Thereby, the sequencer 104 sequentially moves the image sensor 101 to different imaging positions. For example, as shown in FIG. 14, the sequencer 104 sequentially moves the image pickup device 101 to 18 image pickup positions and stops the image pickup device 201 at those image pickup positions. Then, the sequencer 104 outputs pixel data from each of the M×N light receiving units 101b by exposing the image sensor 101 every time the image sensor 201 stops at the imaging position. At this time, the sequencer 104 stores the pixel data output from each of the M×N light receiving units 101 b in the area of the memory 105 .

Specifically, in the first imaging, the sequencer 104 stops the imaging device 101 at the first initial position, which is the first imaging position, and causes the imaging device 101 to perform imaging (that is, exposure) at this time. At the first initial position, for example, as shown in FIG. 14, the G light receiving area is arranged at the upper end of the optical image. Then, the sequencer 104 transfers the M×N pixel data of the image pickup device 201 obtained by the first image pickup to the first area 1a of the memory 105 while moving the image pickup device 101 to the second image pickup position. do. At this time, each of the M×N pieces of pixel data is stored in the first area 1a in association with the pixel position of the picked-up image composed of M×N pieces of pixels.

The RGB decomposing unit 107 decomposes the M×N pixel data into R pixel data, G pixel data, and B pixel data, and outputs the R pixel data, G pixel data, and B pixel data to the synthesizing unit 108 .

When the sequencer 104 moves the image sensor 201 to the second imaging position, as shown in FIGS. 14 and 15, the sequencer 104 moves the image sensor 201 to the left from the first initial position by 1/2 of the horizontal pixel pitch, It is moved upward by the vertical pixel pitch and stopped. Then, the sequencer 104 causes the imaging element 201 to perform the second imaging (that is, exposure). The sequencer 104 transfers the M×N pixel data of the image sensor 201 obtained by the second imaging to the second area 1b of the memory 105 while moving the image sensor 201 to the third imaging position. Also at this time, each of the M×N pieces of pixel data is stored in the second area 1b in association with the pixel position in the captured image.

The shifter 106 shifts the pixel position of each of the M×N pieces of pixel data stored in the second area 1b. As a result, the pixel position of each of the M×N pieces of pixel data obtained by the second imaging is shifted to the pixel position when the imaging element 201 is arranged at the first initial position.

The RGB decomposing unit 107 decomposes the shifted M×N pixel data into R pixel data, G pixel data, and B pixel data, and outputs them to the synthesizing unit 108 .

When the sequencer 104 moves the image pickup device 201 to the third image pickup position, as shown in FIGS. 14 and 15, the sequencer 104 moves the image pickup device 201 from the second image pickup position to the right by the horizontal pixel pitch and stops. Then, the sequencer 104 causes the imaging element 201 to perform the third imaging (that is, exposure). The sequencer 104 transfers the M×N pixel data of the image sensor 201 obtained by the third imaging to the third area 1c of the memory 105 while moving the image sensor 201 to the fourth imaging position. Also at this time, each of the M×N pieces of pixel data is stored in the third area 1c in association with the pixel position in the captured image.

The shifter 106 shifts the pixel positions of the M×N pieces of pixel data stored in the third area 1c. As a result, the pixel positions of the M×N pieces of pixel data obtained by the third imaging are shifted to the pixel positions when the imaging element 101 is arranged at the first initial position.

The RGB decomposing unit 107 decomposes the shifted M×N pixel data into R pixel data, G pixel data, and B pixel data, and outputs them to the synthesizing unit 108 .

By such first to third imaging, R pixel data and G and the pixel data of B are output to the synthesizing unit 108 .

Here, the sequencer 104 moves the imaging element 201 to the second initial position, which is the fourth imaging position. At this time, as shown in FIGS. 14 and 15, the sequencer 104 moves the image sensor 201 from the third image pickup position to the left by a distance of 1/2 of the horizontal pixel pitch, and further downward by the vertical pixel pitch. Move it above and stop it. As shown in FIG. 14, when the image pickup device 201 is at the second initial position, each light receiving region that overlaps with the optical image when the image pickup device 201 is at the first initial position is shifted from the vertical pixel pitch and the horizontal pixel pitch. are also shifted by a short distance.

Then, the sequencer 104 causes the imaging element 201 to perform fourth imaging (that is, exposure). The sequencer 104 transfers the M×N pixel data of the image pickup device 201 obtained by the fourth image pickup to the fourth area 2 a of the memory 105 while moving the image pickup device 201 to the fifth image pickup position. At this time, each of the M×N pieces of pixel data is stored in the fourth area 2a in association with the pixel position of the picked-up image made up of M×N pieces of pixels.

The RGB decomposing unit 107 decomposes the M×N pixel data into R pixel data, G pixel data, and B pixel data, and outputs the R pixel data, G pixel data, and B pixel data to the synthesizing unit 108 .

In order to perform the fifth and sixth imaging, the sequencer 104 performs processing similar to that of the second and third imaging based on the second initial position instead of the first initial position.

As a result of the fourth to sixth imaging, R pixel data and G and the pixel data of B are output to the synthesizing unit 108 .

Next, the sequencer 104 moves the image sensor 201 to the third initial position, which is the seventh initial position. Then, in order to perform the seventh to ninth imaging, the sequencer 104 performs the same processing as the first to third imaging based on the third initial position instead of the first initial position.

By performing the seventh to ninth imaging, R pixel data and G and the pixel data of B are output to the synthesizing unit 108 .

Next, the sequencer 104 moves the image sensor 201 to the fourth initial position, which is the tenth initial position. Then, in order to perform the 10th to 12th imaging, the sequencer 104 performs the same processing as the 1st to 3rd imaging based on the fourth initial position instead of the first initial position.

As a result of the 10th to 12th imaging, R pixel data and G and the pixel data of B are output to the synthesizing unit 108 .

Next, the sequencer 104 moves the image sensor 201 to the fifth initial position, which is the 13th initial position. Then, in order to perform the 13th to 15th imaging, the sequencer 104 performs the same processing as for the 1st to 3rd imaging based on the fifth initial position instead of the first initial position.

By such 13th to 15th imaging, R pixel data and G and the pixel data of B are output to the synthesizing unit 108 .

Then, the sequencer 104 moves the image sensor 201 to the sixth initial position, which is the sixteenth initial position. Then, in order to perform the 16th to 18th imaging, the sequencer 104 performs the same processing as for the 1st to 3rd imaging based on the sixth initial position instead of the first initial position.

By such 16th to 18th imaging, R pixel data and G and the pixel data of B are output to the synthesizing unit 108 .

FIG. 16 is a diagram showing the shift of the light receiving area within the pixel area in this modified example.

The sequencer 104 moves the imaging device 201 so that the light receiving area of G, for example, shifts to the positions shown in (a) to (f) of FIG. 16 with respect to the pixel area in the optical image. In the example shown in FIG. 16, the sequencer 104 sequentially arranges the G light-receiving regions at six positions within the pixel region by controlling the vertical actuator 103a and the horizontal actuator 103b. These six positions are separated from each other by a distance shorter than the distance between the plurality of light receiving portions 101b, that is, the vertical pixel pitch and the horizontal pixel pitch.

When the G light receiving regions are arranged at different positions in the pixel region as described above, the synthesizing unit 108 outputs the shifted G pixel data from the light receiving unit 101b corresponding to the light receiving regions. is added. Thereby, the synthesizing unit 108 calculates the pixel value of G in the pixel area as the pixel value included in the synthesized image. Similarly, the synthesizing unit 108 calculates the R pixel value and the B pixel value in the pixel region as pixel values included in the synthesized image.

As described above, even in the imaging apparatus 200 including the imaging device 201 in the delta arrangement, the light receiving area is shifted multiple times within the pixel area, and exposure is performed at the shifted positions. Therefore, similarly to the above embodiment, the aperture ratio of the image sensor 201 with respect to the pixel area can be equivalently increased. As a result, even if a positioning error occurs in the imaging element 201, it is possible to suppress the occurrence of color unevenness.

In the above embodiment and modifications, each component such as the sequencer 104, the shifter 106, the RGB decomposition unit 107, and the synthesis unit 108 is configured by dedicated hardware, or a software program suitable for each component. may be implemented by executing Each component may be realized by reading and executing a software program recorded in a recording medium such as a hard disk or a semiconductor memory by a program execution unit such as a CPU or processor. Here, the software that implements the image capturing apparatus of the above embodiments and modified examples is a program that causes a computer to execute each step of the flowchart shown in FIG. 11 .

As described above, the imaging apparatus according to the present disclosure has been described based on the embodiment and its modification, but the present disclosure is not limited to this embodiment and its modification. As long as it does not deviate from the spirit of the present disclosure, various modifications that can be conceived by those skilled in the art are applied to the present embodiment, and forms constructed by combining the components of different embodiments are also included within the scope of the present disclosure. may

For example, in the above-described embodiments and their modifications, the imaging device includes the imaging elements of the Bayer array or the delta array, but instead of these imaging elements, it may include imaging elements of other arrays. . Even in such a case, the aperture ratio of the image sensor can be increased, and the occurrence of color unevenness can be suppressed. Also, the order of movement of the imaging elements shown in FIGS. 8 and 14 is an example, and the imaging elements may be moved in another order.

In the above embodiment and its modification, the sequencer 104 moves the image sensor by controlling the vertical actuator 103a and the horizontal actuator 103b, but the optical system including the lens 102 may be moved. That is, in the present disclosure, the imaging device may be moved or the optical system may be moved in order to change the relative position of the imaging surface of the imaging device with respect to the optical image. Alternatively, both the imaging element and the optical system may be moved.

In addition, in the above-described embodiment and its modification, the color of the color filter 101a is RGB, but it is not limited to this, and may be, for example, cyan, magenta, yellow, green, or the like.

The imaging device according to the present disclosure has the effect of improving image quality, and can be used for cameras, for example.

Reference Signs List 100, 200 imaging device 101, 201 imaging device 101a color filter 101b light receiving unit 102 lens 103a vertical actuator (driving unit)
103b Lateral actuator (drive unit)
104 sequencer 105 memory 106 shifter 107 RGB decomposing unit 108 synthesizing unit 111 translucent unit

Claims (8)

an image pickup device having a plurality of light receiving portions that are spaced apart from each other at every pixel pitch along one surface and receive light that has passed through a light transmitting portion of any one of RGB colors included in the color filter;
a driving unit that changes a relative position within the imaging plane with respect to an optical image projected on the imaging plane of the imaging device;
a synthesizing unit that generates a synthesized image by synthesizing pixel data output from each of the plurality of light receiving units of the imaging element each time the imaging element is exposed;
a light-receiving region of the first light-receiving unit corresponding to a first color that is one of the RGB colors is located within one pixel region of the optical image;
The drive unit
The light receiving regions of the first light receiving unit are arranged in sequence at a plurality of positions separated from each other by a distance shorter than the pixel pitch in the one pixel region,
The synthesizing unit
generating the composite image based on pixel data output from the first light receiving unit when the light receiving areas of the first light receiving unit are arranged in order at each of the plurality of positions in the pixel area;
The pixel area in the optical image is
It has a size that overlaps the entire light receiving area of one light receiving unit on the imaging surface and a non-light receiving area around the light receiving area, and does not overlap the light receiving area of another light receiving unit.
Imaging device. The drive unit
arranging the light receiving regions of the first light receiving unit in order at each of a plurality of positions outside the one pixel region;
The synthesizing unit
The composite image is generated based on pixel data output from the first light receiving unit when the light receiving regions of the first light receiving unit are sequentially arranged at each of the plurality of positions within the pixel region and outside the pixel region. The imaging device according to claim 1, wherein the image is generated. The imaging device further comprises
a memory for storing pixel data output from each of the plurality of light receiving units of the imaging device;
a shifter that shifts the pixel position of the pixel data stored in the memory;
The drive unit further
By changing the relative position to the outside of the pixel area, the light receiving area of the second light receiving section corresponding to a second color different from the first color among the RGB is shifted to the plurality of light receiving areas within the pixel area. placed in each of the positions in turn,
The shifter is
When the pixel data output from the second light receiving section is transferred to the memory, the pixel position of the pixel data of the second light receiving section stored in the memory is replaced with the pixel position of the pixel data of the first light receiving section. Shift to pixel position,
The synthesizing unit
By adding the shifted pixel data output from the second light receiving unit when the light receiving regions of the second light receiving unit are arranged in order at each of the plurality of positions in the pixel region, The imaging device according to claim 2, wherein the pixel value of the second color in the pixel area is calculated as the pixel value included in the composite image. an image pickup device having a plurality of light receiving portions that are spaced apart from each other at every pixel pitch along one surface and receive light that has passed through a light transmitting portion of any one of RGB colors included in the color filter;
a driving unit that changes a relative position within the imaging plane with respect to an optical image projected on the imaging plane of the imaging device;
a synthesizing unit that generates a synthesized image by synthesizing pixel data output from each of the plurality of light receiving units of the imaging element each time the imaging element is exposed;
The arrangement of the plurality of light-transmitting parts included in the color filter is a Bayer arrangement,
The drive unit
a plurality of light receiving regions of the first light receiving unit corresponding to a first color, which is one of the RGB colors, separated from each other by a distance shorter than the pixel pitch in one pixel region of the optical image; and sequentially arranging the light-receiving regions of the first light-receiving unit at each of a plurality of positions outside the one pixel region,
Arrangement within the pixel region is performed four times as one set, and arrangement outside the pixel region is performed four times, for a total of 16 times ;
The synthesizing unit
The composite image is generated based on pixel data output from the first light receiving unit when the light receiving regions of the first light receiving unit are sequentially arranged at each of the plurality of positions within the pixel region and outside the pixel region. generate,
Imaging device. an image pickup device having a plurality of light receiving portions that are spaced apart from each other at every pixel pitch along one surface and receive light that has passed through a light transmitting portion of any one of RGB colors included in the color filter; ,
a driving unit that changes a relative position within the imaging plane with respect to an optical image projected on the imaging plane of the imaging device;
a synthesizing unit configured to generate a synthesized image by synthesizing pixel data output from each of the plurality of light receiving units of the imaging element each time the imaging element is exposed;
a memory for storing pixel data output from each of the plurality of light receiving units of the imaging device;
a shifter that shifts the pixel position of the pixel data stored in the memory;
The light receiving area of the first light receiving unit corresponding to R, which is the first color, is located within one pixel area of the optical image,
The color filters are each composed of a plurality of sets each including an R light-transmitting portion, two G light-transmitting portions, and a B light-transmitting portion,
By changing the relative position, the drive unit
The light-receiving regions of the first light-receiving unit are sequentially arranged at four positions separated from each other by a distance shorter than the pixel pitch in the one pixel region,
The light-receiving regions of the second light-receiving unit corresponding to B, which is the second color, are arranged in order at each of the four positions,
The light-receiving regions of the third light-receiving unit corresponding to G, which is the third color, are arranged in order at each of the four positions,
arranging the light receiving areas of the fourth light receiving unit corresponding to G, which is the fourth color, at each of the four positions in order;
The shifter is
When the pixel data output from the second light receiving section is transferred to the memory, the pixel position of the pixel data of the second light receiving section stored in the memory is replaced with the pixel position of the pixel data of the first light receiving section. Shift to pixel position,
When the pixel data output from the third light receiving section is transferred to the memory, the pixel position of the pixel data of the third light receiving section stored in the memory is replaced with the pixel position of the pixel data of the first light receiving section. Shift to pixel position,
When the pixel data output from the fourth light receiving section is transferred to the memory, the pixel position of the pixel data of the fourth light receiving section stored in the memory is replaced with the pixel position of the pixel data of the first light receiving section. Shift to pixel position,
The synthesizing unit
Based on the pixel data output from the first light receiving unit when the light receiving regions of the first light receiving unit are arranged in order at each of the four positions in the pixel region, the first light receiving unit in the pixel region calculating the pixel value of the color as the pixel value included in the composite image;
the pixel region based on pixel data that is output from the second light receiving portion and shifted when the light receiving regions of the second light receiving portion are arranged in order at each of the four positions in the pixel region; calculating the pixel value of the second color in as the pixel value included in the composite image;
When the light receiving areas of the third light receiving section are arranged in order in the four positions in the pixel area, the pixel is output from the third light receiving section and shifted based on the pixel data. calculating pixel values of the third color in the region;
When the light receiving areas of the fourth light receiving section are arranged in order in the four positions in the pixel area, the pixel is output from the fourth light receiving section and shifted based on the pixel data. calculating pixel values of the fourth color in the region;
calculating a pixel value of G included in the composite image based on the pixel value of the third color and the pixel value of the fourth color ;
Imaging device. an image pickup device having a plurality of light receiving portions that are spaced apart from each other at every pixel pitch along one surface and receive light that has passed through a light transmitting portion of any one of RGB colors included in the color filter;
a driving unit that changes a relative position within the imaging plane with respect to an optical image projected on the imaging plane of the imaging device;
a synthesizing unit that generates a synthesized image by synthesizing pixel data output from each of the plurality of light receiving units of the imaging element each time the imaging element is exposed;
The arrangement of the plurality of light-transmitting portions included in the color filter is a delta arrangement,
The drive unit
a plurality of light receiving regions of the first light receiving unit corresponding to a first color, which is one of the RGB colors, separated from each other by a distance shorter than the pixel pitch in one pixel region of the optical image; and sequentially arranging the light-receiving regions of the first light-receiving unit at each of a plurality of positions outside the one pixel region,
Placement within the pixel region is performed 6 times as one set, and placement outside the pixel region is performed 3 times, for a total of 18 times ,
The synthesizing unit
The composite image is generated based on pixel data output from the first light receiving unit when the light receiving regions of the first light receiving unit are sequentially arranged at each of the plurality of positions within the pixel region and outside the pixel region. generate,
Imaging device. An image pickup device is used which has a plurality of light receiving portions which are spaced apart from each other at every pixel pitch along one surface and receive light transmitted through the light transmitting portions of any one of RGB colors included in the color filter. An imaging method,
changing the relative position in the imaging plane with respect to the optical image projected on the imaging plane of the imaging element;
generating a composite image by synthesizing pixel data output from each of the plurality of light receiving units of the imaging device each time the imaging device is exposed;
a light-receiving region of the first light-receiving unit corresponding to a first color that is one of the RGB colors is located within one pixel region of the optical image;
In the change of the relative position of the imaging plane,
The light receiving regions of the first light receiving unit are arranged in sequence at a plurality of positions separated from each other by a distance shorter than the pixel pitch in the one pixel region,
In generating the composite image,
generating the composite image based on pixel data output from the first light receiving unit when the light receiving areas of the first light receiving unit are arranged in order at each of the plurality of positions in the pixel area;
The pixel area in the optical image is
It has a size that overlaps the entire light receiving area of one light receiving unit on the imaging surface and a non-light receiving area around the light receiving area, and does not overlap the light receiving area of another light receiving unit.
Imaging method. An imaging device is used which has a plurality of light-receiving portions which are spaced apart from each other at every pixel pitch along one surface and which receive light transmitted through the light-transmitting portions of any one of RGB colors included in the color filter. A program for imaging with
changing the relative position in the imaging plane with respect to the optical image projected on the imaging plane of the imaging element;
Generating a composite image by synthesizing pixel data output from each of the plurality of light receiving units of the imaging device each time the imaging device is exposed,
let the computer run
a light-receiving region of the first light-receiving unit corresponding to a first color that is one of the RGB colors is located within one pixel region of the optical image;
In the change of the relative position of the imaging plane,
The light receiving regions of the first light receiving unit are arranged in sequence at a plurality of positions separated from each other by a distance shorter than the pixel pitch in the one pixel region,
In generating the composite image,
generating the composite image based on pixel data output from the first light receiving unit when the light receiving areas of the first light receiving unit are arranged in order at each of the plurality of positions in the pixel area;
The pixel area in the optical image is
It has a size that overlaps the entire light receiving area of one light receiving unit on the imaging surface and a non-light receiving area around the light receiving area, and does not overlap the light receiving area of another light receiving unit.
program.

Images