JP2023174740A - Imaging apparatus - Google Patents

Abstract

To achieve high accuracy in noise reduction.SOLUTION: An imaging apparatus includes an imaging unit that images a subject and outputs first image data and second image data that is temporally later than the first image data, a calculation unit that calculates a first weight of the first image data on the basis of first noise-removed image data from which noise has been removed from the first image data and the second image data, an afterimage reduction processing unit that reduces afterimages in the second image data, and a removing unit that removes noise from the second image data that has been subjected to afterimage reduction processing by the afterimage reduction processing unit, on the basis of the first noise-removed image data and the first weight calculated by the calculation unit.SELECTED DRAWING: Figure 6

Description

The present invention relates to an imaging device.

As a noise removal method for moving images, there is weighted averaging processing of past frames and current frames (for example, see Patent Document 1 below). However, in the noise reduction system of Patent Document 1, in order to further improve accuracy, it is necessary to hold many past frames in the image memory, but there is also a limit to the amount of memory that can be used in the image memory. On the other hand, if the amount of memory used is limited, no improvement in noise reduction accuracy can be expected.

Japanese Patent Application Publication No. 06-062283

An imaging device according to a first aspect of the invention disclosed in this application images a subject and uses first image data, second image data temporally later than the first image data, and the second image data. an imaging unit that outputs third image data that is temporally later than the first image data; A first weight is calculated, and a second weight of the second image data is calculated based on the second denoised image data and the third image data from which noise is removed from the second image data, and the first weight is calculated. and a calculation unit that calculates a third weight based on the second weight, and a removal unit that removes noise from the third image data based on the third weight calculated by the calculation unit.

An imaging device according to a second aspect of the invention disclosed in this application includes an imaging unit that images a subject and outputs first image data and second image data that is temporally later than the first image data. , an extraction unit that extracts first spatial information indicating characteristics of a spatial structure included in the image from at least one of the first noise-removed image data and the second image data, which are noise-removed from the first image data; a calculation unit that calculates a first weight of the first image data based on the first spatial information extracted by the extraction unit; the first noise-removed image data; and the first weight calculated by the calculation unit; , a removing unit that removes noise from the second image data based on .

An imaging device according to a third aspect of the invention disclosed in the present application includes an imaging unit that images a subject and outputs first image data and second image data that is temporally later than the first image data. , a calculation unit that calculates a first weight of the first image data based on the first noise-removed image data and the second image data from which noise has been removed from the first image data; second image data that has been subjected to afterimage reduction processing by the afterimage reduction processing unit based on the afterimage reduction processing unit that reduces afterimages, the first noise removed image data, and the first weight calculated by the calculation unit; and a removal unit that removes noise from the noise.

An imaging device according to a fourth aspect of the invention disclosed in this application includes an imaging unit that images a subject and outputs first image data and second image data that is temporally later than the first image data. , a calculation unit that calculates a first weight of the first image data based on the first noise-removed image data and the second image data from which noise has been removed from the first image data; and the first noise-removed image data. and a first weight calculated by the calculation unit, a removal unit that removes noise from the second image data, and a second noise removal unit that removes noise from the second image data by the removal unit. and an afterimage reduction processing unit that reduces afterimages in image data.

An imaging device according to a fifth aspect of the invention disclosed in the present application includes an input section that receives an input of a frame rate when imaging a subject, and a first input section that sets an exposure time corresponding to the frame rate input by the input section. an adjustment unit that adjusts the exposure time to a second exposure time that is longer than the first exposure time; and an adjustment unit that images the subject based on the frame rate and the second exposure time adjusted by the adjustment unit; an imaging unit that outputs image data and second image data that is temporally later than the first image data; a calculation unit that calculates a first weight of the first image data based on the first noise removed image data, and a first weight calculated by the calculation unit, from the second image data. and a removing section that removes noise.

An image processing device according to a sixth aspect of the invention disclosed in the present application includes first image data, second image data captured temporally later than the first image data, and temporally larger image data than the second image data. an acquisition unit that acquires third image data captured later in the first image data; 1 weight is calculated, and a second weight of the second image data is calculated based on the second noise-removed image data and the third image data in which noise is removed from the second image data, and the second weight is calculated based on the first weight and the third image data. The image forming apparatus includes a calculation section that calculates a third weight based on the second weight, and a removal section that removes noise from the third image data based on the third weight calculated by the calculation section.

An image processing device according to a seventh aspect of the invention disclosed in the present application includes an acquisition unit that acquires first image data and second image data captured temporally later than the first image data; an extraction unit that extracts first spatial information indicating characteristics of a spatial structure included in the image from at least one of the first noise-removed image data and the second image data from which noise has been removed from the first image data; a calculation unit that calculates a first weight of the first image data based on the extracted first spatial information; and a first weight calculated by the first noise-removed image data and the calculation unit. and a removing unit that removes noise from the second image data.

An image processing device according to an eighth aspect of the invention disclosed in the present application includes an acquisition unit that acquires first image data and second image data captured temporally later than the first image data; a calculation unit that calculates a first weight of the first image data based on the first noise-removed image data and the second image data from which noise has been removed from the first image data; and reducing afterimages in the second image data. noise is removed from the second image data that has been subjected to afterimage reduction processing by the afterimage reduction processing section based on the afterimage reduction processing section that performs afterimage reduction processing, the first noise removed image data, and the first weight calculated by the calculation section. and a removing section for removing.

An image processing device according to a ninth aspect of the invention disclosed in the present application includes an acquisition unit that acquires first image data and second image data captured temporally later than the first image data; a calculation unit that calculates a first weight of the first image data based on the first noise-removed image data and the second image data from which noise has been removed from one image data; the first noise-removed image data; a removing unit that removes noise from the second image data based on the first weight calculated by the calculating unit; and a second noise-removed image data from which noise is removed from the second image data by the removing unit. and an afterimage reduction processing section that reduces afterimages of.

An image processing program according to a tenth aspect of the invention disclosed in the present application includes first image data, second image data captured temporally later than the first image data, and second image data captured temporally later than the second image data. an acquisition process of acquiring third image data captured after the first image data; 1 weight is calculated, and a second weight of the second image data is calculated based on the second noise-removed image data and the third image data in which noise is removed from the second image data, and the second weight is calculated based on the first weight and the third image data. A processor executes a calculation process of calculating a third weight based on the second weight, and a removal process of removing noise from the third image data based on the third weight calculated by the calculation process. let

An image processing program according to an eleventh aspect of the invention disclosed in this application includes an acquisition process of acquiring first image data and second image data captured temporally later than the first image data; an extraction process for extracting first spatial information indicating a feature of a spatial structure included in the image from at least one of the first noise-removed image data and the second image data from which noise has been removed from the first image data, and the extraction process a calculation process of calculating a first weight of the first image data based on the extracted first spatial information; and a first weight calculated by the first noise-removed image data and the calculation process. and causes the processor to perform a removal process of removing noise from the second image data.

An image processing program according to a twelfth aspect of the invention disclosed in this application includes an acquisition process of acquiring first image data and second image data captured temporally later than the first image data; A calculation process of calculating a first weight of the first image data based on the first noise-removed image data and the second image data from which noise has been removed from the first image data, and reducing afterimages in the second image data. noise is removed from the second image data that has been subjected to the afterimage reduction process by the afterimage reduction process, based on the afterimage reduction process performed by the afterimage reduction process, the first noise removed image data, and the first weight calculated by the calculation process. The processor executes the removal process.

An image processing program according to a thirteenth aspect of the invention disclosed in this application includes an acquisition process of acquiring first image data and second image data captured temporally later than the first image data; a calculation process of calculating a first weight of the first image data based on the first noise-removed image data and the second image data from which noise has been removed from one image data; the first noise-removed image data; a first weight calculated by the calculation process, and a removal process for removing noise from the second image data based on the first weight; The processor is caused to execute afterimage reduction processing for reducing afterimages of.

FIG. 1 is a block diagram showing an example of the hardware configuration of an imaging device. FIG. 2 is a block diagram showing an example of the functional configuration of the imaging device according to the first embodiment. FIG. 3 is a flowchart illustrating an example of a recursive noise removal procedure by the imaging device according to the first embodiment. FIG. 4 is an explanatory diagram showing an example of recursive noise removal using adjusted weights. FIG. 5 is an explanatory diagram showing an example of recursive noise removal without applying adjusted weights. FIG. 6 is a block diagram showing an example of the functional configuration of the imaging device according to the second embodiment. FIG. 7 is a flowchart illustrating an example of a recursive noise removal procedure by the imaging device according to the second embodiment. FIG. 8 is a block diagram showing an example of the functional configuration of an imaging device according to the third embodiment. FIG. 9 is a flowchart illustrating an example of a recursive noise removal procedure by the imaging device according to the third embodiment. FIG. 10 is a block diagram showing an example of the functional configuration of the imaging device according to the fourth embodiment. FIG. 11 is an explanatory diagram showing an example of interpolation processing by the afterimage reduction processing section according to the fourth embodiment. FIG. 12 is a flowchart illustrating an example of a recursive noise removal procedure by the imaging device according to the fourth embodiment. FIG. 13 is a block diagram showing an example of the functional configuration of an imaging device according to the fifth embodiment. FIG. 14 is a flowchart illustrating an example of a recursive noise removal procedure by the imaging device according to the fifth embodiment. FIG. 15 is an explanatory diagram showing an example of afterimage reduction processing according to the sixth embodiment. FIG. 16 is a block diagram showing an example of the functional configuration of an imaging device according to the sixth embodiment. FIG. 17 is a flowchart illustrating an example of a recursive noise removal procedure by the imaging device according to the sixth embodiment. FIG. 18 is an explanatory diagram showing an example of moving image capturing according to the seventh embodiment. FIG. 19 is a block diagram showing an example of the functional configuration of an imaging device according to the seventh embodiment. FIG. 20 is a flowchart illustrating an example of an imaging processing procedure by the imaging device.

Example 1 will be explained. As a method for removing noise from moving images, there is weighted averaging processing of past image data from which noise has been removed and current image data from which noise has not been removed. In order to improve noise removal accuracy, it is necessary to store a large number of past frames in memory, but this increases memory usage.

Therefore, in the first embodiment, in order to suppress the number of noise-removed image data to the necessary minimum, the imaging device performs noise removal by recursively using the weights used for noise removal. Improve accuracy and reduce memory usage.

The weight is the reliability of the preceding image data with respect to the succeeding image data for each area when temporally continuous preceding image data and subsequent image data are compared in the same area in a moving image. A region is a pixel set of one or more pixels. The region may be the entire image data.

The weight ranges, for example, from 0.0 to 1.0, and the larger the value, the higher the reliability. The more similar the same areas of the preceding image data and the subsequent image data are, the larger the weight value becomes. By using weights recursively, the value of weight becomes smaller for afterimages of past image data, and afterimages are reduced. Example 1 will be described in detail below.

<Example of hardware configuration of imaging device>
FIG. 1 is a block diagram showing an example of the hardware configuration of an imaging device. The imaging device 100 is a device capable of capturing moving images, and specifically, for example, is a digital camera, a digital video camera, a smartphone, a tablet, a personal computer, or a game console. In FIG. 1, a digital camera will be described as an example of an imaging device.

The imaging device 100 includes a processor 101, a storage device 102, a drive unit 103, an optical system 104, an image sensor 105, an AFE (Analog Front End) 106, an LSI (Large Scale Integration) 107, and an operation device 108. , a sensor 109 , a display device 110 , a communication IF (Interface) 111 , and a bus 112 . The processor 101 , storage device 102 , drive unit 103 , LSI 107 , operation device 108 , sensor 109 , display device 110 , and communication IF 111 are connected to a bus 112 .

A processor 101 controls the imaging device 100. The storage device 102 becomes a work area for the processor 101. Furthermore, the storage device 102 is a non-temporary or temporary recording medium that stores various programs and data. Examples of the storage device 102 include ROM (Read Only Memory), RAM (Random Access Memory), HDD (Hard Disk Drive), and flash memory. A plurality of storage devices 102 may be installed in the imaging apparatus 100, and at least one of them may be detachably attached to the imaging apparatus 100.

The drive unit 103 drives and controls the optical system 104. The drive unit 103 includes a drive circuit 103a and a drive source 103b. The drive circuit 103a controls the drive source 103b according to instructions from the processor 101. The drive source 103b is, for example, a motor, and controls the zooming lens 141b and the focusing lens 141c in the optical system 104 in the optical axis direction and controls the opening and closing of the diaphragm 142 under the control of the drive circuit 103a.

Optical system 104 includes a plurality of lenses (lens 141a, zooming lens 141b, and focusing lens 141c) arranged in the optical axis direction, and an aperture 142. The optical system 104 collects object light and emits it to the image sensor 105 .

The image sensor 105 receives object light from the optical system 104 and converts it into an electrical signal. The image sensor 105 may be, for example, an XY addressing type solid-state image sensor (for example, CMOS (Complementary Metal-Oxide Semiconductor)), or a sequential scanning type solid-state image sensor (for example, CCD (Charge Coupled Device)). There may be.

On the light-receiving surface of the image sensor 105, a plurality of light-receiving elements (pixels) are arranged in a matrix. In each pixel of the image sensor 105, a plurality of types of color filters, each of which transmits light of a different color component, are arranged according to a predetermined color arrangement (eg, Bayer arrangement). Therefore, each pixel of the image sensor 105 outputs an analog electrical signal corresponding to each color component by color separation using a color filter.

The AFE 106 is an analog front end circuit that performs signal processing on analog electrical signals from the image sensor 105. The AFE 106 generates RAW image data by sequentially performing electrical signal gain adjustment, analog signal processing (correlated double sampling, black level correction, etc.), A/D conversion processing, and digital signal processing (defective pixel correction, etc.). , output to LSI. The drive unit 103, optical system 104, image sensor 105, and AFE 106 described above constitute an image capture unit 120.

The LSI 107 performs specific processing on the RAW image data from the AFE 106, such as image processing such as color interpolation, white balance adjustment, edge enhancement, gamma correction, and gradation conversion, as well as encoding processing, decoding processing, and compression/expansion processing. It is an integrated circuit. Specifically, the LSI 107 is a PLD (Programmable Logic D) such as an ASIC (Application Specific Integrated Circuit) or an FPGA (Field-Programmable Gate Array). evice).

The operating device 108 inputs commands and data. Examples of the operation device 108 include various buttons including a release button, switches, dials, and touch panels. A sensor is a device that detects information, and includes, for example, an AF (Automatic Focus) sensor, an AE (Automatic Exposure) sensor, a gyro sensor, an acceleration sensor, a temperature sensor, and the like. The display device 110 displays image data and a setting screen. The display device 110 includes a rear monitor on the back of the imaging device 100 and an electronic viewfinder. The communication IF 111 connects to a network and transmits and receives data.

<Functional configuration example of imaging device 100>
FIG. 2 is a block diagram showing an example of the functional configuration of the imaging device 100 according to the first embodiment. The imaging device 100 includes an acquisition section 201, a calculation section 202, a removal section 203, an image storage section 204, and a first memory 205. The acquisition unit 201, the calculation unit 202, the removal unit 203, the image storage unit 204, and the first memory 205 constitute the image processing device 200.

The acquisition unit 201 acquires a series of temporally continuous image data captured by the imaging unit 120. The acquisition unit 201 is, for example, a buffer memory that is one of the storage devices 102. Note that when the imaging device 100 is an image processing device 200 without the imaging unit 120, a series of image data is acquired from the storage device 102 or the communication IF 111. The acquisition unit 201 is, for example, a buffer memory that is one of the storage devices 102.

Specifically, the calculation unit 202, removal unit 203, and image storage unit 204 are realized, for example, by causing the processor 101 to execute a program stored in the storage device 102 shown in FIG. 1, or by the LSI 107. . Further, the first memory 205 is one of the storage devices 102.

Here, a series of image data will be explained using consecutive first to third image data as an example. It is assumed that the imaging unit 120 images the subject and outputs the first image data to the third image data in this order. The first image data is the temporally oldest image data output from the imaging unit 120 among the consecutive first to third image data. The second image data is image data output from the imaging unit 120 temporally later than the first image data. The third image data is image data output from the imaging unit 120 temporally later than the second image data.

The calculation unit 202 includes a weight calculation unit 221 , a weight adjustment unit 222 , a weight storage unit 223 , and a second memory 224 . The weight calculation unit 221 calculates the first weight of the first image data based on the first noise-removed image data and the second image data from which noise has been removed from the first image data by the removal unit 203. When calculating the first weight, the calculation unit 202 reads the first noise-removed image data from the first memory 205 as past image data preceding the second image data. The first image data is preceding image data of the second image data, and the second image data is subsequent image data of the first image data.

Further, the calculation unit 202 calculates the second weight of the second image data based on the second noise-removed image data and the third image data from which noise has been removed from the second image data by the removal unit 203. When calculating the second weight, the calculation unit 202 reads the second noise-removed image data from the first memory 205 as past image data preceding the third image data. The second image data is preceding image data of the third image data, and the third image data is subsequent image data of the second image data.

As described above, the weight is the reliability of the preceding image data relative to the succeeding image data for each area when temporally continuous preceding image data and subsequent image data are compared in the same area. A region is a pixel set of one or more pixels. The region may be the entire image data.

Here, image data subjected to noise removal processing before time t is written as Iold, and image data at time t is written as It. When Iold and It are not distinguished, they are simply expressed as image data I. The image data I is, for example, a matrix corresponding to each value of a matrix-like pixel group. Therefore, the weights are also expressed by a matrix and are denoted W. Each element of the weight W corresponds to a region (x, y) in the image data I, and the weight w of that region ranges from, for example, 0.0 to 1.0.

The higher the weight (reliability) w in a certain area, the less blurring of the image between the preceding image data and the subsequent image data in that area, that is, the less afterimage. The lower the value, the more blurring occurs between the preceding image data and the subsequent image data in the region, that is, the more afterimages there are.

The weight w of the area in the preceding image data Iold with respect to the succeeding image data at time t is calculated by the following equation (1).

w=f(It,Iold)...(1)

The function f(·) is a function for determining the weight w of the area in the preceding image data Iold. Note that if the area is the entire preceding image data Iold, the weight w will be one value. Further, the weight calculation unit 221 may calculate a plurality of weights W according to the number of channels of the preceding image data Iold. It is preferable that the weight w of the region of the preceding image data Iold be increased as the degree of similarity of the subsequent image data It is greater. An example of how to obtain the weight w of a region is the following equation (2).

The function fx,y means calculation of the weight w of the region (x,y). x indicates the position of the area of the preceding image data Iold in the column direction, and y indicates the position in the row direction. Further, the function φx,y is a function that obtains partial image data of a neighboring area centered on the area (x, y). Also, ||·|| means L2 norm. Further, σ is a parameter that adjusts the weight calculation result. σ is a parameter for adjusting the sensitivity to the image difference, which is the value of the numerator of the above equation (2), and when the value of σ is increased, a larger weight is calculated even if the image difference is large.

The weight adjustment unit 222 adjusts the weight W calculated by the weight calculation unit 221 based on the latest weight Wold stored in the second memory 224. Let the adjusted weight W be a weight W'. The weight storage unit 223 stores the weight W before adjustment by the weight adjustment unit 222 in the second memory 224 as Wold (see equation (4)). The second memory 224 is configured by the storage device 102.

The adjusted weight W' is calculated by the following formula (3).

W′=g(W,Wold)...(3)

The function g(·) is a function that adjusts the weight W. Wold is updated by the weight W before adjustment, as shown in equation (4) below. Here, the weight W is directly updated to Wold, but the size of the weight W may be reduced and updated to Wold. The weight is called a past weight Wold.

Wold=W...(4)

Further, the right side of equation (3) is expressed, for example, by equation (5) below.

Further, the adjusted weight W' may be set with an upper limit or a lower limit, or may be multiplied by a uniform adjustment coefficient. Further, the right side of equation (3) may be expressed by, for example, equation (6) below.

g(W,Wold)=min(W,Wold)...(6)

In the above equation (6), the weight W and the past weight Wold are compared for each element, and the weight w having the smaller value is adopted. In this way, the weight w' of the area of the adjusted weight W' becomes smaller than the weight w before adjustment.

Here, adjustment of the weight W will be explained using the above-mentioned first image data to third image data as examples. Note that the weight W in the case where the first image data is the preceding image data and the first noise-removed image data obtained by removing noise from the first image data is the past image Iold is defined as the weight W1. Further, the weight W when the second image data is the preceding image data and the second noise-removed image data obtained by removing noise from the second image data is the past image Iold is set as the weight W2. Further, the weight W when the third image data is the preceding image data and the third noise-removed image data obtained by removing noise from the third image data is the past image Iold is set as the weight W3.

Note that when the first image data is the first image data of the moving image data, there is no image data preceding the first image data, so the weight W1 is not adjusted by the weight adjustment unit 222 but is adjusted by the weight storage unit 223. It is stored in the second memory 224 as a past weight Wold. When the weight W2 is calculated based on the first image data and the second image data, the weight adjustment unit 222 reads the past weight Wold (=weight W1) from the second memory 224, and uses the above formula (5) to calculate the weight W2. The adjusted weight W2' is calculated by multiplying by The weight storage unit 223 stores the weight W2 before adjustment in the second memory 224 as a past weight Wold.

Further, when the weight W3 is calculated based on the second image data and the third image data, the weight adjustment unit 222 reads the weight Wold (=weight W2) from the second memory 224, and uses the above equation (5) to calculate the weight W3. The adjusted weight W3' is calculated by multiplying by the weight W3. The weight storage unit 223 stores the weight W3 before adjustment in the second memory 224 as Wold.

The removal unit 203 removes noise from the image data to be noise removed based on the adjusted weight W', and outputs noise removed image data. Specifically, for example, the removal unit 203 removes noise from the image data to be noise removed by weighted averaging using the adjusted weight W′. The weighted average is expressed, for example, by the following equation (7).

In the above equation (7), Iout is the image data from which noise has been removed (noise removed image data).

The image storage unit 204 stores the noise-removed image data Iout output from the removal unit 203 in the first memory 205 as past image data Iold. The first memory 205 is configured by the storage device 102.

<Recursive noise removal process>
FIG. 3 is a flowchart illustrating an example of a recursive noise removal procedure by the imaging device 100 according to the first embodiment. FIG. 4 is an explanatory diagram showing an example of recursive noise removal using adjusted weights W'. As a comparative example of FIG. 4, FIG. 5 is illustrated. FIG. 5 is an explanatory diagram showing an example of recursive noise removal without applying the adjusted weight W'. Note that in FIGS. 4 and 5, the steps corresponding to the steps in FIG. 3 are given the step numbers in FIG. Note that in FIG. 5, since the adjusted weight W' is not applied, steps S304 to S306 do not exist. Furthermore, in FIGS. 4 and 5, an example will be described in which the imaging device 100 captures moving image data in which the imaging direction is fixed and the subject image moves from left to right.

First, the left side of the center dotted line in FIGS. 4 and 5 will be described using FIG. 3. The imaging device 100 uses the acquisition unit 201 to acquire and read current image data (step S301), and reads past image data from the first memory 205 (step S302). Here, the current image data is the subsequent image data I(t-1) at time t-1, and the past image data is the preceding image data at time (t-2) before time (t-1). Iold is denoised image data from which noise has been removed from Iold. It is assumed that the object image OB(t-2) exists in Iold.

The imaging device 100 uses the weight calculation unit 221 to calculate the weight W(t-1) by applying the past image data Iold and the current image data I(t-1) to equation (2) (step S303). In the past image data Iold and the current image data I(t-1), the more similar the images in the same area are, the larger the weight becomes, and the less similar they are, the smaller the weight becomes. Therefore, in the past image data Iold and the current image data I(t-1), the area where neither the subject images OB(t-2) and OB(t-1) are present becomes almost the same image, and the image of the area concerned is The weight w(t-1) is, for example, the maximum value of 1.0.

On the other hand, in the past image data Iold and the current image data I(t-1), the area where either the subject image OB(t-2) or OB(t-1) exists is a different image, and the area The weight w(t-1) becomes low. Here, as an example, the weight w(t-1) of the area is set to 0.0. Note that the area at the boundary between the subject images OB(t-2) and OB(t-1) is larger than the area where either one of the subject images OB(t-2) or OB(t-1) exists. Similar. Here, as an example, the weight w(t-1) of the area is set to 0.5.

Next, the imaging device 100 reads the past weight Wold from the second memory 224 (step S304). Taking this past weight Wold as an example, it is assumed that it is a matrix in which all elements have values of "1.0". The imaging device 100 uses the weight adjustment unit 222 to apply the past weight Wold from the second memory 224 and the weight W(t-1) calculated in step S303 to equation (3), and calculates the adjusted weight W. ' is output (step S305).

The adjusted weight W' is obtained by, for example, the product of the past weight Wold and the weight W(t), as shown in equation (5). Since the past weight Wold is a matrix in which all elements have values of "1.0", the adjusted weight W' is the same as the calculated weight W(t-1).

Then, the imaging device 100 stores the weight W(t-1) before adjustment in the second memory 224, and sets it as a new past weight Wold (step S306).

The imaging device 100 uses the removal unit 203 to apply the past image data Iold, the current image data I(t-1), and the adjusted weight W' to equation (7), and then obtains the current image data I(t-1). Noise is removed from the image data (step S307), and noise-removed image data Iout(t-1) is output (step S308). The output noise-removed image data Iout(t-1) is displayed on the display device 110. In the noise-removed image data Iout(t-1), the subject image OB(t-1) exists in the same position as the current image data I(t-1), and the subject image OB(t-2) of the past image data Iold exists. An afterimage A(t-2) exists at the same position as . The afterimage A(t-2) is mainly image data in which the outline of the object image OB(t-2) is prominent.

Furthermore, the image capturing device 100 outputs the noise-removed image data Iout(t-1) to the first memory 205 by the image storage unit 204, and updates the past image data Iold (step S309). After that, the imaging apparatus 100 determines whether or not the image processing is to be ended, for example, based on the user's operation input to end the motion imaging (step S310). If the process does not end (step S310: No), the imaging device 100 updates the reading position of the buffer memory in the acquisition unit 201 (step S311), and returns to step S301.

Next, the right side of the center dotted line in FIGS. 4 and 5 will be described using FIG. 3. The imaging device 100 uses the acquisition unit 201 to acquire and read current image data I(t) at time t (step S301), and reads past image data Iold from the first memory 205 (step S302). Here, the past image data Iold is the noise-removed image data Iout(t-1) obtained in step S309.

The imaging device 100 uses the weight calculation unit 221 to calculate the weight W(t) by applying the past image data Iold and the current image data I(t) to equation (2) (step S303). In the past image data Iold and the current image data I(t), the more similar the images of the same area are, the larger the weight becomes, and the less similar the images are, the smaller the weight becomes. Therefore, in past image data Iold and current image data I(t), an area where neither object image OB(t-1) or OB(t) exists becomes almost the same image, and the weight of the area w(t ) has a maximum value of 1.0, for example.

On the other hand, in the past image data Iold and the current image data I(t), an area where either the subject image OB(t-1) or OB(t) exists becomes a different image, and the weight of the area w( t) becomes lower. Here, as an example, the weight w(t) of the region is set to 0.0. Note that the area at the boundary between the subject images OB(t-1) and OB(t) is more similar than the area where either one of the subject images OB(t-1) and OB(t) exists. . Here, as an example, the weight w(t) of the region is set to 0.5. Furthermore, in the past image data Iold (noise removed image data Iout(t-1)), the weight of the area where the afterimage A(t-2) exists is set to 0.75.

Next, the imaging device 100 reads the past weight Wold from the second memory 224 (step S304). This past weight Wold is the weight W(t-1) previously calculated in step S306. The imaging device 100 uses the weight adjustment unit 222 to apply the past weight Wold from the second memory 224 and the weight W(t) calculated in step S303 to equation (3) to obtain the adjusted weight W'. Output (step S305).

The adjusted weight W' is obtained by, for example, the product of the past weight Wold and the weight W(t), as shown in equation (5). In this case, the past weight Wold is the weight W(t-1) calculated earlier in step S306, so the weight w(t) inside the area where the afterimage A(t-2) exists is 0, The contour weight w(t) is 0.375 (=0.75×0.5). Also, the weight w(t) inside the area where the afterimage A(t-1) exists remains 0 (=0.0×0.0), but the weight w(t) of the contour is 0.25( =0.5×0.5).

Then, the imaging apparatus 100 stores the weight W(t) before adjustment in the second memory 224, and sets it as a new past weight Wold (step S306).

The imaging device 100 uses the removal unit 203 to apply the past image data Iold, the current image data I(t), and the adjusted weight W' to equation (7), and removes noise from the current image data I(t). (step S307), and outputs noise-removed image data Iout(t) (step S308). The output noise-removed image data Iout(t) is displayed on the display device 110. In the noise-removed image data Iout(t), a subject image OB(t) exists at the same position as the current image data I(t), and an afterimage A exists at the same position as the subject image OB(t-1) of the past image data Iold. (t-1) exists, and furthermore, an afterimage A(t-2) exists at the same position as the object image OB(t-2).

Further, the image capturing device 100 outputs the noise-removed image data Iout(t) to the first memory 205 by the image storage unit 204, and updates the past image data Iold (step S309). After that, the imaging apparatus 100 determines whether or not the image processing is to be ended, for example, based on the user's operation input to end the motion imaging (step S310). If it ends (step S310: Yes), the imaging device 100 ends the image processing.

In this way, in the area where there is no conversion between the past image data Iold and the current image data I(t-1), I(t), the weights W(t-1), W for referring to the past image data Iold are Since (t) becomes larger, it is possible to increase the noise removal rate while reducing the amount of memory used.

However, in the case of the example shown in FIG. 5, if there is an area where the weight calculation is not successful between the past image data Iold and the current image data I(t-1), I(t), that is, there is an area where the calculation has failed. , the results are also retained. Therefore, the failure remains for a long time. This causes a significant deterioration in image quality as an afterimage.

For example, as in the afterimage A(t-2) of the outline of the object image OB(t-2) indicated by the thick arrow 5A1 in FIG. 5, step S303 in the next frame (image data I(t)) The weight calculation result becomes large as shown by the thick arrow 5a1 (w(t-1)=0.75), and as a result, an afterimage A(t-2) of the failed contour remains as shown by the thick arrow 5A2. It's stored away.

On the other hand, the imaging apparatus 100 of the first embodiment retains the past image data Iold with the weight calculated at that time attached, and reuses the past image data Iold. As a result, the weights W(t-1) and W(t) for referring to past image data Iold tend to become smaller in areas with low reliability where afterimages occur, resulting in afterimages. can be suppressed.

For example, while the contour weight w(t-1) of the object image OB(t-2) indicated by the thick arrow 4B1 in FIG. In the multiplied thick arrow 4b2, the weight w(t) is small (0.375=0.5×0.75), and as a result, the afterimage A(t-2) of the failed contour as shown in the thick arrow 5B2. The remaining condition has been reduced.

In this way, in the case of FIG. 5, the noise removal accuracy is lower than that in FIG. 4, so in order to achieve the same noise removal accuracy as in FIG. It is necessary to hold a large amount in the memory 205, which increases memory usage. On the other hand, in FIG. 4, it is sufficient to hold the most recent past image data in the first memory 205, so it is possible to both improve noise removal accuracy and save memory.

Embodiment 2 is an example of suppressing the occurrence of afterimages by using spatial information included in image data. In the second embodiment, differences from the first embodiment will be mainly explained, and the same reference numerals will be used for the parts common to the first embodiment, and the explanation will be omitted.

<Functional configuration example of imaging device 100>
FIG. 6 is a block diagram showing an example of the functional configuration of the imaging device 100 according to the second embodiment. The calculation unit 602 of the second embodiment includes a spatial information extraction unit 621 and a weight calculation unit 622. The spatial information extraction unit 621 extracts spatial information from image data to be extracted. The image data to be extracted is at least one of the past image data Iold and the current image data It. When using both image data Iold and It, the spatial information extraction unit 621 extracts spatial information for each image data I, calculates the average of the two pieces of spatial information, selects the maximum value, or performs α blending. may be calculated.

Spatial information is information indicating characteristics of a spatial structure included in image data to be extracted, and specifically includes, for example, a spatial frequency component. In the case of spatial frequency components, for example, the spatial information extraction unit 621 extracts a low frequency component Slow and a high frequency component High using low-pass filters of two different sizes, as shown in equations (8) to (13) below. I can do it.

I in equations (8) and (9) is image data to be extracted. Furthermore, the function abs(·) in equation (11) is a function that calculates the absolute value for each region (x, y). Equation (13) is substituted into equations (8) and (9). When substituted into equation (8), equation (13) becomes Hσ1(x,y), and σ becomes σ1. Similarly, when substituted into equation (9), equation (13) becomes Hσ2(x,y) and σ becomes σ2.

Note that the image data I to be extracted may be converted in advance to make it easier to extract spatial information. Furthermore, the spatial information extraction unit 621 may calculate not only the two components of the low frequency component Slow and the high frequency component High, but also frequency components of a plurality of bands.

The weight calculation unit 622 calculates the weight W of the past image data based on the current image data, the past image data, and the spatial information extracted by the spatial information extraction unit 621. Areas where afterimages are likely to be noticeable are often bright areas or near edges (contours of subject images in past image data). Therefore, it is better to reduce the weight of such areas.

Therefore, in order to first determine a bright area, the weight calculation unit 622 executes processing such as the following equations (14) and (15) using the low frequency component Slow.

Slow, It, Slow, and Iold in Equation (14) mean the low frequency component Slow of the current image data It and past image data Iold, respectively. max(·) in equation (14) is a function that calculates the maximum value for each region (x, y). S′low in equation (15) is the maximum value of the low frequency component Slow of the current image data It and past image data Iold, but may be an average value.

Further, σ(·) is a sigmoid function that calculates a value for each region (x, y), and the larger the input value, the closer it approaches 1. Tlow is a parameter representing a threshold value. As a result, the adjustment value Klow based on the low frequency component Slow in equation (15) becomes a smaller value as the low frequency component Slow becomes larger, that is, the brighter the area.

Next, in order to determine the vicinity of the edge, the weight calculation unit 622 executes processing such as the following equations (16) and (17) using the high frequency component High.

High, It, High, and Iold in Equation (16) mean the high frequency component High of the current image data It and past image data Iold, respectively. max(·) in equation (16) is a function that calculates the maximum value for each region (x, y). S'high in equation (17) is the maximum value of the high frequency components High of the current image data It and past image data Iold, but may be an average value.

Further, σ(·) is a sigmoid function that calculates a value for each region (x, y), and the larger the input value, the closer it approaches 1. Thigh is a parameter representing a threshold value. As a result, the adjustment value Khigh based on the high frequency component High in equation (17) becomes a smaller value as the high frequency component High becomes larger, that is, closer to the edge.

Using the obtained adjustment value Klow based on the low frequency component Slow and the adjustment value Khigh based on the high frequency component Shigh, the finally updated weight Wnew is obtained by the following equation (18). The removal unit 203 removes noise from the current image data It using the updated weight Wnew in the same process as in the first embodiment, and outputs it as noise-removed image data. The image storage unit 204 also stores the noise-removed image data in the first memory 205 as past image data Iold.

In equation (18), W on the right side is a weight for referring to past image data Iold. By updating the weight W to the weight Wnew in equation (18), the weight in bright areas and near edges, where afterimages are likely to be noticeable, becomes smaller, so it is possible to suppress the occurrence of afterimages. Although the low frequency component Slow and the high frequency component High of the current image data It and the past image data Iold are used here, only one of the low frequency component Slow and the high frequency component High may be used.

<Recursive noise removal process>
FIG. 7 is a flowchart illustrating an example of a recursive noise removal procedure by the imaging device 100 according to the second embodiment. In the second embodiment, when compared with FIG. 3, steps S303 to S306 do not exist, and after step S302, the imaging device 100 executes spatial information extraction processing by the spatial information extraction unit 621 (step S701). Next, the imaging device 100 uses the weight calculation unit 622 to calculate the adjustment weight Wnew (step S702). After this, the imaging device 100 uses the adjustment weight Wnew to remove noise from the current image data It by the removal unit 203 (Step S307), and outputs the noise-removed image data Iout to the display device 110 (Step S308). .

The image capturing device 100 uses the image storage unit 204 to store the noise-removed image data Iout in the first memory 205 as past image data Iold (step S309). By performing such processing recursively until the imaging is completed, the weight in areas where afterimages are likely to be noticeable is adjusted to be low, and as a result, the occurrence of afterimages can be suppressed.

Embodiment 3 is an example in which the spatial information extraction unit 621 of Embodiment 2 is applied to Embodiment 1. In the third embodiment, the differences from the first and second embodiments will be mainly explained, and the same reference numerals will be used for the parts common to the first and second embodiments, and the explanation will be omitted.

<Functional configuration example of imaging device 100>
FIG. 8 is a block diagram showing an example of the functional configuration of the imaging device 100 according to the third embodiment. In the case of the third embodiment, the calculation unit 802 includes a spatial information extraction unit 621, a weight calculation unit 622, a weight adjustment unit 222, a weight storage unit 223, and a second memory 224. The weight adjustment unit 222 adjusts the weight Wnew updated by the weight calculation unit 622 based on the latest weight Wold stored in the second memory 224.

Let the adjusted updated weight Wnew be an adjusted weight W'. The weight storage unit 223 stores the weight Wnew before adjustment by the weight adjustment unit 222 in the second memory 224 as a past weight Wold. Specifically, for example, the weight adjustment unit 222 adjusts the weight Wnew by replacing W in equations (3) to (7) used by the weight adjustment unit 222 with Wnew.

<Recursive noise removal process>
FIG. 9 is a flowchart illustrating an example of a recursive noise removal procedure by the imaging device 100 according to the third embodiment. In Example 3, when compared with FIG. 3, step S303 does not exist, and after step S302, the imaging device 100 executes spatial information extraction processing by the spatial information extraction unit 621 (step S701). Next, the imaging device 100 uses the weight calculation unit 622 to calculate the weight Wnew (step S702).

The imaging device 100 reads the past weight Wold from the second memory 224 (step S304), uses the weight adjustment unit 222 to adjust the weight Wnew with the past weight Wold, and outputs the adjusted weight W′ to the removal unit 203. The weight Wnew before adjustment is output to the weight storage unit 223 (step S305). Then, the weight storage unit 223 stores the weight Wnew before adjustment in the second memory 224, and updates the past weight Wold (step S306).

After that, the imaging device 100 uses the adjusted weight W′ to remove noise from the current image data It (step S307), and outputs the noise-removed image data Iout to the display device 110 (step S307). S308). The image capturing device 100 uses the image storage unit 204 to store the noise-removed image data Iout in the first memory 205 as past image data Iold (step S309). By performing such processing recursively until the imaging is completed, the weight in areas where afterimages are likely to be noticeable is adjusted to be low, and as a result, the occurrence of afterimages can be suppressed.

In this way, by extracting spatial information, calculating weights using the spatial information, and then adjusting the weights, it is possible to suppress afterimages before and after adjusting the weights, and to improve the efficiency of suppressing the occurrence of afterimages. I can do it.

The fourth embodiment is an example in which the removal unit 203 performs afterimage reduction processing on the current image data before removing noise from the current image data using weights. The removal unit 203 performs noise removal using weights from the current image data after the afterimage reduction process. The imaging device 100 of the fourth embodiment aims to suppress the occurrence of afterimages by reducing the afterimages of the current image data before noise removal. In the fourth embodiment, the explanation will focus on the differences from the first to third embodiments, and the same reference numerals will be used for the parts common to the first to third embodiments, and the explanation will be omitted.

<Functional configuration example>
FIG. 10 is a block diagram showing an example of the functional configuration of the imaging device 100 according to the fourth embodiment. In the case of the fourth embodiment, the removal unit 203 acquires the weights from the calculation unit 202. The weight from the calculation unit 202 is the adjusted weight W' in the first and third embodiments, and the updated weight Wnew in the second embodiment. Further, when the calculation unit 202 has the configuration shown in FIG. 5, the weight from the calculation unit 202 is calculated by the weight calculation unit 221 by applying the past image data Iold and the current image data I to the above equation (2). is the weight W. In the fourth embodiment, these weights are collectively referred to simply as weight W unless otherwise specified.

The removal unit 203 includes an afterimage reduction processing unit 1001 and a noise removal processing unit 1002. The afterimage reduction processing unit 1001 executes processing to reduce afterimages of current image data. Specifically, for example, the afterimage reduction processing unit 1001 blurs the outline of a moving subject image that becomes an afterimage by applying a low-pass filter to the current image data. This makes the outline of the moving subject image less noticeable. In addition, the afterimage reduction processing unit 1001 performs interpolation between the moving subject images for two pieces of image data I(t-1) and I(t) that are continuous in the time direction, thereby eliminating the moving subject image that becomes the afterimage. Blur the outline (interpolation process).

The noise removal processing unit 1002 removes noise from the image data to be noise removed, based on the weight W from the calculation unit 202, on the image data that has been subjected to afterimage reduction processing by the afterimage reduction processing unit 1001, and generates a noise removed image. Output data. Specifically, for example, the removal unit 203 removes noise from the image data to be noise-removed using a weighted average by substituting the weight W from the calculation unit 202 into W′ in the above equation (7). The image data to be noise removed is image data that has been subjected to a low-pass filter, or image data that has been generated by interpolation processing.

<Interpolation processing example>
FIG. 11 is an explanatory diagram showing an example of interpolation processing by the afterimage reduction processing unit 1001 according to the fourth embodiment. In FIG. 11, a rectangle is a group of image data that is continuous in the time direction, that is, moving image data. (A) shows unprocessed image data Ia (t-3), Ia (t-2), Ia (t- 1), Ia(t) is shown. In image data Ia(t-3), Ia(t-2), Ia(t-1), Ia(t), object images OBa(t-3), OBa(t-2), OBa(t-1 ), OBa(t) are assumed to be moving from left to right.

(B) shows the composition of image data. The afterimage reduction processing unit 1001 combines two temporally consecutive pieces of image data and detects a motion vector. Specifically, for example, the afterimage reduction processing unit 1001 synthesizes image data Ia(t-3) and Ia(t-2) to generate composite image data Ib(t-2), and generates the object image OBa( A motion vector mv(t-2) between OBa(t-3) and OBa(t-2) is detected.

Similarly, the afterimage reduction processing unit 1001 synthesizes the image data Ia(t-2) and Ia(t-1) to generate composite image data Ib(t-1), and generates the object image OBa(t-2). , OBa(t-1) is detected. The afterimage reduction processing unit 1001 synthesizes image data Ia(t-1) and Ia(t) to generate composite image data Ib(t), and generates composite image data Ib(t) between object images OBa(t-1) and OBa(t). Detect a motion vector mv(t).

(C) shows interpolation of composite image data Ib(t-2), Ib(t-1), and Ib(t). Specifically, for example, the afterimage reduction processing unit 1001 calculates a motion vector mv(t- 2) by duplicating the contour on the terminal side (left half arc) in the movement direction, interpolation is performed between object images OBa(t-3) and OBa(t-2). Thereby, the afterimage reduction processing unit 1001 generates interpolated composite image data Ic(t-2) including the interpolated object image OBc(t-2).

Similarly, the afterimage reduction processing unit 1001 selects the terminal end of the motion vector mv(t-1) among the contours of the temporally older object image OBa(t-2) in the composite image data Ib(t-1). By duplicating the side contour (left half circular arc) in the movement direction, interpolation is performed between object images OBa(t-2) and OBa(t-1). Thereby, the afterimage reduction processing unit 1001 generates interpolated composite image data Ic(t-1) including the interpolated object image OBc(t-1).

In addition, the afterimage reduction processing unit 1001 also processes, in the synthesized image data Ib(t), the contour on the terminal side of the motion vector mv(t) (left By duplicating the half circular arc) in the moving direction, interpolation is performed between the object images OBa(t-1) and OBa(t). Thereby, the afterimage reduction processing unit 1001 generates interpolated composite image data Ic(t) including the interpolated object image OBc(t). In FIG. 11, the outline to be copied is the outline of the temporally older subject image, but it may be the outline of the temporally newer subject image.

In this way, the imaging device 100 uses interpolation processing to convert afterimages that appear discretely in each image data into a moving trajectory of the subject image as if the shutter speed was slowed down, and continuously displays the resulting trajectory. Thereby, the afterimage can be reduced by making the user see the afterimage as if it does not exist.

<Recursive noise removal process>
FIG. 12 is a flowchart illustrating an example of a recursive noise removal procedure by the imaging device 100 according to the fourth embodiment. In Example 4, after step S302, the imaging apparatus 100 calculates weights according to any of Examples 1 to 3 and FIG. 5 (Step S1203). Next, the imaging apparatus 100 causes the afterimage reduction processing unit 1001 to perform afterimage reduction processing using a low-pass filter or interpolation processing (step S1204). Then, in the imaging apparatus 100, the noise removal processing unit 1002 uses the weight W calculated in step S1203 to remove noise from the image data after the afterimage reduction process (step S307).

After that, the imaging device 100 outputs the noise-removed image data Iout to the display device 110 (step S308), and the image storage unit 204 stores the noise-removed image data Iout in the first memory 205 as past image data Iold ( Step S309). By performing such processing recursively until the imaging is completed, it is possible to continuously make afterimages less noticeable and, as a result, to suppress the occurrence of afterimages.

In the fourth embodiment, an example was described in which the removal unit 203 performs afterimage reduction processing on the current image data before removing noise from the current image data using weights. This is an example in which the removal unit 203 removes noise from the current image data using weights, and then performs afterimage reduction processing on the noise-removed image data. The imaging device 100 of the fifth embodiment aims to suppress the occurrence of afterimages by reducing the afterimages of the current image data after noise removal. In the fifth embodiment, differences from the fourth embodiment will be mainly explained, and the same reference numerals will be used for the parts common to the first to fourth embodiments, and the explanation will be omitted.

<Functional configuration example>
FIG. 13 is a block diagram showing an example of the functional configuration of the imaging device 100 according to the fifth embodiment. The removal unit 203 acquires the weights from the calculation unit 202. The weight from the calculation unit 202 is the adjusted weight W' in the first and third embodiments, and the updated weight Wnew in the second embodiment. Further, when the calculation unit 202 has the configuration shown in FIG. 5, the weight from the calculation unit 202 is calculated by the weight calculation unit 221 by applying the past image data Iold and the current image data I to the above equation (2). is the weight W. In the fifth embodiment, these weights are collectively referred to simply as weight W unless otherwise specified.

The removal unit 203 includes a noise removal processing unit 1002 and an afterimage reduction processing unit 1001. The noise removal processing unit 1002 removes noise from the image data to be noise-removed based on the weight W from the calculation unit 202 for the image data from the acquisition unit 201, and outputs noise-removed image data. Specifically, for example, the removal unit 203 removes noise from the image data using a weighted average by substituting the weight W from the calculation unit 202 into W' in the above equation (7).

The afterimage reduction processing unit 1001 executes processing to reduce the afterimage of the noise-removed image data. Specifically, for example, the afterimage reduction processing unit 1001 blurs the outline of a moving subject image that becomes an afterimage by applying a low-pass filter to the noise-removed image data. This makes the outline of the moving subject image less noticeable. Furthermore, as shown in FIG. 11, the afterimage reduction processing unit 1001 performs interpolation between the moving subject images for two pieces of noise-removed image data I(t-1) and I(t) that are continuous in the time direction. By doing so, the outline of the moving subject image, which becomes an afterimage, is blurred (interpolation processing).

<Recursive noise removal process>
FIG. 14 is a flowchart illustrating an example of a recursive noise removal procedure by the imaging device 100 according to the fifth embodiment. In Example 5, after step S302, the imaging apparatus 100 calculates weights according to any of Examples 1 to 3 and FIG. 5 (Step S1203). Next, the imaging device 100 uses the weight W calculated in step S1203 to remove noise from the noise-removed image data by the noise removal processing unit 1002 (step S307). Then, in the imaging apparatus 100, the afterimage reduction processing unit 1001 executes afterimage reduction processing using a low-pass filter or interpolation processing (step S1204).

After that, the imaging device 100 outputs the noise-removed image data Iout to the display device 110 (step S308), and the image storage unit 204 stores the noise-removed image data Iout in the first memory 205 as past image data Iold ( Step S309). By performing such processing recursively until the imaging is completed, it is possible to continuously make afterimages less noticeable and, as a result, to suppress the occurrence of afterimages. Further, by performing noise removal prior to afterimage reduction processing, blurring and interpolation of noise can be suppressed.

Embodiment 6 is an example in which a region where an afterimage occurs (an afterimage occurrence region) is detected from image data, and the afterimage is removed from the detected afterimage occurrence region (detection region). Thereby, the range for reducing afterimages can be narrowed down to the afterimage generation area, and afterimage reduction processing can be efficiently executed. Embodiment 6 will be mainly described with respect to the differences from Embodiment 5, and the same reference numerals will be used for parts common to Embodiments 1 to 5, and the explanation will be omitted.

FIG. 15 is an explanatory diagram showing an example of afterimage reduction processing according to the sixth embodiment. In FIG. 15, rectangles indicate image data. The shaded range within the rectangle indicates noise N. Although noise may exist uniformly in the entire image data, here, in order to simplify the explanation, it is assumed that noise N exists at a specific position. (A) shows an example where the subject image does not move, and (B) shows an example where the subject image moves.

In (A), image data Ia(t-1) and Ia(t) are image data sequences that are continuous in the time direction. Image data Ia(t-1) and Ia(t) both have noise N and object images OB(t-1) and OB(t) at the same position. The noise-removed image data NRIa(t-1) is image data from which noise N has been removed from the image data Ia(t-1) based on the noise-removed image data at time t-2 (past image data Iold). The noise-removed image data NRIa(t) is image data from which noise N has been removed from the image data Ia(t) based on the noise-removed image data NRIa(t-1) at time t-1 (past image data Iold). be.

The imaging device 100 uses a weighted average of image data Ia(t) at time t, which is current image data, and noise-removed image data NRIa(t-1) at time t-1 (past image data Iold). Noise removed image data NRIa(t) from which noise N is removed from data Ia(t) is generated. Further, the differential image data DIa(t) is image data obtained by calculating the difference between the image data Ia(t) and the noise-removed image data NRIa(t). In this case, the subject image OB(t) and its background cancel each other out, and the noise N remains as a difference.

In (B), image data Ib(t-1) and Ib(t) are image data sequences that are continuous in the time direction. The image data Ib(t-1) and Ib(t) have noise N at the same position, but the positions of the subject images OB(t-1) and OB(t) are different. That is, it shows that the subject is moving from left to right.

The noise-removed image data NRIb(t-1) is image data from which noise N has been removed from the image data Ib(t-1) based on the noise-removed image data at time t-2 (past image data Iold). It is assumed that the noise-removed image data NRIb(t-1) includes an afterimage A(t-2) of the subject image that existed in the image data at time t-2.

The noise-removed image data NRIb(t) is image data from which noise N has been removed from the image data Ib(t) based on the noise-removed image data NRIb1(t-1) (past image data Iold) at time t-1. be. However, the noise-removed image data NRIb1(t) includes an afterimage A(t-1) at the position of the object image OB(t-1) in the noise-removed image data NRIb(t-1).

The imaging device 100 uses a weighted average of image data Ib(t) at time t, which is current image data, and noise-removed image data NRIb(t-1) at time t-1 (past image data Iold). Noise removed image data NRIb1(t) from which noise N has been removed from data Ib(t) is generated. Further, the differential image data DIb(t) is image data obtained by calculating the difference between the image data Ib(t) and the noise-removed image data NRIb1(t). In this case, the object image OB(t) and its background cancel each other out, and the noise N and the afterimage A(t-1) remain as a difference.

The imaging device 100 refers to the differential image data DIb(t) and identifies the area of the afterimage A(t-1) as the afterimage occurrence area 1500. Next, the imaging device 100 extracts image data 1501 at the same position as the image data of the afterimage occurrence area (in this case, the contour afterimage A(t-1)) from the image data Ib(t) at time t. . The imaging device 100 then replaces the afterimage A(t-1) in the noise-removed image data NRIb1(t) with the extracted image data 1501. Thereby, the imaging device 100 can generate noise-removed image data NRIb2(t) by removing the afterimage A(t-1) from the noise-removed image data NRIb1(t).

Note that the imaging device 100 extracts image data 1501 at the same position as the image data of the afterimage occurrence area 1500 (in this case, the contour afterimage A(t-1)) from the image data Ib(t) at time t. However, the range to be extracted is not limited to the same position as the afterimage generation area 1500, but may be an area including the afterimage generation area 1500 and its vicinity. Further, here, since the afterimage is often the outline of the subject image, the afterimage generation area 1500 is defined as an area corresponding to the outline of the subject image, but the afterimage generation area 1500 may also be the inside surrounded by the outline. Thereby, it is possible to reduce the omission of removal of afterimages.

<Functional configuration example>
FIG. 16 is a block diagram showing an example of the functional configuration of the imaging device 100 according to the sixth embodiment. The afterimage reduction unit obtains noise-removed image data from the noise removal processing unit 1002, as shown in FIG. Further, as shown in FIG. 15, the afterimage reduction processing unit 1001 obtains the current image data Ib(t) from the obtaining unit 201, for example. The afterimage reduction processing unit 1001 also acquires past image data Iold from the first memory 205. The past image data Iold is, for example, the noise-removed image data NRIb(t-1) at the previous time t-1.

As shown in FIG. 15, the afterimage reduction processing unit 1001 generates noise-removed image data NRIb(t) for the current image data Ib(t) by a weighted average of the current image data Ib(t) and the past image data Iold. and generates differential image data DIb(t) between the current image data Ib(t) and the generated noise-removed image data NRIb(t).

Then, the afterimage reduction processing unit 1001 detects the afterimage occurrence area 1500 from the difference image data DIb(t), and identifies image data 1501 at the same position as the afterimage occurrence area 1500 from the current image data Ib(t). Then, the afterimage reduction processing unit 1001 generates noise-removed image data NRIb2(t) from which the afterimage A(t-1) has been removed by combining the image data 1501 with the noise-removed image data NRIb1(t).

The afterimage reduction processing unit 1001 outputs the generated noise-removed image data NRIb2(t) to the display device 110 and also to the image storage unit 204. The image storage unit 204 stores the acquired noise-removed image data NRIb2(t) in the first memory 205 as new past image data Iold.

Note that the afterimage reduction processing unit 1001 may apply the low-pass filter and interpolation processing shown in the fifth embodiment to the sixth embodiment. Specifically, for example, the afterimage reduction processing unit 1001 applies a low-pass filter or interpolation processing to the current image data Ib(t), and uses the current image data Ib(t) after the application and the generated noise-removed image data NRIb. (t) and differential image data DIb(t) may be generated.

The afterimage reduction processing unit 1001 also applies a low-pass filter and interpolation processing to the generated noise-removed image data NRIb2(t), and displays the applied noise-removed image data NRIb2(t) on the display device 110 and the image storage unit 204. You can also output to

<Recursive noise removal process>
FIG. 17 is a flowchart illustrating an example of a recursive noise removal procedure by the imaging device 100 according to the sixth embodiment. In Example 6, after step S302, the imaging apparatus 100 calculates weights according to any one of Examples 1 to 3 and FIG. 5 (step S1203). Next, the imaging device 100 uses the weight W calculated in step S1203 to remove noise from the noise-removed image data by the noise removal processing unit 1002 (step S307). Then, the imaging apparatus 100 causes the afterimage reduction processing unit 1001 to perform afterimage reduction processing using a low-pass filter or interpolation processing (steps S1771 to S1773).

Specifically, for example, the afterimage reduction processing unit 1001 generates differential image data DIb(t) between the current image data Ib(t) and the noise-removed image data NRIb1(t) (step S1771), and Afterimage generation area 1500 is detected from DIb(t) (step S1772).

Then, the afterimage reduction processing unit 1001 extracts image data 1501 at the same position as the afterimage generation area 1500 from the current image data Ib(t), and combines it with the noise-removed image data NRIb1(t), thereby creating a noise-removed image. Data NRIb2(t) is generated and afterimage A(t-1) is removed (step S1773). Thereafter, the afterimage reduction processing unit 1001 outputs the noise-removed image data NRIb2(t) to the display device 110 as output image data Iout (step S308), and outputs it to the image storage unit 204 (step S309).

By performing such processing recursively until the imaging is completed, it is possible to continuously make afterimages less noticeable and, as a result, to suppress the occurrence of afterimages. In this way, the range for reducing afterimages can be narrowed down to the afterimage occurrence area, and the afterimage reduction process can be efficiently executed. In other words, it is not necessary to perform afterimage reduction processing on areas where afterimages do not occur, so it is possible to avoid afterimage reduction processing on areas where afterimages do not need to be reduced, and the original image can be maintained. be able to.

Example 7 is an example in which afterimages are made less noticeable by adjusting the conditions for capturing a moving image in which afterimages are less likely to occur.

<Video imaging example>
FIG. 18 is an explanatory diagram showing an example of moving image capturing according to the seventh embodiment. FIG. 18 is an example of capturing a moving image of a certain subject, and the rectangles are image data sequences that are continuous in the time direction, that is, moving image data. Further, for convenience of explanation, the frame rate is assumed to be 3 [fps] as an example. Further, among times t1 to t3, time t1 is the oldest time, and time t3 is the latest time. (A) is an example of video imaging in which the imaging conditions are not adjusted according to the seventh embodiment, and (B) is an example of video imaging in which the imaging conditions are adjusted in the seventh embodiment. Therefore, one piece of image data is generated at 1/3 [fps] in both (A) and (B).

Note that (t1) to (t3) are added to the end of the code to distinguish between times t1 to t3, but if the times t1 to t3 are not to be distinguished, they are simply written as (t). Although times t1 to t3 are not distinguished, they are simply expressed as time t.

In (A), et(t) is the exposure time. gp(t) is the gap time between successive exposure times. The exposure time is the time during which the image sensor 105 is exposed to object light from the lens system from the open state of the shutter (for example, rolling shutter) to the closed state.

The imaging device 100 images a subject from time t1 to t3 and generates image data Ia(t1), Ia(t2), and Ia(t3). Image data Ia(t1), Ia(t2), and Ia(t3) include object images OBa(t1), OBa(t2), and OBa(t3), respectively.

When capturing an image of a subject, first, the imaging device 100 performs exposure for 1/3 [s] starting from time t with an exposure time et(t) starting from time t. Next, the imaging apparatus 100 reads an analog electrical signal from the imaging element 105 at a gap time gp(t) after the exposure time et(t) has elapsed, and the AFE 106 executes signal processing. As a result, image data Ia(t) is generated.

The imaging device 100 performs noise removal recursively at every time t. For example, at time t3, the imaging device 100 performs noise removal on the image data Ia(t3) obtained at time t3. Specifically, for example, the imaging device 100 obtains the image data Ia(t2) by taking a weighted average of the noise-removed image data obtained by removing noise from the image data Ia(t2) and the image data Ia(t3). Noise-removed image data NRIa (t3) from which noise has been removed from t3) is generated.

In the noise-removed image data NRIa(t3), the afterimages Aa(t2) and Aa(t1) are converted into object images OBa(t2) and OBa( This is image data taken over from t1), and the older the object images OBa(t2) and OBa(t1), the thinner the object images become.

In addition, in the noise-removed image data NRIa (t3), the object image OBa (t3) and afterimages Aa (t2) and Aa (t1) appear discretely in the direction of movement of the object, resulting in an image in which the movement of the object is unnatural. becomes. That is, when the exposure time et(t) is short, the gap time gp(t), which is the non-exposed time, becomes relatively long, causing a shift between the previous and subsequent images and making discontinuity noticeable.

In (B), ET(t) is the exposure time. GP(t) is the gap time between successive exposure times. The exposure time ET(t) becomes longer than the exposure time et(t) due to the imaging condition setting according to the seventh embodiment, and thus the gap time GP(t) becomes shorter than the gap time GP(t). .

The imaging device 100 images a subject from time t1 to t3 and generates image data Ib(t1), Ib(t2), and Ib(t3). Image data Ib(t1), Ib(t2), and Ib(t3) include subject images OBb(t1), OBb(t2), and OBb(t3), respectively.

When capturing an image of a subject, first, the imaging device 100 performs exposure for 1/3 [s] starting from time t with an exposure time ET(t) starting from time t. Next, the imaging apparatus 100 reads an analog electrical signal from the imaging element 105 at a gap time GP(t) after the exposure time ET(t) has elapsed, and the AFE 106 executes signal processing. As a result, image data Ib(t) is generated.

Since the exposure time ET(t) is longer than the exposure time et(t) in (A), it is possible to capture an image of the subject moving. In this case, the image data Ib(t) includes an object image OBb(t) and a trajectory TJ(t) that follows the object image OBb(t) as the object moves from left to right. The trajectory TJ(t) is a time-series afterimage of the subject image OBb(t) obtained as the exposure time ET(t) passes.

The imaging device 100 performs noise removal recursively at every time t. For example, at time t3, the imaging device 100 performs noise removal on the image data Ib(t3) obtained at time t3. Specifically, for example, the imaging device 100 obtains the image data Ib(t2) by taking a weighted average of the noise-removed image data obtained by removing noise from the image data Ib(t2) and the image data Ib(t3). Noise-removed image data NRIb(t3) from which noise has been removed from t3) is generated.

In the noise-removed image data NRIb(t3), the afterimages Ab(t2) and Ab(t1) are converted into object images OBa(t2) and OBa( This is image data taken over from the trajectory TJ(t2) and TJ(t1) of t1), and the older the object images OBb(t2) and OBb(t1), the thinner the object images are.

In the noise-removed image data NRIb(t3), the subject image OBb(t3), its trajectory TJ(t3), and afterimages Ab(t2) and Ab(t1) appear continuously in the moving direction of the subject, so the subject's movement is becomes a natural image. In other words, when the exposure time ET(t) is shorter than the exposure time et(t), the gap time GP(t), which is the non-exposed time, becomes relatively short, and the shift between the previous and subsequent images is reduced, resulting in discontinuity. becomes less noticeable.

<Functional configuration example of imaging device 100>
FIG. 19 is a block diagram showing an example of the functional configuration of the imaging device 100 according to the seventh embodiment. The imaging device 100 includes a conversion table 1900, an input section 1901, a setting section 1902, an adjustment section 1903, an imaging section 120, and an image processing section 1905. Conversion table 1900 is specifically realized, for example, by storage device 102 shown in FIG. 1. Further, the input unit 1901, the setting unit 1902, the adjustment unit 1903, and the image processing unit 1905 specifically, for example, cause the processor 101 to execute a program stored in the storage device 102 shown in FIG. Alternatively, it is realized by the LSI 107.

The conversion table 1900 is a table for converting the first imaging condition to the second imaging condition. The first imaging condition 1952 is a default imaging condition, and is, for example, an exposure time according to the frame rate 1951. In the imaging device 100, for example, when the frame rate 1951 is set to "FRi", the exposure time is set to "eti". In the example of FIG. 18, when the frame rate 1951 in (A) is 3 [fps], the exposure time is et(t).

The second imaging condition 1953 is an imaging condition for reducing afterimages, and corresponds to the first imaging condition 1952. For example, the second imaging condition 1953, FCi, is an exposure time ETi that is longer than the default exposure time eti when the frame rate 1951 is FRi. Furthermore, in order to suppress overexposure due to the extension of the exposure time from eti to ETi, the second imaging condition 1953, FCi, may include the amount of decrease in ISO sensitivity (the ISO sensitivity after decrease may also be used).

The input unit 1901 receives from the operating device 108 an input of a frame rate 1951 when capturing an image of a subject. Note that when the shooting mode is selected from the operating device 108, the input unit 1901 may receive the frame rate 1951 defined in the shooting mode. Note that the exposure time may differ even with the same frame rate 1951 depending on the shooting mode.

The setting unit 1902 sets a second imaging condition 1953 for reducing afterimages based on the frame rate 1951 input by the input unit 1901 and the exposure time corresponding to the frame rate 1951. Specifically, for example, the setting unit 1902 reads the second imaging condition 1953 corresponding to the frame rate 1951 from the input unit 1901 from the conversion table 1900. When only the frame rate 1951 is given from the input unit 1901, the setting unit 1902 reads the corresponding second imaging condition 1953 from the conversion table 1900, setting the exposure time of the first imaging condition 1952 as the default exposure time.

Further, when the frame rate 1951 and exposure time are specified by the shooting mode from the input unit 1901, the setting unit 1902 converts the second imaging condition 1953 corresponding to the frame rate 1951 and exposure time from the conversion table 1900. read out. Furthermore, if the second imaging condition 1953 includes the amount of decrease in ISO sensitivity (or the ISO sensitivity after the decrease), the setting unit 1902 also reads the amount of decrease in the ISO sensitivity (or the ISO sensitivity after the decrease). .

The adjustment unit 1903 adjusts the exposure time corresponding to the frame rate 1951 inputted by the input unit 1901 to a longer exposure time. Specifically, as shown in FIG. 18, for example, if the frame rate 1951 is 3 [fps] and the exposure time is et(t), the adjustment unit 1903 converts the exposure time et(t). The exposure time is changed to the exposure time ET(t) obtained from the table 1900.

The imaging unit 120 images the subject based on the frame rate 1951 and the exposure time adjusted by the adjustment unit 1903, and outputs an image data string that is continuous in the time direction. Specifically, for example, the imaging unit 120 generates image data Ib(t) as shown in FIG. 18(B).

The image processing unit 1905 performs image processing on the image data Ib(t) from the imaging unit 120. Specifically, for example, the image processing section 1905 includes the acquisition section 201, the calculation section 202, the removal section 203, the image storage section 204, and the first memory 205 shown in Examples 1 to 6. Therefore, the recursive noise processing shown in Example 1 generates noise-removed image data NRIb(t3) as shown in FIG. 18(B).

<Example of imaging processing procedure>
FIG. 20 is a flowchart illustrating an example of an imaging processing procedure by the imaging apparatus 100. The imaging apparatus 100 receives the input of the frame rate 1951 through the input unit 1901 (step S2001), and sets the second imaging condition 1953 through the setting unit 1902 (step S2002).

In the imaging apparatus 100, the adjustment unit 1903 adjusts the exposure time under the second imaging condition 1953 (step S2003), and the imaging unit 120 images the subject to generate an image data string (step S2004). The imaging device 100 uses the image processing unit 1905 to execute image processing including the recursive noise removal processing (FIGS. 3, 7, 9, 12, 14, and 17) shown in Examples 1 to 6. (Step S2005). Note that the image processing (step S2005) may include recursive noise removal processing as shown in FIG.

Note that in the above example, the exposure time, which is the light amount information, is the subject of adjustment, but the light amount information that is the subject of adjustment is not limited to the exposure time, but may be the F value or the ISO sensitivity. When the F value is the adjustment target, the exposure time of the first imaging condition 1952 becomes the F value, and the exposure time of the second imaging condition 1953 becomes an F value smaller than the F value of the first imaging condition 1952. In this case as well, in order to reduce overexposure, the second imaging condition 1953 may include a value that makes the exposure time shorter than the default exposure time or lowers the ISO sensitivity.

Further, when the ISO sensitivity is the adjustment target, the exposure time of the first imaging condition 1952 becomes the ISO sensitivity, and the exposure time of the second imaging condition 1953 becomes the ISO sensitivity higher than the ISO sensitivity of the first imaging condition 1952. In this case as well, in order to reduce overexposure, the second imaging conditions 1953 may include values such as making the exposure time shorter than the default exposure time or increasing the F value.

In this way, according to the seventh embodiment, it is possible to generate an image data sequence in which afterimages are less noticeable. Further, by applying these image data strings to the image processing unit 1905, the noise removal results as in Examples 1 to 6 can be obtained recursively.

100 imaging device, 101 processor, 102 storage device, 105 imaging element, 120 imaging unit, 200 image processing device, 201 acquisition unit, 202 calculation unit, 203 removal unit, 204 image storage unit, 205 first memory, 221 weight calculation unit , 222 weight adjustment unit, 223 weight storage unit, 224 second memory, 602 calculation unit, 621 spatial information extraction unit, 622 weight calculation unit, 802 calculation unit, 1001 afterimage reduction processing unit, 1002 noise removal processing unit, 1900 conversion table , 1901 input section, 1902 setting section, 1903 adjustment section, 1905 image processing section

Claims (14)

an imaging unit that images a subject and outputs first image data and second image data that is temporally later than the first image data;
a calculation unit that calculates a first weight of the first image data based on the first noise-removed image data and the second image data from which noise has been removed from the first image data;
an afterimage reduction processing unit that reduces afterimages in the second image data;
a removing unit that removes noise from second image data that has been subjected to afterimage reduction processing by the afterimage reduction processing unit, based on the first noise-removed image data and the first weight calculated by the calculation unit;
An imaging device having: The imaging device according to claim 1,
The afterimage reduction processing section is an imaging device that reduces afterimages in the second image data using a low-pass filter. The imaging device according to claim 1,
The afterimage reduction processing unit combines the first image data and the second image data, and in the combined image data, a subject image in the first image data and a subject image in the second image data are combined. An imaging device that interpolates between. The imaging device according to claim 2,
The imaging unit outputs third image data that is temporally later than the second image data,
The calculation unit calculates a second weight of the second image data based on the second noise-removed image data and the third image data from which noise has been removed from the second image data by the removal unit, and 1 weight and the second weight, calculate a third weight,
The afterimage reduction processing unit reduces afterimages in the third image data,
The removal unit removes noise from the third image data that has been subjected to afterimage reduction processing by the afterimage reduction processing unit, based on the second noise-removed image data and the third weight calculated by the calculation unit. , imaging device. The imaging device according to claim 1,
an extraction unit that extracts first spatial information indicating characteristics of a spatial structure included in the image from at least one of the first noise-removed image data and the second image data;
The calculation unit is an imaging device that calculates the first weight based on the first spatial information extracted by the extraction unit. an imaging unit that images a subject and outputs first image data and second image data that is temporally later than the first image data;
a calculation unit that calculates a first weight of the first image data based on the first noise-removed image data and the second image data from which noise has been removed from the first image data;
a removal unit that removes noise from the second image data based on the first noise-removed image data and the first weight calculated by the calculation unit;
an afterimage reduction processing unit that reduces afterimages in second noise-removed image data from which noise has been removed from the second image data by the removal unit;
An imaging device having: The imaging device according to claim 6,
The afterimage reduction processing unit is an imaging device that reduces afterimages in the second noise-removed image data using a low-pass filter. The imaging device according to claim 6,
The afterimage reduction processing unit combines the first noise-removed image data and the second noise-removed image data, and in the combined image data, the subject image in the first noise-removed image data and the second noise An imaging device that interpolates between the removed image data and a subject image. The imaging device according to claim 7,
It has an adjustment section that adjusts the weight,
The imaging unit outputs third image data that is temporally later than the second image data,
The calculation unit calculates a second weight of the second image data based on the second noise-removed image data and the third image data that have been subjected to afterimage reduction processing by the afterimage reduction processing unit, and calculates a second weight of the second image data, and Calculating a third weight based on the second weight,
The removal unit removes noise from the third image data based on the second noise-removed image data that has undergone the afterimage reduction process and the third weight calculated by the calculation unit,
The afterimage reduction processing section is an imaging device that reduces afterimages in third noise-removed image data from which noise has been removed from the third image data by the removal section. The imaging device according to claim 6,
an extraction unit that extracts first spatial information indicating characteristics of a spatial structure included in the image from at least one of the first noise-removed image data and the second image data;
The calculation unit is an imaging device that calculates the first weight based on the first spatial information extracted by the extraction unit. The imaging device according to claim 6,
The afterimage reduction processing unit detects image data of an afterimage existing in the second noise removed image data based on the difference between the first noise removed image data and the second image data, and An imaging device that replaces data with image data of a region within the second image data that corresponds to the afterimage detection region. an input section that accepts input of a frame rate when capturing an image of a subject;
an adjustment unit that adjusts first light amount information corresponding to the frame rate input by the input unit to second light amount information larger than the first light amount information;
Based on the frame rate and second light amount information adjusted by the adjustment section, image the subject and output first image data and second image data that is temporally later than the first image data. an imaging unit to
a calculation unit that calculates a first weight of the first image data based on the first noise-removed image data and the second image data from which noise has been removed from the first image data;
a removal unit that removes noise from the second image data based on the first noise-removed image data and the first weight calculated by the calculation unit;
An imaging device having: The imaging device according to claim 12,
a setting unit configured to set imaging conditions for reducing afterimages based on the frame rate and the first light amount information;
The imaging unit images the subject based on the frame rate, the second light amount information, and the imaging conditions set by the setting unit, and outputs the first image data and the second image data. , imaging device. The imaging device according to claim 12,
It has an adjustment section that adjusts the weight,
The imaging unit outputs third image data that is temporally later than the second image data,
The calculation unit calculates a second weight of the second image data based on the second noise-removed image data and the third image data from which noise has been removed by the removal unit, and calculates a second weight of the second image data, and Calculate a third weight based on the weight,
The removing unit is an imaging device that removes noise from the third image data based on the second noise-removed image data and a third weight calculated by the calculating unit.

Images