WO2011033675A1

WO2011033675A1 - Image processing apparatus and image display apparatus

Info

Publication number: WO2011033675A1
Application number: PCT/JP2009/066446
Authority: WO
Inventors: 信幸松本; 孝井田; 賢造五十川
Original assignee: 株式会社東芝
Priority date: 2009-09-18
Filing date: 2009-09-18
Publication date: 2011-03-24

Abstract

A motion-blurred image can be generated with a suppression amount calculated from a motion width of a subject at image pickup time, a camera aperture time and a motion width of the subject at display time. An image-pickup motion blur amount setting unit (101) sets a first motion blur filter from a motion width of a subject occurring at image pickup time. A display motion blur amount setting unit (102) sets a second motion blur filter from a motion width of the subject occurring at display time. A motion blur suppression amount calculating unit (103) calculates a pixel value conversion filter from the first motion blur filter and an inverse filter of the second motion blur filter. A storage unit (104) stores a first image the initial pixel values of which are equal to the pixel values of an input image. A conversion unit (105) corrects the pixel values of the first image such that the difference between the pixel value of each pixel of a second image, which is obtained by use of the first image and the pixel value conversion filter, and the pixel value of the respective corresponding pixel of the input image is small.

Description

Image processing apparatus and image display apparatus

The present invention relates to an image processing apparatus and an image display apparatus that suppress, for example, image blurring.

[Images taken with a digital camera or moving images displayed on a TV may be blurred due to subject movement or camera movement. In order to make it easy to see images and videos with motion blur, the motion blur is suppressed by a sharpening process. For example, Japanese Patent Laid-Open No. 2006-81150 discloses a technique for suppressing motion blur by inverse filter processing or sharpening processing using an unsharp mask based on a parameter indicating imaging blur characteristics.

Japanese Unexamined Patent Publication No. 2006-81150

However, the above-described conventional technique has a problem in that motion blur is empirically suppressed only in accordance with the movement width of the subject at the time of shooting, and an unnatural image is generated.

The present invention provides an image processing apparatus and an image display apparatus capable of suppressing blur with a natural feeling.

Means for Solving the Invention

An image processing apparatus according to an aspect of the present invention calculates a shooting motion blur amount of a subject according to a motion width of the subject in an input image, and obtains a first motion blur filter that reproduces the shooting motion blur amount. A first setting unit, a second setting unit that determines a display motion blur amount that is a desired motion blur amount smaller than the shooting motion blur amount, and obtains a second motion blur filter that reproduces the display motion blur amount; A calculation unit that calculates a pixel value conversion filter from a first motion blur filter and an inverse filter of the second motion blur filter, and a memory that stores a first image having a pixel value of the input image as an initial value And the first image is converted by the pixel value conversion filter to obtain a second image, and the difference between the pixel value of the input image and the pixel value of the corresponding pixel of the second image is reduced. The image of the first image By correcting the value, it comprises a conversion unit for generating a motion blur suppressing image.

According to the present invention, it is possible to suppress blur with a natural feeling.

1 is a block diagram of an image processing apparatus according to a first embodiment. The figure explaining the imaging | movement motion blur amount and a display motion blur amount. FIG. 6 is a diagram for explaining the operation of a conversion unit 105. 6 is a flowchart showing a flow of operations performed by the image processing apparatus according to the first embodiment. FIG. 5 is a block diagram of an image processing apparatus according to a second embodiment. 9 is a flowchart showing a flow of operations performed by the image processing apparatus according to the second embodiment. FIG. 9 is a block diagram of an image display device according to a third embodiment.

Hereinafter, embodiments of the present invention will be described in detail. In addition, the same code | symbol is attached | subjected to the structure and process which mutually perform the same operation | movement, and the overlapping description is abbreviate | omitted.

[First embodiment]
FIG. 1 is a block diagram of the image processing apparatus of the present embodiment.

The image processing apparatus according to this embodiment includes a shooting motion blur amount setting unit (first setting unit) 101, a display motion blur amount setting unit (second setting unit) 102, a motion blur suppression amount calculating unit (calculating unit) 103, and a storage. Unit 104 and conversion unit 105. This image processing apparatus processes an input image y1 obtained by photographing a subject, and generates and outputs an image in which the blur amount of the subject in the input image y1 is suppressed.

The shooting motion blur amount setting unit 101 receives a signal representing the movement width of the subject in the input image (the movement width of the subject when shooting with the imaging device). The shooting motion blur amount setting unit 101 calculates the blur amount of the subject (shooting motion blur amount) in the input image from the motion range indicated by the input signal, the camera opening time during shooting, and the shooting interval time during shooting. Then, a first motion blur filter that reproduces the shooting motion blur amount is set. The signal representing the movement width may be received from the outside through user input, or may be read from a recording medium such as a memory. Information on the camera opening time and the shooting interval time may be given in advance or may be given from the outside.

The left side of FIG. 2 is a diagram showing the relationship between the movement width of the subject at the time of shooting, the shooting motion blur amount, the camera opening time, and the shooting interval time. The movement width of the subject at the time of shooting refers to the width of the moving subject or the entire camera within the time (frame interval time) during which a moving image frame is shot. If the camera at the time of shooting is fully open (that is, the shutter is open), and the shooting interval time between frames of the movie is equal to the opening time of the shutter, the movement width of the subject at the time of shooting The amount of motion blur is the same. Also, if the opening time is shorter than the shooting interval time, the shooting motion blur amount is
Shooting motion blur amount = subject movement width during shooting x (opening time / shooting interval time)
Can be calculated as

Here, y _{1 is} the vector in which the pixel values of the motion blur-suppressed image of the subject at the time of shooting are arranged in raster scan order, x is the vector in which the pixel values of the image without motion blur are arranged in raster scan order, and the first motion blur filter If the matrix representation of W ₁ is W ₁ , the relationship of y ₁ , x, W ₁ is

It becomes. For example, when the shooting motion blur amount of the subject at the time of shooting is a motion of 8 pixels in the horizontal direction, the filter W ₁ is

It becomes a circular matrix expressed as

The panning operation, which is the main camera work, can be formulated with this expression because the movement of the entire screen is in the horizontal direction. If the movement direction is an oblique direction, the above-described movement in the horizontal direction can be formulated by rotating the entire screen so as to correct the movement direction in the horizontal direction.

The imaging motion blur amount setting unit 101 sends the set first motion blur filter W1 to the motion blur suppression amount calculation unit 103. Here, the shooting motion blur amount setting unit 101 calculates the shooting motion blur amount based on the movement width of the subject at the time of shooting, the opening time, and the shooting interval time, but the opening time and the shooting interval time are the same. In some cases, the first motion blur filter W1 may be set simply using the movement width of the subject at the time of shooting as the shooting motion blur amount. Also, if the shooting motion blur amount is known in advance, a signal indicating the shooting motion blur amount is given to the shooting motion blur amount setting unit 101, and the shooting motion blur amount setting unit 101 performs a first operation based on the given signal. One motion blur filter W1 may be set.

The display motion blur amount setting unit 102 receives an input of a signal indicating the movement width of the subject at the time of display. The display motion blur amount setting unit 102 determines a display motion blur amount that is a desired blur amount at the time of display according to the motion width indicated in the signal, and a second motion blur filter that reproduces the display motion blur amount Set. More specifically, the display motion blur amount is an amount indicating how much motion blur is to be generated on each frame and still image frame of a moving image. In general, the movement width of the subject at the time of display can be used as the display movement blur amount, and therefore, the input movement width can be used as the display movement blur amount as it is. The signal may be received from the outside via a user input, or may be read from a recording medium such as a memory.

Here, the amount of display motion blur can be set based on the rate conversion rate and the range of motion of the subject at the time of shooting. For example, if the movement width of the subject at the time of shooting is 8 pixels in the horizontal direction and the frame rate conversion rate is double speed of 60 [Hz] → 120 [Hz], the display motion blur amount is 8 pixels / Double speed = 4 pixels in the horizontal direction should be set. Further, for example, when the movement width of the subject at the time of shooting is a movement of 8 pixels in the horizontal direction and the frame rate conversion rate is a quadruple speed of 60 [Hz] → 240 [Hz], the display motion blur amount is 8 Pixel / 4x speed = set to movement of 2 pixels in the horizontal direction.

Also, the display motion blur amount can be a user-specified display motion blur amount input from the outside in order to reflect the motion blur amount intended by the creator. As a result, it is possible to perform processing such as converting motion blur in a still image photograph into a dynamic expression desired by the creator.

Here, y _{2 is} a vector in which the pixel values of an image (motion blur-suppressed image) having an object movement width (display motion blur amount) at the time of display are arranged in raster scan order, and the pixel value of an image without motion blur is raster scanned. the vectors ordered x (same as shooting motion blur amount setting unit), when the matrix representation of the second motion blur filter to reproduce the display motion blur and W _2, y _2, x, the relationship of W ₂ is ,

It becomes. The matrix W ₂ of the second motion blur filter, for example, when the movement width of the subject at the time of display is a movement of 4 pixels in the horizontal direction,

It becomes a circular matrix expressed as

Display motion blur quantity setting unit 102, second motion blur filter W ₂ set, and sends the motion blur suppression amount calculating unit 103.

Motion blur suppression amount calculation unit 103 calculates a first motion blur filter W _1, the pixel value conversion filter and a second motion blur inverse filter of the filter W _2. Hereinafter, the pixel value conversion filter will be described.

First, formula 2 is transformed into formula

Substituting this into Equation 1,

Get. here

Then, the following formula 4 is obtained.

A matrix W in Expression 3 represents a pixel value conversion filter according to the present embodiment.

For example, four pixels matrix, the second motion blur filter W ₂ is the horizontal direction of the first motion-blur filter W ₁ (if to reproduce the 8 pixels of the motion blur) laterally in the case of eight pixels of the motion Pixel value conversion filter W based on Equation 3, if it is a matrix in the case of motion for minutes (when reproducing motion blur for 4 pixels)

It becomes a circular matrix expressed as

The motion blur suppression amount calculation unit 103 sends the calculated pixel value conversion filter to the conversion unit 105.

Here, a pixel value conversion filter is calculated and stored in advance for combinations of various values of the shooting motion blur amount of the subject at the time of shooting and the motion width (display motion blur amount) of the subject at the time of display. Also good. In this case, the motion blur suppression amount calculation unit 103 receives data indicating the shooting motion blur amount from the shooting motion blur amount setting unit 101, and receives data indicating the display motion blur amount from the display motion blur amount setting unit 102. The corresponding pixel value conversion filter is selected from the pixel value conversion filter group calculated in advance based on the received data. Thereby, the amount of processing can be reduced. When the opening time and the shooting interval time are equal, the movement width of the subject at the time of shooting may be used as the shooting motion blur amount.

Here, the pixels of the temporary motion blur suppression image corresponding to the pixels of the input image are referred to as corresponding pixels. “Corresponding” indicates a relationship between pixels that are relatively in the same position on the image plane. The temporary motion blur-suppressed image is sent to the conversion unit 105. The storage unit 104 is configured by a readable / writable storage medium such as a DRAM (memory) of a PC (personal computer), an HDD (hard disk), or an LSI frame memory.

By using such a correction method, it is possible to obtain an output image having a desired display blur while suppressing the influence of noise included in the input image. Hereinafter, this reason will be described in detail.

FIG. 4 is a flowchart for explaining an operation in which the image processing apparatus of the present embodiment generates an output image in which motion blur is suppressed from an input image.

First, the shooting motion blur amount setting unit 101 determines the shooting motion blur amount in accordance with the movement width of the subject at the time of shooting, and sets a first motion blur filter that reproduces the shooting motion blur amount (step S201).

Next, the display motion blur amount setting unit 102 determines the display motion blur amount according to the movement width of the subject at the time of display, and sets a second motion blur filter that reproduces the display motion blur amount (step S202). .

Next, the motion blur suppression amount calculation unit 103 calculates a pixel value conversion filter from the first motion blur filter and the inverse filter of the second motion blur filter (step S203). Specifically, the pixel value conversion filter is calculated by multiplication of the first motion blur filter and the inverse filter of the second motion blur filter.

Next, the storage unit 104 stores the input image, and generates and stores a temporary motion blur suppression image having each pixel value of the input image as an initial pixel value (step S204).

Next, the conversion unit 105 performs a conversion process of the temporary motion blur suppression image stored in the storage unit 104 based on the input image stored in the storage unit 104 and the pixel value conversion filter (step S205). Specifically, first, one of the pixels of the input image is set as the target pixel. A pixel value of the provisional motion blur suppression image is converted by a pixel value conversion filter to calculate a trial calculation pixel value corresponding to the target pixel. Then, the pixel value of the temporary motion blur suppression image is converted (corrected) so that the difference amount between the pixel value of the target pixel and the trial calculation pixel value becomes small. Each other pixel of the input image is also set as a pixel of interest sequentially, and the above conversion process is repeated.

Next, the conversion unit 105 determines whether the conversion process has been performed for a predetermined number of repetitions (step S206). Specifically, the conversion unit 105 determines whether the conversion process using the target pixel has been performed a predetermined number of times. If a determination result is NO, it will return to Step S205. If the determination result is YES, the provisional motion blur suppression image at that stage is output as a motion blur suppression image, and the process ends.

The operation of the present embodiment described so far is particularly effective when the entire screen is moving, such as camera panning. When the background is fixed and only the subject moves, the motion blur of only the subject can be suitably suppressed by performing the following operation. Two methods of operation are shown.

(1) A moving area (a set of pixels) is detected from the input image, and y ₁ , y ₂ , W ₁ , W ₂ , x are defined only in that area, and conversion processing is performed. In other words, the shooting blur amount setting unit sets the first motion blur filter W1 targeting only the region, and the display blur amount setting unit sets the second motion blur filter W2 targeting only the region. In the conversion unit, the conversion process is performed only on the pixels in the region of the input image and the pixels in the region of the temporary motion blur suppression image. Information on the area may be notified from an external device to the imaging blur amount setting unit, the display blur amount setting unit, and the conversion unit, or a processing unit that detects the region is added, and the processing unit performs detection. May be.

(2) y ₁ , y ₂ , W ₁ , W ₂ , and x are defined on the entire screen as before, but a filter that reproduces shooting motion blur and a display motion blur are reproduced in the background and subject portions. Switch the matrix component with the filter. That is, the values of the filters obtained by the methods described so far are used as they are in the rows W ₁ and W ₂ for the pixels of the subject portion. On the other hand, values of W ₁ and W ₂ for the pixels in the background portion are set so as to be an identity transformation (that is, y ₁ = x, y ₂ = x). The photographing blur amount setting unit and the display blur amount setting unit may operate in this way.

The methods (1) and (2) described here can be similarly applied to the second and third embodiments described below.

As described above, according to the present embodiment, the motion blur suppression is performed by calculating the pixel value conversion filter from the motion width of the subject at the time of shooting, the opening time of the camera, and the motion width of the subject at the time of display (display motion blur amount). By performing the processing, it is possible to suppress the blur amount according to the movement width of the subject at the time of shooting, the opening time of the camera, and the movement width of the subject at the time of display, and a natural image can be generated.

[Second Embodiment]
FIG. 5 is a block diagram of the image processing apparatus of the present embodiment.
The image processing apparatus of this embodiment is obtained by adding a subject motion width estimation unit 401 to the image processing apparatus of the first embodiment shown in FIG.

The subject movement width estimation unit 401 estimates the movement width of the subject from the input image. For example, the motion width of the subject at the time of shooting is estimated by using a general motion vector calculation process such as a gradient method or a region matching method between two input images that are temporally continuous. Since the other components 101 to 105 are the same as those in the first embodiment, a duplicate description is omitted.

FIG. 6 is a flowchart for explaining an operation in which the image processing apparatus of the present embodiment outputs an output image in which motion blur is suppressed from the input image.

First, the subject motion width estimation unit 401 estimates the subject motion width from the input image (step S501).
Next, the shooting motion blur amount setting unit (first setting unit) 101 determines the shooting motion blur amount according to the estimated movement width of the subject, and the first motion blur for reproducing the shooting motion blur amount A filter is set (step S502).
Next, the display motion blur amount setting unit (second setting unit) 102 determines the display motion blur amount according to the movement width of the subject at the time of display, and the second motion blur for reproducing the display motion blur amount. A filter is set (step S503).
Next, the motion blur suppression amount calculation unit (calculation unit) 103 calculates a pixel value conversion filter from the first motion blur filter and the inverse filter of the second motion blur filter (step S504).

Next, the storage unit 104 stores the input image, and generates and stores a temporary motion blur suppression image having each pixel value of the input image as an initial value (step S505).
Next, based on the input image stored in the storage unit 104 and the pixel value conversion filter, the conversion unit 105 performs pixel value conversion processing of the temporary motion blur suppression image stored in the storage unit 104 (step S506). ). The conversion method is as described in the first embodiment.
Next, the conversion unit 105 determines whether the conversion process has been performed for a predetermined number of repetitions (step S507). If the determination result is NO, the process returns to step S506, and if YES, the temporary motion blur suppression image at that stage is output as a motion blur suppression image, and the process ends.

In this way, by calculating the pixel value conversion filter from the movement width of the subject at the time of shooting estimated from the input image, the opening time of the camera, and the movement width of the subject at the time of display, and performing motion blur suppression processing, The blur amount can be suppressed according to the movement width of the subject at the time of shooting, the opening time of the camera, and the movement width of the subject at the time of display, and a natural image can be generated.

[Third embodiment]
FIG. 7 is a block diagram of the image display apparatus of the present embodiment. This image display device includes an image processing device 100 and an image display unit 200.

The image processing apparatus 100 has the same configuration as the image processing apparatus shown in FIG.

The image display unit 200 displays the output image obtained by the image processing apparatus 100.

In this embodiment, the image display unit 200 is connected to the image processing apparatus 100 of the first embodiment, but is connected to the image processing apparatus of the second embodiment, and displays an output image obtained by the image processing apparatus. May be.

Note that the present invention is not limited to the above-described embodiment as it is, and can be embodied by modifying constituent elements without departing from the scope of the invention in the implementation stage. In addition, various inventions can be formed by appropriately combining a plurality of components disclosed in the embodiment. For example, some components may be deleted from all the components shown in the embodiment. Furthermore, constituent elements over different embodiments may be appropriately combined.

In addition, the image processing apparatuses in the first and second embodiments and the image display apparatus in the third embodiment described above can also be realized by using, for example, a general-purpose computer device as basic hardware. Is possible. The program to be executed has a module configuration including each function described above. The program is an installable or executable file that is recorded on a computer-readable recording medium such as a CD-ROM, CD-R, DVD, etc. Also good.

101 ... Shooting motion blur amount setting unit (first setting unit), 102 ... Display motion blur amount setting unit (second setting unit), 103 ... Motion blur suppression amount calculation unit (calculation unit), 104 ... Storage unit, 105 ... Conversion unit, 401 ... Subject motion width estimation unit (motion width estimation unit), 100 ... Image processing device, 200 ... Image display unit

Claims

A first setting unit that calculates a shooting motion blur amount of the subject according to a motion width of the subject in the input image, and obtains a first motion blur filter that reproduces the shooting motion blur amount;
A second setting unit for determining a display motion blur amount that is a desired motion blur amount smaller than the shooting motion blur amount, and obtaining a second motion blur filter that reproduces the display motion blur amount;
A calculation unit that calculates a pixel value conversion filter from the first motion blur filter and an inverse filter of the second motion blur filter;
A storage unit for storing a first image having an initial value of a pixel value of the input image;
The first image is converted by the pixel value conversion filter to obtain a second image, and the difference between the pixel value of the input image and the pixel value of the corresponding pixel of the second image is reduced. A conversion unit that generates a motion blur-suppressed image by correcting the pixel value of the first image;
An image processing apparatus.
The first setting unit calculates the first motion blur amount from the movement width of the subject in the input image, the opening time of the imaging device at the time of shooting the subject, and the shooting interval time of the imaging device. 2. The image processing apparatus according to claim 1, wherein:
The second setting unit calculates the movement width of the subject after the frame rate conversion of the input image from the movement width of the subject in the input image and the frame rate conversion rate of the input image given in advance, 3. The image processing apparatus according to claim 2, wherein the second motion blur amount is determined according to the calculated movement width of the subject.
3. The image processing apparatus according to claim 2, wherein the second setting unit uses a user-specified motion blur amount input from the outside as the second motion blur amount.
A movement width estimation unit for estimating a movement width of the subject from the input image;
The first setting unit includes the first motion blur amount based on the motion width estimated by the motion width estimation unit, the opening time of the imaging device at the time of photographing the subject, and the photographing interval time of the imaging device. 2. The image processing apparatus according to claim 1, characterized in that:
A first setting unit that calculates a shooting motion blur amount of the subject according to a motion width of the subject in the input image, and obtains a first motion blur filter that reproduces the shooting motion blur amount;
A second setting unit for determining a display motion blur amount that is a desired motion blur amount smaller than the shooting motion blur amount, and obtaining a second motion blur filter that reproduces the display motion blur amount;
A calculation unit that calculates a pixel value conversion filter from the first motion blur filter and an inverse filter of the second motion blur filter;
A storage unit for storing a first image having an initial value of a pixel value of the input image;
The first image is converted by the pixel value conversion filter to obtain a second image, and the difference between the pixel value of the input image and the pixel value of the corresponding pixel of the second image is reduced. A conversion unit that generates a motion blur-suppressed image by correcting the pixel value of the first image;
An image display device comprising: an image display unit that displays the motion blur suppression image.