JP2005175841A - Imaging apparatus and method, and program - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To easily realize hyperresolution effect. <P>SOLUTION: An imaging part 11 performs conversion to an imaging frame Fa of an internal data format with the number of pixels being (ix×jy) and a frame rate of F when the frame rate which an output part 13 can output is F while the number of output pixels is (x×y). An image converter 12 converts the resolution of the imaging frame Fa supplied from the imaging part 11 into such resolution as the output part 13 can display for generating an output frame Fb ((x×y) pixels). The image converter 12, at that time, performs a prescribed pixel number conversion process based on the moving speed of an image for each prescribed block, so that such visual effect as an observer who sees the image of output frame Fb recognizes at a resolution that exceeds that resolution. <P>COPYRIGHT: (C)2005,JPO&NCIPI

Description

  The present invention relates to an imaging apparatus, method, and program, and more particularly, to an imaging apparatus, method, and program for converting a resolution of a captured image for each predetermined area based on the amount of movement of the captured image.

  Recent digital video camera devices are equipped with an image sensor having a larger number of pixels than the number of recorded pixels of a moving image, mainly for the purpose of photographing a high-definition still image. In such an apparatus, when a moving image is recorded, conversion from the number of imaging pixels to the number of recording pixels is necessary (reduction of the number of pixels).

  As a method of reducing the number of pixels in a solid-state imaging device such as a CCD (Charge Coupled Device) imaging device or a CMOS (Complementary Metal Oxide Semiconductor) imaging device, the number of imaging pixels is thinned and read by controlling the driving method. A method and a method of adding and reading out signals of a plurality of pixels as in Patent Documents 1 and 2 below have been put into practical use.

  It is known that a CCD image sensor and a CMOS image sensor can realize high frame rate imaging by driving at high speed.

Japanese Patent Laid-Open No. 11-331706 JP 2003-78919 A

  However, in the pixel number conversion method by thinning, there is a problem that aliasing occurs due to a decrease in the sampling rate and the moving image quality is deteriorated. Further, the pixel number conversion method by pixel addition has a problem that the resolution is rate-controlled by the number of output pixels.

  The present invention has been made in view of such a situation. By performing adaptive signal processing when outputting a moving image, an observer of the output image does not recognize image quality deterioration due to the reduction in the number of pixels. Further, the captured image can be converted into an output image.

  An imaging step of obtaining an image signal having a first resolution by photoelectric conversion of the present invention, and an image signal obtained by the processing of the imaging step is converted into an image signal having a second resolution lower than the first resolution. An imaging method having a resolution conversion step and an output step of outputting an image corresponding to the resolution-converted image signal at a predetermined frame rate, wherein the resolution conversion step is at least a partial area of the image obtained in the imaging step. The amount of movement is detected, and the resolution of the region is converted based on the detected amount of movement.

  The resolution conversion step may include a spatial thinning process that provides a visual effect such that the display unit that displays the image signal output from the output unit at a frame rate has a resolution that exceeds the second resolution.

  The amount of space thinning in the space thinning process can be determined based on the amount of movement of the region.

  A horizontal thinning amount in the horizontal direction can be determined based on the horizontal movement amount of the region, and a vertical spatial thinning amount can be determined based on the vertical movement amount of the region.

  When the frame rate is high, the spatial thinning amount in the spatial thinning process can be increased.

  Spatial decimation processing of an arbitrary frame may determine the amount of spatial decimation so that all the aliasing components of the decimation-processed signal in a predetermined number of adjacent frames preceding and following the frame cancel each other. it can.

  In the resolution conversion step, spatial thinning processing can be performed on a signal obtained by performing spatial filtering on the image signal obtained in the imaging step processing.

  In the resolution conversion step, the image obtained by the processing in the imaging step can be divided into a plurality of regions, and the movement amount can be detected for each of the divided regions.

  An imaging step of obtaining an image signal having a first resolution by photoelectric conversion of the present invention, and an image signal obtained by the processing of the imaging step is converted into an image signal having a second resolution lower than the first resolution. An imaging method having a resolution conversion step and an output step for outputting an image corresponding to the resolution-converted image signal at a predetermined frame rate, wherein the resolution conversion step outputs a plurality of image signals obtained by the processing of the imaging step. The area is divided into areas, the movement amount for each area is detected, the spatial thinning amount of each area is set based on the detected movement amount, and the spatial filter is applied to each area of the image signal obtained in the imaging step. After performing the processing, the spatial thinning processing can be performed for each region based on the set spatial thinning amount.

  Imaging means for obtaining an image signal having a first resolution by photoelectric conversion, and resolution conversion for converting the image signal obtained by the imaging means into an image signal having a second resolution lower than the first resolution And an output device that outputs an image signal whose resolution has been converted by the resolution conversion unit at a predetermined frame rate. The imaging apparatus includes: a moving unit configured to move at least a part of an area of an image obtained by the imaging unit. A movement amount detection unit that detects the amount and a resolution conversion processing unit that converts the resolution of the region based on the detected movement amount can be provided.

  The resolution conversion means can perform spatial thinning processing that gives a visual effect such that the display means for displaying the image signal output from the output means at a frame rate has a resolution exceeding the second resolution.

  The resolution conversion unit further includes a storage unit for storing a table of spatial thinning amounts corresponding to the movement amount, and the spatial thinning amount can be determined by referring to the table.

  The resolution conversion unit can perform spatial thinning processing on a signal obtained by performing spatial filtering on the image signal obtained by the imaging unit.

  The movement amount detection unit can divide the image obtained by the imaging unit into a plurality of regions and detect the movement amount for each of the divided regions.

  Imaging means for obtaining an image signal having a first resolution by photoelectric conversion, and resolution conversion for converting the image signal obtained by the imaging means into an image signal having a second resolution lower than the first resolution And an output unit that outputs the image signal converted in resolution by the resolution conversion unit at a predetermined frame rate. The resolution conversion unit divides the image signal obtained by the imaging unit into a plurality of regions. A movement amount detection unit that detects a movement amount for each region, and sets a spatial thinning amount of each region based on the detected movement amount, and a spatial filter for each region of the image signal obtained by the imaging unit And a resolution conversion processing unit that performs spatial thinning processing for each region based on a set spatial thinning amount after performing the processing.

  An imaging step of obtaining an image signal having a first resolution by photoelectric conversion of the present invention, and an image signal obtained by the processing of the imaging step is converted into an image signal having a second resolution lower than the first resolution. An imaging program comprising a resolution conversion step and an output step for outputting an image corresponding to the resolution-converted image signal at a predetermined frame rate. The resolution conversion step includes at least a part of the image obtained in the imaging step. The amount of movement of the area is detected, and the resolution of the area is converted based on the detected amount of movement.

  In the present invention, an image signal having a first resolution is obtained by photoelectric conversion, the image signal is converted into an image signal having a second resolution lower than the first resolution, and the resolution-converted image signal is converted to an image signal. A corresponding image is output at a predetermined frame rate, the movement amount of at least a part of the obtained image is detected, and resolution conversion of the region is performed based on the detected movement amount.

  According to the present invention, a captured image can be converted into an output image so that an observer of the reproduced output image can recognize a resolution exceeding the maximum resolution that can be expressed in the number of output pixels.

  Embodiments of the present invention will be described below. The correspondence relationship between the invention described in this specification and the embodiments of the invention is exemplified as follows. This description is intended to confirm that the embodiments supporting the invention described in this specification are described in this specification. Therefore, although there is an embodiment which is described in the embodiment of the invention but is not described here as corresponding to the invention, it means that the embodiment is not It does not mean that it does not correspond to the invention. Conversely, even if an embodiment is described herein as corresponding to an invention, that means that the embodiment does not correspond to an invention other than the invention. Absent.

  Further, this description does not mean all the inventions described in this specification. In other words, this description is for the invention described in the present specification, which is not claimed in this application, that is, for the invention that will be applied for in the future or that will appear and be added by amendment. It does not deny existence.

  The imaging step of obtaining an image signal having the first resolution by photoelectric conversion (for example, the processing of the imaging unit 11 in FIG. 18) of the present invention, and the image signal obtained by the processing of the imaging step from the first resolution A resolution conversion step (for example, processing in steps S4 to S7 in FIG. 23) for converting to an image signal having a low second resolution, and an output step for outputting an image corresponding to the resolution-converted image signal at a predetermined frame rate (For example, the processing of the output unit 13 in FIG. 18), the resolution conversion step detects the movement amount of at least a partial region of the image obtained in the imaging step, and the detected movement amount Based on this, the resolution conversion of the area is performed.

  The imaging step of obtaining an image signal having the first resolution by photoelectric conversion (processing of the imaging unit 11 in FIG. 18) of the present invention, and the image signal obtained by the processing of the imaging step are lower than the first resolution. A resolution conversion step for converting the image signal to a resolution of 2 and an output step for outputting an image corresponding to the resolution-converted image signal at a predetermined frame rate (for example, processing of the output unit 13 in FIG. 18). In the imaging method, the resolution conversion step divides the image signal obtained in the imaging step processing into a plurality of regions and detects the movement amount for each region (for example, the processing in step S3 in FIG. 23). The spatial thinning amount of each area is set based on the amount of movement (for example, steps S4 and S5 in FIG. 23), and the spatial filter is applied to each area of the image signal obtained in the imaging step. Processing (for example, step S6 in FIG. 23) after subjected to, it is possible to perform spatial decimation processing for each area based on the spatial decimation amount set (for example, step S7 in FIG. 23).

  The image pickup means (for example, the image pickup unit 11 in FIG. 18) that obtains an image signal having the first resolution by photoelectric conversion, and the image signal obtained by the image pickup means are converted into a second lower than the first resolution. Resolution conversion means (for example, the image conversion unit 12 in FIG. 18) for converting the image signal having resolution, and output means (for example, in FIG. 18) that outputs the image signal whose resolution has been converted by the resolution conversion means at a predetermined frame rate. In the imaging apparatus including the output unit 13), the resolution conversion unit detects a movement amount of at least a partial area of the image obtained by the imaging unit (for example, the movement amount detection unit 21 in FIG. 18). ) And a resolution conversion processing unit (for example, the pixel number conversion unit 22 in FIG. 18) that converts the resolution of the region based on the detected movement amount.

  The image pickup means (for example, the image pickup unit 11 in FIG. 18) that obtains an image signal having the first resolution by photoelectric conversion, and the image signal obtained by the image pickup means are converted into a second lower than the first resolution. Resolution conversion means (for example, the image conversion unit 12 in FIG. 18) for converting the image signal having resolution, and output means (for example, in FIG. 18) that outputs the image signal whose resolution has been converted by the resolution conversion means at a predetermined frame rate. In the imaging apparatus including the output unit 13), the resolution conversion unit divides the image signal obtained by the imaging unit into a plurality of regions and detects a movement amount for each region (for example, FIG. 18). The movement amount detection unit 21) sets the spatial thinning amount of each region based on the detected movement amount (for example, the control unit 44 in FIG. 19), and for each region of the image signal obtained by the imaging means Spatial filtering After performing (for example, the spatial filter processing unit 41), a resolution conversion processing unit (for example, FIG. 19) that performs spatial thinning processing (for example, the spatial thinning processing unit 42) for each region based on the set spatial thinning amount. And a pixel number conversion unit 22).

  An imaging apparatus to which the present invention is applied converts a moving image obtained as a result of imaging into an image having the number of output pixels, and outputs the converted image using a super-resolution effect based on a predetermined visual characteristic. By performing the processing, it is possible to prevent the observer from recognizing image quality deterioration due to pixel number conversion.

  First, the visual characteristics and super-resolution effect will be described.

  Human vision has a function of perceiving light when the sum of received light stimuli reaches a certain threshold (hereinafter referred to as a temporal integration function). That is, the perception of light follows the sum of the light integrated over time regardless of the distribution state of the light stimulus within the presentation time. Further, the stimulus (threshold) that can perceive light becomes smaller as the presentation time of the stimulus becomes longer, and becomes larger as the presentation time becomes shorter.

This relationship is known as the Block's law, and the following equation holds: Where I is the intensity of the stimulus as a threshold, T is the stimulus presentation time, and k is a constant.
I × T = k

  This relationship can be expressed as shown in FIG. 1, where the horizontal axis is the stimulus presentation time T and the vertical axis is the threshold (intensity I). This curve is known as the threshold presentation time curve. According to the threshold value presentation time curve of FIG. 1, when light of intensity Ia is presented in impulse for time Ta, light of intensity Ib 1 / n of Ia continues for time Tb that is n times Ta. The person feels the same brightness when presented.

  It should be noted that the rule of the block is valid up to a time when the stimulus is presented (time TL in the example of FIG. 1) (although it becomes a straight line to the right until time TL), the threshold value of the stimulus is exceeded when time TL is exceeded. It depends only on the intensity (it does not change with the presentation time, and as a result, the threshold presentation time curve shows a characteristic like a broken line). The maximum stimulus presentation time TL for which the block law holds is called the critical presentation time. Although this time TL varies depending on the stimulus presentation conditions such as the intensity of the background light, there is a report that it is approximately 25 ms to 100 ms.

  Details of the block rule are described in, for example, “Visual Information Processing Handbook, Japanese Visual Society, pp.219-220”.

  The human vision also has a function (hereinafter referred to as a sensory memory function) that, when a stimulus is perceived, the stimulus is stored for a certain period of time after the presentation of the stimulus ends. There are many reports that this time is 10 ms to 200 ms. This function is also called iconic memory or visual persistence, and is described in, for example, “Visual Information Handbook, Japanese Visual Society, pp.229-230”.

  Next, the resolution effect realized based on visual characteristics will be described. Note that the super-resolution effect in the present invention uses a visual characteristic that an observer perceives a sum of a plurality of images within a certain time. This is considered to be caused by the complicated relationship between the temporal integration function and the sensory memory function. In the following description, this is based on the temporal integration function for convenience.

  For example, when a subject moving in parallel in the horizontal direction is photographed at a predetermined frame rate (hereinafter referred to as an input image frame rate) and a predetermined sampling rate (hereinafter referred to as an input image sampling rate), as shown in FIG. 2A. An input frame Fa in which the subject image Wa moves rightward (X-axis direction) toward the drawing at a speed v (pixel / frame) is obtained. FIG. 2A shows four consecutive input frames Fa1 to Fa4.

  The input frame Fa obtained in this way is sampled in the X-axis direction (moving direction of the subject image Wa) at a sampling rate (hereinafter referred to as a display image sampling rate) that is 1 / m of the input image sampling rate. (Thinning shall be performed with the thinning amount m). In the case of FIG. 2A, since the input frame Fa is thinned by the thinning amount 4, the number of pixels in the X-axis direction is reduced to 1/4 (rough in the X-axis direction) as shown in FIG. Display frame Fb is obtained. The display frame Fb includes an image obtained by thinning the subject image Wa of the input frame Fa with a thinning amount 4 (hereinafter referred to as a display subject image Wb).

  The display frame Fb thus obtained is displayed at a predetermined frame rate (hereinafter referred to as display image frame rate). As a result, the observer perceives integrated images of a plurality of display frames Fb displayed within the integration time in the temporal integration function described above.

  Here, it is assumed that the observer's line of sight follows the display subject image Wb on the display frame Fb displayed in this way. In this case, since the viewpoint of the observer is always located at the center of the display subject image Wb, the display subject image Wb on the observer's retina is almost stationary. The coordinate axes Vx and Vy shown in FIG. 2B indicate coordinates on the retina, and the coordinate axes X and Y indicate coordinates on the frame (both are shown on the display frame Fb1 in the figure, but the display frame Fb2 The illustration of Fb4 to Fb4 is omitted). As for the coordinate axes Vx and Vy, since a reverse image of the real image is formed on the retina, the direction of the coordinate system is opposite to that of the coordinate axes X and Y.

  In addition, as shown by the dotted line in FIG. 3, the display frame Fb is sampled at a certain position on the frame (in this example, a position at an interval of 4 pixels). Therefore, when the movement amount does not match the multiple of the sampling interval, the position of the sampled subject image Wa is shifted by v for each frame, so that each display subject image Wb of the display frame Fb is the sampling position of the subject image Wa. It is formed in a part different by the amount of deviation.

  For example, when the moving speed v of the subject image Wa is 1 (pixel / frame), the amount of movement (1 pixel) between frames does not match a multiple of the sampling interval (4 pixels), and the sampled subject image Wa is sampled. Is shifted by one pixel in the X-axis direction. Therefore, in this case, each display subject image Wb of the display frame Fb is formed from a different portion of the subject image Wa.

  In this way, when the display subject image Wb is formed from a portion that differs from the subject image Wa by the deviation of the sampling position, the display subject image Wb is integrated over a plurality of frames in the visual system. An image in which the pixels are denser than Wb (an image having a higher resolution than the resolution of the display subject image Wb (hereinafter referred to as super-resolution)) is perceived.

  For example, when the integration time in the visual characteristic corresponds to the display time of the four display frames Fb in FIG. 2B and the four display subject images Wb in the display frames Fa1 to Fa4 are integrated, as shown in FIG. 2C. In addition, an image having a resolution of about four times the resolution of the display subject image Wb, that is, about the same resolution as the subject image Wa is perceived.

  The super-resolution effect is realized by this principle. However, when the thinning process is performed, a aliasing component is generated, which becomes aliasing distortion and image quality deteriorates. Therefore, in the present invention, as described below, a device for removing the folding component is devised.

・・・（１）

・・・（２） Expression (1) represents a signal f _s (x) obtained by discretizing a one-dimensional original signal f (x) with an interval X. In equation (1), δ (x) is a delta function. Equation (2) represents the Fourier transform F _s (ω) of the discretized signal f _s (x). In equation (2), F (ω) is the Fourier transform of the original signal f (x), and ω _s represents the sampling angular frequency.

... (1)

... (2)

・・・（３） Expression (3) represents the Fourier transform F _sφ (ω) of the signal f _sφ (x) _obtained by discretizing the original signal f (x) by an interval X while shifting the original signal f (x) in the real space.

... (3)

Equation (3) indicates that the fundamental wave of k = 0 is the same as the original signal, and the n-th harmonic of k = n is out of phase by 2πnφ. As described above, assuming that the subject image Wa is moving in parallel at a certain moving speed v and the sampling is sampled at 1 / m in the moving direction, the original signal is a band of m times the Nyquist frequency of the display frame Fb. have. Therefore, the sampling signal f _sφ (x) thinned and sampled at 1 / m has an aliasing component, and in Equation (3), k = 0 is the original signal component, and k = 1, 2 _,. (m-1) is a folding component.

FIG. 4 shows the Fourier transform F _sφ (ω) when the thinning amount m = 2. At this time, the band of the original signal is twice the Nyquist frequency, and the first harmonic return component exists in the sampling signal f _sφ (x) sampled by 1 / m. As can be seen from this figure, the sampling signal f _sφ (x) has the Fourier transform F (ω) component of the original signal f (x) as it is, and the first harmonic F (ω−ω _s ) at k = 1. And F (ω + ω _s ) are folded back by −2πφ and 2πφ, respectively.

When the thinning sampling interval is 1 / m, the sampling signal f _sφ (x) thinned by 1 / m includes 1st to (m−1) -order folding components, and each phase is 2πkφ. It will be shifted. Since the sampling signal f _sφ (x) is a signal _{obtained by} sampling the original signal f (x) shifted by φ by 1 / m, it is considered to correspond to any one display frame Fb in FIG. 2B.

・・・（４） Here, consider the signals of the display frames Fb that are different in time in FIG. 2B. When the subject (original signal f (x)) is translated at the speed v, the phase of the sample points is shifted for each frame as shown in FIG. Therefore, the sampling point shift amount φ in the equation (3) is a function of the time t, and depends on the speed v (pixel / frame) and the thinning-out amount m (pixel) as in the equation (4). Become. In Expression (4), T represents a time interval and is the reciprocal of the frame rate.

... (4)

Expression (4) represents that the shift amount φ ₀ becomes 0 when t = 0, and the shift amount increases by v / m as t = T, 2T, 3T. When equation (4) is applied to equation (3), the phase of the aliasing component at each time is obtained. FIG. 5 shows the phases of the primary folding components at time t = 0, T, 2T, 3T,. 6 shows the phase of the second-order aliasing component, FIG. 7 shows the phase of the third-order aliasing component, and FIG. 8 shows the phase of the fourth-order aliasing component at time t = 0, T, 2T, 3T,. Represents each.

  In this way, the k-th order folding component rotates at equal intervals (2πkφT intervals) as time advances, that is, the frame, and returns to phase 0 at time t = (m / v) T. Further, the phase rotation interval doubles as the order of the aliasing component increases.

  In this way, the phase of the next aliasing component of k (= 1, 2,..., (M−1)) generated by the thinning process (thinning sampling) with the thinning amount m rotates at 2πkφT. Depending on the direction and the number of images to be integrated (the number of folded components to be combined), the folded components may cancel each other. In other words, since φt depends on the moving speed v and the thinning amount m as shown in the equation (4), the aliasing components may be canceled out depending on the moving speed v and the thinning amount m and the number of images to be integrated. is there.

  For example, when v = 1, when m = 4 is thinned, the image of the display frame Fb has 0 (= 2π × 1 × [(1/4) × 0 / T] as shown in FIG. ), Π / 2 (= 2π × 1 × [(1/4) × (T / T)], π (= 2π × 1 × [(1/4) × 2T / T], 3 / 2π (= 2π .Times.1.times. [(1/4) .times.3T / T]), ..., there is a primary folding component whose phase changes (the phase changes at intervals of .pi. / 2). The folding components after 4T are not shown, and the same applies to FIGS.

  In the image of the display frame Fb, as shown in FIG. 10, 0 (= 2π × 2 × [1/4 × 0 / T]), π (= 2π × 2 × [1/4 × T / T] ), 2π (= 2π × 2 × [(1/4) × (2T / T)]), 3π (= 2π × 2 × [(1/4) × 3T / T]),. A second-order aliasing component that changes (the phase changes at an interval of π), and 0 (= 2π × 3 × [(1/4) × (0 / T)], 3π / 2 ( = 2π × 3 × [(1/4) × (T / T)], 3π (= 2π × 3 × [(1/4) × (2T × T)]), 9π / 2 (= 2π × 3 ×) [(1/4) × (3T / T)]), ..., there is a third-order aliasing component whose phase changes (the phase changes at intervals of 3π / 2).

  In this case, since the vectors of the primary to tertiary folding components at t = 0, T, 2T, and 3T are directed to cancel each other, as shown in FIGS. When four display frames Fb are integrated, they are all canceled out.

・・・（５）

・・・（６）

・・・（７） If the condition for canceling the k-th order aliasing component is expressed by an expression, the expression (5) is obtained. If the expression (5) is expanded by Euler's formula, the expressions (6) and (7) are obtained.

... (5)

... (6)

... (7)

  That is, in the present invention, the aliasing component is removed by determining the thinning-out amount m so that aliasing components cancel each other are generated according to the moving speed v of the subject image Wa.

Here, considering the case where the discretized signal f _sφ (x) is reduced to 1 / m by a band-limited digital filter, the original signal f (x) shifted by φ is a band at the Nyquist frequency so that no aliasing occurs. Limited. Therefore, for example, when m = 2, the Fourier space is as shown in FIG. 12, and each frame image corresponding to the signal reduced to 1 / m is a low-resolution image that does not include the aliasing component. Therefore, in this case, the fundamental wave of the reduced signal is a signal different from the original signal, and it is not possible to express a frequency component equal to or higher than the Nyquist frequency no matter how the addition processing is performed on images of a plurality of frames. The super resolution effect cannot be obtained. Therefore, in order to obtain the super-resolution effect, it is important not to limit the band of the original signal, and it is optimal to thin out and sample the original image having a wideband spatial frequency component.

  In the above, for the sake of simplicity, the case where the original signal is a one-dimensional signal has been described as an example, but the same applies to a two-dimensional image. Although the movement of the subject image Wa in the X-axis direction has been described as an example with reference to FIG. 2, the same applies to the movement in the Y-axis direction.

  Next, conditions for aliasing components that cancel each other (conditions for aliasing components of an image that can obtain a super-resolution effect) will be described.

  The condition for obtaining the super-resolution effect is that Expression (5) is satisfied, that is, Expression (6) and Expression (7) are satisfied. As shown in FIG. 13, if the k-th order folding component at time t is a vector Zk (t), the sum of the vectors Zk (t) in the integration range of the visual system is zero. The establishment of this condition depends on the integration time, but it is known that this integration time varies depending on the observation environment, and it is difficult to measure it accurately. It is difficult to limit.

On the other hand, for example, a subject image Wa moving in the X-axis direction or the Y-axis direction at a predetermined moving speed v is sampled at a predetermined thinning amount m and displayed at a predetermined frame rate, and the displayed display subject image Wb is displayed. From an experiment to confirm whether the observer actually perceived the image at the super resolution, if the frame rate is high, that is, if the number of images to be integrated is large, the super resolution effect can be obtained even if the thinning amount m increases. I know that. At this time, the condition for obtaining the super-resolution effect depends on the moving speed v, and it is considered that the relationship is approximately represented by Expression (8).
2πn + α ≦ 2πkφT ≦ 2π (n + 1) −α (8)

  As described above, the phase of each aliasing component rotates at intervals of 2πkφT, but Equation (8) indicates that the super-resolution effect cannot be obtained when the phase rotation interval of each subsequent aliasing component is close to a multiple of 2π. Represents. As shown in FIG. 14, the fact that the phase rotation interval is close to a multiple of 2π means that the phase of the aliasing component hardly changes even when the time t changes, and the aliasing component remains without being canceled. It is because it ends.

  For example, when the conditions for establishing Equation (8) are examined for the primary to tertiary folding components generated when m = 4, the shaded moving speed v (pixel / frame) in FIG. In the range, equation (8) does not hold and the super-resolution effect cannot be obtained.

  For this primary folded component, for example, when v = 4, the phase rotation interval of the folded component = 2π × 1 × (4/4) (2πkφT), and the phase rotation interval of the folded component is 1 × 2π. Therefore, in a certain range centered on the speed v = 4 (a range of speed where the phase rotation interval is a range of 2α centered on a multiple of 2π), the folding component of the linear expression is not canceled. That is, when v = 4n (n = 0, 1, 2, 3,...), The phase rotation interval is n times 2π. Therefore, in a certain range centered on v = 4n, the linear expression The folded component is not negated.

  For the second-order aliasing component, for example, when v = 2, the phase rotation interval = 2π × 2 × (2/4) (1 × 2π), and when v = 4, the phase rotation interval = 2π × Since 2 × (4/4) (twice 2π), a constant range centered on the velocity v = 2, 4 (a range of velocity where the phase rotation interval is a range of 2α centered on a multiple of 2π) , The folding component of the quadratic expression is not canceled out. That is, when v = 2n, the phase rotation interval is n times 2π. Therefore, the folding component of the quadratic expression is not canceled within a certain range centered on v = 2n.

  For the third-order aliasing component, for example, when v = 4/3, the phase rotation interval = 2π × 3 × (4/3) / 4 (1 × 2π), and when v = 8/3, the phase Rotation interval = 2π × 3 × (8/3) / 4 (2 × 2π), and when v = 4, phase rotation interval = 2π × 3 × 4/4 (3 × 2π), In a certain range centered on the speed v = 4/3, 8/3, 4 (the range of speed where the phase rotation interval is 2α centered on a multiple of 2π), the folding component of the cubic equation is canceled out. Disappear. That is, when v = (4/3) n, the phase rotation interval is n times 2π, so that the third-order aliasing component is canceled within a certain range centered on v = (4/3) n. Absent.

Since the phase rotation interval 2πkφT = 0 when the speed v = 0, each of the primary to tertiary folding components is in a certain range (0 to v _α1 , 0 to v _α2 , 0 to v _α3 ) no longer cancels.

  The primary and secondary aliasing components that exist when m = 3 (FIG. 16) and the primary aliasing component that exists when m = 2 (FIG. 17) are also described above by taking m = 4 as an example. In addition, at the speed at which the phase rotation interval is in the range of 2α centered on a multiple of 2π, the aliasing components of the following equations are not canceled out.

Further, as shown in FIG. 13, as the order of the aliasing component increases, the phase rotation interval at each order increases to 2 times and 3 times. If the phase rotation interval is θ, the moving speed v of the subject image Wa is small, and when the phase rotation interval θ is smaller than α, equation (8) does not hold and the super-resolution effect cannot be obtained. When the moving speed v of the subject image Wa increases and the phase rotation interval reaches α, super resolution is obtained. From this, it is considered that α is a critical point (phase rotation interval) at which the super-resolution effect is obtained. This α varies depending on the display image frame rate, and tends to decrease as the display image frame rate increases. If the moving speed of the subject image Wa at the critical point is v _α , Equation (9) is obtained, and if it is transformed, Equation (10) is obtained. Therefore, when the display image frame rate increases and α decreases, the velocity v _α (v _α1 , v _α2 , or v _{α3 in} the example of FIG. 15) decreases, and as a result, even if the movement amount is small, the super-resolution is achieved. An effect is obtained.

Also, it can be seen from equation (10) that v _{α at} the critical point depends on the thinning amount m and the order k of the aliasing component, and the critical point speed v _α increases as the thinning amount m increases. Since the velocity v _{α at the} critical point decreases as the order k increases (in the example of FIG. 15, v _α2 is smaller than v _α1 and v _α3 is smaller than v _α2 ), the higher-order folding component It can be seen that the region where the super-resolution effect is not obtained becomes narrow.

In summary, the following can be said about the super-resolution effect in the visual system.
The critical point α at which the super-resolution effect can be obtained becomes small in high frame rate display.
When the thinning amount is m, the 1st to m−1 order folding components need to satisfy the formula (8).
When the thinning amount m decreases, the speed v _α of the subject image Wa at the critical point decreases (when the thinning amount m is small, the super-resolution effect can be obtained even if the movement amount is small).

  From the above, it can be seen that the super-resolution effect can be realized by performing the thinning process according to the moving speed (size and direction) of the subject.

  Increasing the display image frame rate in this way is advantageous for obtaining the super-resolution effect, but increasing the display image frame rate can also improve image quality degradation such as motion blur and jerkiness. It will be advantageous.

  Next, a configuration example of the moving image conversion apparatus 1 to which the present invention is applied will be described with reference to FIG. The moving image conversion apparatus 1 performs a resolution conversion process using the above-described super-resolution effect, so that the observer does not recognize image quality degradation due to the conversion to the number of output pixels, It can be converted into an image having.

  The imaging unit 11 includes a solid-state imaging device (such as a CCD imaging device or a CMOS imaging device) capable of imaging with a larger number of pixels than the number of pixels of the image that can be output by the output unit 13.

  The imaging unit 11 performs a moving image imaging process at a predetermined frame rate and a predetermined spatial sampling rate (hereinafter simply referred to as a sampling rate), and converts an image obtained as a result of the imaging process into an internal data format. For example, when the frame rate of the image that can be output by the output unit 13 is F and the number of pixels is (x × y), the frame rate is F and the number of pixels is (ix × jy). This is converted into an imaging frame Fa in the internal data format. i, j, x, and y are each a positive number.

  The imaging unit 11 supplies the converted imaging frame Fa ((ix × jy) pixels) to the image conversion unit 12.

  Note that when the image signal output from the solid-state image sensor is an analog signal, the imaging unit 11 performs conversion into a digital signal according to an internal data format by an analog / digital converter (not shown).

  Further, as described above, since the super-resolution effect is more easily obtained at a high frame rate, the imaging unit 11 can cope with the high frame rate. As for the solid-state imaging device, the CMOS imaging device is more suitable for high frame rate imaging because of the difference in driving method. In this case, the imaging unit 11 includes the CMOS imaging device.

  The image conversion unit 12 outputs the imaging frame Fa ((ix × jy) pixels) supplied from the imaging unit 11 as an output frame Fb ((x × y)) with the number of pixels (number of output pixels) that the output unit 13 can output. Pixel). At that time, the image conversion unit 12 executes the resolution conversion process using the above-described super-resolution effect for each block of a predetermined size, and when the output frame Fb is displayed at the predetermined frame rate, the observation is performed. A visual effect (super-resolution effect) that a person perceives at a resolution exceeding that resolution (super-resolution) is realized.

  The image conversion unit 12 supplies the generated output frame Fb to the output unit 13.

  The output unit 13 outputs a moving image signal. As described above, since the super-resolution effect is more easily obtained at a high frame rate, the output unit 13 can output an image at a high frame rate.

  Next, the configuration of the image conversion unit 12 will be described. An image signal in the internal data format of (ix × jy) pixels is input to the image conversion unit 12.

  The movement amount detection unit 21 of the image conversion unit 12 is input the imaging frame Fa ((ix × jy) pixels) (current frame Fa) supplied from the imaging unit 11 and one frame time or several frame times before. The motion vector of the block image ((ip × jq) pixels) of the current frame Fa is detected by block matching processing with the captured frame Fa (previous frame Fa). p and q are both positive integers. In this case, the total number of blocks of the imaging frame Fa is (x / p × y / q).

  The movement amount detection unit 21 supplies the imaging frame Fa to the pixel number conversion unit 22 together with the detected motion vector in units of blocks.

  The pixel number conversion unit 22 applies the block image ((ip × jq) pixels) of the imaging frame Fa supplied from the movement amount detection unit 21 for each frame in the X-axis direction and the Y-axis direction obtained from the motion vector. Filtering or sampling (thinning-out processing) in the X-axis direction and Y-axis direction according to the moving amount v (pixel) (moving speed v (pixel / frame)). The pixel number conversion unit 22 collects the block image ((p × q) pixels) of the output frame Fb obtained as a result of the processing by the total number of blocks (x / p × y / q), and outputs a single output frame Fb. ((X × y) pixels) is generated and output to the output unit 13.

  Next, the configuration of the movement amount detection unit 21 and the pixel number conversion unit 22 of the image conversion unit 12 will be described with reference to FIG.

  The frame memory 31 of the movement amount detection unit 21 stores the imaging frame Fa (current frame Fa) supplied from the imaging unit 11. The delay circuit 32 delays the previous frame Fa input one frame time or several frame times before the current frame Fa so that the phase thereof matches the phase of the current frame Fa. The delay circuit 32 supplies the delayed previous frame Fa to the frame memory 33. The frame memory 33 stores the previous frame Fa from the delay circuit 32.

  The block matching circuit 34 performs block matching in units of blocks of (ip × jq) pixels between the current frame Fa stored in the frame memory 31 and the previous frame Fa stored in the frame memory 33. A motion vector of a block image ((ip × jq) pixels) of the current frame Fa is detected. The block matching circuit 34 supplies the block image itself of the current frame Fa to the spatial filter processing unit 41 of the pixel number conversion unit 22 and the detected motion vector of the block image to the control unit 44 of the pixel number conversion unit 22. Supply.

  The configuration of the movement amount detection unit 21 is an example of implementation, and other configurations may be adopted as long as the motion vector of each block of the imaging frame Fa can be detected.

  A block image ((ip × jq) pixels) (see the left side of FIG. 20) of the imaging frame Fa from the block matching circuit 34 of the movement amount detection unit 21 is input to the spatial filter processing unit 41 of the pixel number conversion unit 22. The The spatial filter processing unit 41 is a digital filter that limits the bandwidth of the spatial resolution, reduces the aliasing component, and converts it into a desired number of pixels given from the control unit 44.

  Specifically, the spatial filter processing unit 41 thins out the block image ((ip × jq) pixels) of the imaging frame Fa with the thinning amounts mx and my in the subsequent spatial thinning processing unit 42 according to the control by the control unit 44. As a result, spatial filtering is performed so that a block image ((p × q) pixels) of the output frame Fb is obtained, and the block image is converted into an image of (mxp × myp) pixels (see the left side in FIG. 20).

  The spatial filter processing unit 41 supplies the converted (mxp × myq) pixel image to the spatial thinning processing unit 42. Note that mx and my are both positive integers.

  The spatial thinning processing unit 42 performs thinning sampling on the original image without converting the bandwidth of the spatial resolution, and converts the original image into a desired number of pixels given from the control unit 44.

  Specifically, for the image of (mxp × myq) pixels (see the right side of FIG. 20) supplied from the spatial filter processing unit 41, the thinning amount mx in the X-axis direction notified from the control unit 44, and the Y-axis According to the direction thinning amount my, thinning is performed in the X-axis direction and the Y-axis direction to generate an image of (p × q) pixels (block image of the output frame Fb) (see the right side in FIG. 20). The spatial thinning processing unit 42 supplies the generated (p × q) pixel image to the frame memory 43.

  As in the example of FIG. 20, when mx <i, my <j, the thinning amount is mx and my notified from the control unit 44, but when mx> i, my> j, thinning is performed. The quantity is i or j.

  Returning to FIG. 19, the frame memory 43 stores the block image ((p × q) pixels) supplied by the spatial thinning processing unit 42 and generates one output frame Fb ((x × y) pixels). .

  A motion vector of the block image from the movement amount detection unit 21 is input to the control unit 44. The control unit 44 calculates the thinning amounts (maximum values) mx and my satisfying the equation (8) from the moving speed v in the X-axis direction and the Y-axis direction obtained from the motion vector. Detect (detect a thinning amount that satisfies a condition for obtaining the super-resolution effect).

  Note that α in the equation (8) varies depending on the number of images integrated within the integration time in the visual system as described above, and it is difficult to determine the value as appropriate. Therefore, in the present invention, for example, a subject image Wa moving in the X-axis direction or the Y-axis direction at a predetermined moving speed v is sampled and displayed with a predetermined thinning amount m, and the displayed display subject image Wb is observed. An experiment is performed in advance to confirm whether the person can actually perceive at the super resolution, and the thinning amount m is determined from the relationship between the moving speed v and the thinning amount m when there is a super resolution effect as a result of the experiment. It shall be.

  FIG. 21 shows an experimental result when the above-described experiment is performed on the subject image Wa moving in the horizontal direction.

  In FIG. 21, when the thinning-out amount m = 4, the moving speed within a certain range centered on the speed v = 4/3, 2, 8/3, 4 and when the thinning-out amount m = 3, the speed v = 3. It is shown that the super-resolution effect cannot be obtained at a moving speed within a certain range centered at / 2,3, and at a moving speed within a certain range centered at speed m = 2,4 at a thinning amount m = 2. Has been. When the moving speed v is within a predetermined range P0 starting from 0 (below the speed represented by the speed v0 in the figure), the super-resolution effect can be obtained in any of m = 2, 3 and 4. Not shown.

  This corresponds to the range of the moving speed v in which Expression (8) is not satisfied, which has been described with reference to FIGS. 15 to 17.

  For example, if m = 4 in FIG. 21 is associated with FIG. 15, it is as shown in FIG. Regarding the primary folding component, the folding component of the primary expression is not canceled within a certain range centered on v = 4n. Regarding the secondary folding component, the folding component of the quadratic expression is not canceled within a certain range centered on v = 2n. As for the third-order folding component, the third-order folding component is not canceled within a certain range centered on v = (4/3) n. Even in a certain range near v = 0, the first-order to third-order aliasing components are not canceled out.

  In other words, if any one of the first-order to third-order aliasing components is not canceled, the super-resolution effect cannot be obtained by thinning out at m = 4 within the range of the moving speed v. In FIG. 22, ranges in which such a super-resolution effect cannot be obtained are indicated by ranges U4a to U4d. Further, in FIG. 22, ranges where the super-resolution effect can be obtained when m = 4 are indicated by ranges P4a to P4d.

  Also in FIG. 21, ranges where the super-resolution effect is obtained when m = 4 are indicated by ranges P4a to P4d. Similarly, in FIG. 21, ranges where the super-resolution effect is obtained when m = 3 are shown as ranges P3a to P3c, and ranges where the super-resolution effect is obtained when m = 2 are shown as ranges P2a, This is indicated by P2b.

  Accordingly, in this example, for example, when the moving speed v in the X-axis direction of the block image of the imaging frame Fa is the speed indicated by va in FIG. It is assumed that the effect is 2 (pixel), and the X-axis direction of the block image is thinned by the thinning amount 2. Also, for example, when the moving speed v in the X-axis direction of the block image of the imaging frame Fa is a speed indicated by vb faster than va in the range of 0 to 4/3, for example, a super-resolution effect can be obtained m = 2. The maximum 4 of 3 and 4 is set, and the X-axis direction of the block image is thinned by the thinning amount 4.

  When the moving speed v in the X-axis direction of the block image of the imaging frame Fa is within the range P0 (when the super resolution effect cannot be obtained), the thinning amount m is set to 1 and the number of pixels All conversions are performed by spatial filtering. In other words, in the present invention, for a region where the moving speed v of the subject image Wa is high, the thinning process is performed so as to include the aliasing component to enable the super resolution effect, and the moving speed is low (range P0) (super resolution effect). For a region in a range in which (a) is not obtained, band limitation is performed by spatial filter processing so that no aliasing occurs.

  If the frame rate is high, the range in which the super-resolution effect cannot be obtained is narrowed as described above (the range of speed at which the super-resolution effect can be obtained becomes large), and thinning can be performed with a larger thinning amount. .

  Summarizing the selection of the thinning amount, the table in FIG. 21 is obtained. That is, the control unit 44 determines the thinning amount based on this table (the same table for the Y-axis direction).

  For example, the control unit 44 stores a table as shown in FIG. 21 in the storage unit 44A, and detects the thinning amount m (mx, my) corresponding to the moving speed v in the table. The control unit 44 controls the spatial filter processing unit 41 to convert the block image ((ip × jq) pixels) of the imaging frame Fa into (mxp × myq) pixels (decimation by spatial filter processing for band limitation). In addition, the spatial thinning processing unit 42 is controlled to perform thinning with the thinning amounts mx and my on the (mxp × myq) pixel image supplied from the spatial filter processing unit 41 (bandwidth). Let me thin out without restriction).

  When the moving speed v is within the range P0, the control unit 44 cannot obtain the super-resolution effect in any of m = 2, 3 and 4. In this case, the pixel conversion is performed for all the pixel conversions with the thinning amount set to 1. The filtering is performed by the processing unit 41.

  Next, the operation of the image conversion unit 12 will be described with reference to the flowchart of FIG.

  In step S1, the block matching circuit 34 of the movement amount detection unit 21 of the image conversion unit 12 initializes a block number (for example, 1), and in step S2, detects a motion vector for all blocks of the imaging frame Fa. Determine whether or not.

  If it is determined in step S2 that motion vectors have not been detected for all blocks, the process proceeds to step S3, where the block matching circuit 34 of the movement amount detection unit 21 performs block matching processing on the block image indicated by the block number. And the motion vector of the block image is detected. The total number of blocks is (x / p × y / q), and each block image of the imaging frame Fa can be identified by block numbers 1 to (x / p × y / q). Yes.

  Specifically, the block matching circuit 34 compares the target block image of the current frame Fa with an arbitrary block image in the search area of the previous frame Fa, and detects the block position of the previous frame Fa that minimizes the mean square error. A vector connecting the target block of the current frame Fa and the detected block of the previous frame Fa is defined as a motion vector. The block matching circuit 34 supplies the block image of the current frame Fa to the spatial filter processing unit 41 of the pixel number conversion unit 22 and supplies the motion vector of the block image to the control unit 44.

  Next, in step S4, the control unit 44 of the pixel number conversion unit 22 moves the moving speed v (moving amount v in the X-axis direction) based on the motion vector supplied from the moving amount detecting unit 21 (block matching circuit 34). ) Then, the control unit 44 detects the thinning amount mx corresponding to the moving speed v (moving amount v) from the table representing the relationship between the moving speed v and the thinning amount m shown in FIG.

  In step S5, the control unit 44 detects the movement speed v (movement amount v) in the Y-axis direction based on the motion vector supplied from the movement amount detection unit 21 (block matching circuit 34). Then, the control unit 44 detects the thinning amount my corresponding to the moving speed v (moving amount v) from the table representing the relationship between the moving speed v in the Y-axis direction and the thinning amount m.

  Next, in step S <b> 6, the control unit 44 of the pixel number conversion unit 22 controls the spatial filter processing unit 41 to block the block image ((ip × jq) pixels of the imaging frame Fa supplied from the movement amount detection unit 21. ) Is converted into an image of (mxp × myq) pixels (see the left side of FIG. 20) (the block image is subjected to spatial filter processing).

  For example, when the thinning amount mx = 4, the control unit 44 controls the spatial filter processing unit 41 to perform spatial filter processing for obtaining an image of (4p × q) pixels. In addition, when the thinning amount mx = 3, the control unit 44 controls the spatial filter processing unit 41 to perform spatial filter processing for obtaining an image of (3p × q) pixels. Here, the pixel in the X-axis direction has been described as an example, but the same applies to the pixel in the Y-axis direction.

  Next, in step S7, the control unit 44 controls the spatial thinning processing unit 42 and is supplied from the spatial filter processing unit 41 according to the thinning amounts mx and my determined in steps S4 and S5 (mxp × myq). The pixel image is thinned out. Thus, the spatial thinning processing unit 42 uses the thinning amount mx in the X-axis direction and the thinning-out operation in the Y-axis direction with respect to the block image of (mxp × myq) pixels supplied from the spatial filter processing unit 41. A thinning process is performed with the amount my. As a result, a block image of (p × q) pixels is obtained (see the right side of FIG. 20).

  For example, when the thinning amount mx = 4, the thinning processing is performed on the (4p × q) pixels supplied by the spatial filter processing unit 41 with the thinning amount 4 in the X-axis direction, and (p × q) A block image of pixels is generated. When the thinning amount mx = 3, the thinning processing is performed on the (3p × q) pixel image supplied by the spatial filter processing unit 41 with the thinning amount 3 in the X-axis direction, and (p × q ) A block image of pixels is generated.

  When space thinning is performed, aliasing components are generated. However, since the thinning amounts mx and my are thinning amounts satisfying Expression (8), the aliasing components do not cancel each other.

  The spatial thinning processing unit 42 supplies a block image of (p × q) pixels obtained as a result of the thinning processing to the frame memory 43. The frame memory 43 stores the block image at a predetermined position and synthesizes one output frame Fb.

  In step S8, the block matching circuit 34 of the movement amount detector 21 increments the block number by one.

  Thereafter, the process returns to step S2, and the subsequent processing is performed.

  If it is determined in step S2 that motion vectors have been detected for all blocks, the process ends.

  As described above, the resolution of one imaging frame Fa is converted for each block by the processing in steps S3 to S7, so that one output frame Fb is generated and displayed at a predetermined frame rate. Is output from the output unit 13. As a result, when the image output from the output unit 13 is reproduced and displayed, the observer can display a resolution higher than the resolution that can be expressed by the number of pixels actually displayed (the display subject image Wb). (The resolution exceeding the resolution) can be perceived, but the spatial resolution perceived by the observer due to the super-resolution effect is mx times the X-axis direction of the block image ((p × q) pixels) of the output frame Fb, and the Y-axis This corresponds to the resolution of an image of (mxp × myq) pixels whose direction is my times.

  As described above, the motion is detected for each block of the image, and appropriate spatial filter processing and spatial thinning processing are performed in the spatial filter processing unit 41 and the spatial thinning processing unit 42, respectively. With respect to the block, the super-resolution effect is obtained, and with respect to the block having a small motion, the aliasing distortion is reduced, and as a whole, a high-quality image can be perceived.

  FIG. 24 shows a configuration example of another imaging apparatus to which the present invention is applied. An image conversion function is implemented in the imaging apparatus.

  This imaging apparatus includes an imaging element 51 and an output unit 52. The image sensor 51 is realized by a CMOS type. In a CMOS image sensor, it is possible to simultaneously mount a light receiving element and an arithmetic element on one device.

  The image sensor 51 includes a light receiving unit 61, a movement amount detection unit 62, and a pixel number conversion unit 63.

  In the light receiving unit 61, light receiving elements that receive light and convert it into an electrical signal are two-dimensionally arranged. Further, the photoelectrically converted signal is converted into a digital signal according to an internal data format by analog / digital conversion.

  In the movement amount detection unit 16, a circuit similar to the movement amount detection unit 21 in the example of FIG. 18 is mounted on the image sensor. In the pixel number conversion unit 63, a circuit similar to the pixel number conversion unit 22 in the example of FIG. 18 is mounted on the image sensor.

  Since the imaging device according to the present invention has a feature that exhibits an effect in high frame rate display, the imaging device of the light receiving unit 61 can perform high frame rate imaging.

  The output unit 52 has the same function as the output unit 13 in the example of FIG.

  The series of processes described above can be realized by hardware, but can also be realized by software. When a series of processing is realized by software, a program constituting the software is installed in a computer, and the management device 2 described above is functionally realized by executing the program by the computer.

  FIG. 25 is a block diagram illustrating a configuration of an embodiment of a computer 101 that functions as the imaging device 1 and the imaging device 51 as described above. An input / output interface 116 is connected to a CPU (Central Processing Unit) 111 via a bus 115, and the CPU 111 receives commands from an input unit 118 such as a keyboard and a mouse via the input / output interface 116. When input, it is stored in a recording medium such as a ROM (Read Only Memory) 112, a hard disk 114, or a magnetic disk 131, an optical disk 132, a magneto-optical disk 133, or a semiconductor memory 134 attached to the drive 120, for example. The program is loaded into a RAM (Random Access Memory) 113 and executed. Thereby, the various processes described above (for example, the process shown by the flowchart of FIG. 23) are performed. Further, the CPU 111 outputs the processing result to a display unit 117 such as an LCD (Liquid Crystal Display) via the input / output interface 116 as necessary. The program is stored in advance in the hard disk 114 or ROM 112 and provided to the user integrally with the computer 101, or as a package medium such as the magnetic disk 131, the optical disk 132, the magneto-optical disk 133, and the semiconductor memory 134. Or can be provided to the hard disk 114 via the communication unit 119 from a satellite, a network, or the like.

  In the present specification, the step of describing the program provided by the recording medium is not limited to the processing performed in chronological order according to the described order, but is not necessarily performed in chronological order. It also includes processes that are executed individually.

It is a figure explaining the rule of a block. It is a figure explaining the principle of resolution conversion processing. It is a figure explaining a sampling position. It is a figure explaining a return component. It is a figure explaining the change of the phase of a return component. It is another figure explaining the change of the phase of a return component. It is another figure explaining the change of the phase of a return component. It is another figure explaining the change of the phase of a return component. It is another figure explaining the change of the phase of a return component. It is a figure explaining the rotation interval of the phase of a return component. It is another figure explaining the rotation interval of the phase of a return component. It is another figure explaining a return component. It is another figure explaining the rotation interval of the phase of a return component. It is a figure explaining the phase of the return component for obtaining a super-resolution effect. It is a figure which shows the speed range which cannot acquire a super-resolution effect. It is another figure which shows the speed range which cannot acquire a super-resolution effect. It is another figure which shows the speed range which cannot acquire a super-resolution effect. It is a block diagram which shows the structural example of the imaging device to which this invention is applied. It is a block diagram which shows the structural example of the movement amount detection part of FIG. 18, and a pixel number conversion part. It is a figure explaining the operation | movement of the pixel number conversion part of FIG. It is a figure which shows the relationship between the moving speed of an image in case a super-resolution effect can be acquired, and the thinning-out amount. It is another figure which shows the relationship between the moving speed of an image in case a super-resolution effect can be acquired, and the thinning-out amount. 20 is a flowchart for explaining the operation of the image conversion unit in FIG. 19. It is a block diagram which shows the structural example of the other imaging device to which this invention is applied. And FIG. 16 is a block diagram illustrating a configuration example of a personal computer.

Explanation of symbols

  DESCRIPTION OF SYMBOLS 1 Imaging device, 11 Imaging part, 12 Image conversion part, 13 Output part, 21 Movement amount detection part, 22 Pixel number conversion part, 31 Frame memory, 32 Delay circuit, 33 Frame memory, 34 Block matching circuit, 41 Spatial filter processing Unit, 42 spatial thinning processing unit, 43 frame memory, 44 control unit, 51 image sensor, 52 output unit, 61 light receiving unit, 62 movement amount detection unit, 63 pixel number conversion unit

Claims (22)

An imaging step of obtaining an image signal having a first resolution by photoelectric conversion;
A resolution conversion step of converting the image signal obtained by the processing of the imaging step into an image signal having a second resolution lower than the first resolution;
An output step of outputting an image corresponding to the image signal whose resolution has been converted at a predetermined frame rate,
The resolution conversion step detects an amount of movement of at least a partial region of the image obtained in the imaging step, and performs resolution conversion of the region based on the detected amount of movement. . The resolution conversion step includes a spatial thinning process that provides a visual effect such that display means for displaying the image signal output from the output means at the frame rate has a resolution exceeding the second resolution. The imaging method according to claim 1. The imaging method according to claim 2, wherein a spatial thinning amount in the spatial thinning process is determined based on a movement amount of the region. The horizontal thinning amount in the horizontal direction is determined based on the horizontal movement amount of the region, and the vertical spatial thinning amount is determined based on the vertical movement amount of the region. Item 4. The imaging method according to Item 3. The imaging method according to claim 2, wherein when the frame rate is high, an amount of space thinning in the space thinning process is increased. In the spatial thinning process for an arbitrary frame, the spatial thinning amount is determined so that all the aliasing components of the signal subjected to the thinning process in a predetermined number of adjacent frames before and after the frame cancel each other. The imaging method according to claim 2. The imaging method according to claim 2, wherein the resolution conversion step performs the spatial thinning process on a signal obtained by subjecting the image signal obtained by the imaging step process to a spatial filter process. The imaging method according to claim 1, wherein the resolution conversion step divides an image obtained by the processing of the imaging step into a plurality of regions, and detects the movement amount for each of the divided regions. An imaging step of obtaining an image signal having a first resolution by photoelectric conversion;
A resolution conversion step of converting the image signal obtained by the processing of the imaging step into an image signal having a second resolution lower than the first resolution;
An output step of outputting an image corresponding to the image signal whose resolution has been converted at a predetermined frame rate,
The resolution conversion step divides the image signal obtained by the processing of the imaging step into a plurality of regions, detects a movement amount for each region, and based on the detected movement amount, a spatial thinning amount of each region And performing spatial filtering on each area of the image signal obtained in the imaging step, and then performing spatial thinning for each area based on the set spatial thinning amount An imaging method characterized by the above. Imaging means for obtaining an image signal having a first resolution by photoelectric conversion;
Resolution conversion means for converting the image signal obtained by the imaging means into an image signal having a second resolution lower than the first resolution;
An image pickup apparatus comprising: output means for outputting an image signal whose resolution is converted by the resolution conversion means at a predetermined frame rate;
The resolution conversion unit includes a movement amount detection unit that detects a movement amount of at least a partial region of the image obtained by the imaging unit, and a resolution conversion that performs resolution conversion of the region based on the detected movement amount. An imaging apparatus comprising: a processing unit. The resolution conversion unit performs a spatial thinning process that provides a visual effect such that a display unit that displays the image signal output from the output unit at the frame rate has a resolution that exceeds the second resolution. The imaging device according to claim 10. The imaging apparatus according to claim 11, wherein a spatial thinning amount in the spatial thinning process is determined based on a movement amount of the region. The horizontal spatial thinning amount is determined based on the horizontal movement amount of the region, and the vertical spatial thinning amount is determined based on the vertical movement amount of the region. The imaging device according to claim 12. The imaging apparatus according to claim 10, wherein when the frame rate is high, an amount of space thinning in the space thinning process is increased. In the spatial thinning process for an arbitrary frame, the spatial thinning amount is determined so that all the aliasing components of the signal subjected to the thinning process in a predetermined number of adjacent frames before and after the frame cancel each other. The imaging apparatus according to claim 11. The resolution conversion unit further includes a storage unit for storing a table of spatial thinning amounts corresponding to the movement amount, and the spatial thinning amount is determined by referring to the table. The imaging device according to claim 10. The imaging apparatus according to claim 10, wherein the resolution conversion unit performs the spatial thinning process on a signal obtained by performing spatial filter processing on the image signal obtained by the imaging unit. The imaging apparatus according to claim 10, wherein the movement amount detection unit divides an image obtained by the imaging unit into a plurality of regions and detects a movement amount for each of the divided regions. The imaging apparatus according to claim 1, wherein the resolution conversion unit is mounted on a semiconductor device. Imaging means for obtaining an image signal having a first resolution by photoelectric conversion;
Resolution conversion means for converting the image signal obtained by the imaging means into an image signal having a second resolution lower than the first resolution;
An image pickup apparatus comprising: output means for outputting an image signal whose resolution is converted by the resolution conversion means at a predetermined frame rate;
The resolution conversion means includes
A movement amount detection unit that divides the image signal obtained by the imaging unit into a plurality of regions and detects a movement amount for each region;
The spatial thinning amount of each area is set based on the detected movement amount, and after the spatial filtering process is performed on each area of the image signal obtained by the imaging unit, the set space is set. An imaging apparatus comprising: a resolution conversion processing unit that performs a spatial thinning process for each region based on a thinning amount. The imaging apparatus according to claim 20, wherein the resolution conversion unit is mounted on a semiconductor device. An imaging step of obtaining an image signal having a first resolution by photoelectric conversion;
A resolution conversion step of converting the image signal obtained by the processing of the imaging step into an image signal having a second resolution lower than the first resolution;
An output step for outputting an image corresponding to the image signal whose resolution has been converted at a predetermined frame rate;
The resolution conversion step detects a movement amount of at least a partial region of the image obtained in the imaging step, and performs resolution conversion of the region based on the detected movement amount.

Images