JP6516695B2 - Image processing system and image processing apparatus - Google Patents

Description

  The present invention relates to an image processing system and an image processing apparatus that perform processing relating to the data size and image quality of an image.

  An image receiving terminal that simultaneously transmits images (moving images and still images) captured by a camera via a communication network such as the Internet, an intranet, a public network, etc., such as a videophone, teleconferencing, or network surveillance camera. It is common practice to display.

  On the other hand, the number of pixels of a camera used for imaging has been increasing with the progress of technology, and now, for example, a full HD size image configured of horizontal 1920 pixels × vertical 1080 pixels is generally used. However, if an application that transmits an image transmits an image of a data size that exceeds the transmission network bandwidth limit of the communication network, the received image may deteriorate or freeze due to a communication error or data delay of the application. Sometimes. In addition, when data size is large when recording transmitted image data in a medium such as a hard disk (magnetic disk) or an optical disk and the like, the medium with limited recording capacity can only record in a short time if the data size is large. There is a problem that the cost of the recording apparatus is increased due to the need for a large capacity recording medium.

  Therefore, in a system for realizing an application for transmitting or recording an image, the data size of the image is reduced according to the transmission band and the recording capacity. For example, the number of pixels in a captured image frame is reduced It is generally carried out to reduce the data size by encoding and compressing after reducing the overall size of and reducing and displaying the reduced image when reproducing the image.

  However, there is a correlation between the encoded data size of the image and the image quality, and the image quality is degraded as the encoded data size is reduced. Therefore, development of technology aimed at reducing the encoded data size while suppressing deterioration in image quality is being carried out, and as a technology relating to image sharpening, for example, patent document 1 discloses two series of digital There is disclosed a technique for obtaining a wide band output signal exceeding the Nyquist frequency of an input signal by canceling a aliasing distortion in a one-dimensional direction using the signal. In addition, Non-Patent Document 1 combines a plurality of image frames into one frame and performs a successive approximation process with an iterative operation to restore a high frequency component exceeding the Nyquist frequency of an input image and output a high resolution. Techniques for obtaining images are disclosed.

Japanese Patent Application Laid-Open 10-280685

S. C. Park, M. K. Park, and M. G. King, "Super-Resolution Image Reconstruction: A Technical Overview," IEEE Signal Processing Magazine, vol. 20, no. 3, pp. 21-36, 2003.

  By the way, a personal computer (hereinafter abbreviated as PC), a mobile terminal or the like as a means for realizing an application for transmitting, recording, and displaying an image such as a videophone, a teleconference, or a network surveillance camera as described above. Is often used. In these terminals, software processing is performed by signal processing using a general-purpose central processing unit (CPU), micro processing unit (MPU), general purpose computing on graphics processing unit (GPGPU), digital signal processor (DSP), or the like. Image encoding, decoding, reduction, enlargement, sharpening, etc. can be realized without using special hardware.

  On the other hand, the super-resolution processing exemplified in the above-mentioned prior art has much more calculation amount required for processing than the edge emphasis processing etc. which only amplifies the high frequency component contained in the image. For example, in the super-resolution technique in which an output image is obtained by successive approximation processing using input images of a plurality of frames as described in Non-Patent Document 1, high-precision motion search in subpixel units for each pixel is performed. It is necessary to iteratively process (interroll) the interframe registration (registration) involved, the enlargement of the image, the reduction of the enlarged image, the difference detection between the reduced image and the input image, and the correction process of the output image. That is, in the prior art, since the motion search and iteration for each pixel has a very large amount of calculation, the processing limit of the computational resources such as the CPU and memory is exceeded, and a frame drop occurs where the image motion is jerky. It is possible that the entire process may stop, or the user input from the mouse or the keyboard may not be accepted and the response may not be received.

  The present invention has been made in view of the above, and provides an image processing system and an image processing apparatus capable of reducing the size of encoded data while suppressing an increase in the amount of calculation related to image processing and a decrease in image quality. The purpose is

  In order to achieve the above object, the present invention provides a first image position shift unit which shifts the position of an image to any one of a plurality of predetermined shift positions with respect to each of a plurality of temporally continuous images. An image reduction unit for reducing and reducing the number of pixels of the image shifted in position by the first image position shift unit; and an encoding unit for encoding the image reduced in the image reduction unit to generate an encoded image A decoding unit that decodes the encoded image sent through the communication network to generate a decoded image, and an image that is enlarged by increasing the number of pixels of the decoded image decoded by the decoding unit An enlargement unit, a second image position shift unit for shifting the position of the image enlarged by the image enlargement unit so as to cancel the position shift performed by the first image position shift unit, and the second image position Position shifted by shift The aliasing distortion of the image, and that a folded distortion reduction unit which performs aliasing distortion reduction processing for reducing by using information on the position shifted other images in the second image position shifting unit.

  It is possible to reduce the encoded data size while suppressing an increase in the amount of calculations and a decrease in image quality relating to image processing, and high-speed image transmission and image sharpening processing can be performed even with relatively weak computing resources.

FIG. 1 is a functional block diagram schematically showing an entire configuration of an image processing system according to a first embodiment. It is a figure showing roughly an example of image position shift processing. It is a figure which shows roughly another example of an image position shift process. FIG. 3 is a functional block diagram schematically showing an example of the configuration of an image enlargement / sharpening unit of the image reception device. It is a functional block diagram which shows the structure of an image reduction part roughly. FIG. 6 is a functional block diagram schematically showing the configuration of a horizontal processing unit and a vertical processing unit of the image reduction unit. FIG. 5 is a view schematically showing an example of a configuration for realizing the function of the image enlargement and sharpening process in the image enlargement and sharpening unit shown in FIG. 4. It is a figure which shows typically another example of the structure which implement | achieves the function of the image expansion / sharpening process in the image expansion / sharpening part shown in FIG. It is a figure which shows typically another example of the structure which implement | achieves the function of the image expansion / sharpening process in the image expansion / sharpening part shown in FIG. It is a figure which shows typically another example of the structure which implement | achieves the function of the image expansion / sharpening process in the image expansion / sharpening part shown in FIG. It is a figure which shows the structure which carried out equivalent conversion of the structure of FIG. It is a block diagram which shows an example of a structure of the one-dimensional asymmetric filter which the image expansion and sharpening part shown in FIG. 11 has. It is a figure which shows the filter factor of the one-dimensional asymmetric filter in the structure of FIG. It is a block diagram which shows an example of a structure in the two-dimensional asymmetric filter which the image expansion and sharpening part in FIG. 9 has. It is a block diagram which shows the other example of a structure of the image expansion / sharpening part in the image receiver of FIG. It is a block diagram which shows an example of a structure in the motion detection part which the image expansion and sharpening part of FIG. 15 has. It is a figure which shows the structural example in the case of performing synchronization using the flame | frame phase information generation part of an image transmission apparatus, and the flame | frame phase information acquisition part of an image receiver. It is a figure which shows the structural example in the case of performing synchronization using the flame | frame phase information generation part of an image receiver, and the flame | frame phase information acquisition part of an image transmitter. It is a figure which shows the structural example in the case of synchronizing using the flame | frame phase information estimation part of an image receiving part. FIG. 20 is an explanatory diagram showing an example of operation of frame phase information estimation processing of the frame phase information estimation unit shown in FIG. 19; It is a flowchart which shows the flame | frame phase estimation process in a flame | frame phase information estimation part. It is a figure which shows the mode of arrangement | positioning of the pixel in a Bayer array color filter. FIG. 2 is a functional block diagram schematically showing a configuration example of an image storage device. It is a figure explaining an example of the filter coefficient used in the image position shift function of FIG. It is a functional block diagram showing roughly an example of 1 composition of an image transmitting device concerning a 2nd embodiment. It is a figure which shows an example of the menu display screen which the control part which an image receiver and an image storage apparatus have generate | occur | produce and display on a display part. It is a functional block diagram showing roughly an example of 1 composition of an image transmitting device concerning a 3rd embodiment. It is a functional block diagram which shows roughly one structural example of the image receiving apparatus concerning 3rd Embodiment. It is a figure explaining an example of the filter factor used in the image position shift part of FIG. It is a flow chart which shows an example of processing in an image receiving device concerning a 4th embodiment.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings.

First Embodiment
A first embodiment of the present invention will be described with reference to FIGS.

First, an outline of image sharpening realized by the present embodiment will be described.
When sampling an image signal, frequency components higher than the Nyquist frequency which are uniquely determined according to the number of pixels constituting one frame of an image become aliasing distortion. In an image processing system (see FIG. 1 and the like described later), a cutoff of a low-pass filter (see FIG. 5 and the like later) when reducing an image in the image reduction unit (104) of the image transmission apparatus (101) If the frequency is set lower than the Nyquist frequency, an effect of reducing aliasing distortion can be expected, but on the other hand, the image after reduction tends to be a blurred image because high frequency components are attenuated. On the other hand, when the cutoff frequency of the low pass filter is set higher than the Nyquist frequency, in the image after reduction, the aliasing distortion component is mixed with the baseband component originally contained in the image before reduction. Since the aliasing distortion looks like moire (interference fringe) noise, the image quality may be greatly degraded.

  Therefore, in the present embodiment, in the image enlargement / sharpening unit (111) of the image reception device (109), when enlarging the transmission image after reduction, aliasing distortion is generated using a plurality of image frames. By reducing the noise, a wide band baseband component is extracted to make the image clearer. In order to reduce the aliasing distortion, if the sampling phase is changed with respect to the same input signal and resampling is performed, the property that the phase of the aliasing distortion generated is changed is used. This property is used in interlaced scanning and the like. For example, in a typical 2: 1 interlace, when imaging, an image is scanned by skipping from the top every scanning line, and a first field consisting of an image of odd-numbered scanning lines and an even-numbered image In the second field of At the time of display, it is converted from interlaced scan to progressive scan by signal processing, or by utilizing the fact that the first field and the second field are synthesized and perceived in the brain by the afterimage effect of the human eye. , Get a clear frame image.

  In addition, since it is possible to convert only the number of pixels of "divisor" of the number of imaged pixels, that is, "1 / N" (where N is a positive integer) only by the concept of interlaced scanning, for example, As in the case of conversion from full HD size (1920 horizontal pixels × 1080 vertical pixels) to D1 size (horizontal 704 pixels × vertical 480 pixels), reduction to “non-divisor” pixel count can not be performed.

  Therefore, in the present embodiment, first, the image position shift unit (103) shifts the position of the image for each frame, and then the image reduction unit (104) reduces the image to obtain a reduced image (a transmission image The phase of the aliasing distortion included in) is intentionally changed. At this time, if the subject is stationary, the phase of aliasing distortion can be uniquely determined from the method of image shift (shift direction and shift amount). For example, when the position of an image in full HD size (horizontal 1920 pixels × vertical 1080 pixels) is shifted by 1 pixel in the horizontal direction, 0 in the horizontal direction in an image of D1 size (horizontal 704 pixels × vertical 480 pixels) after image reduction. Since the .37 (= 704/1920) pixel shift occurs, the phase of the aliasing distortion rotates by 11π / 15 (= 2π × 704/1920) radians. The amount of phase rotation can be determined constant and in advance in the entire frame, thereby eliminating the need for the motion search in units of sub-pixels described above. Thereafter, the aliasing distortion is reduced based on the operation principle to be described later, so that the image can be sharpened even when the number of pixels is reduced to a non-divided number.

  Details of the present embodiment based on the above findings are described below.

  FIG. 1 is a functional block diagram schematically showing an entire configuration of an image processing system according to the present embodiment.

  In FIG. 1, the image processing system includes an image transmission apparatus (101), a communication network (107), an image storage apparatus (108), an image reception apparatus (109), an operation unit (114), an image reception apparatus (109), and It is roughly comprised by the display part (113).

  The image transmitting apparatus (101) comprises: a camera (102) for picking up an image (still image, moving image); an image position shifting part (103) for carrying out image position shifting processing on the image picked up by the camera (102); An image reduction unit (104) that performs image reduction processing on an image subjected to shift processing, an encoding unit (105) that performs encoding processing on an image that has undergone image reduction processing, and an image via a communication network (107) An image transmission apparatus (101) including a camera (102), an image position shift unit (103), an image reduction unit (104), and an encoding unit (105) based on commands and data transmitted from the reception apparatus (109). ) And a control unit (106) for controlling the overall operation. Also, the image receiving apparatus (109) performs the decoding process on the image transmitted from the image transmitting apparatus (101) via the communication network (107) and the decoding unit (110). It has an image enlargement / sharpening unit (111) that performs image enlargement / sharpening processing on an image, and a control unit (112) that controls the overall operation of the image receiving apparatus (109).

  In the image processing system according to the present embodiment, the transmission data size is reduced by reducing the number of pixels constituting one frame of the image in the image reduction unit (104) of the image transmission apparatus (101). By enlarging the image in the image enlargement / sharpening unit (111) of (109), it is possible to display a clear image with less blur. The image having the third number of pixels output from the image receiving apparatus (109) is further enlarged or reduced using an image enlarging unit or an image reducing unit (not shown) to obtain an image having a fourth number of pixels (not shown). After conversion, the display unit (113) may be configured to display.

  The camera (102) may be, for example, an imaging unit including a photoelectric conversion element such as a charge coupled device (CCD) or a complementary metal-oxide semiconductor (CMOS) (not shown) and a lens, signal level adjustment, contrast adjustment, brightness adjustment, white balance An image (still image, moving image) captured by the camera (102) is sent to the image position shift unit (103) in the subsequent stage. In the following description, it is assumed that the number of pixels of an image is represented by a combination of the number of pixels in the horizontal direction (the number of horizontal pixels) and the number of pixels in the vertical direction (the number of vertical pixels). The frame is described below as being composed of a first number of pixels.

  The image position shift unit (103) performs image position shift processing to shift the position of the image obtained by the camera (102) up, down, left, and right in units of integer pixels. Here, the image position shift processing in the image position shift unit (103) will be described with reference to FIG. 2 and FIG.

  FIG. 2 is a diagram schematically showing an example of the image position shift process.

  In FIG. 2, the image in which three persons are subjects is shown in a simulated manner, the broken line shifts the original image frame (200) taken by the camera (102), and the solid line shifts by the image position shift processing. The illustrated image frames (201) to (204) are shown. The arrows indicate state transitions for each frame time (for example, 1/30 seconds). In FIG. 2, image position shift processing is performed to shift the image in a four-frame cycle, which is referred to as “four-frame type image position shift”.

  In the state of the image frame (201) after the image position shift processing, the position of the original image frame (200) is shifted to the upper left, and predetermined pixel values at the right end and the lower end of the image (for example, pixel value = 0 representing black) ) Is inserted. Similarly, in the state of the image frame (202), the position of the original image frame (200) is shifted to the upper right, and black (pixel value = 0) is inserted at the left end and the lower end of the image. In the state, the position of the original image frame (200) is shifted to the lower right and black (pixel value = 0) is inserted at the left end and the upper end of the image, and in the state of the image frame (204), the original image frame ( The position 200) is shifted to the lower left, and black (pixel value = 0) is inserted at the right end and the upper end of the image. That is, in the four-frame type image position shift, the image position shift process is performed by causing the state transition of the image frames (201) to (204) in order of one frame time. The shift direction and shift amount in the image position shift unit (103) are controlled by the control unit (106).

  FIG. 3 is a diagram schematically showing another example of the image position shift process.

  In FIG. 3 as in FIG. 2, the broken lines indicate the original image frame (200) captured by the camera (102), and the solid lines indicate the image frames (205) and (205) shifted by the image position shift process. ing. The arrows indicate state transitions for each frame time (for example, 1/30 seconds). In FIG. 3, image position shift processing for shifting an image in two frame cycles is performed, which is referred to as “two frame type image position shift”.

  In the state of the image frame (205) after the image position shift processing, the position of the original image frame (200) is shifted to the left and black (pixel value = 0) is inserted at the right end of the image. Similarly, in the state of the image frame (206), the position of the original image frame (200) is shifted to the right to insert black (pixel value = 0) at the left end of the image. That is, in the two-frame type image position shift, the image position shift processing is performed by causing the state transition of the image frames (205) and (206) in order of one frame time.

  The image position shift processing is not limited to the examples shown in FIGS. 2 and 3. For example, in the four-frame position shift shown in FIG. 2, the sequence of “image frame (201) → image frame (204) → image frame (203) → image frame (202) → upper left (201) →. Position shift, position shift in the order of "image frame (201) → image frame (202) → image frame (204) → image frame (203) → image frame (201) → ...", or "image frame (201)-> image frame (204)-> image frame (202)-> image frame (203)-> image frame (201)-> ... "position shift or position shift in a random (random) order You may Further, for example, with the position of the image frame (200) as one state of position shift, "image frame (200) → shift to the right position shift → shift to the lower right position shift → shift downward to the position → image frame (200) → Position shift may be performed as "."

  Similarly, for example, in the two-frame position shift shown in FIG. 3, in the state of the image frame (205), the position shift is not performed (that is, the position is fixed as the image frame (200)). (200) → image frame (206) → image frame (200) →...

  In addition, image position shift processing including position shift in the vertical direction or oblique direction (not shown) may be performed.

  Furthermore, the present invention is not limited to one that performs position shift in a four-frame cycle, such as four-frame type position shift, or one that performs position shift in a two-frame cycle, such as two-frame type position shift. Position shift, “6 frame period” position shift combining 3 frame position shift in the horizontal direction and 2 frame position shift in the vertical direction, or position shift of 3 frames in the horizontal direction Position shift of “9 frame period” combining the position shift of three frames in the vertical direction and the like may be performed.

  In the image enlargement and clear processing in the image enlargement and sharpening unit (111) described in detail later, the state of the image clear changes according to the direction of the position shift by the image position shift processing. For example, when the “four-frame type position shift” shown in FIG. 2 is performed, the resolution (sharpness) in the horizontal direction and the vertical direction of the image after the image enlargement / sharpening process is improved, as shown in FIG. When the "two-frame position shift" is performed, only the resolution (sharpness) in the horizontal direction is improved. In other words, resolution in a desired direction can be improved by performing an image position shift process that shifts the image frame in the direction in which the resolution is to be improved.

  FIG. 5 is a functional block diagram schematically showing the configuration of the image reduction unit. FIG. 6 is a functional block diagram schematically showing the configuration of the horizontal processing unit and the vertical processing unit of the image reduction unit.

  In FIG. 5, an image reduction unit (104) performs an image reduction process to convert the number of pixels constituting one frame of an image into a second number of pixels smaller than the first number of pixels. This configuration is also called an interpolation filter, and performs horizontal image reduction processing (401-H) that performs horizontal image reduction processing on the image input from the image position shift unit (103) and vertical image reduction processing. And a vertical processing unit (401-V). The order of processing for the input image may be switched between the horizontal processing unit (401-H) and the vertical processing unit (401-V).

  In FIG. 6, the horizontal processing unit (401-H) and the vertical processing unit (401-V) can be realized with the same configuration, and a pixel insertion unit (402) performing m-fold upsampling and removing high frequency components or It has a low pass filter (403) to reduce and a pixel thinning part (404) which performs 1 / n down sampling.

  In the horizontal processing unit (401-H) and the vertical processing unit (401-V), after performing m-fold upsampling (zero insertion) on the input image by the pixel insertion unit (402), a low-pass filter (403 By removing or reducing unnecessary high frequency components and performing 1 / n downsampling by the pixel thinning unit (404) to multiply the number of pixels in the horizontal direction (or vertical direction) of the image by m / n (but m , N can be increased or decreased to a positive integer). Note that the image enlargement processing by the image enlargement unit (301) described later can be realized with the same configuration by setting the constants m and n of the image reduction unit 104.

  That is, for example, when the full HD size (horizontal 1920 pixels × vertical 1080 pixels) and the D1 size (horizontal 704 pixels × vertical 480 pixels) are considered as the number of pixels constituting an image frame, m = 11 and n = 30 If settings are used, horizontal reduction (1920 pixels to 704 pixels) can be realized in the image reduction unit (104). If m = 2 and n = 1 are set, horizontal conditions in the image enlargement unit (301) can be obtained. Directional enlargement (704 pixels → 1408 pixels) can be realized. The same applies to vertical reduction and enlargement. The number of output pixels (second number of pixels) in the image reduction unit (104) is controlled by the control unit 106 by setting m and n.

  In the low-pass filter (403), a window function (such as a Hanning window) is multiplied by a sinc function (= sin (x) / x) obtained by inverse Fourier transform of an ideal frequency characteristic with the Nyquist frequency as a cutoff frequency. Use the filter coefficients. When the image position shift with sub-pixel accuracy is performed, the center position of the sinc function is moved by t to realize sin (x−t) / (x−t). In addition, on the premise of performing processing to reduce aliasing distortion in the processing at the post stage, it is also considered to set the cutoff frequency of each of the horizontal and vertical low-pass filters (403) to a frequency higher than the Nyquist frequency. Be The operation equivalent to the configuration shown in FIG. 6 can be realized by a polyphase filter (polyphase filter).

  The encoding unit (105) encodes and compresses the image input from the image reduction unit (104), and transmits the image to the image reception device (109) or the image storage device (108) via the communication network 107. is there. The processing configuration for connecting the image transmitting apparatus (101) or the image receiving apparatus (109) and the communication network (107) is not shown as it uses a general technology. Also, the decoding unit 110 of the image reception device (109) performs a decoding process according to the coding method used by the coding unit (105).

  As an encoding method used by the encoding unit 105, for example, MPEG (Moving Picture Expert Group) -1, MPEG-2, MPEG-4, H.264, and so on. H.264. Standardized encoding such as H.265, VC-1, Joint Photographic Experts Group (JPEG), Motion JPEG, JPEG-2000 may be performed, or non-standard encoding may be performed. The coding method in the coding unit (105) is controlled by the control unit (106).

  The communication network (107) may be wired or wireless, and is a network for communicating digital data using a general communication protocol such as IP (Internet Protocol).

  FIG. 4 is a functional block diagram schematically showing an example of the configuration of the image enlargement / sharpening unit of the image reception apparatus.

  In FIG. 4, an image enlargement / sharpening unit (111) performs an image enlargement process on an image having the second number of pixels decoded by the decoding unit (110), and an image enlargement unit. It has an image position shift unit (302) that performs image position shift processing on the processed image, and a aliasing distortion reduction unit (303) that performs aliasing reduction processing on the image that has undergone the image position shift processing.

  In each of the following signal processing, the same processing may be performed on all color image signals (RGB (red, green, blue), YUV (brightness Y, color difference UV), etc.). In addition, the following signal processing may be performed only on the luminance signal Y, and may not be performed on the color difference signal (UV), and only the image enlargement processing may be performed.

  The image enlargement unit (301) is an image having a third number of pixels, which is an image having a second number of pixels (for example, D1 size (horizontal 704 pixels × vertical 480 pixels)) input from the decoding unit (110). The image enlargement processing for converting the image data into the image data can be realized by the same configuration as the image reduction unit (104) as described above. The purpose of the image enlargement processing is to increase the number of pixels of the image (that is, to increase the sampling frequency) in order to prevent the aliasing distortion reduced in the subsequent processing block from folding back again in the frequency domain. The number of pixels of may be a number of pixels larger than the second number of pixels. Therefore, for simplicity of description, the third pixel number will be described as twice the second pixel number (ie, 1408 horizontal pixels × 960 vertical pixels).

  The image position shift unit (302) performs the image enlargement processing in the image enlargement unit (301) in the direction opposite to the image position shift processing by the image position shift unit (103) of the image transmission apparatus (101). Image shift between frames is suppressed by shifting the position (that is, shifting the position of the image so as to cancel the position shift performed by the image position shift unit (103)). For example, when the image position shift unit (103) performs an image position shift process to shift one pixel leftward and one pixel upward, the image position shift unit (302) shifts 0.x rightward. If an image position shift process is performed to shift 73 (= 1408/1920) pixels downward by 0.89 (= 960/1080) pixels, the image returns to the original position, and the image blur between frames Is reduced. In image position shift processing in the image position shift unit (302), image shift may be performed so as to cancel the image position shift processing in the image position shift unit (103). The image position shift unit (ie, all the images are shifted to the positions of the centers of the “frame-type image position shift” and “two-frame type image position shift” shown in FIG. The shift direction and shift amount in 302) may be determined. Also, even if all the images are shifted to fixed positions that are not the centers in the image position shift processing of the image position shift unit (103), the shift direction and shift amount in the image position shift unit (302) are determined. Good. The shift direction and shift amount of the image position shift unit (302) are controlled by the control unit (112).

  The aliasing distortion reduction unit (303) performs aliasing distortion reduction processing to reduce aliasing distortion of the image on which the image position shift processing has been performed in the image position shift unit (302), and an output image of the image enlargement / sharpening unit (111) It is said that. The details of the aliasing reduction process will be described in detail later.

  FIG. 7 is a view schematically showing an example of a configuration for realizing the function of the image enlargement and sharpening process in the image enlargement and sharpening unit shown in FIG.

  In FIG. 7, an image enlargement / sharpening unit (501) obtains a wide band output signal exceeding the Nyquist frequency of an input signal by canceling aliasing distortion in a one-dimensional direction using two series of digital signals. One-dimensional enlargement units (502- # 0), (502- # 1), one-dimensional interpolation filters (503- # 0), (503- # 1), Hilbert transformer (506), etc. It is composed of

  In the image enlargement / sharpening unit (501), two series of digital signals (i.e., the input # 0 and the input # 1) are 1 by the respective one-dimensional enlargement units (502- # 0) and (502- # 1). After being expanded in the dimensional direction (horizontal direction or vertical direction), each one-dimensional interpolation filter (503- # 0), (503- # 1) uses the respective positional shift amounts (α0, α1) to The image frame is aligned, the output signal of the adder (504), and the image signal after being multiplied by the coefficient K by the multiplier (507) after passing through the subtractor (505) and the Hilbert transformer (506) After converting the phases of aliasing distortions contained in the two into 180 degrees (opposite phase) with each other, and adding the both by an adder (508) to cancel aliasing distortion in the image. Can be cell.

  Here, the image enlargement unit (301) shown in FIG. 4 corresponds to the one-dimensional enlargement units (502- # 1) and (502- # 2) in FIG. 7, and the image position shift unit (302) It corresponds to the dimensional interpolation filters (503- # 0) and (503- # 1). The image enlargement unit (301) shown in FIG. 4 is equivalent to the interpolation filter (401) as described above, and the sinc function used to determine the coefficients of the low pass filter (403) in it is sin (x−t) ) / (X−t), by combining the image position shift unit (302) with the one-dimensional interpolation filters (503- # 0) and (503- # 1), the one-dimensional enlargement unit (502- #) 1) and (502- # 2) can be omitted. Also, the aliasing distortion reduction unit (303) in FIG. 4 is realized by the adder (504), the subtractor (505), the Hilbert transformer (506), the multiplier (507), and the adder (508) in FIG. Thus, the image enlargement / sharpening unit (111) in FIG. 4 and the image enlargement / sharpening unit (501) in FIG. 7 can be regarded as equivalent.

  The configuration of the image enlargement / sharpening unit (501) shown in FIG. 7 exemplifies the case of performing the image enlargement processing and aliasing reduction processing in the one-dimensional direction, and two-dimensional in the horizontal direction and the vertical direction. The case of performing the image enlargement process and the aliasing reduction process of the direction will be described below with reference to FIG.

  FIG. 8 is a view schematically showing another example of the configuration for realizing the function of the image enlargement and sharpening process in the image enlargement and sharpening unit shown in FIG. 4.

  In FIG. 8, processing blocks (501-HA), (501-HB), and (501-V) having the same configuration as the image enlargement / sharpening unit (501) shown in FIG. 7 are used. The outputs of (501-HA) and (501-HB) are connected in series as the input of the processing block (501-V). The horizontal processing unit (601) composed of processing blocks (501-HA) and (501-HB) performs image enlargement processing and aliasing reduction processing in the horizontal direction, and vertical processing composed of processing blocks (501-V) A part (602) performs image enlargement processing and aliasing reduction processing in the vertical direction. That is, by configuring the image enlargement / sharpening unit (111) in this way, two-dimensional image enlargement processing and aliasing reduction processing can be performed using signals of four systems (that is, inputs # 0 to # 3). It can be realized. The same processing can be realized even if the processing order of the horizontal processing unit (601) and the vertical processing unit (602) is replaced with each other.

  Here, the image position shift shown in FIG. 2 and FIG. 3, the position shift amount (.alpha.0, .alpha.1) shown in FIG. 7, the horizontal position shift amount (.alpha.0, .alpha.1, .alpha.2, .alpha.3) shown in FIG. The relationship with the vertical position shift amount (β0, β1) will be described. In the “four frame type position shift” shown in FIG. 2, the image having the first number of pixels (for example, horizontal 1920 pixels × vertical 1080 pixels) shown in FIG. Two-dimensional coordinates (−h, −v) → (h, −v) → (h, v) → (−h, v) → using the two-dimensional coordinates of the position of (200) as a reference (0, 0) It is assumed that the position is shifted as (−h, −v) →... (Where h = 2 m, v = 2 n and m and n are positive integers). Similarly, in the “two-frame type position shift” shown in FIG. 3, two-dimensional coordinates (−, 0) are taken with the two-dimensional coordinates of the central position of horizontal reciprocation (position of image frame (200)) In the following description, it is assumed that h, 0) → (h, 0) → (−h, 0) →...

  First, the case of “four-frame type position shift” shown in FIG. 2 will be described. In the case of corresponding to the “four frame type position shift”, the two-dimensional processing block shown in FIG. 8 is used. As described above, after an image having a first number of pixels (for example, 1920 horizontal pixels × vertical 1080 pixels) is reduced to a second number of pixels (for example, horizontal 704 pixels × vertical 480 pixels) and transmitted. When the aliasing distortion reduction process is performed by expanding to a third pixel number (for example, 1408 horizontal pixels × 960 vertical pixels), (-h, -v) → (h,) in an image of the first pixel number The position shift of −v) → (h, v) → (−h, v) → (−h, −v) →... Is the image of the third pixel number, p = 1408/1920, q = As 960/1080, it corresponds to a position shift of (-ph, -qv)-> (ph, -qv)-> (ph, qv)-(-ph, qv)-(-ph, -qv)-... . Therefore, by setting the horizontal position shift amount (α0, α1, α2, α3) = (ph, −ph, −ph, ph) and the vertical position shift amount (β0, β1) = (qv, −qv) The image position shift processing in the image position shift unit (103) on the image transmission apparatus (101) side and the image position shift processing in the image position shift unit (302) on the image reception apparatus (109) side And can stop image blurring.

  Therefore, in the horizontal processing unit (601) of FIG. 8, the coefficient Kα = −1 / tan (π (α0−α1)) = − 1 / tan (2πph) = − 1 / tan (2πh × 1408/1920) = − By setting 1 / tan (πh × 22/15), aliasing distortion in the horizontal direction can be reduced. Further, in the vertical processing unit (602) of FIG. 8, the coefficient Kβ = −1 / tan (π (β0−β1)) = − 1 / tan (2πqv) = − 1 / tan (2πv × 960/1080) = − By setting 1 / tan (πv × 16/9), the aliasing distortion in the vertical direction can be reduced.

  Next, the case of “2-frame position shift” shown in FIG. 3 will be described. When this "two-frame type position shift" is supported, image enlargement processing and aliasing reduction processing in the horizontal direction are performed using the one-dimensional processing block shown in FIG. 7 as horizontal processing. On the other hand, for vertical processing, only image enlargement in the vertical direction is performed using the interpolation filter shown in FIG. 6, and horizontal processing and vertical processing are serially connected to perform two-dimensional processing. As described above, an image having a first number of pixels (for example, 1920 horizontal pixels × vertical 1080 pixels) is reduced to a second number of pixels (for example, horizontal 704 pixels × vertical 480 pixels) to form a communication network (107). And (-h) in the image of the first number of pixels when transmission distortion reduction processing is performed after enlargement to a third number of pixels (for example, 1408 horizontal pixels × vertical 960 pixels) after transmission via , 0) → (h, 0) → (−h, 0) →... In the image with the third number of pixels, the position shift is (−ph, 0) → (ph) as p = 1408/1920. , 0) → (−ph, 0) →... Therefore, if the horizontal position shift amount (α0, α1) = (ph, −ph) is set, the image position shift processing by the image position shift unit (103) on the image transmission device (101) side and the image receiving device (109) The effect of the image position shift processing by the image position shift unit (302) on the side can be canceled and canceled, and image blurring can be stopped.

  Therefore, in the image enlargement / sharpening unit (501) of FIG. 7, the coefficient K = −1 / tan (π (α0−α1)) = − 1 / tan (2πph) = − 1 / tan (2πh × 1408/1920) If it is set as =) /-1 / tan (πh × 22/15), aliasing distortion in the horizontal direction can be reduced.

  Even when the number of pixels is reduced to a non-divisible number by the above-described image reduction processing, aliasing distortion included in the image can be reduced, and the image can be sharpened without performing iteration.

  Here, in the configuration of FIG. 8 described above, since the amount of operation is about three times that of the configuration of FIG. Therefore, a configuration that can realize the equivalent operation while reducing the amount of calculation of the configuration shown in FIG. 8 will be described using FIGS. 9 and 10.

  FIGS. 9 and 10 are diagrams schematically showing another example of the configuration for realizing the function of the image enlargement / sharpening processing in the image enlargement / sharpening unit shown in FIG.

  In the configuration of FIG. 7 described above, focusing on the fact that the Hilbert transform is an odd-symmetric filter with fixed coefficients and that the coefficients Kα and Kβ are constant, “adder (504), subtractor (505) , The Hilbert transformer (506), the multiplier (507), and the adder (508) are all linear operations composed of fixed coefficients, and it can be understood that they can be represented as one "one-dimensional asymmetric filter" (described later) . That is, in the configuration of FIG. 7, after passing the one-dimensional interpolation filter (503-# 0) to the input # 0 and then passing the "one-dimensional asymmetric filter" and one-dimensional interpolation to the input # 1. Even if the signal passes through the "one-dimensional asymmetric filter" in which the polarity (positive or negative) of the subtractor (505) is inverted after passing through the filter (503- # 1), the signals are collectively added finally. , The same output will be obtained.

  Therefore, in FIG. 9, the order of “one-dimensional asymmetric filter” in the horizontal processing unit (601) of FIG. 8 and “one-dimensional enlargement unit and one-dimensional interpolation filter” in the vertical processing unit (602) Each vertical one-dimensional enlargement unit is collectively referred to as a two-dimensional enlargement unit (702), each horizontal and vertical one-dimensional interpolation filter is collectively referred to as a two-dimensional interpolation filter (703), and each horizontal and vertical one-dimensional asymmetric filter is A two-dimensional processing unit (701) is configured as a two-dimensional asymmetric filter (704) collectively.

  In this two-dimensional processing unit (701), two-dimensional interpolation is performed according to the horizontal position shift amount (α0, α1, α2, α3) and vertical position shift amount (β0, β1) corresponding to the inputs # 0 to # 3. The respective coefficients of the filter (703) and the two-dimensional asymmetric filter (704) are set to form respective two-dimensional processing units (701-# 0 to # 3).

  Subsequently, each signal passed through each two-dimensional processing unit (701- # 0 to # 3) is passed through each frame memory (705- # 0 to # 3), and then all signals are added by the adder (706). By adding, an output equivalent to the configuration of FIG. 8 can be obtained.

  As described above, in the configuration shown in FIG. 8, the output can not be obtained unless all operations of the horizontal processing unit (601) and the vertical processing unit (602) are performed using the inputs # 0 to # 3. In the configuration of FIG. 9, the processing for the inputs # 0 to # 3 is calculated independently of the other inputs by each two-dimensional processing unit (701-# 0 to # 3). Since each input # 0 to # 3 is input in the order of input # 0, input # 1, input # 2, input # 3, input # 0,... , The calculation result may be stored in the frame memory (705- # 0 to # 3). Furthermore, since each two-dimensional processing unit (701- # 0 to # 3) is not necessary for an image frame that is not input, only one two-dimensional processing unit (701) is installed as shown in FIG. It is configured to change the internal coefficient according to the frame number according to the amount (α0, α1, α2, α3) and the vertical position shift amount (β0, β1), and using the switch (707) The frame memory (705- # 0 to # 3) may be written while being selected for each frame.

  Therefore, by configuring the image enlargement / sharpening unit (111) as shown in FIG. 10, the amount of calculation in the configuration of FIG. 8 is about 1/6 (= 2 (horizontal / vertical) / 3) (horizontal). The amount of computation can be reduced to (horizontal / vertical) × 1/4 (ratio of the number of input frames to be processed)).

  Similarly, the configuration of the image enlargement / sharpening unit (111) in the case of “two-frame type position shift” can equivalently convert the configuration of FIG. 7 into the configuration of FIG. 11. That is, the one-dimensional enlargement units (502- # 0) and (502- # 1) in FIG. 7 are combined into the one-dimensional enlargement unit (801) in FIG. 11, and the one-dimensional interpolation filter (503- # 0) in FIG. (503- # 1) are combined into a one-dimensional interpolation filter (802) in FIG. 11, and the “adder (504), subtractor (505), Hilbert transformer (506), multiplier (507), and "Adder (508)" is integrated into a one-dimensional asymmetric filter (803) in FIG. 11, and is set to change the internal coefficient according to the frame number in accordance with the horizontal position shift amount (.alpha.0, .alpha.1) Using the switch (804) to select and write the frame memories (805- # 0) and (805- # 1) for each frame, and the adder (806) By adding, an output equivalent to the configuration of FIG. 7 can be obtained.

  That is, by configuring the image enlargement / sharpening unit (111) as shown in FIG. 11, it is possible to reduce the amount of computation for one approximately one-dimensional interpolation filter compared to the amount of computation in the configuration of FIG.

  12 to 14 are a block diagram and an operation explanatory view showing an example of the configuration of the asymmetric filter provided in the image enlargement / sharpening unit in FIGS. 5 and 11.

  FIG. 12 is a block diagram showing an example of the configuration of a one-dimensional asymmetric filter included in the image enlargement / sharpening unit of FIG.

  In FIG. 7 described above, after passing through the one-dimensional enlargement unit (502- # 0) and the one-dimensional interpolation filter (503- # 0), the input # 0 "signal passed through the adder (504)" And “a signal that has passed through the subtractor (505), Hilbert transformer (506), and multiplier (507)” are added by the adder (508) to be an output. At this time, if the input # 1 = 0 (no signal), for the input # 0, both the adder (504) and the subtractor (505) have a factor of 1 (that is, no change in the signal). I understand. Therefore, as shown in FIG. 12, the filter coefficient of the one-dimensional asymmetric filter (704) for the input # 0 is the signal multiplied by the coefficient K through the Hilbert transformer (506) and the input signal by the adder (508). The added signal is output.

  On the other hand, when input # 0 = 0 (no signal), for input # 1, the coefficient of the adder (504) is 1, and the coefficient of the subtractor (505) is -1. . Therefore, the filter coefficient of the one-dimensional asymmetric filter (704) with respect to the input # 1 is a value obtained by inverting the polarity (positive or negative) of the coefficient K as compared with the coefficient of the filter coefficient of the one-dimensional asymmetric filter (704) with respect to the input # 0. It becomes.

  The Hilbert transformer (506) is an odd symmetric filter, and the coefficient C (t) = 0 when t = 2 m (where m is an integer) and the coefficient when t = 2 m + 1 (where m is an integer) C (t) = 2 / (πt). Therefore, in the configuration of FIG. 12, the filter coefficients of the one-dimensional asymmetric filter (704) become “asymmetric” operations that are neither even symmetric nor odd symmetric as shown in FIG. The filter coefficient (C (t)) shown in FIG. 13 is an example, and for example, the coefficient of Hilbert transform is multiplied by a general window function (such as a Hanning window) to obtain a coefficient C (t) (where t ≠ 0). ) May reduce the influence of the filter end.

  FIG. 14 is a block diagram showing an example of a two-dimensional asymmetric filter included in the image enlargement / sharpening unit shown in FIG.

  The two-dimensional asymmetric filter (704) is formed by connecting the above-described one-dimensional asymmetric filter (803) in series, and converts the number of pixels in the horizontal direction by the horizontal processing unit (803-H). A two-dimensional pixel number conversion can be realized by converting the number of pixels in the vertical direction by 803-V).

  As described above, by performing independent signal processing for each input image frame using the asymmetric filter, it is possible to realize the reduction of the amount of calculation in the aliasing reduction.

  Next, processing corresponding to an image including motion in a subject will be described.

  By eliminating the need for motion search and iteration for each pixel, the above-described configuration of the image enlargement / sharpening unit (111) reduces the amount of computation and realizes image sharpening that enables high-speed processing. Among them, in order to make the motion search for each pixel unnecessary, the position shift amounts (α n, β n) used in the configurations of FIGS. 7 to 11 are set to fixed values for each frame. At this time, there is no problem if the subject in the image frame does not move, but if there is movement in the subject, there is a problem that multiple images are generated due to the calculation between the frames. Therefore, a configuration example for solving such a problem will be described below.

  FIG. 15 is a block diagram showing another example of the configuration of the image enlargement / sharpening unit in the image reception device of FIG.

  In FIG. 15, a processing unit (1001) including an image enlargement unit (301), an image position shift unit (302), and a aliasing reduction unit (303) is the same as the configuration shown in FIG. The signal of which image blurring between frames is suppressed by the image position shift unit (302) is passed through a low pass filter (1002) for suppressing unnecessary aliasing, and pixels of the image are detected by a motion detection unit (1003) described later. Control signal (m, but 0 (no motion) m m m 1 (with motion)) according to the magnitude of each motion, and using a weighted mixing unit (1004) to generate a low-pass filter (1002) The signal (p1) passed through and the signal (p0) passed through the aliasing reduction section (303) are weighted and mixed according to the control signal m to form an output signal. Here, the weighted mixing unit (1004) includes a subtractor (1005) that generates a control signal (1-m) from the control signal m, multipliers (1006) and (1007), and an adder (1008), and the pixel The processing of “output = p1 × m + p0 × (1-m)” is performed every time. That is, in the area where the value of control signal m is close to 0 (area close to stationary), the signal (p0) passed through the aliasing distortion reduction unit (303) is output, and the area where the value of control signal m is close to 1 (movement is In the large area), the signal (p1) passed through the low pass filter (1002) is controlled to be output.

  FIG. 16 is a block diagram showing an example of a configuration of a motion detection unit included in the image enlargement / sharpening unit of FIG.

  In the case of the “four-frame position shift type” described above in the motion detection unit (1003), images of four consecutive frames input from the low pass filter (1002) are frame numbers (# 0, # 0) by the switch (1101). 1), # 2, # 3, # 0,...), And the data is stored in the frame memory (1102). Subsequently, the image read from each frame memory (1102) is averaged for each pixel using an averaging unit (1103), and an average image and each frame are calculated using a subtractor (1104) and an absolute value unit (1105). After calculating the absolute value of the pixel-by-pixel difference, the maximum value unit (1106) obtains these maximum values for each pixel, and the normalization unit (1107) fits within the range of “0 ≦ m ≦ 1”. Thus, after normalizing the value of m, the control signal is m. Note that this normalization can be realized by a general technique of subtracting or multiplying the output signal of the maximum value unit (1106) by a predetermined fixed value, so detailed illustration and description will be omitted.

  An area in which all the values of the four frame images (# 0, # 1, # 2, # 3) input to the motion detection unit (1003) by the motion detection unit (1003) configured in this way match (ie, In the area where the image is stationary, the value of the control signal m is 0, and the area where the values of the four frame images (# 0, # 1, # 2, # 3) do not match (that is, one of four frames In the region where the image is moving above the frame), the control signal m has a value of 0 <m ≦ 1 according to the magnitude of the absolute value of the difference signal described above.

  In the case of the "two-frame position shift type" described above, the switch (1101), the frame memory (1102), the averaging unit (1103), and the subtractor in the configuration of the motion detection unit (1003) shown in FIG. (1104) and the absolute value unit (1105) can be easily realized only by changing from 4 frames to 2 frames, so the description by the drawings is omitted.

  With the configuration described above, even when the subject in the image has a motion, it is possible to obtain a high quality image in the static region of the image while suppressing the occurrence of the multiple image described above in the moving region of the image. For example, if this configuration is applied to a monitoring camera, a parked vehicle becomes a stationary area of an image, so that a license plate or the like can be viewed with a fine image.

  Next, frame synchronization between the image transmitting apparatus and the image receiving apparatus will be described.

  17 to 19 are block diagrams respectively showing an example of the configuration of the control unit of the image transmitting apparatus and the image receiving apparatus of FIG.

  The image position shift unit (103) of the image transmission apparatus (101) and the image position shift unit (302) of the image reception apparatus (109) operate in exact synchronization with each other in frame units, and are reverse to each other. It is necessary to control to position shift. In order to obtain frame phase information used for this synchronization, any one of the configurations shown in FIGS. 17 to 19 is used. The frame phase information is, for example, one of the frame numbers “# 0, # 1, # 2, # 3” in the case of “four-frame type position shift”, and “two-frame type position shift”. In the case, it is one of the frame numbers "# 0, # 1".

  FIG. 17 is a diagram showing a configuration example in the case of performing synchronization using a frame phase information generation unit of the image transmission apparatus and a frame phase information acquisition unit of the image reception unit.

  In FIG. 17, the frame phase information generation unit (1201) of the control unit (106) of the image transmission apparatus (101) generates frame phase information, and using the multiplexing unit (1202), the encoding unit (105) Multiplexed with the image data output from. After this multiplexed data is transmitted to the image receiving apparatus (109) via the communication network (107) and separated by the separating unit (1203), the image data is sent to the decoding unit (110), and frame phase information is obtained. Are input to the control unit (112). Thus, the frame phase information acquisition unit (1204) of the control unit (112) can obtain the same information as the frame phase information generated by the frame phase information generation unit (1201) of the image transmission apparatus (101). Note that the frame phase information generation unit (1201), the multiplexing unit (1202), the separation unit (1203), and the frame phase information acquisition unit (1204) can be realized by general techniques, and thus detailed illustration is omitted.

  FIG. 18 is a diagram showing a configuration example in the case of performing synchronization using the frame phase information generation unit of the image reception unit and the frame phase information acquisition unit of the image transmission device.

  In FIG. 18, the frame phase information generation unit (1206) of the control unit (112) of the image reception device (109) generates frame phase information, and the image transmission device (101) is generated via the communication network (107). Transmit Thus, the frame phase information acquisition unit (1205) of the control unit (106) of the image transmission apparatus (101) is identical to the frame phase information generated by the frame phase information generation unit (1206) of the image reception apparatus (109). You can get information on The frame phase information generation unit (1206) and the frame phase information acquisition unit (1205) can be realized by a general technique, and thus detailed illustration is omitted.

  FIG. 19 is a diagram illustrating a configuration example in the case of performing synchronization using a frame phase information estimation unit of an image reception unit.

  In FIG. 19, the frame phase information generation unit (1201) of the control unit (106) of the image transmission apparatus (101) generates frame phase information, but this information is not transmitted to the image reception apparatus (109). In the image receiving apparatus (109), the encoded image data transmitted from the image transmitting apparatus (101) is decoded by the decoding unit (110), and a frame held by the control unit (112) based on the decoded image. The frame phase information estimation unit (1207) performs frame phase information estimation processing to obtain frame phase information.

  FIG. 20 is an explanatory drawing showing an example of the operation of the frame phase information estimation processing of the frame phase information estimation unit shown in FIG.

  In FIG. 20, an image (1301) is an image frame (201) position-shifted in the upper left direction in FIG. 2 and D1 size reduced by the image reduction unit (104) of the image transmitting apparatus (101) shown in FIG. It is an image (horizontal 704 pixels × vertical 480 pixels), and is given as an example for explanation. In FIG. 20, an example of a series of pixel values at the position of y = 0, where y is a two-dimensional coordinate on the image (1301) as (x, y) (where x is a horizontal coordinate and y is a vertical coordinate) An example of a series of pixel values at position 1 and an example of a series of pixel values at position y = 2 are shown.

  The image frame (201) is shifted in the upper left direction, and as described above, black (pixel value = 0) is inserted at the right end and the lower end of the image. Therefore, in the image 1301 after reduction, at the left end where the image originally exists, the pixel value P (0, y) at the position x = 0 and the pixel value P (1, y) at the position x = 1 The magnitude relationship of changes randomly according to the pattern. On the other hand, at the right end where no image originally exists, the pixel value P (703, y) at the position x = 703 is larger than the pixel value P (704, y) at the position x = 704 because black is inserted. . By using this property, frame phase information estimation processing is performed to estimate whether the image is shifted to the left or right.

  FIG. 21 is a flowchart showing frame phase estimation processing in the frame phase information estimation unit.

  In FIG. 21, when the frame phase information estimation process is started, the frame phase information estimation unit (1207) first prepares each data (L, R) of the left end counter and the right end counter on the memory. The value is initialized to 0, and data (y) indicating vertical coordinates is prepared on the memory, and y = 0 is initialized (step S1401). Subsequently, the values (P (0, y) and P (1, y)) of the two pixels at the left end of the image and the values (P (703, y) and P (704, y)) of the two pixels at the right end Each counter value (L, R) is updated according to the magnitude relation (step S1402). That is, in step S1402, if "P (0, y) P P (1, y)", the value (L) of the left end counter is increased by 1, and if "P (704, y) P P (703, If it is y), the value (R) of the right end counter is increased by one. Subsequently, it is determined whether the value of the vertical coordinate to be processed has reached the lower end of the image, that is, whether y = 479 has been reached (step S1403), and if the determination result is NO, data (y Is incremented by one to y ← y + 1 (step S1404), and the processes in steps S1404 and S1402 are repeated until the determination result in step S1403 is YES. If the determination result in step S1403 is YES, the image shift direction is estimated according to the value of each counter value (L, R) (step S1405). That is, in step S1405, if "the value of the left end counter (L) 右端 the value of the right end counter (R)", it is estimated that the image is shifted to the left, otherwise the image is right Estimated to be shifted in the direction. When step S1405 ends, the process ends.

  By the above processing, it is possible to estimate whether the image is shifted in position to the left or right. In FIG. 21, only the frame phase information estimation process in the left and right direction is illustrated for simplicity of explanation, but the frame phase information estimation can be performed by performing the same process in the vertical direction as well. is there. That is, by using the procedure of rotating the direction of each data value and numerical update shown in FIG. 21 by 90 degrees with respect to the image, it is possible to similarly estimate which of the upper and lower positions the image is shifted. Moreover, it is obvious that similar estimation can be performed by comparing the variances of pixel values instead of comparing the magnitude relation of pixel values at the edge of an image, and furthermore, it is similar using both magnitude relation and variance. An estimate may be made.

  Here, the image position shift processing in the image position shift unit (103) will be further considered. That is, depending on the type of image captured by the camera (102) of the image transmission apparatus (101), it is necessary to consider the content of the image position shift process. For example, when the camera (102) includes image data of a single-chip imaging device in which an image Bayer array color filter is arranged, it is necessary to add restrictions to the operation of the image position shift processing of the image position shift unit (103).

  FIG. 22 is a diagram showing the arrangement of pixels in the Bayer array color filter.

  As shown in FIG. 22, in the Bayer array color filter (1501), R: red, G: green, B: blue color filters are regularly arranged with one pixel interval of 2 pixels in both horizontal and vertical directions. ing. That is, when an image signal (hereinafter referred to as a Bayer image signal) obtained by photoelectric conversion of light transmitted through the Bayer array color filter (1501) by the single-chip imaging device is input to the image position shift unit (103), If one or both of the direction and the vertical direction are shifted in odd pixel units, the positional relationship of R: red, G: green, B: blue changes, and the color of the image after position shift is It will change.

  Therefore, when the Bayer image signal is input to the image position shift unit (103), R: red, G: green, B by shifting the position in units of two pixels (even pixel units) in both the horizontal direction and the vertical direction. : Perform image position shift processing such that the positional relationship of blue does not change from that before position shift. In addition, only the color signal (RGB, YUV, etc.) converted from the Bayer image signal or the luminance signal Y converted from the Bayer image signal after the position shift in units of two pixels (even-numbered pixels) as described above is performed. Each signal processing shown in the present embodiment can be applied as it is.

  FIG. 23 is a functional block diagram schematically showing one configuration example of the image storage device.

  In FIG. 23, an image storage device (108) is a device for storing (recording) or reproducing encoded image data, and includes a communication interface (1601), a control unit (1602), and a memory (1603). , Storage (1604), output interface (1605), input interface (1606), etc., and a bus (1607) connecting each processing block.

  The image storage device (108) stores an application program in the storage (1604), the control unit (1602) deploys the program from the storage (1604) to the memory (1603), and the control unit (1602) By executing the program, various functions such as recording, reproduction, and search can be realized. In the following description, for the sake of simplicity, various functions realized by the control unit (1602) executing each application program are realized mainly by the “various program function units” expanded in the memory (1603). However, it is also possible to use hardware as a processing unit having the same function, and to realize each function with each processing unit as a main body.

  The application program may be stored in advance in the storage (1604) when the image storage device (108) is shipped, or may be a medium such as an optical medium such as a CD (Compact Disc) or a DVD (Digital Versatile Disc) or a semiconductor memory. And may be installed on the image storage device (108) via a media connection (not shown). It is also possible to download and install from the communication network (107) via the communication interface (1601).

  The communication interface (1601) is connected to the communication network (107), receives image data from the image transmitting apparatus (101) shown in FIG. 1 via the communication network (107), and receives an image receiving apparatus (image receiving apparatus (101). 109) also has a function of transmitting image data. The control unit (1602) controls the communication interface (1601), the memory (1603) (various program function units), the storage (1604), and the input / output interface (1605, 1606). The control unit (1602) also has a function of executing various signal processing in accordance with the processing procedure described later. In the memory (1603), the functional part of the application program stored in the storage (1604) is expanded under the control of the control part (1602). The storage (1604) stores image data from the image transmission apparatus (101), and stores an application program and various information created by the application program. The output interface (1605) has a function of outputting an image as a result of signal processing in the control unit to an external device via the bus (1607). The output image is displayed on an external display unit (1608). The input interface (1606) has a function of receiving a signal from the operation unit (1609) and transmitting the signal to the control unit (1602) via the bus (1607).

  The image storage device (108) configured in this manner follows the following operation sequence at the time of recording. That is, at the time of recording, the image storage device (108) reads the recording application program (not shown) stored in the storage (1604) into the memory (1603), and the control unit (in accordance with the procedure described in the recording application program). 1602) controls each part. First, a connection with the image transmission apparatus (101) shown in FIG. 1 is established via the communication interface (1601) and the communication network (107). After that, the encoded image data transmitted from the image transmission apparatus (101) is received via the communication interface (1601) and the communication network (107), and the encoded data is transferred to the storage (1604) via the bus (1607). Image data is stored. At this time, the received encoded image data may be decoded by the control unit (1602), and the image may be output through the output interface (1605) and displayed by the display unit (1608). Subsequently, an operation sequence at the time of reproduction in the image storage device (108) will be described.

  Further, the image storage device (108) follows the following operation sequence at the time of reproduction. That is, at the time of reproduction, the image storage device (108) reads the reproduction application program (not shown) stored in the storage (1604) into the memory (1603), and the control unit (106) according to the procedure described in the reproduction application program. 1602) controls each part. Thereafter, the encoded image data stored in the storage (1604) is read via the bus (1607), and is transmitted to the image receiving apparatus (109) via the communication interface (1601) and the communication network (107). .

  At the time of transmission, encoded image data to be transmitted may be decoded by the control unit (1602), and an image may be output and displayed on the display unit (1608) via the output interface (1605). In addition, the encoded image data stored in the storage (1604) is read out by the image storage device (108) alone, decoded by the control unit (1602), and displayed through the output interface (1605) to the display unit (1608). Image may be output and displayed. However, as described above, when the image obtained by shifting the image position by the image position shift unit (103) of the image transmission apparatus (101) is displayed as it is on the display unit (1608), the image periodically blurs. Become. Therefore, various program function units expanded in the memory (1603) have a frame phase information estimation function (1610) having the same function as the frame phase information estimation unit (1207) and an image position shift unit (302). An image shift is provided by having an image position shift function (1611) having a function and performing processing of canceling out the effect of the image position shift processing. That is, the image position shift function (1611) is configured to set the third pixel number in the operation of the image position shift unit (302) of FIG. 4 to the pixel number of the coded image data stored in the storage (1604) (second By replacing it with the number of pixels), the operation of the image position shift unit (302) can be realized as it is.

  Also, the image position shift function (1611) can be realized by using a convolution process with a linear filter having asymmetric coefficients.

  FIG. 24 is a diagram for explaining an example of filter coefficients used in the image position shift function of FIG.

  In FIG. 24, coefficients (a) to (d) indicate filter coefficients to be convoluted into the decoded image. In each coefficient (a) to (d), the symbol or numerical value in parentheses (ie, 0, α, 1-α, β, 1-β, etc.) represents the value of each coefficient, and the right shoulder of the parentheses The symbol "T" represents transposition. Also, the symbol "*" represents a convolution operation. That is, coefficients (a) to (d) in FIG. 24 represent coefficients of a horizontal 3-tap × vertical 3-tap two-dimensional filter.

  Here, assuming that α> 0 and β> 0, the center of gravity of the two-dimensional filter represented by the coefficient (a) is shifted in the lower right direction, and an image obtained by convoluting this filter coefficient is in the lower right direction Position shifts. Similarly, the image in which coefficient (b) is convoluted is shifted in the lower left direction, the image in which coefficient (c) is convoluted is shifted in the upper right direction, and the image in which coefficient (d) is convoluted is upper left Position shifts in the direction.

  For example, an image having the first number of pixels (for example, horizontal 1920 pixels × vertical 1080 pixels) in FIG. 1 is based on the two-dimensional coordinates of the central position of rotational motion in “four-frame position shift” in FIG. As the position 0, 0, the two-dimensional coordinates (−h, −v) → (h, −v) → (h, v) → (−h, v) → (−h, −v) →. When the image is shifted, in the image of the number of pixels (second number of pixels) of the encoded image data stored in the storage (1604), p = 704/1920, q = 480/1080, and (−ph, − It corresponds to the position shift of qv) → (ph, −qv) → (ph, qv) → (−ph, qv) → (−ph, −qv) →.

  Therefore, in the filter coefficient, it is determined as α = ph and β = qv, and switching is performed for each frame in the order of coefficient (a) → (b) → (d) → (c) → (a) → ... By collapsing into the stored image, it is possible to suppress blurring of the image. By configuring as described above, it is possible to suppress image blurring when the image stored in the image storage device (108) is reproduced by the display unit (1608).

  Note that the “horizontal 3 taps × vertical 3 taps two-dimensional filter” described above is an example for describing the operation, and the present invention is not limited to this, and a two-dimensional having a different number of taps from this It is clear that a filter may be used. In addition, it is apparent that an image in which blurring is suppressed in this manner may be transmitted to the outside via the communication interface (1601) and the communication network (107).

  The effects of the present embodiment configured as described above will be described.

  The super-resolution processing exemplified in the prior art requires much more computation amount for processing than the edge enhancement processing for amplifying high frequency components contained in an image. For example, in the super-resolution technique in which an output image is obtained by successive approximation processing using input images of a plurality of frames as described in Non-Patent Document 1, high-precision motion search in subpixel units for each pixel is performed. It is necessary to iteratively process (interroll) the interframe registration (registration) involved, the enlargement of the image, the reduction of the enlarged image, the difference detection between the reduced image and the input image, and the correction process of the output image. That is, in the prior art, since the motion search and iteration for each pixel has a very large amount of calculation, the processing limit of the computational resources such as the CPU and memory is exceeded, and a frame drop occurs where the image motion is jerky. It is possible that the entire process may stop, or the user input from the mouse or the keyboard may not be accepted and the response may not be received.

  On the other hand, in the present embodiment, the image position shift unit (103) (106) shifts the position of the image to any of a plurality of predetermined shift positions with respect to each of a plurality of temporally consecutive images. First image position shift unit), an image reduction unit (104) for reducing and reducing the number of pixels of the image shifted in position by the image position shift unit (103), and an image reduced by the image reduction unit (104) A coding unit (105) for coding the image to generate a coded image, a decoding unit (110) for decoding the coded image sent through the communication network (107) to generate a decoded image, and a decoding unit An image enlargement unit (301) which enlarges and enlarges the number of pixels of a decoded image decoded by the encoding unit (110), and an image position shift unit (an image enlargement unit (301)) The position made in 103) An image position shift unit (302) (second image position shift unit) that shifts the position so as to cancel out the offset and the aliasing distortion of the image shifted in position by the image position shift unit (302) Since it is configured to include the aliasing distortion reduction unit (303) that performs aliasing distortion reduction processing for reduction using information of other images position-shifted in (302), an increase in the amount of calculation related to image processing and image quality The coding data size can be reduced while suppressing the decrease in That is, the amount of computation can be reduced by eliminating the need for motion search and iteration for each pixel constituting an image, and image sharpening and transmission data that can be processed at high speed even with relatively weak computing resources. It is possible to realize image transmission with a small size.

Second Embodiment
A second embodiment of the present invention will be described with reference to FIGS. 25 and 26.

  The present embodiment is configured such that the image transmitting apparatus according to the first embodiment has a function of switching the presence or absence of the image position shift processing, and the image position shift processing in which the image position shift processing is canceled and canceled. Even when displayed via an image receiving apparatus or an image storage apparatus that does not have means for performing (that is, means for shifting the position in the reverse direction), it is possible to provide an image that does not blur periodically.

  FIG. 25 is a functional block diagram schematically showing one configuration example of the image transmitting apparatus according to the present embodiment. In the figure, the same reference numerals are given to the same members as in the first embodiment, and the description will be omitted.

  In FIG. 25, an image transmitting apparatus (1802) comprises a camera (102) for capturing an image (still image or moving image), and an image position shift unit (103) for performing image position shift processing on an image captured by the camera (102). And the image signal from the camera (102) is input to the image reduction unit (104) or the image signal from the camera (102) passes through the image position shift unit (103) and then the image reduction unit (104) A switch (1803) for switching between input and output, an image reduction unit (104) for performing image reduction processing on an image input via the switch (1803), and encoding processing for an image subjected to image reduction processing Based on commands and data transmitted from the image receiving apparatus (109) and the image storage apparatus (108) via the communication network (107) A controller (control unit) that controls the overall operation of the image transmission apparatus (1802) including the image processing apparatus (102), the image position shift unit (103), the switch (1803), the image reduction unit (104), and the encoding unit (105). 106).

  FIG. 26 is a diagram showing an example of a menu display screen generated by the control unit of the image receiving apparatus or the image storage apparatus and displayed on the display unit.

  In FIG. 26, the menu display screen (1901) generated by the control unit (112, 1602) of the image receiving apparatus (109) or the image storage apparatus (108) and displayed on the display unit (113, 1608) transmits an image. A message portion (1902) indicating that the menu display is for selecting an operation related to the image position shift processing of the device (1802), a message portion (1903) indicating that the image position shift processing is not performed, and an image position shift A selection unit (1905) for selecting not to perform processing, a message unit (1904) indicating that image position shift processing is to be performed, and a selection unit (1906) for selecting image position shift processing And. Note that FIG. 26 exemplifies the case where the user of the image reception device (109) or the image storage device (108) selects to shift the image position. The message portion (1903) indicating that the image position shift processing is not performed may include a message indicating that the resolution improvement can not be expected. Further, the message portion (1904) indicating that the image position shift processing is to be performed may include a message indicating that the resolution improvement can be expected.

  When the image transmission apparatus (1802) is connected to the image reception apparatus (109) or the image storage apparatus (108) having the image position shift unit (302) via the communication network (107), the user displays a menu. By selecting the selection unit (1906) of the screen (1901), the switch (1803) is switched to the lower side in the drawing, and the image passing through the image position shift unit (103) is reduced by the image reduction unit (104). The data is reduced, encoded by the encoding unit (105), and transmitted to the image reception device (109) or the image storage device (108) through the communication network (107). As a result, the image receiving apparatus (109) can output a high quality image with reduced aliasing distortion.

  On the other hand, when the image transmitting apparatus (1802) is connected to an image receiving apparatus or an image storage apparatus not having the image position shift unit (302) via the communication network (107), the user can By selecting the selection unit (1905) of 1901), the switch (1803) is switched to the upper side in the figure and reduced by the image reduction unit (104) without passing through the image position shift unit (103). The encoding unit (105) encodes and transmits to the image reception device (109) and the image storage device (108) via the communication network (107). As a result, even the image receiving apparatus can output an image free of blurring.

  The switch (1803) is controlled in accordance with the control signal sent from the control unit (1804) according to the setting of the menu display screen (1901), but at this time, the image via the communication network (107) Control may be performed in accordance with a command automatically transmitted from the reception device (109) or the image storage device (108). That is, the image receiving apparatus (109) or the image storage apparatus (108) having the image position shift unit (302) sets in advance to send a command indicating that the image position shift processing is to be performed, and this command If it is not received in 1804), it may be determined that the image receiving apparatus without the image position shift unit (302) is connected, and the switch (1803) may be switched to the upper side in the figure.

  The other configuration is the same as that of the first embodiment.

  Also in this embodiment configured as described above, the same effect as that of the first embodiment can be obtained.

  In addition, in an image subjected to image position shift processing by the image position shift unit (103), means for performing image position shift processing that cancels out and cancels out the image position shift processing (that is, means for performing position shift in the reverse direction) In the image receiving apparatus having no image, the displayed image is periodically blurred. However, the image receiving apparatus according to the present embodiment including the image position shifting unit by providing the configuration for switching the presence or absence of the image position shifting process. Even when connected to any image receiving apparatus etc. which does not have an image position shift unit, etc., an image suitable for each can be displayed.

Third Embodiment
A third embodiment of the present invention will be described with reference to FIGS.

  In this embodiment, the period is displayed even in the case of displaying via an image receiving apparatus that does not have means for performing image position shift processing that cancels out and cancels image position shift processing (that is, means for performing position shift in the reverse direction). 10 shows another embodiment capable of providing an image which is not blurred.

  FIG. 27 is a functional block diagram schematically showing one configuration example of the image transmitting apparatus according to the present embodiment. FIG. 28 is a functional block diagram schematically showing one configuration example of the image receiving apparatus according to the present embodiment. In the figure, the same reference numerals are given to the same members as in the first embodiment, and the description will be omitted.

  In FIG. 27, an image transmitting apparatus (2001) comprises a camera (102) for capturing an image (still image or moving image), and an image position shift unit (103) for performing image position shift processing on an image captured by the camera (102). And an image reduction unit (104) that performs image reduction processing on the image subjected to the image position shift processing, and cancels out and cancels the position shift performed in the image position shift unit (103) on the image subjected to the image reduction processing. An image position shift unit (2002) that performs position shift, a coding unit (105) that performs coding processing on the image subjected to the image position shift processing by the image position shift unit (2002), and a communication network (107). The camera (102), the image position shift unit (103), the image reduction unit (104), the image position shift unit (2002), and It has control unit for controlling an image transmission apparatus (2001) the whole operations including coding portion (105) and (2003).

  The image position shift unit (2002) is for shifting the image in the direction opposite to the image blur generated by the image position shift unit (103) under the control of the control unit (2003). Operates similarly to the image position shift function (1611) of. That is, the operation of the image position shift unit (302) is performed by replacing the third pixel number in the operation of the image position shift unit (302) in FIG. 4 with the pixel number (second pixel number) of the image data to be transmitted. Can be used as it is. The above-mentioned asymmetric filter can also be used as the image position shift unit (2002).

  In FIG. 28, the image receiving apparatus (2004) comprises a decoding unit (110) for decoding the image transmitted from the image transmitting apparatus (2001) via the communication network (107), and a decoding process. The image position shift unit (2005) that performs the same image position shift processing as the image position shift unit 103 on the image that has been subjected to the image position shift processing (2005) And a control unit (2006) for controlling the overall operation of the image receiving apparatus (2004).

  The image position shift unit (2005) operates under the control of the control unit (2006) to offset and cancel the position shift performed by the image position shift unit (2002) of the image transmission apparatus (2001). It is

  FIG. 29 is a view for explaining an example of filter coefficients used in the image position shift unit of FIG.

  In FIG. 29, coefficients (a) to (d) indicate filter coefficients to be convoluted into the decoded image. In each coefficient (a) to (d), the symbol or numerical value in parentheses (ie, 0, α, 1-α, β, 1-β, etc.) represents the value of each coefficient, and the right shoulder of the parentheses The "-1" of represents an inverse characteristic, and the right shoulder symbol "T" of a bracket represents transposition. Also, the symbol "*" represents a convolution operation. That is, coefficients (a) to (d) of FIG. 29 represent the coefficients of the inverse filter of the two-dimensional filter shown in FIG. In addition, since each coefficient of an inverse filter can be calculated | required by a general technique, specific description is abbreviate | omitted.

  The other configuration is the same as that of the first embodiment.

  Also in the present embodiment configured as described above, the same effects as those of the first embodiment can be obtained.

  When the image transmission apparatus (2001) and the image reception apparatus (2004) are used in combination, the image reception apparatus (2004) can output a clear image by the same operation as that of the first embodiment. At the same time, when the image transmission device (2001) is connected to an image reception device or an image storage device without the image position shift unit, blurring of the image generated at the time of display can be suppressed.

Fourth Embodiment
A fourth embodiment of the present invention will be described with reference to FIG.

  In the present embodiment, the image storage device (108) in the first embodiment is used as an image reception device (1612).

  The image processing system according to the present embodiment uses the image transmission apparatus (101) as the image transmission apparatus, and is described in an image reception application program (not shown) stored in the storage (1604) of the image storage apparatus (108). The control unit (1602) controls each part according to the process (described in detail in FIG. 30 later), whereby the image storage device (108) is also used as an image reception device (1612) (see FIG. 23).

  FIG. 30 is a flowchart showing an example of processing in the image receiving apparatus according to the present embodiment.

  In FIG. 30, the image receiving apparatus (1612) first acquires image data encoded via the communication network (107) (step S2201), and the control unit (1602) decodes the image data, It is stored in the memory (1603) (step S2202). Subsequently, frame phase information is acquired (step S2205). Also, in parallel with step S2205, the image is enlarged (step S2203), and the position of the image is shifted (step S2204). These processes correspond to the process contents of the image enlargement unit (301) and the image position shift unit (302) shown in FIG.

  Subsequently, aliasing distortion reduction processing is performed (step S2206). In step S2206, two-dimensional processing is performed (step S2207), the image is stored in a predetermined frame memory (memory (1603) in FIG. 23) (step S2208), and the values of all frame memories are added for each same pixel position. (Step S2209). These processes correspond to the process contents in the configuration shown in FIGS. 9 and 10. In parallel with step S2206, low-pass filter processing for suppressing unnecessary aliasing is performed (step S2210). Subsequently, motion detection processing is performed to obtain a control signal m (step S2211). This process corresponds to the process content in the configuration shown in FIG.

  Subsequently, weighted mixing is performed using the image after aliasing reduction obtained in step S2206, the image after low-pass filter obtained in step S2210, and the control signal m obtained in step S2211 to obtain an output image ( Step S2212). This process corresponds to the operation of the weight mixing unit (1004) shown in FIG. The output image signal obtained in step S2212) is output via the output interface (1605) (step S2213), and the process is ended.

  If the control unit (1602) controls the processing shown in steps S2201 to S2213 described above to be executed each time image data is transmitted from the image transmission device (101) to the image reception device (1612). Good. Alternatively, each time the processing shown in steps S2201 to S2213 is completed, a command instructing transmission of image data may be transmitted from the image reception device (109) to the image transmission device (101). Further, by providing another program position shift function (not shown) in the various program function units expanded in the memory (1603) of FIG. 23, the operation of the image position shift unit (2005) described in FIG. It is obvious that the operation shown in FIG. 29 can be realized, and it is apparent that an operation similar to that shown in FIG. 28 can be realized using the configuration of the image receiving device (1612).

  The other configuration is the same as that of the first embodiment.

  The same effect as that of the first embodiment can be obtained also in the present embodiment configured as described above.

  The present invention is not limited to the above-described embodiments, and includes various modifications. For example, the above-described embodiments are described in detail to explain the present invention in an easy-to-understand manner, and are not necessarily limited to those having all the described configurations. In addition, each of the configurations, functions, and the like described above may be realized by designing a part or all of them with, for example, an integrated circuit. Further, each configuration, function, etc. described above may be realized by software by the processor interpreting and executing a program that realizes each function.

  That is, the function of each embodiment of the present invention can be realized by a program code of software. In this case, a storage medium storing the program code is provided to the system or apparatus, and a computer (or CPU or MPU) of the system or apparatus reads the program code stored in the storage medium. In this case, the program code itself read from the storage medium implements the functions of the above-described embodiments, and the program code itself and the storage medium storing the same constitute the present invention. As a storage medium for supplying such a program code, for example, a flexible disk, a CD-ROM, a DVD-ROM, a hard disk, an optical disk, an optical magnetic disk, a CD-R, a magnetic tape, a non-volatile memory card, a ROM Etc. are used.

  Also, based on instructions of the program code, an OS (Operating System) or the like running on the computer performs a part or all of the actual processing so that the functions of the above-described embodiment can be realized by the processing. May be Furthermore, after the program code read out from the storage medium is written in the memory on the computer, the CPU of the computer or the like performs part or all of the actual processing based on the instruction of the program code, and the processing The functions of the embodiment described above may be realized by

  Furthermore, by distributing the program code of the software for realizing the functions of the embodiment via a network, the storage means such as a hard disk or memory of a system or apparatus or a storage medium such as CD-RW or CD-R The computer (or CPU or MPU) of the system or apparatus may read out and execute the program code stored in the storage means or the storage medium at the time of use.

  Finally, it should be understood that the processes and techniques described herein are not inherently related to any particular device, and may be implemented by any suitable combination of components. Furthermore, various types of general purpose devices may be used in accordance with the teachings described herein. It may prove useful to build a dedicated device to carry out the method steps described here. In addition, various inventions can be formed by appropriately combining a plurality of constituent elements disclosed in the embodiments. For example, some components may be deleted from all the components shown in the embodiment. Furthermore, components in different embodiments may be combined as appropriate. Although the invention has been described in connection with specific examples, these are for the purpose of explanation rather than limitation in all respects. Those skilled in the art will appreciate that there are numerous combinations of hardware, software, and firmware that are suitable for practicing the present invention. For example, the described hardware may be implemented by an application specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or the like, and the described software is an assembler, C / C ++, perl, shell, PHP, Python And may be implemented in a wide range of programs or scripting languages, such as Java.

  Furthermore, in the above-described embodiment, the control lines and the information lines indicate what is considered necessary for the description, and not all the control lines and the information lines in the product are necessarily shown. All configurations may be connected to each other.

  In addition, other implementations of the present invention will be apparent to those skilled in the art from consideration of the specification and embodiments of the present invention disclosed herein. Various aspects and / or components of the described embodiments may be used alone or in any combination in a computerized storage system having the ability to manage data.

101 image transmitting apparatus 102 camera 103 image position shifting section 104 image reducing section 105 encoding section 106 control section 107 communication network 108 image storage apparatus 109 image receiving apparatus 110 decoding section 111 image enlarging and sharpening section 112 control section 113 display section 114 Operation unit

Claims (12)

A first image position shift unit that shifts the position of the image to any of a plurality of predetermined shift positions with respect to each of a plurality of temporally consecutive images;
An image reduction unit for reducing and reducing the number of pixels of the image position-shifted by the first image position shift unit;
An encoding unit that encodes the image reduced by the image reduction unit to generate an encoded image;
A decoding unit that generates a decoded image by decoding the coded images sent via the communication network,
An image enlargement unit which enlarges and enlarges the number of pixels of the decoded image decoded by the decoding unit;
A second image position shift unit that shifts the position of the image enlarged by the image enlargement unit so as to cancel the position shift performed by the first image position shift unit;
A aliasing distortion reduction unit that performs aliasing distortion reduction processing to reduce aliasing distortion of an image shifted in position by the second image position shift unit using information of another image shifted in position by the second image position shift unit and,
A third image position shift unit that shifts the image to a position that cancels out the position shift by the first image position shift unit with respect to the image reduced by the image reduction unit, and sends the image to the encoding unit;
A fourth image position shift unit that shifts the image to a position that cancels out the position shift by the third image position shift unit with respect to the decoded image decoded by the decoding unit and sends the image to the image enlargement unit; An image processing system comprising: A first image position shift unit that shifts the position of the image to any of a plurality of predetermined shift positions with respect to each of a plurality of temporally consecutive images;
An image reduction unit for reducing and reducing the number of pixels of the image position-shifted by the first image position shift unit;
An encoding unit that encodes the image reduced by the image reduction unit to generate an encoded image;
A decoding unit that generates a decoded image by decoding the coded images sent via the communication network,
An image enlargement unit which enlarges and enlarges the number of pixels of the decoded image decoded by the decoding unit;
A second image position shift unit that shifts the position of the image enlarged by the image enlargement unit so as to cancel the position shift performed by the first image position shift unit;
A aliasing distortion reduction unit that performs aliasing distortion reduction processing to reduce aliasing distortion of an image shifted in position by the second image position shift unit using information of another image shifted in position by the second image position shift unit Equipped with
The aliasing distortion reduction unit is provided with a motion adaptive processing unit that performs aliasing distortion reduction processing on a still area of the image and outputs an image shifted in position by the second image position shift unit in a moving area of the image. An image processing system characterized by In the image processing system according to claim 1,
The image processing system according to claim 1, wherein the image reduction unit reduces the number of pixels of the original image to a pixel number that is a non-divisor of the number of pixels of the original image. In the image processing system according to claim 1,
The image processing system characterized in that the aliasing reduction unit uses an asymmetric filter having an asymmetric coefficient. In the image processing system according to claim 1,
The image processing system according to claim 1, wherein the second image position shift unit performs position shift based on frame phase information generated based on the position shift in the first image position shift unit. In the image processing system according to claim 1,
The image processing system according to claim 1, wherein the second image position shift unit performs position shift based on a decoded image decoded by the decoding unit. In the image processing system according to claim 1,
The image processing system according to claim 1, wherein the first image position shift unit performs position shift based on command information received via the communication network. In the image processing system according to claim 7,
An image processing system comprising: a determination unit that determines the command information. A plurality of temporally continuous images, each of which is position shifted to any one of a plurality of predetermined shift positions, reduced in number of pixels and reduced , shifted to a position canceling the position shift, and encoded a decoding unit for generating a decoded picture by decoding the coded images sent via the communication network,
Respect decoded decoded image by the decoding unit, and shifts the image to further cancel position shift to a position to cancel the position shift, the first image position shifting unit for sending the image enlargement unit first An image enlargement unit for enlarging the number of pixels of the decoded image shifted in position by the image position shift unit ;
Respect enlarged image by the image enlargement portion, and a second image position shifting unit positioned shifted to cancel the positional shift by the first image position shifting unit,
Wherein the aliasing distortion of the position shifted by the image the second image position shifting unit, the aliasing distortion reducing unit for performing aliasing reduction processing of reducing by using information on the position shifted other images in the second image position shifting unit And an image processing apparatus characterized by comprising: A first image position shift unit that shifts the position of the image to any of a plurality of predetermined shift positions with respect to each of a plurality of temporally consecutive images;
An image reduction unit for reducing and reducing the number of pixels of the image position-shifted by the first image position shift unit;
A second image position shift unit that shifts the image to a position that cancels the position shift by the first image position shift unit with respect to the image reduced by the image reduction unit;
And an encoding unit that encodes the image position-shifted by the second image position shift unit to generate an encoded image.
A decoding unit that generates a decoded image by decoding the encoded image, with respect to the decoded decoded image by the decoding unit, the image in a position to cancel the positional shift by the second image position shifting unit A third image position shift unit for shifting the image, an image enlargement unit for enlarging the number of pixels of the decoded image shifted in position by the third image position shift unit and expanding the image, and the image enlarged in the image enlargement unit On the other hand, a fourth image position shift unit that shifts the position so as to cancel the position shift performed by the third image position shift unit, and aliasing distortion of the image shifted by the fourth image position shift unit, a processing apparatus having a aliasing reduction unit which performs aliasing distortion reduction processing for reducing by using information on the position shifted another image in the fourth image position shifting unit via said communication network coded image The image processing apparatus characterized by sending. A plurality of temporally consecutive images, each of which is position-shifted to any of a plurality of predetermined shift positions, reduced in number of pixels, reduced, encoded, and sent via a communication network A decoding unit that decodes a coded image to generate a decoded image;
An image enlargement unit which enlarges and enlarges the number of pixels of the decoded image decoded by the decoding unit;
An image position shift unit that shifts the position of the image enlarged by the image enlargement unit so as to cancel the position shift;
Wherein the aliasing distortion of the position shifted by the image the image position shift unit, and a folded distortion reduction unit which performs aliasing distortion reduction processing for reducing by using information on the position shifted other images by the image position shift unit,
The aliasing distortion reduction unit is provided with a motion adaptive processing unit that performs aliasing distortion reduction processing on a still area of the image and outputs an image shifted in position by the image position shift unit in a moving area of the image. An image processing apparatus characterized by A first image position shift unit that shifts the position of the image to any of a plurality of predetermined shift positions with respect to each of a plurality of temporally consecutive images;
An image reduction unit for reducing and reducing the number of pixels of the image position-shifted by the first image position shift unit;
And an encoding unit that encodes the image reduced by the image reduction unit to generate an encoded image.
A decoding unit that generates a decoded image by decoding the encoded image, and an image enlarging unit for enlarging by increasing the number of pixels decoded decoded image by the decoding unit, in the image enlargement portion A second image position shift unit for performing position shift so as to cancel the position shift performed by the first image position shift unit with respect to the enlarged image; and an image of the image shifted by the second image position shift unit A folding distortion reduction process for reducing aliasing distortion using information of another image position-shifted by the second image position shift unit is performed on the still region of the image, and the second region of the moving image of the image a processing apparatus having a aliasing reduction unit having a motion adaptive processing section for outputting a position shifted image by the image position shift unit, an image processing, characterized in that sending the encoded images via a communication network Location.

Images