JP2013172321A - Image processing apparatus and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To emphasize a video the image features of which change.SOLUTION: The image processing apparatus includes an extraction means for receiving a frame included in a video and extracting the image features for each frame, a delay means for storing and delaying the image features extracted by the extraction means, a determination means for comparing the image features extracted by the extraction means with the image features delayed by the delay means, and determining change of the image features, a storage means for storing the frame in which the image features changed, when a determination is made that the image features changed by the determination means, an output means for outputting any one of a frame input or a frame stored in the storage means, and a control means for controlling the output means to output a predetermined number of frames stored in the storage means continuously to the storage means, when a determination is made that the image features changed by the determination means.

Description

  The present invention relates to an image processing apparatus and program for enhancing video.

  As a method of emphasizing a part of the video, effects such as slow playback, frame advance, and pause are performed in broadcast and video program production. These effects extend the presentation time of particularly important scenes in order to improve the visibility of particularly fast moving scenes.

  As a technique for emphasizing a partial area of an image, there is an effect of performing partial enlargement. There is a technique for automating this partial enlargement, for example, identifying a subject region by tracking the subject or recognizing the subject, and automatically presenting the subject with partial enlargement.

  Still cameras, video cameras, and printers have been put to practical use that recognize specific subjects in video and images, and automatically perform image correction so that the exposure state and color tone of these subject areas are appropriate. (Patent Document 1).

JP 2008-263320 A

  However, in the prior art, effects such as slow playback, frame advance, and pause are performed manually, and the operation requires skill and effort.

  In addition, the method of performing partial enlargement is effective for enhancing a small subject, but does not perform instantaneous scene enhancement.

  In addition, a technique for recognizing a specific subject in a video or image and automatically correcting the image so that the exposure state and color tone of the subject area are appropriate is suitable for photographing a specific subject in a good state. It is an effective method. However, when the illumination changes suddenly, the operation is performed to suppress the change, so that the luminance information sensed on the sensor at the time of the sudden illumination change is discarded.

  Therefore, the conventional technique has a problem that it is impossible to emphasize a video in which image characteristics such as luminance change.

  Accordingly, the present invention has been made in view of the above problems, and an object thereof is to provide an image processing apparatus and program capable of enhancing a video in which the characteristics of an image change.

  An image processing apparatus according to an aspect of the present invention includes an extraction unit that inputs a frame included in a video, extracts an image feature for each frame, and a delay unit that stores and delays the image feature extracted by the extraction unit And the image feature extracted by the extraction unit and the image feature delayed by the delay unit to determine a change in the image feature, and the determination unit determines that the image feature has changed. The image feature is changed by the storage means for storing the frame whose image feature has changed, the output means for outputting any of the input frame and the frame stored in the storage means, and the determination means. Control means for controlling the output means so as to continuously output a predetermined number of frames stored in the storage means.

  In addition, the storage unit stores a predetermined number of frames after the frame in which the image feature has changed, and the control unit continuously outputs the frame in which the image feature has been stored stored in the storage unit. The predetermined number of frames stored in the storage unit may be output.

  The predetermined number of frames is a frame after the frame in which the image feature has changed, and may be a discontinuous frame.

  Further, the image processing apparatus may further include a preprocessing unit that rearranges the input frames in the order of video encoding and outputs the rearranged frames to the extraction unit.

  The image feature may be a value related to a luminance value.

  According to another aspect of the present invention, there is provided a program for inputting a frame included in a video, extracting an image feature for each frame, and a delaying step for buffering and delaying the extracted image feature. A step of comparing the extracted image feature with the delayed image feature to determine a change in the image feature; and, if it is determined that the image feature has changed, a frame in which the image feature has changed is stored. A storage step of storing in the means, an output step of outputting any of the input frame and the frame stored in the storage means, and if it is determined that the image feature has changed, the storage means stores A control step for controlling to output a predetermined number of the frames continuously, and causing the computer to execute.

  According to the present invention, it is possible to emphasize a video in which the feature of an image changes.

1 is a block diagram illustrating an example of a configuration of an image processing device according to Embodiment 1. FIG. FIG. 6 is a diagram for explaining a specific example of image processing in the first embodiment. 3 is a flowchart illustrating an example of image processing of the image processing apparatus according to the first exemplary embodiment. FIG. 10 is a diagram for explaining a specific example of image processing in the second embodiment. 9 is a flowchart illustrating an example of image processing of an image processing apparatus according to a second embodiment. FIG. 9 is a block diagram illustrating an example of a configuration of an image processing apparatus according to a third embodiment. FIG. 10 is a diagram for explaining a specific example of image processing in the third embodiment. 10 is a flowchart illustrating an example of image processing of an image processing apparatus according to a third embodiment. FIG. 10 is a block diagram illustrating an example of a configuration of an image processing apparatus according to a fourth embodiment.

  The present invention will be described below with reference to the accompanying drawings. An image processing apparatus described below emphasizes an input image (input video) A with an image (frame) whose image characteristics have changed, and outputs an image (moving image) B as a result.

[Example 1]
<Configuration>
FIG. 1 is a block diagram illustrating an example of the configuration of the image processing apparatus 1 according to the first embodiment. In the example illustrated in FIG. 1, the image processing apparatus 1 includes a feature extraction unit 101, a delay unit 102, a determination unit 103, an image storage unit 104, an output unit 105, and a control unit 106.

  The feature extraction unit 101 analyzes the frame A (t) at time t of the input video A, and extracts an image feature F (t) used to determine whether or not the frame is to be emphasized.

  The image feature F (t) is, for example, a value related to a luminance value in a frame (dispersion, average, or sum of luminance values), a representative value of hue (average hue component of color vectors (R, G, B)), The average of motion vectors and the average of luminance values of a specific area are used.

  By using the value related to the brightness value, it is possible to identify a frame in which a sudden brightness increase (flash or the like), a sudden picture change (explosion scene or the like), and the like has occurred. By using the representative value of the hue, it is possible to identify a frame in which a sudden change in hue (such as a blinking neon sign) has occurred.

  Further, if the average of motion vectors is used, it is possible to specify a frame in which a sudden change in motion (the moment of occurrence or explosion) occurs. By using the average of the luminance values in the specific area, it is possible to specify a frame in which a moment when an object enters the certain area (moment when an animal crosses the center of the screen) or the like has occurred.

  Hereinafter, a case where a value related to a luminance value is used as the image feature F (t) will be described, but the present invention is not limited to this example. Using values related to luminance values as image features, for example, in a video for reporting, I want to emphasize an image in which a person (criminal) is reflected by a flash from a dark video (a night video, a video of a car in which the criminal rides) It is suitable for occasions.

  For example, the feature extraction unit 101 calculates an image feature F (t) based on the luminance value of the frame A (t) in order to determine whether or not the current frame A (t) is an image at the time of illumination variation.

  Specifically, the feature extraction unit 101 calculates, for example, the sum of the luminance values of the current frame A (t) as the image feature F (t). For example, an average luminance value obtained by dividing the total luminance value by the total number of pixels of the frame may be used as the total luminance value.

  When the input video A is a monochrome image, the feature extraction unit 101 sets the pixel value at the image coordinates (x, y) of the frame A (t) at time t as A (t; x, y). The sum of luminance values is obtained by the following equation (1).

In addition, when the input image A is a color or the like, the feature extraction unit 101 includes, for example, a pixel value A (t; x, y) that is a vector value when the input image A includes a plurality of color components and wavelength (band) components. Thus, the sum of luminance values is obtained by the following equation (2).

Here, the pixel value at the image coordinates (x, y) of the frame at time t of the input video A is expressed as A (t; x, y).

  In addition, when using a conversion matrix (row vector) M from a color vector to a luminance value, the feature extraction unit 101 obtains the sum of the luminance values by Expression (3).

Here, it is assumed that the pixel value A (t; x, y) is a three-dimensional column vector, the first component represents the red component, the second component represents the green component, and the third component represents the blue component. . In this case, as the transformation matrix M, for example, the following ITU-R BT. A matrix (row vector) for luminance signal calculation defined in the 601 standard is used.

Further, as the transformation matrix M, the following ITU-R BT. A matrix (row vector) for calculating a luminance signal defined in the 709 standard may be used.

The feature extraction unit 101 outputs the calculated image feature F (t) to the delay unit 102 and the determination unit 103.

  The delay means 102 temporarily stores (buffers) the image feature F (t) and outputs past image features. For example, the delay unit 102 outputs the image feature F (t−1) one point before to the determination unit 103.

  The determination unit 103 compares the image feature F (t) at the current time t with a past image feature (for example, the image feature F (t−1) before one time point), and whether or not there is a temporal change in the feature. Determine. The determination unit 103 determines the output value (determination flag) C (t) depending on, for example, whether or not this temporal change has exceeded the first threshold value.

  The first threshold value may be set in advance by an experiment, or may be set by adjustment by the user.

  For example, it is assumed that the determination flag C (t) takes a value “1” in the case of “with time change” and a value “0” in the case of “no time change”.

  For example, the determination unit 103 determines that the absolute value of the difference between the image feature F (t) at the current time t and the past image feature F (t−d) (d is a natural number; for example, d = 1) is the first threshold value. When θ (θ is a positive real number) is exceeded, it is determined that “there is a temporal change” (Formula (4)).

The determination unit 103 sets the determination flag C (t) to “1” at the moment when the luminance value suddenly increases or decreases by using any one of Expressions (1) to (3) and Expression (4). can do.

  Further, for example, the determination unit 103 has the first difference value between the image feature F (t) at the current time t and the past image feature F (t−d) (d is a natural number; −for example, d = 1). When the threshold value θ (θ is a positive real number) is exceeded, it may be determined that “there is a temporal change” (Formula (5)).

The determination unit 103 sets the determination flag C (t) to “1” at the moment when the luminance value suddenly increases by using any one of Expressions (1) to (3) and Expression (5). Can do.

  In addition, for example, the determination unit 103 determines that the difference value between the image feature F (t) at the current time t and the past image feature F (t−d) (d is a natural number; for example, d = 1) is the first threshold value. When it is less than θ (θ is a negative real number), it may be determined that “there is a temporal change” (Formula (6)).

The determination unit 103 sets the determination flag C (t) to “1” at the moment when the luminance value decreases rapidly by using any one of the expressions (1) to (3) and the expression (6). Can do.

  The determination unit 103 outputs a determination flag, which is a determination result of determining a temporal change, to the image storage unit 104 and the control unit 106.

  The image storage unit 104 displays the frame A (t) of the input video A at the time when the determination flag C (t) output by the determination unit 103 becomes “with time change” (that is, C (t) = 1). Remember.

Further, the image storage unit 104 outputs the frame A (t ₁ ) at the time point (t ₁ ; t ₁ ≦ t) at which it is determined that “there is a time change” in the past (including the current time t). Output to. That is, the image storage unit 104 outputs the frame A (t ₁ ) to the output unit 105 for a predetermined number of frames from the time point t ₁ at which it is determined that “there is a temporal change”.

The output means 105 is either the current frame A (t) of the input video or the most recent frame A (t ₁ ) “with time change” output from the image storage means 104 (including the current time t). Output one. The output means 105 selects any frame in response to a control signal S (t) from the control means 106 described later, and outputs the selected frame as a frame B (t) at time t of the output video B ( Formula (7)).

The control unit 106 generates a control signal S (t) for controlling the output unit 105 based on the determination flag C (t) output by the determination unit 103 and outputs the control signal S (t) to the output unit 105.

  For example, the control unit 106 determines that the determination flag C (t) output by the determination unit 103 is 1 (including the current time) from the latest 1 to the time point e (e is referred to as the duration); S (t) is set to 1 only for the above integer) (formula (8)).

With the above configuration, it is possible to continuously display frames whose image characteristics have been converted for the duration, and to emphasize the frames.

<Specific example>
Next, a specific example of image processing in the first embodiment will be described. In this specific example, for example, what the output video looks when the duration e = 2 is described.

  FIG. 2 is a diagram for explaining a specific example of image processing in the first embodiment. In the example shown in FIG. 2, the input video is the upper frame group (A (1) to A (8) shown in fm11 to fm18), and the output video is the lower frame group (B (1) to B (1) to fm21 to fm28). B (8)).

  In the input image shown in FIG. 2, the input frame A (3) indicated by fm13 is captured brightly by the flash of the flash. On the other hand, the other input frame groups (A (1), A (2) and A (4) to A (8) shown in fm11, fm12 and fm14 to fm18) are imaged darkly.

  At this time, when the determination unit 103 described above determines a temporal change using the equation (5), it is assumed that the difference between the input frame A (2) and the input frame A (3) is larger than the first threshold θ. At time t = 3, the determination flag C (3) becomes 1.

  In addition, the determination unit 103 determines that the difference in the sum of the luminance values between frames is equal to or less than the first threshold θ at other time points, and sets C (t) = 0. Therefore, the image storage unit 104 stores the frame A (3) at time t = 3 based on the determination flag C (3).

  Next, the control means 106 outputs the frame A (3) output from the image storage means 104 from the time point t = 3 for the duration e = 2 (that is, only during t = 3, 4, 5). At other time t, control is performed so as to output the input frame A (t) at time t.

  Here, when video coding including intra coding is performed, the image quality is likely to deteriorate in a frame in which the characteristics of the image have changed, and it is difficult to emphasize this frame. However, according to the first embodiment, for example, it is expected that the image quality of the frame B (5) is improved compared to the frame B (3), and this frame B (5) is temporarily stopped. It becomes possible to highlight and display.

<Operation>
Next, the operation of the image processing apparatus 1 in the first embodiment will be described. FIG. 3 is a flowchart illustrating an example of image processing of the image processing apparatus 1 according to the first embodiment. In step S101 shown in FIG. 3, the feature extraction unit 101 extracts an image feature F (t) from the input video. The feature extraction unit 101 may extract the image features using, for example, any one of the expressions (1) to (3).

  In step S102, the delay means 102 stores the extracted image feature F (t) and delays it for a predetermined time. The predetermined time is, for example, one frame.

  In step S <b> 103, the determination unit 103 compares the image feature F (t) of the current frame with the image feature F (t−d) delayed by the delay unit 102.

  In step S104, the determination unit 103 determines whether or not there has been a temporal change in the image feature as a result of the comparison. For example, the determination unit 103 may use any one of Expressions (4) to (6).

  If the determination result has a temporal change (C (t) = 1) (step S104-YES), the process proceeds to step S105, and if the determination result has no temporal change (C (t) = 0) (step S104). -NO) Proceed to step S106.

  In step S105, the image storage unit 104 stores the input frame (input frame) because the determination flag C (t) from the determination unit 103 is “1”.

  In step S106, the control means 106 determines whether or not there is a feature change (temporal change) within the past predetermined number of frames (duration e). If there is a feature change within the past predetermined number of frames (step S106-YES), the process proceeds to step S107, and if there is no feature change within the past predetermined number of frames (step S106-NO), the process proceeds to step S108.

  In step S <b> 107, the output unit 105 outputs the frame stored in the image storage unit 104 based on the control signal S (t) (S (t) = 1) from the control unit 106.

  In step S108, the output unit 105 outputs an input frame based on the control signal S (t) (S (t) = 0) from the control unit 106.

  In step S109, the image processing apparatus 1 determines whether or not the processing of all the videos has been completed. If all the video processing has been completed (step S109—YES), this image processing is terminated. If all video processing has not been completed (step S109—NO), the processing returns to step S101.

  As described above, according to the first embodiment, it is possible to emphasize a video in which the feature of an image changes. For example, it is possible to emphasize a frame by specifying a frame in which the feature of the image changes with time and continuing this frame for a duration. Therefore, by continuously outputting the video of the decisive moment when the state of the image has changed, even if irreversible video coding is performed in the subsequent stage of this device, the image quality of the decisive moment of the video is reduced. Can be prevented.

[Example 2]
Next, an image processing apparatus according to the second embodiment will be described. In the second embodiment, the frame when the image characteristics change and the subsequent frame are stored, and the stored frame is output after the continued frame. This makes it possible to smoothly switch to the input frame using the subsequent frames for the playback timing shifted due to the continuous output of the same frame.

<Configuration>
Since the configuration of the image processing apparatus according to the second embodiment is the same as that of the image processing apparatus 1 according to the first embodiment, the description will be made using the same reference numerals. The configuration other than the configuration described below is the same as the configuration of the first embodiment.

  The image storage unit 104 stores a frame at the time when the feature of the image “changes with time” and a predetermined number of frames after this frame (hereinafter referred to as subsequent frames). The subsequent frames may be a predetermined number of frames after the frame at the time of the change, or a predetermined number of frames that are not continuous (for example, skipped) after the frame at the time of the change. Also good.

For example, the image storage unit 104 stores the frame A (t ₁ ) and the frame A (t ₁ +2), where t ₁ is the time point when it is determined that “there is a temporal change”.

In addition, the image storage unit 104 sets the video frame A (t ₁ ) at the time point (t ₁ ; t ₁ ≦ t) at which it is determined that “there is a temporal change” in the past (including the current time t) as the time point. A (t ₁ ) is output over t ₁ and a predetermined duration e (e is an integer equal to or greater than ₁ ).

Further, the image storage unit 104 may sequentially store the subsequent frames stored from the time point t ₁ + e + 1 to the time point t ₁ + e + f (hereinafter, f is referred to as the number of subsequent frames; f is an integer of 1 or more). ) Output to the output means 105.

When the number of subsequent frames is f = 1, for example, the image storage unit 104 outputs the subsequent frame A (t ₁ +2) at the time point t ₁ + e + 1.

  In this way, the frame output from the image storage unit 104 at the time point t is set to D (t).

  The output unit 105 outputs either the current frame A (t) of the input video or the frame D (t) output from the image storage unit 104. The output unit 105 selects, for example, according to a control signal S (t) from the control unit 106 described later, and outputs the selected frame as a frame B (t) at the time t of the output video B (formula (9)). ).

The control means 106 controls to output a “with time change” frame for the duration e, and outputs a predetermined frame stored in the image storage means 104 for the subsequent f frames.

  For example, the control unit 106 generates a control signal S (t) for controlling the output unit 105 based on the determination flag C (t) output by the determination unit 103 and outputs the control signal S (t) to the output unit 105.

  For example, the control unit 106 sets S (t) to 1 only after the determination flag C (t) output by the determination unit 103 becomes 1 immediately (including the current time) until the e + f time point (expression) (10)).

With the above configuration, it is possible to continuously display frames whose image characteristics have been converted for the duration of time, and emphasize this frame and smoothly switch from the stored frame to the input frame. To be able to do that.

<Specific example>
Next, a specific example of image processing in the second embodiment will be described. In this specific example, for example, the output video when the duration time e = 2 and f = 1 will be described.

  FIG. 4 is a diagram for explaining a specific example of image processing in the second embodiment. In the example shown in FIG. 4, the input video is the upper frame group (A (1) to A (8) shown in fm11 to fm18), and the output video is the lower frame group (B (1) to B (1) to fm31 to fm38). B (8)).

  In the input video shown in FIG. 4, the input frame A (3) indicated by fm13 is captured brightly by the flash of the flash. On the other hand, the other input frame groups (A (1), A (2) and A (4) to A (8) shown in fm11, fm12 and fm14 to fm18) are imaged darkly.

  At this time, when the determination unit 103 described above determines a temporal change using the equation (5), it is assumed that the difference between the input frame A (2) and the input frame A (3) is larger than the first threshold θ. At time t = 3, the determination flag C (3) becomes 1.

  In addition, the determination unit 103 determines that the difference in the sum of the luminance values between frames is equal to or less than the first threshold θ at other time points, and sets C (t) = 0. Therefore, the image storage unit 104 stores the frame A (3) at time t = 3 and the subsequent frame, for example, the frame A (5) after 2 frames, based on the determination flag C (3).

  Next, the control means 106 outputs from the image storage means 104 for a duration e = 2 from the time point t = 3 (that is, only between t = 3, 4, and 5 shown in fm33 to fm35 shown in FIG. 4). Control to output D (t) = A (3).

  The control means 106 controls to output D (t) = A (5) output from the image storage means 104 over the subsequent f = 1 frame (that is, only t = 6 shown in fm36 shown in FIG. 4). . The control means 106 controls to output the input frame A (t) at the time point t at other time points t.

  This makes it possible to smoothly switch to the input frame using the subsequent frames for the playback timing shifted due to the continuous output of the same frame.

<Operation>
Next, the operation of the image processing apparatus according to the second embodiment will be described. FIG. 5 is a flowchart illustrating an example of image processing of the image processing apparatus according to the second embodiment. The processing in steps S201 to S203 shown in FIG. 5 is the same as the processing in steps S101 to S103 shown in FIG.

  In step S <b> 204, the determination unit 103 determines whether or not there has been a temporal change in image characteristics within the past first number of frames as a result of the comparison. The first frame is, for example, the number of subsequent frames.

  If the determination result has a temporal change (C (t) = 1) (step S204-YES), the process proceeds to step S205, and if the determination result has no temporal change (C (t) = 0) (step S204). -NO) Proceed to step S206.

  In step S205, the image storage unit 104 stores the input frame (input frame) because the determination flag C (t) from the determination unit 103 is “1”.

  In step S206, the control means 106 determines whether or not there is a feature change (temporal change) within the past second number of frames (continuation time e). If there is a feature change within the past second number of frames (step S206—YES), the process proceeds to step S207, and if there is no feature change within the past second number of frames (step S206—NO), the process proceeds to step S208.

  In step S207, the output unit 105 determines the frame stored in the image storage unit 104 based on the control signal S (t) (S (t) = 1) from the control unit 106 (a frame having a temporal change). Is output.

  In step S208, the control means 106 determines whether or not the current frame is within the third number of frames (e + f) from the frame that has changed over time. If it is within the third frame number (step S208—YES), the process proceeds to step S209, and if it is not within the third frame number (step S208—NO), the process proceeds to step S210.

  In step S209, the output unit 105 outputs the subsequent frame stored in the image storage unit 104 based on the control signal S (t) (S (t) = 1) from the control unit 106.

  In step S210, the output unit 105 outputs an input frame based on the control signal S (t) (S (t) = 0) from the control unit 106.

  In step S <b> 211, the image processing apparatus 1 determines whether or not the processing of all videos has been completed. If all the video processing has been completed (step S211—YES), this image processing is terminated. If all video processing has not been completed (step S211—NO), the processing returns to step S201.

  As described above, according to the second embodiment, it is possible to emphasize a video in which the feature of an image changes. For example, by specifying a frame in which the characteristics of the image have changed over time, and continuing this frame for the duration, this frame is emphasized, and at the same time, the subsequent frame is shifted with respect to the playback timing shifted by the continuous output of the same frame. Can be used to smoothly switch to the input frame.

[Example 3]
Next, an image processing apparatus according to the third embodiment will be described. In the third embodiment, assuming that video encoding is performed in the subsequent stage of the present apparatus, the processing of the first and second embodiments is performed according to the encoding order of the GOP (Group Of Picture) structure. Thereby, since the prediction accuracy of inter-frame prediction in video coding is improved, it is possible to improve image quality and rate distortion characteristics at the same bit rate.

<Configuration>
FIG. 6 is a block diagram illustrating an example of the configuration of the image processing apparatus 3 according to the third embodiment. In the example illustrated in FIG. 6, the image processing apparatus 3 includes a preprocessing unit 301, a feature extraction unit 101, a delay unit 102, a determination unit 103, an image storage unit 104, an output unit 105, and a control unit 106. And post-processing means 302.

  In the configuration of the image processing apparatus 3 in the third embodiment, the same components as those in the first and second embodiments are denoted by the same reference numerals, and the description thereof is omitted. Note that the post-processing unit 302 is not necessarily required because the video encoding process is performed in the subsequent stage of the present apparatus.

  The preprocessing unit 301 rearranges the input video in the encoding order in consideration of the GOP structure in the video encoding technique. The preprocessing unit 301 outputs each rearranged frame as an input video to the feature extraction unit 101 or the like.

  Each configuration of the feature extraction unit 101 to the control unit 106 executes the processing described in the first and second embodiments.

  The post-processing unit 302 rearranges the frames of the video output by the output unit 105 in the display order and outputs the rearranged video as an output video. Note that the post-processing unit 302 is not necessarily required in consideration of the video encoding performed in the subsequent stage.

  When the post-processing unit 302 is not provided in the image processing apparatus 3, video encoding at the subsequent stage of the apparatus can be performed in the order output by the output unit 105. Further, the image processing device 3 may include a video encoding circuit.

  Examples of video encoding techniques include MPEG (Moving Picture Experts Group) 2 and H.264. H.264 / AVC (Advanced Video Coding).

  With the above configuration, it is possible to continuously display frames whose image characteristics have been converted for a continuous time in consideration of video encoding. This frame is emphasized and image quality is improved. Will be able to.

<Specific example>
Next, a specific example of image processing in the third embodiment will be described. In this specific example, for example, when the duration e = 2, the output video when rearranged in the coding order and processed will be described.

FIG. 7 is a diagram for explaining a specific example of image processing in the third embodiment. In the example shown in FIG. 7, for example, a GOP structure as IBPBPBPBPBPBPBP as shown in fm11 to fm19 is adopted.
I: Intra picture or slice P: Forward prediction picture or slice B: Bidirectional prediction picture or slice In this case, the preprocessing means 301 is not the input frame order, but as shown in fm41 to fm49 shown in FIG. Rearrange to the encoding processing order (IPBPBPBPB). Note that the numbers in each frame shown in FIG. 7 indicate the input order.

  Each unit after the feature extraction unit 101 executes, for example, the image processing described in the first embodiment in the ordered frame order.

  Reference numerals fm51 to fm59 shown in FIG. 7 indicate the results of performing the image processing described in the first embodiment. As shown in FIG. 7, the frames fm52 to fm54 are the same frame.

  Next, the post-processing unit 302 rearranges fm51 to fm59 shown in FIG. 7 in the order of the input video to obtain an output video. Frame groups after rearrangement by the post-processing means 302 are shown as fm61 to fm69.

  Note that the rearrangement of the frames may be performed by rearranging the logical order in the image processing apparatus 3 instead of actually rearranging the frame order.

<Operation>
Next, the operation of the image processing apparatus 3 in the third embodiment will be described. FIG. 8 is a flowchart illustrating an example of image processing of the image processing apparatus 3 according to the third embodiment. The processing shown in FIG. 8 is based on the image processing in the first embodiment, but may be based on the image processing in the second embodiment.

  In step S301, the preprocessing unit 301 rearranges the frames of the input video in the order of video encoding and outputs the images.

  Each process of steps S302 to S310 is the same as each process of steps S101 to S109 shown in FIG.

  In step S311, the post-processing unit 302 rearranges the frames of the video output from the output unit 105 in the original input order, and outputs the rearranged video as an output video.

  As described above, according to the third embodiment, it is possible to emphasize a video in which the feature of an image changes. For example, a frame in which the characteristics of the image have changed over time is identified, this frame is continuously emphasized for the duration, and the prediction accuracy of inter-frame prediction in video coding is improved. And rate distortion characteristics can be improved.

[Example 4]
FIG. 9 is a block diagram illustrating an example of the configuration of the image processing device 4 according to the fourth embodiment. An image processing apparatus 4 shown in FIG. 9 is an example of an apparatus in which each image processing described in the first to third embodiments is implemented by software.

  As illustrated in FIG. 9, the image processing apparatus 4 includes a control unit 401, a main storage unit 402, an auxiliary storage unit 403, a drive device 404, a network I / F unit 406, an input unit 407, and a display unit 408. These components are connected to each other via a bus so as to be able to transmit and receive data.

  The control unit 401 is a CPU that controls each device, calculates data, and processes in a computer. The control unit 401 is an arithmetic device that executes an image processing program stored in the main storage unit 402 or the auxiliary storage unit 403. The control unit 401 receives data from the input unit 407 and the storage device, calculates and processes the data, and then outputs the data to the display unit 408 and the storage device.

  The control unit 401 can realize the processing described in each embodiment by executing the above-described image processing program.

  The main storage unit 402 is a ROM (Read Only Memory), a RAM (Random Access Memory), or the like. The main storage unit 402 is a storage device that stores or temporarily stores programs and data such as OS (Operating System) and application software that are basic software executed by the control unit 401.

  The auxiliary storage unit 403 is an HDD (Hard Disk Drive) or the like, and is a storage device that stores data related to application software or the like.

  The drive device 404 reads the program from the recording medium 405, for example, a flexible disk, and installs it in the storage unit.

  A predetermined program is stored in the recording medium 405, and the program stored in the recording medium 405 is installed in the image processing apparatus 4 through the drive device 404. The installed predetermined program can be executed by the image processing apparatus 4.

  The network I / F unit 406 is a peripheral having a communication function connected via a network such as a LAN (Local Area Network) or a WAN (Wide Area Network) constructed by a data transmission path such as a wired and / or wireless line. This is an interface between the device and the image processing apparatus 4.

  The input unit 407 includes a keyboard having cursor keys, numeric input, various function keys, and the like, and a mouse and a slice pad for performing key selection on the display screen of the display unit 408. The input unit 407 is a user interface for a user to give an operation instruction to the control unit 401 or input data.

  The display unit 408 is configured by a CRT (Cathode Ray Tube), an LCD (Liquid Crystal Display), or the like, and performs display according to display data input from the control unit 401.

  The delay unit and the image storage unit described in the first to third embodiments can be realized by the main storage unit 402 or the auxiliary storage unit 403, for example. The configuration other than the delay unit and the image storage unit described in the first to third embodiments can be realized by the control unit 401 and the main storage unit 402 as a work memory, for example.

  The program executed by the image processing apparatus 4 has a module configuration including each unit described in the first to third embodiments. As actual hardware, when the control unit 401 reads out and executes a program from the auxiliary storage unit 403, one or more of the above-described units are loaded onto the main storage unit 402, and one or more of the respective units are loaded. Are generated on the main storage unit 402.

  As described above, each image processing (video enhancement processing) described in the first to third embodiments may be realized as a program for causing a computer to execute. The processing described in each embodiment can be realized by installing this program from a server or the like and causing the computer to execute the program.

  It is also possible to record the program in the recording medium 405 and cause the computer or portable terminal to read the recording medium 405 on which the program is recorded, thereby realizing the above-described image processing. The recording medium 405 is a recording medium that records information optically, electrically, or magnetically, such as a CD-ROM, a flexible disk, a magneto-optical disk, etc. Various types of recording media such as a semiconductor memory for recording can be used.

  Note that each of the means described in the first to third embodiments can employ various integrated circuits and electronic circuits. Moreover, a part of each means demonstrated in Examples 1-3 can also be made into another integrated circuit and an electronic circuit. For example, examples of the integrated circuit include ASIC (Application Specific Integrated Circuit) and FPGA (Field Programmable Gate Array). Examples of the electronic circuit include a central processing unit (CPU) and a micro processing unit (MPU).

  Each embodiment has been described in detail above. However, the present invention is not limited to the specific embodiment, and various modifications and changes other than the above-described modification are possible within the scope described in the claims. .

1, 3, 4 Image processing apparatus 101 Feature extraction means 102 Delay means 103 Determination means 104 Image storage means 105 Output means 106 Control means 301 Preprocessing means 302 Postprocessing means

Claims (6)

Extracting means for inputting a frame included in the video and extracting an image feature for each frame;
Delay means for storing and delaying the image features extracted by the extraction means;
A determination unit that compares the image feature extracted by the extraction unit with the image feature delayed by the delay unit and determines a change in the image feature;
A storage unit that stores a frame in which the image feature has changed when the determination unit determines that the image feature has changed;
Output means for outputting either the input frame or the frame stored in the storage means;
Control means for controlling the output means so as to continuously output a predetermined number of frames stored in the storage means when it is determined by the determination means that the image characteristics have changed;
An image processing apparatus comprising: The storage means
Storing a predetermined number of frames after the frame in which the image feature has changed,
The control means includes
The image processing apparatus according to claim 1, wherein after the frames with the changed image characteristics stored in the storage unit are continuously output, the predetermined number of frames stored in the storage unit are controlled to be output.   The image processing apparatus according to claim 2, wherein the predetermined number of frames is a frame after the frame in which the image feature has changed, and is a discontinuous frame.   4. The image processing apparatus according to claim 1, further comprising a preprocessing unit that rearranges the input frames in the order of video encoding and outputs the rearranged frames to the extraction unit. 5.   The image processing apparatus according to claim 1, wherein the image feature is a value related to a luminance value. An extraction step of inputting a frame included in the video and extracting an image feature for each frame;
A delay step of buffering and delaying the extracted image features;
A determination step of comparing the extracted image feature with the delayed image feature to determine a change in the image feature;
If it is determined that the image feature has changed, a storage step of storing in the storage means the frame in which the image feature has changed;
An output step of outputting either the input frame or the frame stored in the storage means;
A control step for controlling to output a predetermined number of frames stored in the storage means continuously when it is determined that the image feature has changed;
A program that causes a computer to execute.