JP4921543B2 - Trimming control device and trimming control program - Google Patents

Description

  The present invention relates to a trimming control technique, and more particularly, to a trimming control technique that does not impair visibility when viewing video content produced for a program on a small screen of a mobile terminal or the like.

  2. Description of the Related Art Conventionally, mobile terminals such as mobile phones and PDAs (Personal Data Assistants) capable of receiving TV broadcasts have been widely used. Such a portable terminal is limited in the size and transmission capacity of a screen for displaying a moving image. Therefore, for example, H.264 adopted for digital broadcasting for portable terminals (1-segment broadcasting). In the case of the H.264 resolution, it is assumed that the visibility is remarkably deteriorated depending on the program.

  As a method for solving this problem, it is conceivable to enlarge and display the attention area of the moving image. However, it is difficult for a mobile terminal to optimally display a trimmed video so as not to impair visibility.

  On the other hand, various techniques have been disclosed for trimming control for displaying a region of interest of an image. For example, the trimming apparatus of Patent Literature 1 calculates an attention area of an image to be trimmed, obtains an object index, selects a trimming technique based on the object index, and performs a trimming process using the selected trimming technique. . Thereby, trimming of an image can be performed automatically and effectively. As a technique for correcting camera shake of a moving image, for example, the camera shake control apparatus disclosed in Patent Document 2 detects a temporal image motion in a video field, calculates a camera shake amount, and uses the determined clipping frame as a hand. By moving according to the amount of blur, camera shake correction is realized. In this case, the camera shake amount is calculated by dividing the video field into four blocks, comparing the blocks, and using a representative matching method.

Japanese Patent Application Laid-Open No. 2004-228995 JP-A-6-46316

  However, the above-described technique obtains the center coordinates of the trimming screen (hereinafter referred to as trimming coordinates) and the center coordinates of the display screen during camera shake correction so that the image can be easily viewed from the viewpoint of the viewer. I have not done it. In other words, when an image is output to a mobile terminal, an image that is easy to see is not necessarily provided because processing that considers visibility is not performed.

  Conventionally, there has been a concern about shortage of contents due to the progress of digitalization and multi-channel broadcasting. For this reason, each broadcast station and program production company must produce a program using limited resources, and a new video generation technique has been demanded.

  Therefore, the present invention has been made to solve such a problem, and the purpose thereof is to realize a natural camera work when displaying content on a terminal or the like having a small display screen, and a screen that does not impair visibility. Is to provide a trimming control device and a trimming control program.

The trimming control device according to the present invention receives a trimming control by inputting an original video composed of a plurality of frames , and generates a trimming screen by cutting out a screen having a predetermined size centered on a predetermined trimming coordinate from the original video of the processing target frame. An object of interest, which is a device, receives reference data indicating object coordinates of interest, which is associated with at least one frame of an input original video composed of a plurality of frames, and detects object coordinates of interest based on the reference data calculating a coordinate detection unit, on the basis of the detected object of interest coordinates, the target object position and target object movement amount of the trimming window to the original image of the n-th frame at the position of the trimming window in the n-1 frame of the original picture Trimming at least the determining the offset amount of the target object from the target object positions of the trimming window with an object of interest moving amount from the center of the original screen, to determine the coordinates obtained by adding the offset amount to the object of interest coordinates in the trimming coordinates A coordinate determining unit, and a trimming processing unit that generates a trimming screen by cutting out a part of the original image from the original screen centered on the determined trimming coordinates by trimming, and outputs the image of the trimming screen; , Provided.

The trimming control apparatus according to the present invention is characterized in that the trimming coordinate determination unit determines the offset amount from the object position of the trimming screen, the target object coordinates, and the target object movement amount .

Further, the trimming control device according to the present invention includes a smoothing processing unit that smoothes a plurality of frames of the detected target object coordinates and uses the smoothed target object coordinates as new target object coordinates. The determination unit determines the offset amount using the new attention object coordinates .

In the trimming control apparatus according to the present invention, the smoothing processing unit calculates a moving average of the detected object coordinates of a plurality of frames, and uses the object object coordinates of the moving average as the new object object coordinates. It is characterized by doing.

Further, in the trimming control device according to the present invention, the smoothing processing unit performs integration processing for a plurality of frames of the detected object coordinates of interest using a low-pass filter, and the object object coordinates subjected to the integration processing are added to the new object coordinates. It is characterized by the attention object coordinates .

Also, the trimming control device in the present invention, the attention object coordinate detection unit receives the metadata including the target object coordinate, and detecting a target object coordinates from the metadata.

Further, the trimming control program according to the present invention inputs an original video composed of a plurality of frames, and trims and outputs a trimming screen having a predetermined size centered on a predetermined trimming coordinate from the original video of the processing target frame. A program executed by the apparatus, the computer constituting the trimming control apparatus, the reference data indicating the object coordinates of interest associated with at least one frame of the input original video consisting of a plurality of frames, a process for detecting the target object coordinates based on said reference data, based on the detected object of interest coordinates, the trimming window to the original image of the n-th frame at the position of the trimming window in the n-1 frame of the original picture The attention object position and the attention object movement amount are calculated, and the offset amount of the attention object from the center of the original screen is determined from at least the attention object position and the attention object movement amount of the trimming screen, and the offset amount is determined as the attention object coordinates. A process for determining the coordinates added to the trimming coordinates, and by trimming, a part of the screen is cut out from the original screen of the input video around the determined trimming coordinates, and a trimming screen is generated. And a process of outputting a video.

  In general, in program shooting, a cameraman comprehensively determines the orientation, moving direction, surrounding environment, and the like of an object that is a subject, and determines camera work such as zoom and pan / tilt. For example, an image shot with a composition (see FIG. 14) that opens in front of the subject according to the direction and direction of movement of the subject feels more natural than an image shot with a composition that always places the subject at the center. There are many cases. In the present invention, the object coordinates of interest are not used as trimming coordinates as they are, but the trimming coordinates are determined in consideration of the traveling direction of the object of interest, and the trimming process is performed using the trimming coordinates. Thus, when content is displayed on a terminal or the like having a small display screen, natural camera work can be realized, and a screen that does not impair visibility can be displayed, and the trimmed video can be easily viewed.

It is a figure which shows the structure and flowchart of Example 1 of the trimming control apparatus by this invention. It is a figure which shows the structure and flowchart of Example 2 of the trimming control apparatus by this invention. It is a figure explaining the offset process of Example 2. FIG. It is a figure which shows the structure and flowchart of Example 3 of the trimming control apparatus by this invention. It is a figure explaining the fuzzy control of Example 3. FIG. FIG. 10 is a flowchart showing a detailed operation of the third embodiment. It is a figure which shows the fuzzy set with respect to the ball position of a trimming screen. It is a figure which shows the fuzzy set with respect to a ball | bowl movement amount. It is a figure which shows the fuzzy set with respect to the movement amount of a trimming coordinate. It is the figure which represented the calculation method of the movement amount of the trimming coordinate by fuzzy control. It is a figure which shows the structure and flowchart of Example 4 of the trimming control apparatus by this invention. FIG. 10 is a flowchart showing a detailed operation of the fourth embodiment. It is a figure which shows the fuzzy set with respect to the ball position of an original screen. It is a figure explaining the front empty screen of Example 4. FIG. It is a figure which shows the structure and flowchart of Example 5 of the trimming control apparatus by this invention.

  Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. The trimming control apparatus according to the embodiment of the present invention inputs video such as video content produced for a program, cuts out and generates a trimming screen from the original screen of the video, and outputs it. When generating this trimming screen, the target object coordinates of interest are detected from the original screen, and the trimming coordinates that are the center coordinates of the trimming screen are determined. Here, when the trimming screen is displayed on a mobile terminal or the like as a video, the present invention is characterized by how to determine trimming coordinates for making a trimmed video that does not impair the visibility.

  The first embodiment of the present invention smoothes fine fluctuations in target object coordinates, and is shown in FIG. The second embodiment uses the coordinates obtained by adding an offset to the object coordinates as trimming coordinates, and is shown in FIGS. 2 and 3. In the third embodiment, trimming coordinates are obtained by fuzzy control, and are shown in FIGS. In the fourth embodiment, trimming coordinates having a front empty screen are obtained by fuzzy control, and are shown in FIGS. The fifth embodiment adds an offset to the trimming coordinates obtained by fuzzy control, and is shown in FIG. Hereinafter, Examples 1 to 5 will be described in detail.

  FIG. 1 is a diagram showing a configuration and a flowchart of Embodiment 1 of a trimming control apparatus according to the present invention. With reference to FIG. 1A, the trimming control device 10 includes a trimming processing unit 11, a target object coordinate detecting unit 12, and a smoothing processing unit 13. The trimming processing unit 11 inputs an image from the outside and trimming coordinates from the smoothing processing unit 13, cuts out a screen of a predetermined size centered on the trimming coordinates from the original screen of the input video, and displays a trimming screen. Generate and output the trimming screen to the outside as a trimmed video. That is, the trimming processing unit 11 reconstructs the scene of the original screen for each frame and displays the reconstructed scene on, for example, a display. The target object coordinate detection unit 12 inputs reference data (metadata) from the outside, detects the coordinates of the target object (target object coordinates) in the original screen, and outputs the target object coordinates to the smoothing processing unit 13. To do. In this case, the target object coordinate detection unit 12 may input the video itself from the outside and detect the target object coordinates by an image processing technique such as pattern matching using normalized correlation tracking. The smoothing processing unit 13 receives the target object coordinates from the target object coordinate detection unit 12, and stores the target object coordinates for each frame in a storage unit (not shown). Then, the target object coordinates for the past several frames are read from the storage means, a moving average is obtained, and the value of the moving average is output to the trimming processing unit 11 as trimming coordinates.

  The operation of the trimming control apparatus 10 will be described with reference to FIG. When the trimming control device 10 inputs video and reference data from the outside, the target object coordinate detection unit 12 detects target object coordinates based on the input reference data (step S101). When the smoothing processing unit 13 receives the target object coordinates from the target object coordinate detection unit 12, the smoothing processing unit 13 stores the object coordinates of a predetermined number of frames (step S102). Then, the smoothing processing unit 13 calculates a moving average based on the stored object coordinates (step S103), and determines a value of the moving average as a trimming coordinate (step S104). The trimming processing unit 11 cuts out a screen having a predetermined size centered on the trimming coordinates from the original screen of the input video, and generates a trimming screen (step S105).

  As described above, according to the trimming control device 10 of the first embodiment, the smoothing processing unit 13 determines the moving average value of the target object coordinates for the past several frames as the trimming coordinates. As a result, even if the object of interest causes a fine coordinate fluctuation, the trimming coordinates do not follow the fluctuation more than necessary.

  The smoothing processing unit 13 described above outputs the moving average values of the object coordinates of interest for the past several frames as trimming coordinates. However, the trimming coordinates are obtained and output using an integration circuit (low-pass filter). It may be. Thereby, the high frequency component produced by the coordinate fluctuation | variation of an attention object can be removed.

  FIG. 2 is a diagram showing a configuration and a flowchart of Embodiment 2 of the trimming control apparatus according to the present invention. With reference to FIG. 2A, the trimming control device 20 includes a trimming processing unit 21, a target object coordinate detecting unit 22, and an offset amount adding unit 23. The trimming processing unit 21 inputs a video from the outside and trimming coordinates from the offset amount adding unit 23, cuts out a screen of a predetermined size centered on the trimming coordinates from the original screen of the input video, and displays a trimming screen. Generate and output the trimming screen to the outside as a trimmed video. The target object coordinate detection unit 22 detects the target object coordinates and outputs the target object coordinates to the offset amount addition unit 23 in the same manner as the target object coordinate detection unit 12 of the first embodiment shown in FIG. The offset amount adding unit 23 inputs the target object coordinates from the target object coordinate detection unit 22, compares the target object coordinates with the center coordinates of the original screen, and obtains the position (distance and direction) from the center of the original screen. An offset amount proportional to the position from the center is determined. In this case, a preset value may be determined as the offset amount. Then, an offset amount is added to the object coordinates to obtain trimming coordinates, and the trimming coordinates are output to the trimming processing unit 21.

  FIG. 3 shows an example in which the object of interest in the original screen is a ball, and the ball itself is located on the left side from the center of the original screen. As shown in FIG. 3, for example, the offset amount adding unit 23 determines the offset amount so that the ball is positioned on the left side of the trimming screen even when the trimming process is performed.

  The operation of the trimming control device 20 will be described with reference to FIG. When the trimming control device 20 inputs video and reference data from the outside, the target object coordinate detection unit 22 detects target object coordinates based on the input reference data (step S201). When the target object coordinates are input from the target object coordinate detection unit 22, the offset amount adding unit 23 detects the position of the target object based on the target object coordinates and the center coordinates of the original screen (step S202), and based on the position. The offset amount is determined (step S203), and the coordinate obtained by adding the offset amount to the target object coordinate is determined as the trimming coordinate (step S204). The trimming processing unit 21 cuts out a screen having a predetermined size centered on the trimming coordinates from the original screen of the input video, and generates a trimming screen (step S205).

  As described above, according to the trimming control device 20 of the second embodiment, the offset amount adding unit 23 compares the target object coordinates with the center coordinates of the original screen, and according to the position of the target object from the center of the original screen. The offset amount is determined, and the coordinates obtained by adding the offset amount to the object coordinates are determined as the trimming coordinates. As a result, a trimming screen reflecting the position of the object of interest on the original screen can be generated. That is, a camera work similar to the camera work in the original video can be realized, and a trimmed video that does not impair visibility can be realized.

  Note that the offset amount adding unit 23 described above compares the target object coordinates with the center coordinates of the original screen and determines the offset amount according to the position of the target object from the center of the original screen. The offset amount may be determined by predicting the moving direction and the moving amount. Specifically, when the ball itself is located on the left side from the center of the original screen, the offset amount adding unit 23 obtains a difference between the ball position of the previous frame and the ball position of the current frame, The direction in which the ball will move and the amount of movement will be predicted. If the ball can be predicted to move further to the left, the offset amount is determined so that the ball is positioned at the center of the trimming screen when performing the trimming process.

  Further, an offset amount in consideration of attack and defense may be determined using information on the team that controls the ball. Specifically, the offset amount adding unit 23 extracts the identification information of the team that controls the ball from the reference data input from the outside, and determines that the attacking team controls the ball. Then, an offset amount of about ¼ of the trimming screen width is determined in the direction of the defensive team. Thereby, a trimming screen having a space (previous free space) in the attack direction can be generated.

  FIG. 4 is a diagram showing a configuration and a flowchart of Embodiment 3 of the trimming control apparatus according to the present invention. With reference to FIG. 4A, the trimming control device 30 includes a trimming processing unit 31, a target object coordinate detection unit 32, and a fuzzy control unit 33. The trimming processing unit 31 inputs a video from the outside and trimming coordinates from the fuzzy control unit 33, cuts out a screen of a predetermined size centered on the trimming coordinates from the original screen of the input video, and generates a trimming screen The trimming screen is output to the outside as a trimmed video. The target object coordinate detection unit 32 detects the target object coordinates and outputs the target object coordinates to the fuzzy control unit 33 in the same manner as the target object coordinate detection unit 12 of the first embodiment shown in FIG. The fuzzy control unit 33 inputs the target object coordinates from the target object coordinate detection unit 32 and calculates the amount of movement of the object. Also, the current object position (object position on the trimming screen) when the previous frame is continuously trimmed at the same position is calculated. The fuzzy control unit 33 uses the object position and the amount of movement of the trimming screen as input values, the amount of movement of the trimming coordinates as an output value, and performs trimming according to the fuzzy rule set so that the object moves to the center of the trimming screen. The coordinates are determined and output to the trimming processing unit 31.

  The operation of the trimming control device 30 will be described with reference to FIG. Although details of the operation of the trimming control device 30 will be described later, an outline of the operation will be described here. When the trimming control device 30 inputs video and reference data from the outside, the target object coordinate detection unit 32 detects target object coordinates based on the input reference data (step S301). When the target object coordinates are input from the target object coordinate detection unit 32, the fuzzy control unit 33 calculates the movement amount of the object (step S302), calculates the object position on the trimming screen (step S303), and performs the trimming coordinates by fuzzy control. Is calculated, and trimming coordinates are determined (step S304). The trimming processing unit 31 cuts out a screen having a predetermined size centered on the trimming coordinates from the original screen of the input video, and generates a trimming screen (step S305).

  Next, the operation of the fuzzy control unit 33 will be described in detail. For example, when a ball is an object of interest in soccer relay, the current coordinates of the ball (the ball position on the trimming screen) and the movement amount of the ball are used as input values. Referring to FIG. 5, a screen in a frame centering on the ball in the original screen is defined as a trimming screen with the horizontal direction as the x-axis and the vertical direction as the y-axis, starting from the upper left of the original screen. The screen shown in the upper part of FIG. 5 is an original screen and trimming screen in the (n-1) th frame, and the screen shown in the lower part is an original screen and trimming screen in the nth frame.

Now, on the original screen of the nt frame, the ball position is (xa [n-1], ya [n-1]), and the center coordinates to be trimmed are (xt [n-1], yt [n-1]). In the original screen of n frames, the ball position is (xa [n], ya [n]), and the center coordinates to be trimmed are (xt [n], yt [n]). In this case, if the ball position is (xbt [n], ybt [n]) on the screen after trimming (trimming screen), Equation (1) is obtained with the trimming coordinates as the starting point (0, 0).
(Xbt [n], ybt [n]) = (xa [n] −xt [n−1], ya [n] −yt [n−1]) (1)
When the ball movement amount is (xbm [n], ybm [n]), the equation (2) is obtained.
(Xbm [n], ybm [n]) = (xa [n] −xa [n−1], ya [n] −ya [n−1]) (2)

Using the current ball position (see equation (1)) and the ball movement amount (see equation (2)) when trimming the previous frame is continued at the same position as the input value for fuzzy control, The amount of movement of the trimming coordinates (Δx [n], Δy [n]) is obtained as an output value. Accordingly, the center coordinates to be actually trimmed are expressed by equation (3).
(Xt [n], yt [n]) = (xt [n−1] + Δx [n], yt [n−1] + Δy [n]) (3)
Note that the initial value of the trimming coordinates is represented by equation (4).
(Xt [0], yt [0]) = (xa [0], ya [0]) (4)

  FIG. 6 is a flowchart showing the detailed operation of the fuzzy control unit 33. This flowchart shows steps 302 to 304 of the flowchart shown in FIG. 4B in more detail. Referring to FIG. 6, in order to cause the trimming processing unit 31 to perform the trimming process of the first frame (n = 0), the fuzzy control unit 33 trims the ball coordinates (xa [0], xa [0]). The coordinates are output (step S601). That is, the trimming coordinates are as described in the above-described equation (4). Then, the process proceeds to the next frame (n = 1) (step S602).

The fuzzy control unit 33 inputs the ball position (xa [n], ya [n]) detected by the target object coordinate detection unit 32 based on the reference data (metadata) (step S603). In addition, the ball position (see equation (1)) and the ball movement amount (see equation (2)) on the trimming screen in the previous frame are respectively obtained (steps S604 and 605), and the trimming coordinates that are the control amount from the fuzzy inference Is obtained as shown in equation (5) (step S606).
(Δx [n], Δy [n]) = Fuzzy {(xbt [n], ybt [n]), (xbm [n], ybm [n])} (5)
Further, coordinates obtained by adding a control amount to the trimming coordinates of the previous frame are output to the trimming processing unit 31 as trimming coordinates (step S607). Then, the process proceeds to the next frame (n ← n + 1) (step S608), and the process proceeds to step 603. In this way, each process from step 603 to step 608 is executed for each frame.

  For example, when a video image with a resolution of VGA (Video Graphics Array) (640 × 480) is trimmed with a QVGA (Quarter VGA) (320 × 240) size, the ball position on the QVGA screen after trimming is −160 ≦ xbt ≦ 160, −120 ≦ ybt ≦ 120, and the ball movement amount takes values in the range of −640 ≦ xbm ≦ 640 and −480 ≦ ybm ≦ 480.

  Next, the calculation method of the control amount (movement amount of object coordinates) by fuzzy inference in step 606 shown in FIG. 6 will be described in detail. In general, fuzzy control is performed using a plurality of IF-THEN rules. This fuzzy rule is shown in Table 1.

In Table 1, each row is a ball position, each column is a ball movement amount, and an intersecting portion thereof corresponds to an output value (movement amount of trimming coordinates). The symbols “NB”, “NS”, “ZO”, “PS”, and “PB” in the table respectively indicate “negatively large value”, “negatively small value”, “zero”, “positively small value”, and “positively large value”. Show. For example, the first row and third column
“IF The ball position is NB (leftward) and the ball movement is ZO (zero) THEN The movement of the trimming coordinate is PS (moves to the right)”
Shows the rule. In order to perform fuzzy control, fuzzy sets corresponding to “NB”, “NS”, “ZO”, “PS”, and “PB” are determined. 7 is a diagram showing a fuzzy set with respect to the ball position on the trimming screen, FIG. 8 is a diagram showing a fuzzy set with respect to the ball movement amount, and FIG. 9 is a diagram showing a fuzzy set with respect to the movement amount of the trimming coordinates. As described above, the fuzzy set is represented as a membership function as shown in FIGS.

For example, the operation when the x coordinate of the ball position is −90 and the ball movement amount in the x axis direction is 15 will be described. FIG. 10 is a diagram schematically illustrating a method of calculating the amount of movement of trimming coordinates by fuzzy control. Referring to FIG. 10, NS = 0.6 and ZO = 0.4 by the ball position and the membership function shown in FIG. 7, and ZO = 0 by the ball movement amount and the membership function shown in FIG. .8, PS = 0.2 is obtained. The fuzzy rules shown in Table 1 are as follows.
"IF NS AND ZO THEN PS"
"IF NS AND PS THEN PS"
"IF ZO AND ZO THEN ZO"
"IF ZO AND PS THEN ZO"
Thus, when general fuzzy control using the simplified fuzzy inference method is performed, the centroid value of the membership function is obtained by the following equation (6).
(0.4 × 0 + 0.6 × 50) / (0.4 + 0.6) = 30 (6)
Therefore, it is only necessary to move the trimming coordinates by 30 in the right direction. Similarly, the movement amount of the trimming coordinates is calculated also in the y-axis direction.

  As described above, according to the trimming control device 30 of the third embodiment, the fuzzy control unit 33 uses the fuzzy rules shown in Table 1 to move the trimming coordinate movement amount based on the ball position and the ball movement amount on the trimming screen. Was calculated. In this case, the fuzzy rules shown in Table 1 are set so that when the ball position is near the center of the screen, the movement amount of the trimming coordinates, which is the output value, is close to 0 without depending on the ball movement amount. Thus, the more the ball moves to the end of the trimming screen, the more control can be performed so that the ball returns to the center of the screen. In addition, it does not react sensitively to fluctuations in the ball position, and gentle control can be realized. That is, even if the object of interest causes a fine coordinate variation, the trimming coordinates do not follow this variation more than necessary. Therefore, trimming control can be realized in a form close to the camerawork know-how possessed by the cameraman.

  FIG. 11 is a diagram showing a configuration and a flowchart of Embodiment 4 of the trimming control apparatus according to the present invention. With reference to FIG. 11A, the trimming control device 40 includes a trimming processing unit 41, a target object coordinate detection unit 42, and a fuzzy control unit 43. When this trimming control device 40 is compared with the trimming control device 30 of the third embodiment shown in FIG. 4, the trimming processing unit 41, the trimming processing unit 31, the target object coordinate detection unit 42, and the target object coordinate detection unit 32 are They are the same in that they have equivalent functions. On the other hand, the fuzzy control unit 43 uses the object position on the original screen, the object position on the trimming screen, and the object movement amount as input values, while the fuzzy control unit 33 sets the object position and object movement amount on the trimming screen. It differs in that it is an input value.

  In the fuzzy rule of Example 3 shown in Table 1, the trimming process is performed so that the ball is positioned at the center of the screen. By the way, for example, when soccer players are attacking in the right direction of the screen, the cameraman generally tends to take a camera work to position the ball on the left side of the screen. Conversely, when the player is attacking in the left direction of the screen, there is a tendency to take a camera work such that the ball is positioned on the right side of the screen. This is to create a natural front vacant image that clearly shows the situation in the direction of travel of the ball. In the fourth embodiment, paying attention to the fact that the original screen takes into account the camera work by such a cameraman, fuzzy control is performed using the ball position of the original screen, the ball position of the trimming screen, and the ball movement amount as input values. By doing so, a previous free image is obtained.

  Specifically, the fuzzy control unit 43 inputs the target object coordinates from the target object coordinate detection unit 42, sets the target object coordinates as the object position on the original screen, and calculates the movement amount of the object. Also, the current object position (object position on the trimming screen) when the previous frame is continuously trimmed at the same position is calculated. Then, the fuzzy control unit 43 uses the object position on the original screen, the object position on the trimming screen, and the amount of movement of the object as input values and the amount of movement of the trimming coordinates as output values, and sets the object position on the original screen as the object position on the trimming screen. The trimming coordinates are determined according to the fuzzy rule set so that the object moves to the center of the trimming screen, and is output to the trimming processing unit 31.

  The operation of the trimming control device 40 will be described with reference to FIG. Although details of the operation of the trimming control device 40 will be described later, an outline of the operation will be described here. When the trimming control device 40 inputs video and reference data from the outside, the target object coordinate detection unit 42 detects target object coordinates based on the input reference data (step S401). When the target object coordinates are input from the target object coordinate detection unit 42, the fuzzy control unit 43 sets the target object coordinates as the object coordinates of the original screen (step S402), calculates the amount of movement of the object (step S403), and the trimming screen Is calculated (step S404), the amount of movement of the trimming coordinates is calculated by fuzzy control, and the trimming coordinates are determined (step S405). The trimming processing unit 41 cuts out a screen having a predetermined size centered on the trimming coordinates from the original screen of the input video, and generates a trimming screen (step S406).

Next, the operation of the fuzzy control unit 43 will be described in detail. For example, when a ball is an object of interest in a soccer broadcast, the ball position on the original screen, the ball position on the trimming screen, and the movement amount of the ball are used as input values. FIG. 12 is a flowchart showing the detailed operation of the fuzzy control unit 43. This flowchart shows steps 402 to 405 of the flowchart shown in FIG. 11B in more detail. Comparing this flowchart with the flowchart of the third embodiment shown in FIG. 6, step 1201 and step 601, step 1202 and step 602, step 1204 and step 604, step 1205 and step 605, step 1207 and step 607, and Step 1208 and step 608 are the same in that they are the same process. On the other hand, Step 1203 and Step 603 and Step 1206 and Step 606 are different in that they are different processes. That is, in step 1203, the fuzzy control unit 43 inputs the ball position (xa [n], ya [n]) detected by the target object coordinate detection unit 42 based on the reference data (metadata), and the ball position Is used as the ball position on the original screen and as an input value for fuzzy control. FIG. 13 shows a fuzzy set for the ball position on the original screen. In step 1206, the fuzzy control unit 43 obtains the movement amount of the trimming coordinate, which is the control amount, from the fuzzy inference as shown in the equation (7).
(Δx [n], Δy [n]) = Fuzzy {(xa [n], ya [n]), (xbt [n], ybt [n]), (xbm [n], ybm [n])} (7)

  Table 2 shows the fuzzy rules used for calculating the control amount (movement amount of the object coordinates) by the fuzzy inference in step 1206 shown in FIG. Comparing this fuzzy rule with the fuzzy rule of Example 3 shown in Table 1, the fuzzy rule in Table 2 is different in that the ball position on the original screen is also used as an input value.

  As described above, according to the trimming control device 40 of the fourth embodiment, the fuzzy control unit 43 uses the fuzzy rules shown in Table 2 based on the ball position on the original screen, the ball position on the trimming screen, and the ball movement amount. The amount of movement of the trimming coordinates was calculated. In this case, the fuzzy rules shown in Table 2 are such that when the ball is positioned on the left side of the original screen, the fuzzy rule is positioned on the left side of the trimming screen, and when the ball is positioned on the right side of the original screen, Is set to be located. As a result, as shown in FIG. 14, a space can be made in the traveling direction of the ball corresponding to the original screen, the situation of the traveling direction of the ball can be understood well, and a natural front empty image can be realized. It becomes possible. That is, it is possible to generate a trimming screen in consideration of camera work by a cameraman.

  FIG. 15 is a diagram showing a configuration and a flowchart of Embodiment 5 of the trimming control apparatus according to the present invention. Referring to FIG. 15A, the trimming control device 50 includes a trimming processing unit 51, a target object coordinate detection unit 52, a fuzzy control unit 53, and an offset amount adding unit 54. When this trimming control device 50 is compared with the trimming control device 30 of the third embodiment shown in FIG. 4, the trimming processing unit 51 and the trimming processing unit 41, and the fuzzy control unit 33 and the fuzzy control unit 53 have equivalent functions. It is the same in that it has. On the other hand, the target object coordinate detection unit 52 outputs the target object coordinates not only to the fuzzy control unit 53 but also to the offset amount addition unit 54, and the offset amount addition unit 54 receives the trimming coordinates from the fuzzy control unit 53. The target object coordinates are respectively input from the object coordinate detection unit 52, the offset amount is determined based on the target object coordinates, the offset amount is added to the trimming coordinates to obtain new trimming coordinates, and the new trimming coordinates are used as the trimming processing unit 51. It differs in that it is output to. The offset amount adding unit 54 has the same function as the offset amount adding unit 23 of the second embodiment shown in FIG.

  The operation of the trimming control device 50 will be described with reference to FIG. When the trimming control device 50 inputs video and reference data from outside, the target object coordinate detection unit 52 detects target object coordinates based on the input reference data (step S501). When the target object coordinates are input from the target object coordinate detection unit 52, the fuzzy control unit 53 calculates the amount of movement of the object (step S502), calculates the object position of the trimming screen (step S503), and trims the coordinates by fuzzy control. Is calculated, and trimming coordinates are determined (step S504). When the offset amount adding unit 54 receives the target object coordinates from the target object coordinate detection unit 52 and the trimming coordinates from the fuzzy control unit 53, the offset amount adding unit 54 detects the position of the target object based on the target object coordinates and the center coordinates of the original screen. Then, an offset amount is determined based on the position, and a new trimming coordinate is determined by adding the offset amount to the trimming coordinate (step S506). The trimming processing unit 51 cuts out and generates a trimming screen having a predetermined size centered on a new trimming coordinate from the original screen of the input video (step S507).

  As described above, according to the trimming control device 50 of the fifth embodiment, the fuzzy control unit 53 determines the trimming coordinates according to the fuzzy rules shown in Table 1 based on the ball position and the ball movement amount on the trimming screen. The offset amount adding unit 54 determines an offset amount according to the position of the object of interest from the center of the original screen, and uses the coordinates obtained by adding the offset amount to the trimming coordinates determined by the fuzzy control unit 53 as new trimming coordinates. I decided to decide. Thereby, the effect equivalent to Example 2 and Example 3 mentioned above can be acquired.

  The above-described fuzzy control unit 53 has the same function as the fuzzy control unit 33 shown in the third embodiment, but has the same function as the fuzzy control unit 43 shown in the fourth embodiment. Also good. In this case, in addition to the effect of the second embodiment, the same effect as that of the fourth embodiment can be obtained. Further, instead of the fuzzy control unit 53, the smoothing processing unit 13 shown in the first embodiment may be included. In this case, in addition to the effect of the second embodiment, the same effect as that of the first embodiment can be obtained.

  As mentioned above, although this invention was demonstrated by Example 1-5, this invention is not limited to the said Examples 1-5, A various deformation | transformation is possible in the range which does not deviate from the technical thought. For example, the trimming control devices 10 to 50 may receive (input) video and reference data by broadcast waves or via the Internet network. In this case, the video and reference data may be received by broadcast waves, or may be received via the Internet network. Further, the video may be received via a broadcast wave and the reference data may be received via the Internet network, or the video may be received via the Internet network and the reference data may be received via a broadcast wave.

  Also, in the case of soccer broadcast, only one high-precision camera such as a high-definition camera is installed (or two from both sides) so that the entire soccer field can be seen. The video of the entire soccer field photographed by the camera may be input, and the trimming processing unit 11 may realize a function equivalent to conventional pan / tilt or switching by a plurality of cameras by trimming processing. The tracking of the ball and the player is performed by image processing such as normalized correlation tracking or human correction, etc., and by performing the trimming processing of the first to fifth embodiments described above, even when the object of interest moves violently, The trimming coordinates can be prevented from changing suddenly.

  When the target object is a soccer relay ball, the target object coordinate detection units 12 to 52 of the trimming control devices 10 to 50 detect the target object coordinates by an image processing technique such as normalized correlation tracking. I did it. However, the transmission side detects the ball position coordinates (target object coordinates) in advance, stores the target object coordinates in the program metadata as data for each frame, and transmits the program metadata including the target object coordinates to the broadcast wave or the Internet. The target object coordinate detection units 12 to 52 may transmit the target object coordinates from the transmitted program metadata by transmitting them to the receiving side trimming control devices 10 to 50 through a network or the like.

  Further, the trimming processing units 11 to 51 of the trimming control devices 10 to 50 cut out a screen having a predetermined size centered on the trimming coordinates from the original screen of the input video, generate a trimming screen, and perform the trimming. Although the screen is output to the outside as a trimmed video, the screen cut out from the original screen may be enlarged and the enlarged screen may be used as the trimming screen, or the cut out screen may be used as it is as the trimming screen.

  The trimming control devices 10 to 50 described above include a volatile storage medium such as a CPU and a RAM, a non-volatile storage medium such as a ROM, an input device such as a keyboard, a display device that displays data, and an external device. You may make it each be comprised with the computer apparatus provided with the interface for communicating. In this case, the trimming processing units 11 to 51, the target object coordinate detection units 12 to 52, the smoothing processing unit 13, the offset amount adding units 23 and 54, and the fuzzy control units 33, 43, and 53 included in the trimming control devices 10 to 50 are provided. Each of these functions is realized by causing the CPU to execute a program describing the function. These programs can also be stored and distributed in a storage medium such as a magnetic disk (floppy disk, hard disk, etc.), optical disk (CD-ROM, DVD, etc.), semiconductor memory, or the like.

10, 20, 30, 40, 50 Trimming control device 11, 21, 31, 41, 51 Trimming processing unit 12, 22, 32, 42, 52 Target object coordinate detection unit 13 Smoothing processing unit 23, 54 Offset amount adding unit 33, 43, 53 Fuzzy controller

Claims (7)

A trimming control device that inputs an original video composed of a plurality of frames, generates a trimming screen by cutting out a screen of a predetermined size centered on a predetermined trimming coordinate from the original video of a processing target frame,
A target object coordinate detection unit that inputs reference data indicating target object coordinates, which is associated with at least one of the input original videos, and detects target object coordinates based on the reference data ;
Based on the detected target object coordinates, the target object position and target object movement amount of the trimming screen with respect to the original video of the nth frame are calculated at the position of the trimming screen of the original video of the (n-1) th frame, and at least the trimming determining the offset amount of the target object from the center of the original screen from the target object position and target object moving amount of the screen, and a trimming coordinate determination unit for determining the coordinates obtained by adding the offset amount to the object of interest coordinates in the trimming coordinates ,
A trimming processing unit that cuts out a part of the screen from the original screen of the input video by trimming, and generates a trimming screen and outputs the video of the trimming screen;
A trimming control apparatus comprising: The trimming control device according to claim 1,
The trimming coordinate determination unit, trimming control apparatus characterized by determining the offset amount from the target object moving amount and the object position of the trimming window and the target object coordinate. In the trimming control device according to claim 1 or 2 ,
Smoothing the plurality of frames of the detected object of interest coordinates, comprising a smoothing unit for the object of interest coordinates the smoothed as a new object of interest coordinates,
The trimming coordinate determination unit, trimming control apparatus characterized by determining the offset amount using the new attention object coordinates. In the trimming control device according to claim 3 ,
The smoothing processing unit, the detected calculates the moving average of a plurality of frames of the attention object coordinates, trimming control device, characterized in that the object of interest coordinates of the moving average and the new attention object coordinates. In the trimming control device according to claim 3 ,
The smoothing processing unit, a plurality of frames of the detected object of interest coordinates subjected to integration processing by the low-pass filter, characterized in that the object of interest coordinates subjected to the integrating process as the new target object coordinate Trimming control device. In the trimming control device according to any one of claims 1 to 5 ,
The attention object coordinate detection unit receives the metadata including the target object coordinate, trimming control device and detecting a target object coordinates from the metadata. A program executed by a trimming control apparatus that inputs an original video composed of a plurality of frames , cuts out and outputs a trimming screen having a predetermined size centered on a predetermined trimming coordinate from the original video of a processing target frame ,
In a computer constituting the trimming control device,
To the associated at least one frame of the original image consisting of a plurality of frames that are input, a process of type the reference data indicating the target object coordinate, detects the target object coordinates based on said reference data,
Based on the detected target object coordinates, the target object position and target object movement amount of the trimming screen with respect to the original video of the nth frame are calculated at the position of the trimming screen of the original video of the (n-1) th frame, and at least the trimming a process of determining the offset amount of the target object from the center of the original screen from the target object position and target object moving amount of the screen, to determine the coordinates obtained by adding the offset amount to the object of interest coordinates to the trimming coordinates,
A process of cutting out a part of the original image from the original screen, generating a trimming screen, and outputting the image of the trimming screen, centering on the determined trimming coordinates by trimming;
Trimming control program to execute.